Sheet 05 · PIL-003
Standards knowledge search
Engineers cannot reliably find the current standard, so they ask colleagues and answers vary between offices.
Record summary
- Stage
- Measuring
- Risk pathway
- Structured review
- Data classification
- Internal
- Participants
- 60
Current status
Measuring against the scale threshold. Accuracy 91% against a 95% target; the library cleanup that would close the gap is scheduled.
- Next decision
- Scale, extend the measurement window, or hold pending library cleanup
- Where it is decided
- AI Governance Committee, 2026-09-15
The record
- Expected outcome
- Engineers find the current standard in under two minutes, with citations accurate enough to trust.
- Business owner
- Tomas Reyes, Data & Reporting Manager
- Program coordinator
- AI Program Manager — coordinates and records; does not approve
- Participating teams
- Structural Practice Group, Civil Practice Group, Data & Reporting
- Tool
- Cadence Search — Cadence Data Systems TL-02
- Originating use case
- Standards library search for engineering staff UC-007
- Related decisions
- DEC-2026-03
- Start date
- 13 Apr 2026
- Decision date
- 15 Sep 2026 (in 13 days)
- Cost to date
- $14,800 — licensing, vendor fees, and third-party support. Participant time is not included and is the larger cost.
Measures
Defined before the pilot started. That is what makes a result a finding rather than an opinion formed afterwards.
| Measure | Target | Current | Evidence |
|---|---|---|---|
| Citation accuracy on the 100-question test set | 95% before firmwide scale | 91% | Measured |
| Median time to locate a current standard | Under 2 minutes, from a 6.5-minute baseline | 1.8 minutes | Measured |
| Weekly active users among pilot participants | 60% of 60 licensed users | 41 of 60 | Measured |
What the pilot has surfaced
Open questions and lessons sit here rather than in a closing report, because they are what the next pilot needs and the next pilot starts before this one ends.
Open questions
- Do the remaining 9% of citation errors cluster in one document generation? If so, this is a cleanup task, not a tool limitation.
- What happens to accuracy when the corpus doubles?
Lessons learned
- The accuracy gap is mostly a records problem. Eight of the nine failing questions cite documents that should have been retired years ago.
- Adoption tracked office proximity to a champion more than it tracked role or seniority.
Guardrails and dependencies
Guardrails
- Internal standards library only — no project or client documents indexed
- Every answer shows its source document and revision
- Permissions mirror the source library; the tool grants no new access
Dependencies
- Retirement of superseded standards revisions (owner: Data & Reporting, due 2026-09-05)
- Vendor penetration-test summary (covered by EXC-02, expires 2026-10-15)