Peak Phase 3 Accuracy Deep Dive: Detection-First After the 402-Miss Problem

ERRATUM (updated 2026-08-07): the annotated drawing images previously shown in this post were not real. scripts/annotate-peak-audit.ts contained an offline demo branch that painted boxes on a hard-coded grid and drew "pipe runs" whose position was a hash of the description text, and it ran even in live mode because the vision calls returned empty and the empty result was cached. The captions describing them as matches, mismatches and misses were therefore incorrect. The images have been withdrawn. The replacement renderer has no demo path, refuses empty inputs, reads actual measurement polylines, and does not colour page pins with a project-level SOR verdict. All numeric results in this post came from scored benchmark artefacts, not from those images, and are unaffected.

SUPERSEDED FOR PRODUCT CLAIMS: After Phase 5, legacy soft among soft-eligible pairs is not the programme headline. Soft figures below (pooled soft 60.0%, and per-sheet soft %) are historical Phase 3 scores on a filtered paired set — not fixed-denominator accuracy on every drawable SOR line. For Detection and QA@20 on all drawable ground truth, read /blog/peak-phase5-measuring-what-we-ship and /blog/peak-phase6-accuracy-sprint. Imagery erratum above remains authoritative.

Peak Phase 1 answered classification: 90.2% of 512 qty>0 Schedule C rows got a takeoff family. Phase 2 asked whether AI quantities match Peak-SOR: soft accuracy among soft-eligible pairs was ~68.6% overall, but detection was only 23.8% because 402 of 463 drawable ground-truth rows never paired (miss / gt_only). Soft among the small paired set was already ~80%. Phase 3 therefore took a first-principles stance: stop scoring EL without EL drawings, raise pairing on plumbing and fire, cut extract noise, and scale partial-floor quantities when title-block coverage is known. This post reports Tasks R–Y against verified Phase 3 artefacts — numbers are not re-run for the write-up.

Paired soft was not the bottleneck. Detection was. Soft >80% overall needs pairing mass first; ±20% tuning on a tiny paired set cannot move the global denominator.

Metric definitions

Same denominators as Phase 2. Phase 3 overall soft and detection exclude EL&ELV when it is honest N/A (no Peak EL PDFs) — not scored as 0%.

Phase 2 vs Phase 3 per sheet

Phase 2 floors are the published Phase 2 report. Phase 3 numbers come from Task V rescore into .tmp/peak-phase3/ and the Task X report. Soft % among soft-eligible pairs; detection and exact on drawable GT.

Goal checklist: overall detection >50% — yes (54.3% weighted). Overall soft >80% — no (60.0% pooled). FS soft >70% — yes (70.7%). P&D detection >50% — no (32.4%). EL detection >30% or N/A — yes (N/A). Detection-first partly worked; the soft >80% miss is driven almost entirely by a material P&D soft regression (−54.6 pp vs published Phase 2).

Root-cause narrative (four failure modes)

Phase 2 evidence mapped cleanly onto four modes. Phase 3 assigned one workstream each — no invented recovery when a gate failed.

• F1 EL routing — Phase 2 scored EL at 0% soft/detection with 266 miss and 49 plumbing ai_only rows because the harness fed P/D/FS/AC PDFs into EL SOR. Peak training has no dedicated EL drawing. Fix: empty EL drawing map, refuse cross-discipline pairing, discipline gate on AI families, metrics null (N/A) and excluded from overall.
• F2 Parent heading (P&D) — bare-diameter SOR children (e.g. “15 mm dia.” under a Gate valve heading) failed description match. Pass 4 enriches with parentHeading and rematches with a diameter gate. Detection rose 23.2%→32.4% but stayed short of >50%; soft on the expanded pair set collapsed (4/23) — pairing honesty without quantity agreement.
• F3 FS noise — Phase 2 soft 60.9% with 53 ai_only (many other). Vocabulary filter + subtype lock dropped low-similarity other labels; soft 70.7%, detection 81.4%, exact 47.5%. Soft >70% goal met.
• F4 Floor scale — partial plans vs whole-building metres were length_scope. When title text implies coverage (e.g. 1 of 5 floors), --scale-floors multiplies AI qty; never scale up when AI already overcounts. P&D stayed 1× (full multi-floor plans); FS/MVAC used 5× on partial roof/elev sets.

What shipped (Tasks R–U)

• R — EL honest N/A: empty EL PDF map; auto-discover only EL/ELV filenames; filterAiItemsForSorSheet + EL family gate; null metrics excluded from overallExcludingNa.
• S — Parent-heading Pass 4: enrichDescriptionWithParent + rematch with diameter gate in ground-truth comparison; unit tests in domain package.
• T — FS extract noise: filterFsExtractNoise drops low-similarity other on FS; subtype guidance hardened; soft/det/exact gains above.
• U — Floor coverage: parseFloorCoverageFromText / computeFloorScaleFactor / applyFloorScaleToAiQty; CLI --scale-floors; OCR pageSummary asks for floor phrases; overcount guard.
• V–X — Full rescore to .tmp/peak-phase3/, regression vs Phase 2 floors (Task W: FAIL on P&D soft only), report docs/PEAK_PHASE3_ACCURACY_REPORT.md. Classification offline still 90.2%.

Remaining gaps + Phase 4 directions

Task W regression gate failed on P&D soft 17.4% vs Phase 2 floor 72.0%. Detection rose on the same AI raw under Pass 4, but soft-eligible quantity agreement collapsed. We do not lower floors or invent recovery. Prioritized research (no implementation claimed here):

• P0 — Root-cause P&D soft collapse: soft-eligible denominators, greedy reassignment, parent-heading false pairs that pass description but fail qty. Optional live re-extract only after pairing diagnostics.
• P1 — Network-aware length: merge connected polylines at junctions; top absolute-Δ failures remain CI / uPVC / GI / refrigerant metres.
• P2 — EL drawing acquisition: Peak still has zero EL PDFs; keep N/A until an EL schedule-bearing drawing lands; never re-poison EL with plumbing families.
• P3 — Legend-seeded symbol clustering for remaining count misses (fittings, valves, gullies).
• P4 — Corrections bank few-shot from top absolute-Δ fails; product KPI: page-comparable soft primary, building-scope linear separate; CI per-sheet min soft and min detection.

Conclusion

Peak Phase 3 is complete as a research cycle. Detection-first cleared weighted detection 54.3% (>50%) with EL honest N/A, FS soft 70.7%, and MVAC soft 80.8% / detection 100%. Overall soft pooled 60.0% missed >80%, almost entirely from P&D soft 17.4% after Pass 4 expanded pairing without holding quantity quality. Classification remains 90.2%. Verification-first means publishing both wins and the regression gate failure. Companion posts: Phase 1 classification /blog/peak-full-sor-audit; Phase 2 quantity /blog/peak-phase2-quantity-audit. Full internal report: docs/PEAK_PHASE3_ACCURACY_REPORT.md.