Peak Full-SOR Accuracy Audit: 512 Items × 18 Pages — Classification from 41% to 90%

ERRATUM (2026-08-06): the annotated drawing images previously shown in this post were not real. scripts/annotate-peak-audit.ts contained an offline demo branch that painted boxes on a hard-coded grid and drew "pipe runs" whose position was a hash of the description text, and it ran even in live mode because the vision calls returned empty and the empty result was cached. The captions describing them as matches, mismatches and misses were therefore incorrect. The images have been withdrawn; a corrected renderer and regenerated imagery are described in Peak Phase 5. All numeric results in this post came from the scored benchmark artefacts, not from those images, and are unaffected.

RESEARCH BENCHMARK — NOT A LIVE PRODUCT SLA: Classification (41.4%→90.2%) and the Detection / Exact / Soft columns in this post are offline or research-benchmark results against Peak-SOR, not a shipped accuracy guarantee. The annotated gallery imagery erratum above remains authoritative — withdrawn overlays must not be reintroduced or cited as evidence. For fixed-denominator quantity honesty (QA@20 / Detection on all drawable rows), see /blog/peak-phase5-measuring-what-we-ship.

A single accuracy percentage is the wrong product goal — but a disciplined, sheet-by-sheet audit against a real priced tender is how you earn the right to improve. This post documents the Peak full-SOR benchmark: all 512 quantity rows across four discipline sheets in Peak-SOR.xlsx, evaluated against 18 pages from The Peak training drawings (plumbing P, drainage D, fire services FS, air-conditioning AC). Starting from a Phase 1 offline baseline of 41.4% classification, Tasks F–J raised classification to 90.2% overall, shipped cross-page aggregation, and measured 97% live detection on MVAC. Task K produced the annotated page gallery below.

Methodology

The benchmark script (scripts/benchmark-peak-full.ts) runs in two phases. Phase 1 is offline and deterministic: parse Peak-SOR.xlsx, classify every row with the BS-standard-driven takeoff family classifier, and report per-sheet and per-family rates — no vision API required. Phase 2 (when GEMINI_API_KEY is set) renders drawing pages, runs drawingExtract + lengthTakeoff, aggregates AI items across pages (Task J), then compares against ground truth with compareAiToGroundTruth(). Provisional rows (temporary works, shop drawings, T&C) are counted separately: they are correctly non-drawable and should not be forced into symbol families. Numbers in this post prefer Phase 1 classification for all four sheets, plus Task I live MVAC detection; we did not re-run a full multi-sheet Gemini live pass for this write-up.

• Ground truth: Peak-SOR.xlsx sheets EL&ELV (266), P&D (146), FS (64), MVAC (36) — qty>0 rows only.
• Drawings: 18 pages — P (5) + D (6) + FS (5) + AC (2); discipline routing via filename prefixes.
• Task J aggregation: same equipment tag → max qty; same pipe family+diameter → sum metres; legend vs plan dedup heuristics.
• Annotated PNGs were withdrawn on 2026-08-06 — see the erratum above. The generator (scripts/annotate-peak-audit.ts) has been replaced by scripts/annotate-benchmark.ts, which has no offline demo mode.

Per-sheet results

Classification is the Phase 1 headline metric (family assigned, not “other”) — it does not measure whether AI quantities match SOR quantities. Detection / Exact / Soft columns below are filled from the completed Peak Phase 2 quantity audit (Tasks L–P). Soft % is among soft-eligible pairs after length_scope exclusions — not every SOR line. Full write-up: /blog/peak-phase2-quantity-audit.

Baseline (pre Tasks F–I) was 41.4% overall (212/512). After classifier expansion: 90.2% (462/512), with 49 provisional and only 1 residual unclassified row. Every discipline exceeded its ≥70% (MVAC ≥85%) classification target.

Before vs after classification

Top families by SOR item count

Phase 1 family landscape across all 512 rows (classification coverage). Detection columns for multi-sheet live extract are intentionally deferred; MVAC equipment/pipe families were validated live under Task I.

Annotated page gallery

This section previously showed representative pages with overlay annotations, described as marking matches, quantity mismatches and misses. Those images were fabricated by a demo code path and have been withdrawn — see the erratum at the top of this post. Corrected, reproducible imagery is in Peak Phase 5.

Item-level analysis — what works, what does not

• Works: BS/DIN-driven pipe families (GI, uPVC, CI, copper, refrigerant, condensate) with Chinese + English keywords and unit gates.
• Works: Electrical board / isolator / cable / conduit families after Task F — 85.7% EL&ELV classification vs 33.5% baseline.
• Works: P&D fittings/valves/gullies/fixtures with parent-heading context — 97.3% classification.
• Works: FS pumps, alarm panels, hose reels, hydrants, sprinklers — 92.2% classification.
• Works: MVAC outdoor tags + FCU kW + refrigerant/condensate mm-pass + derived controllers (Task I) — 97% live detection.
• Works: Cross-page aggregation (Task J) — equipment max-dedup, pipe length sum, legend vs plan heuristics reduce false positives.
• Hard: Exact quantity on multi-page buildings — 2 AC pages cannot reconstruct whole-building pipe metres; soft ±20% is more honest than exact match.
• Hard: Dense EL symbol grids (70 luminaires) — VLM counting remains weak; legend-seeded template matching is the next research path.
• Hard: GI ductwork area (m²) — not present as qty>0 on Peak MVAC; future SORs need hatch-region area takeoff, not lengthTakeoff.
• Residual: 1 unclassified EL row (“20mm thick wooden”) + 49 provisional — ceiling without redefining provisionals is ~90.4%.

Deep dive: EL&ELV (Task F)

Largest sheet (266 items). Baseline classification was 33.5% — MCCBs with bare amp ratings, changeover switches, conduits, trunking, and light fittings fell through to “other”. Task F expanded families: luminaire (70), distribution_board (43), switch_isolator (27), electrical_cable (27), conduit (18), circuit_breaker (10), cable_tray (9), junction_box (9), plus data_outlet, earthing, busbar, socket_outlet, transfer_switch. Result: 228 classified (85.7%), 37 provisional, 1 unclassified.

Deep dive: P&D (Task G)

Highest classification rate of the four sheets: 97.3% (142/146). Parent-heading context unlocked bare-diameter fittings; sanitary fixtures, manholes, gullies, and valves gained dedicated families. P + D drawings both feed the same SOR sheet — Task J sums pipe lengths across drawings and takes max for identical fixtures when they are duplicates.

Deep dive: FS (Task H)

Fire services rose from 35.9% to 92.2% classification. Families cover GI fire mains, fittings, fire pumps (including jockey/xFJP patterns), valves, alarm panels, manual call points, hydrants, sprinklers, extinguishers, detectors, and hose reels. Length items still need geometric takeoff; count items need legend-aware multi-page sum (Task J).

Deep dive: MVAC (Task I)

Classification ceiling held at 91.7% (33/36) — the three provisional rows (temporary power, shop drawings/O&M, testing & commissioning) are correctly non-drawable. Live Gemini on two AC pages reached 97% detection after Task I fixes: outdoor CU tag recall, 100% refrigerant/condensate mm-pass, 100% FCU kW bins, and derived central controller + control wiring + slim duct cover. Exact qty accuracy remains ~39–42% because two pages do not contain whole-building pipe runs; soft ±20% accuracy ~50–55% is the fairer operational metric until multi-floor aggregation is complete.

Task J — cross-page aggregation

Without aggregation, the same outdoor unit on a legend page and a plan page becomes a false double-count; pipe runs on consecutive pages under-count if not summed. packages/domain ships aggregateAiItems() with: (1) equipment tag + family → max quantity; (2) pipe family + diameter → sum metres; (3) count items by normalized description → sum; (4) legend vs plan heuristics; (5) P+D multi-drawing rules. The benchmark logs “Before aggregation: X → After: Y” and merge methods for audit.

Next research directions

• Legend-seeded template matching for dense EL symbol grids (luminaires, sockets) — crop one legend instance, match connected components by shape similarity.
• Area measurement for ductwork (m²) — hatch-region / areaEstimate path when future SORs include GI duct area rows.
• Fitting detection via parent-heading context on live extract (not only classification) — diameter tokens alone are ambiguous without section context.
• Multi-scale extraction — zoomed tiles for small symbols that vanish at full-page OCR resolution.
• Cross-reference P and D drawings for P&D sheet with network-aware polyline merge at junctions.
• Phase 2 quantity audit completed — detection + exact/soft published for all four sheets; see /blog/peak-phase2-quantity-audit (soft ~68.6% overall among soft-eligible pairs).

Conclusion

Peak full-SOR classification moved from 41.4% to 90.2% across 512 items — well above the 70–80% classification target — while MVAC live detection reached 97% with honest soft accuracy on multi-page lengths. The residual gap is no longer “unknown family,” but vision counting density, whole-building length reconstruction, and area takeoff for ductwork. Annotated pages make that gap inspectable: green and yellow overlays show what the model saw; red markers show what a QS still needs to catch. Verification-first means publishing both the wins and the red markers.