Reading Drawings Is Not Reviewing Drawings

Every vendor in the document-AI category makes some version of the same claim: our system reads drawings. Reddy's framework points out what that single sentence conceals -- at least four distinct processes, each with its own failure mode: reading (decoding what is on the sheet), comprehension (interpreting one detail correctly on its own), understanding (connecting that detail across sheets, disciplines, and revisions), and application (turning understanding into the right action for the right person in the right form). A fifth level, outcomes, sits downstream: what actually happened on the project, and who is accountable for it.

The framework's sharpest point is that competence at one level says nothing about the level above. A system can decode every symbol and still misread a detail. It can interpret every detail correctly in isolation and still miss the coordination conflict that only exists across sheets. It can catch that conflict and still recommend the wrong action, because it does not know a bulletin already resolved it.

That is the conceptual ladder. What has been missing is empirical footing -- and that is what AEC-Bench quietly provides.

AEC-Bench was not designed against Reddy's framework -- the benchmark predates the essay -- which makes the fit more interesting, not less. When you place its nine task families on the ladder, a clear shape appears:

Two findings fall out of this mapping. First: the benchmark's difficulty curve follows the ladder. Published scores are strongest on intra-sheet (comprehension) tasks, weaker as scope widens to cross-sheet understanding, and weakest on submittal review -- the one task family that approaches application. The best configuration in the original paper scored 23.1% on submittal review; the paper notes agents "tend to over-generate findings," producing false positives. The framework predicted this shape before the data existed.

Second: everything above level three is currently unmeasured. AEC-Bench stops where application begins -- it has no project context, no roles, no trade sequencing, no addendum that already resolved the flag. That is not a flaw in the benchmark; it is a boundary. The problem is that vendor marketing routinely operates two levels above where the evidence sits.

The practical payoff of the mapping is a checklist. When a vendor demos a drawing-review tool, identify which level the claim actually sits at, then ask for the evidence that matches that level -- nothing higher will do.

Level 1 -- Reading

Ask for degradation data, not headline accuracy: scanned sheets, hand-marked redlines, dense overlapping annotations, non-standard symbol legends. Clean CAD-exported PDFs are the easy case and prove little.

Level 2 -- Comprehension

Ask for AEC-Bench intra-sheet scores, or run the tool on single details from your own past sets. This level is genuinely strong today; a vendor who cannot show it here has a problem.

Level 3 -- Understanding

Ask for cross-document scores (AEC-Bench intra-drawing and intra-project families), and run one test no benchmark covers well yet: give the tool a set with an addendum and ask which version of a detail governs. Geometric clash detection does not count as understanding -- knowing what the set fails to show is the harder half.

Level 4 -- Application

No benchmark tests this today, so the only honest evidence is a pilot on one live project, scoped to a named function, role, and method -- submittal review for the architect of record via a hold-point log, not "drawing intelligence" in the abstract. Track the false-positive burden explicitly: an over-flagging tool costs your team the review time it promised to save.

Level 5 -- Outcomes

This level is evaluated by deployment design, not by the tool. Where is the named, accountable sign-off before AI output drives an action? The professional structure already exists -- stamps, submittal logs, RFI approvals -- and it maps directly onto the AIA's risk-tier guidance: the higher the stakes, the more senior the required reviewer, up to the architect of record for anything touching life safety or stamped deliverables. A deployment that designed the sign-off out is a liability, whatever its benchmark score.

AEC-Bench's authors have named their next directions -- 3D model interaction, calculations, takeoffs -- and the benchmark being open means the community will push task coverage upward. Expect level-four-shaped tasks (context-aware action recommendations) within a generation or two of the benchmark. As that happens, the gap between measured capability and marketed capability should narrow, which is good for everyone except vendors relying on the gap.

Until then, the discipline is simple: name the level, demand the matching evidence, and never let a comprehension score sell you an application promise. The ladder is Reddy's. The data is Nomic's. The habit of holding claims against both is yours to build.