Industry data · Updated July 2026
AEC AI Benchmark Tracker
The industry finally has a common, reproducible test for AI agents on real construction documents. This page is an independent record: what the benchmark measures, who has published results, and how to read the numbers before trusting them.
What AEC-Bench is
AEC-Bench is the first open benchmark for evaluating AI agents on real architecture, engineering, and construction coordination work — released March 2026 by Nomic (authors: Harsh Mankodiya, Chase Gallik, Theodoros Galanos, and Andriy Mulyar). It contains 196 task instances across nine task families, built on real 2D drawing sets and specifications, scored automatically against ground-truth defects placed by domain experts.
An agent gets the documents, works the task, and either gets it right or does not (scored Pass@1 — first attempt, no retries). The entire benchmark — dataset, agent harness, and evaluation code — is released under an Apache 2.0 license, which means any vendor, firm, or researcher can run it and publish results. That is what makes it matter: for the first time, "does this AI actually work on a real drawing set?" has a common test instead of a vendor demo.
What it tests
Three complexity scopes, from reading one sheet correctly to coordinating across an entire document set. If you have ever spent an evening tracing a detail reference across a 300-page set, you will recognize these tasks.
| Scope | Task family | Tasks | What it tests |
|---|---|---|---|
| Intra-Sheet | Detail Technical Review | 14 | Localized technical questions about a single detail view |
| Intra-Sheet | Detail Title Accuracy | 15 | Whether detail titles match what is actually drawn |
| Intra-Sheet | Note Callout Accuracy | 14 | Whether callout text corresponds to referenced elements |
| Intra-Drawing | Cross-Reference Resolution | 51 | Finding cross-references that point to non-existent targets |
| Intra-Drawing | Cross-Reference Tracing | 24 | Tracing all locations that reference a given detail |
| Intra-Drawing | Sheet Index Consistency | 14 | Checking sheet index entries match actual sheets |
| Intra-Project | Drawing Navigation | 12 | Locating the correct file, sheet, and detail for a query |
| Intra-Project | Spec-Drawing Sync | 16 | Finding conflicts between specifications and drawings |
| Intra-Project | Submittal Review | 36 | Evaluating submittals for spec and drawing compliance |
Task taxonomy per the AEC-Bench paper and published materials. The benchmark covers 2D drawings and specifications only — no 3D/BIM model interaction, quantity takeoff, or engineering calculations yet; the authors have named those as future directions.
Published results
Nomic (benchmark authors)
March 31, 2026Baseline results across several agent harnesses, including Claude Code and Codex configurations, published in the AEC-Bench paper. Aggregate scores by complexity scope; key finding is that domain-specific harness design improves performance more than model size. Best single-configuration score on submittal review, the hardest task family: 23.1%.
Structured AI
April 13, 2026Self-reported 84.7% overall across all 196 tasks, claiming the highest score in every complexity scope. Standout claim: 75.0% on submittal review versus the paper's best 23.1%. Per-task methodology published on their blog.
Know of a published AEC-Bench result we have missed, or want to submit yours? Email nate@aechub.org with a public source. We list results with attribution and status; we do not rank what we have not verified.
How to read these numbers
Self-reported is not verified. The benchmark is open, which cuts both ways: anyone can run it, and anyone can tune against it. A vendor reporting its own score on a public test set deserves more credit than a vendor with no benchmark at all — and less than an independently replicated result. We label each entry accordingly.
High scores still mean real error rates. An 84.7% overall score is genuinely impressive against the baselines — and it still means roughly one task in seven comes back wrong. That is useful the way a second pair of eyes is useful: run it after your reviewer, not instead of your reviewer. It is not yet the reliability you would stake a submittal approval on unsupervised.
The interesting result is the shape, not the headline. The paper's core finding is that harness design — how the agent navigates documents, what tools it has, how it verifies itself — moves scores more than raw model size. And the hardest tasks are exactly the cross-document ones (submittal review, spec-drawing sync) where a coordinator spends days and errors carry the most consequence. Watch the intra-project scores, not the averages.
Use it in procurement. If you are evaluating document-AI tools, you now have a sharper question than "can we see a demo": ask for the vendor's AEC-Bench score and how it was produced. A vendor who has not run a free, open, replicable benchmark relevant to their product is telling you something. Pair the answer with your own workflow audit so you know what the workflow is worth before you price the tool.
Why we track this
AEC Hub is independent — no vendor pays for placement, and benchmark listings are not endorsements. We track published results because evidence beats demos, and because the industry deciding its AI spend deserves a scorekeeper with no score in the game. As more vendors publish, this page will grow; independent replication is on our research roadmap.