How we score a model on CAD

Every number on the leaderboard comes out of the same pipeline. A model is handed a plain-English prompt for a parametric part, it drives the same tools a user's chat session would, and whatever it builds is measured, rendered and judged the same way for every model in the run. This page is the rest of that pipeline.

What a score is

A trial's score blends three signals: process 30%, geometry 50%, AI judge 20%.

• Process asks whether the turn ended with something built at all. No build means zero on everything downstream. Each failed build along the way costs 15% (max(0, 1 − 0.15 × failed builds)), so a model that thrashes before recovering scores below one that builds cleanly.
• Geometry is a set of deterministic checks the case declares: bore diameters, hole counts, wall thickness, whether two bodies touch. They are measured off the built solid, not asked of a language model.
• AI judge is a checklist of 5 to 10 yes/no criteria specific to the case, answered by a language model from the same renders a person would look at, and scored as the fraction answered yes.

Strict pass

The weighted score gives partial credit at every layer, so once every frontier model can do most of a task the averages bunch near 100 and stop separating them. Strict reads the same evidence as a pass or a fail: an attempt counts only if it built, passed every gate, passed every measured check and passed every AI judge criterion. An attempt that errored or timed out is a strict fail, because that is what a user would have got. Hover it on the board for the cost per solved task. Strict does not decide the ranking, and a run published before attempts carried itemised checks shows a dash.

Human review

A second set of numbers comes from one reviewer looking at the parts. For each task the models' first attempts are laid side by side and dragged into order, ties allowed, with a line below which the reviewer would not accept the part as an answer. A model's rank score on a task is 1 − (position − 1) / (n − 1), so first place scores 100% and last 0%. Human rank is the mean across tasks. The reviewer can also answer the AI judges' own criteria, which is what feeds the agreement figures on the model pages. Human review never affects Overall or Strict, and a model shows "n of N" until every ranked task is covered. This is one person's judgement, not a panel's. So far the reviewer has ordered 54 of 54 ranked tasks and recorded a verdict on 275 of 275 first attempts.

Gates

Before geometry is scored a build has to pass three gates: watertight (no gaps in the mesh), holes (the number of through-holes matches what the part should have), and overlaps (no two bodies interpenetrating). One failed gate zeroes the geometry component for that attempt, even if every dimensional check would have passed. A part that leaks air is not a slightly worse version of a correct part, and averaging it in with the passing checks would hide that. Gate pass on the board is the share of a model's attempts that cleared all three.

The AI judges and their criteria

This run was graded independently by Opus 4.8 and GPT-5.6 Sol, given byte-identical inputs. A criterion passes only when both AI judges agree. Where they disagree it counts as failed and is flagged, so a reader can see where the two split rather than have it averaged away. Each case defines its criteria up front, before any model attempts it, as a fixed list of observable facts rather than a free-text rubric an AI judge could read differently from one run to the next.

Reference cases

Some tasks hand the model a real file instead of describing the part in words: a mesh exported from another application, or a photograph of an object on a desk. Each file is checked in with its provenance, and that provenance is printed beside it. 15 of them are in this run. The AI judges see that input first and answer the same five resemblance criteria on every such case: the same kind of object and silhouette, proportions that agree, every major feature present, those features in the right place, and nothing invented. Where the task asks for a change to what the reference shows, the shape criteria name that change, so obeying the prompt cannot cost a model points.

Tiers

Cases carry one of three difficulty tiers. They group the task list and decide how many attempts each case gets; they no longer split the board, because a difficulty label answers a question about the task rather than about the model. Hard cases are built to a harder standard: spec-sheet fidelity graded against a stated datasheet, multi-turn edits where every earlier constraint has to survive the last one, and parametric parts rebuilt at both ends of a control's range. Each case is attempted 2 or 3 times, by tier. This run covers 55 tasks (20 easy, 10 medium, 25 hard).

Cost and time caveats

Cost comes from the same published, versioned token-rate catalog the production billing ledger uses, applied to the tokens each trial consumed. These are reporting estimates, not invoice reconciliation, and they exclude credits, taxes and negotiated billing. Where cache accounting is not fully broken out the estimator prices the write at the more expensive of the two cache lifetimes, so the figure errs high. AI judge spend is shown separately: $339.70 candidate and $48.02 AI judging, with 7 attempts unpriced. 0 of them errored before any usage was recorded, and the rest ran on a model with no published rate. The cost column is a mean over the attempts that carry a cost, and it says how many attempts it covers whenever that is fewer than all of them.

Time is the wait on the model, summed over an attempt's rounds and meaned over the attempts that finished. It leaves out the local build and render time between rounds, which contends with every other trial under the harness's concurrency and so measures the machine rather than the model. Neither cost nor time is part of the weighted score, but they break a tie on score: cost first, then time. Treat time as a regression signal for the pipeline rather than a promise about response time in the product.

The robustness case

One task on the board, create/clamshell-cover, is marked "robustness only, not ranked" and excluded from every ranking column. It asks for a large, detailed part in a single response, and what it measures is whether a model can finish that response at all. Several models have emitted their entire output budget in one burst and built nothing, three attempts out of three, which is a different failure from getting the part wrong. Once a model does finish, its process and geometry scores land at 1.0 regardless of vendor, while the AI judge still marks it down for two details a few pixels across that the wall-thickness check already confirms are present. It is useful for finding models that fall over on a long response and not useful for ranking the ones that complete it.

Calibration log

Case checks are written from nominal dimensions before any model has attempted them. A check that fails every model on an otherwise correct part is read as a bad tolerance, not a bad model. Corrections are logged rather than edited in place: