Which models can actually do CAD?

2 models, 6 tasks, 36 attempts. Each attempt is graded on whether it built, on measured geometry, and on a checklist an AI judge answers from the renders. Click any number to open the attempt behind it.

Quality

against

Sonnet 5.5 sits alone in the top-left corner: first on strict pass and the cheapest to run. The share of attempts that got everything right: built, every gate, every measured check, every AI judge criterion.

These tasks were hand-picked

We chose these six tasks because they were the lowest-scoring hard tasks in an earlier run. That makes this a test of the hardest cases, not a fair overall ranking, and its 79 vs 64 overstates the gap between the two models.

For the overall comparison, see the run over all 29 hard tasks, where Sonnet 5.5 scored 92 and Kimi K3.1 Pro 89.

All models

Rank by

#	Model	Quality	Cost and time	Overall	Strict	Gate pass
1	Sonnet 5.5Anthropic			79	22%	89%
2	Kimi K3.1 ProModal			64	0%	67%

Overall gives partial credit at every layer, so the top of the board bunches. Strict counts only the attempts that got everything right. Gate pass is the share whose part was watertight, hole-correct and overlap-free. Hover any bar for its number. Ties break by cost, then time.

Browse the tasks

Hard

How the score is made

Built 30%, measured geometry 50%, AI judge 20%. A failed gate zeroes geometry for that attempt. The AI judges are Opus 5.5 and GPT-6 Sol. Read the methodology.

This is an older run. See the current results.

Run kimi31-vs-sonnet2026-11-05commit 0f156dfaspend $12.07 + $2.76 AI judgingOther runs: 2026-11-05T06-56-21-kimi31-sonnet-hard, 2026-10-16T18-49-10-hard-compact-2026-10-16, 2026-09-23T06-56-16-hard-baseline-2026-09-23, 2026-09-10T19-44-23-baseline-2026-09, 2026-09-04T20-02-10-stage2-hard-cases, 2026-09-04T19-48-31-screen-14-fixed