Co-Design Bench asks whether coding agents can turn real users’ multimodal requests into working revisions of existing web applications. The arXiv preprint (2610.35138) was submitted on October 26, 2026, at 18:10:21 Asia/Shanghai. Authors are affiliated with Fudan University, Alibaba Token Hub / Alibaba Group, Zhejiang University, CUHK, Shanghai Jiao Tong University and NTU.

The benchmark contains 273 tasks and 2,331 hidden expert acceptance criteria, drawn from 37,463 deduplicated queries across 26,314 coding sessions. Multimodal Editing covers 100 localized edits; Repository Editing covers 173 coordinated multi-file revisions.

Two averages with different denominators

Under a shared Claude Code harness, the strongest of ten endpoints, gpt-6-astra, reaches:

  • Strict Completion 69.9%: for each task, count the share of criteria judged fully PASS, then average those task shares with equal weight across all 273 tasks. This is an average of per-task criterion-pass rates—not the share of tasks that succeed, and not a micro-average over all 2,331 criteria mixed together.
  • Full-Pass Rate 26.4%: the share of tasks in which every criterion is PASS. PARTIAL does not count as full pass.

That gap is the point of the paper: an agent can satisfy most checklist items on average while still leaving unfinished requirements on most tasks. A separate Graded score gives PARTIAL half credit; the best combined graded figure is 75.3%, which should not be mixed with the 69.9% Strict figure. Track-level Strict / Full-Pass figures for the same endpoint are ME 75.4% / 41.0% and RE 66.7% / 17.9%; the combined numbers weight the 100 and 173 tasks accordingly.

Open release scope

The project site links a Hugging Face release described as 96 ME tasks / 521 criteria. The paper still reports 100 ME tasks and 553 ME criteria. The site examples are archived project runs.

Main comparisons hold the harness fixed—up to three hours and 120 turns—and use a judge-then-refute protocol. Two frontend experts reviewed 200 task–model outputs; that is not a full human re-check of all 2,730 pairs. Harness sensitivity elsewhere in the paper also shows model×framework interactions under that shared setup.

For nearby discussion of how harness choice changes measured results, see what a coding harness buys and DeepSeek’s harness-dependent score gaps.

We haven’t independently reproduced Co-Design Bench or re-run its endpoints.