Models: which ones build, and what a working build costs

QuorX runs any model through OpenRouter, so the useful question is not "which model is best" but "which model builds a given job, for how much". This report answers it with numbers from QuorX's own benchmarks, and shows the same numbers for the replies run on this computer, from its logs.

How models are compared

The build benchmark gives every model the same tasks from bench/tasks-build.json: games, a dashboard and forms (snake with portals, a sales dashboard, an invoice builder, 2048, a kanban board and more). Each run goes through the real QuorX agent in a throwaway folder, with its plan, its checks and its reviewer, exactly as in a chat. Then a judge model, which sees only the request, plays the build in a headless browser and scores it from 1 to 5 on function, craft, design and originality. A build passes when all four reach the bar (3 by default).

The numbers that matter for choosing a model:

• Pass rate: the share of runs whose build passed.
• Cost per working build: everything the runs cost, divided by the builds that passed. A cheap model that fails most tasks can cost more per working build than a dearer one that passes.
• Steps and minutes: how long the agent worked on each build.

The builders tried are listed in bench/models-build.json, from free models to frontier ones. Running the benchmark spends real money on OpenRouter, so it only runs when someone chooses to, with a spending cap.

Leaderboard

From 4 build runs (bench-build-2026-10-20T23-13-27.json, bench-build-2026-10-21T00-03-03.json). Each run is one task from bench/tasks-build.json, built by the full QuorX agent in a throwaway folder; a judge model then plays the build and scores it. A build passes when function, craft, design and originality all reach the bar.

The decider experiment

QuorX started from one question: can a cheap model get stronger by asking a quorum? Next to the model that writes the reply, QuorX keeps a panel of deciders: other models it can ask a short question, such as "the page shows a start button" or "the answer is 42". Each decider gives its own probability; the panel's answer is their combined vote, and a split or unsure panel says so instead of guessing.

The agent asks with its own guess and how sure it is. Comparing the guess with the panel's answer sorts every question into one of four buckets:

The known-answer benchmark (bench/tasks.json and bench/tasks-hard.json: trick questions, traps and puzzles with one right answer) measures it directly. The same small model answers alone (setup A), asks the deciders first (setup C), or writes three drafts for the deciders to pick from (setup D); a frontier model alone (setup E) shows the ceiling. If C or D closes the gap to E for much less money, the quorum works.

The decider experiment: the numbers

No run of the known-answer benchmark has been published here yet. It compares the same small model alone (setup A) with the small model asking the deciders (setup C), three drafts picked by the deciders (D), and a frontier model alone (E). It costs money too:

> node scripts/bench.mjs --setups A,C,E --limit 10
> node scripts/news-data.mjs
> 

On this computer

These numbers come from this computer only: every reply QuorX ran here is logged in logs/runs.jsonl, and every question the deciders answered in logs/decisions.jsonl, logs/checks.jsonl and logs/approvals.jsonl. They never leave the computer and are never written into News. Below: the models used here over the last 90 days (the pass rate counts replies whose checks before the final answer passed), then what asking the deciders changed.

What QuorX spent here, per day and per model, over the last 30 days: