The gap in the table

On December 20 ARC Prize published o3's ARC-AGI-1 results: 75.7 percent on the semi-private set in the low-compute setting at about 26 dollars a task, and 87.5 percent at high compute, where a task cost 4,560 dollars. That was the moment the first benchmark stopped being useful as a frontier measure. On March 24 the same organisation launched ARC-AGI-2, and their table now has o3-preview at an estimated 4 percent on the new set at around 200 dollars a task, o1-pro at about 1 percent, and every pure LLM they tested at 0.

The other column in that table is the one that matters. A human panel solved 100 percent of the tasks, in the sense that at least two people got each one, and the average person scored around 60 percent. A benchmark where the best system is at 4 and an ordinary person is at 60 is measuring something models do not have yet. That is the design goal, and it is worth going through how they hit it.

Calibrate on people first

The first choice was to make human solvability a hard inclusion criterion rather than an assumption. ARC Prize tested more than 400 people in controlled sessions, and a task made it into the calibrated sets only if at least two humans solved it in at most two attempts. Tasks that stumped everyone were thrown out no matter how hard they made models look. That rule protects the benchmark from the most common failure of hard evals, which is that they drift into puzzles nobody can actually do.

The sets themselves are 1,000 public training tasks that are not calibrated, then three calibrated evaluation sets of 120 tasks each: a public one, a semi-private one used for testing commercial APIs, and a fully private one for the Kaggle competition. Keeping the semi-private and private splits separate from anything a lab can scrape is what makes the o3 numbers meaningful.

Target the specific failures

The second choice was to write tasks around the ways the strongest systems failed on v1, and the launch post names three. Symbolic interpretation, where a shape has to be treated as meaning something rather than as a pattern to be copied. Compositional reasoning, where several rules interact and have to be applied together. And contextual rule application, where the same visual cue means different things depending on the rest of the grid, and a system that latches onto surface pattern gets it wrong.

The reason this works against o3 in particular is that test-time search over program-like solutions handles single rules well and interacting rules badly. If the space of candidate transformations grows multiplicatively with the number of rules, then brute force sampling at 200 dollars a task stops being enough. The v1 tasks left room for that approach. The v2 tasks were chosen to close it.

Score the cost, not just the answer

The third choice is that efficiency is now part of the score. Every entry in the leaderboard has a cost per task next to it, and the grand prize of 700,000 dollars, out of a one million dollar pool, requires 85 percent on the private set within the Kaggle compute limits. ARC Prize puts the human panel at 17 dollars a task, which reflects what it cost to collect the data, and estimates the true floor at 2 to 5 dollars.

I think this is the most transferable idea in the whole design. o3's 87.5 percent on v1 came with a 172-fold compute multiplier over the 75.7 percent run. A benchmark that reports accuracy alone lets that difference vanish. One that plots accuracy against dollars makes it the whole story, and it makes the December result look like what it was: a demonstration that you can buy a high score, at a price nobody would pay in production.

What still worries me

The obvious risk is the one that got v1. The public training set is large, the format is stable, and the last winner, the ARChitects team, scored 53.5 percent on v1 with test-time training at 25 cents a task. They are at 3 percent on v2 today, but an open competition with a million dollars attached is a strong incentive to find whatever regularity the task authors did not notice. If v2 falls in a year the way v1 did, the question will be whether it fell to general reasoning or to a better v2-specific trick.

What I would like to see, and what the arXiv paper's human data makes possible, is a report of per-task human pass rates alongside model pass rates over the year. If models climb on the tasks people find easy and stay flat on the ones people find hard, that is a different story from the one where both move together, and the difference is the thing a benchmark like this exists to detect.

Sources

  1. ARC Prize, Announcing ARC-AGI-2 and ARC Prize 2025
  2. Chollet, Knoop, Kamradt, Landers and Pinkard, ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems (arXiv 2505.11831)
  3. ARC Prize, OpenAI o3 breakthrough high score on ARC-AGI-Pub