ARC-AGI-2 Explained: How Tiny Colored Grids Test Whether AI Can Think on the Spot
Most AI tests reward a good memory. If an AI model reads millions of textbooks, websites, and code files during training, it can answer hard history or chemistry questions largely by recalling similar examples it already saw.
ARC-AGI-2 ("Abstraction and Reasoning Corpus for AGI, version 2") strips all of that memorized knowledge away. It is a visual logic benchmark that shows an AI 2 to 4 tiny colored grids with Before-and-After examples and zero written instructions, then hands it a brand-new grid and asks what the output grid should look like. Every puzzle is built so a regular person with no special training can figure it out in a couple of minutes, yet an AI cannot look up the answer from memory.
1. Why the Original Puzzle Test Needed an Upgrade
Software researcher François Chollet introduced the original ARC-AGI test in a 2019 paper (arXiv:1911.01547). His goal was to measure fluid intelligence: how quickly a person or computer figures out a brand-new rule from scratch, rather than repeating a practiced skill.
For five years, that first version (ARC-AGI-1) stumped almost every AI model:
- 2019 to early 2024: Standard chat models scored under 20% because the puzzles never appeared in public training data, and predicting the next word did not solve multi-step visual rules (arXiv:2505.11831).
-
Late 2024 (
o3breakthrough): OpenAI's previewo3model jumped to 76% at standard settings (~$200 per puzzle) and 88% at high compute by spending roughly $20,000 per puzzle in test-time compute to generate and check thousands of candidate rules (arXiv:2505.11831).
That result exposed a weak spot in ARC-AGI-1: many of its older puzzles only used a single simple rule (like flipping a shape or filling an outline), so a computer could crack nearly half of them through brute-force search if you threw enough computing money at the problem. When the ARC Prize Foundation launched ARC-AGI-2 in early 2025 (arXiv:2505.11831), accuracy on the exact same o3 medium model dropped overnight from 53.0% down to 3.0%, and every other AI model scored below 5%.
2. What Makes ARC-AGI-2 Easy for Regular People Yet Hard for AI
The ARC Prize Foundation designed ARC-AGI-2 around one rule: easy for everyday humans, hard for AI.
- Tested on 400+ Regular People: Researchers ran controlled in-person sessions in San Diego with more than 400 members of the general public across all kinds of jobs. Every single puzzle kept in the final benchmark was solved by at least two non-expert testers in two tries or fewer, with zero prior training (arXiv:2601.10904), and an average person solves roughly 66% of the puzzles they attempt.
-
Multiple Rules Working Together: Instead of one global change across the whole board,
ARC-AGI-2stacks two or three visual rules that depend on context. In official taske3721c99(Figure 1 of arXiv:2505.11831), colored boxes with 1, 2, or 3 holes act as a color key telling you how to paint matching gray shapes elsewhere on the board. In official task00576224shown below, the solver has to spot both a 3x3 tiling rule and a horizontal mirror flip on the middle two rows at the same time. - Separate Task Buckets to Block Memorization: To stop data contamination and overfitting, the dataset splits puzzles into public practice and evaluation sets on GitHub, 120 semi-private leaderboard tasks used to test commercial API models, and a hidden held-out set of 120 fully private competition tasks that never touch the internet (arXiv:2601.10904).
00576224: the solver sees only colored grids and must spot both the 3x3 tiling and the middle-band horizontal flip. Source: ARC Prize Foundation (arcprize/ARC-AGI-2), Apache 2.0 License.
3. How One Puzzle Run Works Step by Step
Every ARC-AGI-2 task uses grids sized anywhere from 1x1 up to 30x30 squares, with 10 possible colors stored as the numbers 0 through 9 (arXiv:2505.11831). Here is what happens during one live inference run:
- Hand the AI 2 to 4 Before-and-After grids with zero hints: Inside an offline sandbox with internet access turned off, the grading system sends a small JSON file containing few-shot examples (pairs of input and output number grids) plus one new test input grid, as opposed to a zero-shot question with no examples.
- Work out the shared pattern: The AI uses chain of thought (reasoning tokens) to compare each Before grid against its After grid, checking object shapes, symmetry, hole counts, movement, and layering until it finds one rule that fits every example pair.
-
Apply that rule to the unseen test grid: To prove real generalization, the AI must output both the exact height and width of the answer grid and the exact color number (
0to9) of every single square. -
Grade it all-or-nothing under pass@2: Just like the human volunteers in the San Diego study, the AI gets 2 attempts per test grid. If either attempt matches every square 100%, it earns 1 point. If a
20x20grid has 399 right squares and 1 wrong square, it gets 0 points (arXiv:2505.11831).
ed36ccf7 rotates each 3x3 shape 90 degrees counter-clockwise. Language models receive raw 0..9 JSON arrays rather than an image, and rotating clockwise scores 0%. Source: ARC Prize (ARC-AGI-2), Apache 2.0 License.
4. Two Catches Behind Published Logic Scores
When you see an ARC-AGI-2 percentage on a model card or leaderboard, two practical details explain why numbers jump around so much:
-
Thinking Bills Can Swing 30x on the Same Model: A raw model answering in one pass scores much lower than the exact same model wrapped inside a refinement harness that lets it test and revise candidate rules in a loop. In the ARC Prize 2025 Technical Report, adding an outer refinement loop lifted one commercial model from 31% accuracy at $0.81 per puzzle up to 54% accuracy at $31.00 per puzzle. Meanwhile, in the strict offline Kaggle contest capped at $0.20 per puzzle on four NVIDIA L4 GPUs over 12 hours, the 1st-place entry (
NVARC) reached 24.03% (arXiv:2601.10904). Tiny custom neural networks with just 7 million parameters have also beaten giant chat models on fixed-budget runs by refining guesses repeatedly. -
Language Models Read Grids as Numbers, Not Pictures: Text models read rows of comma-separated digits like
[0, 0, 5], which makes vertical and diagonal shapes easy to misread. Giving a model a Python command-line scratchpad for program synthesis (so it can write a short Python script, test that script against the example grids inside a scaffold, and fix mistakes) raises scores sharply: in a 2026 evaluation study, adding a live Python scratchpad jumped an open-weight model from 6.1% to 26.4% and lifted a frontier model from 59.8% to 73.4% using the exact same prompt.
5. Where Models Stand Today (and What Score You Need)
Over the past year, frontier reasoning models have climbed well past the 66% average person mark on BenchLM's ARC-AGI-2 tracker and Modello's Math & Unseen Logic rankings (which blend ARC-AGI-2 with PhD holdout exams):
-
Top Unseen Puzzle & Logic Leaders: GPT-6 Astra leads raw
ARC-AGI-2puzzle solving on Modello at 95.0% (81.1% blended Logic, $5.32 per finished LiveBench 2.0 task, included in $20 ChatGPT Plus). Claude Fable 5.2 leads overall blended Math & Logic at 91.2% ($9.85 per task), and Gemini 4 Carbon scores 84.8% Logic at $2.02 per task on current intro rates ($4.04 standard). - Strong Everyday & Free Options Above the Human Average: Claude Opus 5.5 hits 80.5% Logic on a $20/month subscription, and Claude Sonnet 5.5 reaches 77.0% Logic on a Free Tier, comfortably clearing the 66% human puzzle baseline at zero monthly cost.
-
A Simple Rule of Thumb: If your work involves tricky multi-step logic, math proofs, financial modeling, or untangling messy rules with no template to copy, set your
Min Math & Logicslider to 78%+ in Set Your Bar. For everyday writing, summarizing documents, or routine coding, paying extra thinking-token bills for a 90%+ puzzle solver rarely changes the answer you get back.