LiveBench 2.0 Explained: How Fresh Work and Real Task Costs Test AI
Two problems quietly ruin most AI leaderboards. First, once a fixed test sits on the public internet for a few months, newer AI models read the questions during training and score high from memory. Second, almost every chart reports an accuracy score while hiding whether a model spent 20 cents or 10 dollars of computing time to finish the job.
LiveBench 2.0 tackles both problems at the same time. It is a regularly refreshed benchmark that tests AI models on newly created tasks across coding, web browsers, slide decks, research reports, data tables, and logic puzzles, grades their outputs automatically without biased AI judges on objective tasks, and publishes the exact dollar cost per successful task right beside every score.
1. Why Static Tests and AI Judges Kept Getting Gamed
Researchers from NYU, the University of Maryland, USC, UIUC, and Abacus.AI (including Colin White, Samuel Dooley, Tom Goldstein, and Yann LeCun) introduced the original LiveBench in a June 2024 paper (arXiv:2406.19314, accepted at ICLR 2025) to fix two recurring flaws in older evaluations:
-
Public Question Leakage: Static tests posted on GitHub or academic sites quickly run into data contamination once web crawlers sweep those pages into future training data. To block memorization, LiveBench built tasks from freshly published math competitions (
AMC12andAIME), recent arXiv abstracts, new movies, and newly released datasets on Hugging Face, refreshing questions over time. -
Biased AI Judges: Many open-ended chat boards rely on LLM-as-a-judge scoring, where a model like
GPT-4reads two answers and picks a winner. In Section 4 of the LiveBench paper, the authors showed that AI judges suffer from verbosity bias (rewarding padded bullet lists), favor their own model family's phrasing, and even marked a valid, compiling C++ program wrong because its algorithm looked unfamiliar to the judge.
To avoid judge mistakes, LiveBench 1.0 restricted its 6 original categories (Math, Coding, Reasoning, Language, Instruction Following, and Data Analysis) to questions with one verified ground truth answer, checked by deterministic grading scripts. Over two years, however, models got so good at isolated 1-turn questions that by June 2026, 18 major models bunched into a narrow 12-point band between 71.6 and 83.4 (LiveBench 2.0 release analysis).
2. What Changed in LiveBench 2.0: Grading Finished Work
In October 2026 (with the current leaderboard release dated November 2026 on livebench.ai), the team froze version 1.0 and launched LiveBench 2.0. Instead of short standalone quizzes, LiveBench 2.0 grades 471 items across 7 categories centered on finished work deliverables:
- Long-Horizon Agentic (18 tasks, 2x weight): Multi-hour long-horizon tasks where an AI agent extends an application across several rounds of changing requirements or operates an existing codebase over simulated weeks, graded deterministically.
- Coding (29 tasks, 1.5x weight): Real software bug fixes taken from merged pull requests on a frozen repository, where breaking any existing feature scores 0.
- Documents & Design (25 tasks, 1.5x weight): Building formatted slide decks, PDFs, and structured documents, graded automatically on layout and file-structure checks of the finished file.
- Browser Use (25 tasks, 1x weight): Multi-step browser use workflows inside live web apps. Because the agent must read screen captures to click and navigate, any model that is not multimodal automatically scores 0.
- Research (22 tasks, 1x weight): Open-ended research briefs that produce a full written report, checked against an explicit factual and structural rubric.
-
Reasoning (202 questions, 1x weight): Short logic, spatial geometry, and deduction puzzles (such as Zebra line-up clues, combinatorial cuts on polygons, and
Web of Liestruth-teller chains) with one exact answer. -
Data Analysis (150 questions, 1x weight): Practical table operations across CSV and structured records, including mapping columns across tables, tracking consecutive growth runs, and classifying column types (such as identifying meteorite codes as
recclassin livebench/data_analysis).
Instead of counting every category equally, LiveBench 2.0 computes its Overall score as a weighted mean: Long-Horizon Agentic counts 2x, while Coding and Documents & Design count 1.5x each, meaning those three multi-step build suites account for 56% of a model's total score (livebench.ai). Switching from short questions to multi-step work immediately separated the field: the same 18 models that sat compressed between 71.6 and 83.4 on v1 spread out across more than 41 points (from 43.9 to 85.7) on LiveBench 2.0 (vakati-tech.ai).
3. How One Run Grades Accuracy and Dollar Cost
Every model on LiveBench 2.0 is tested in a single pass@1 inference run at its highest supported effort setting. Here is what happens from start to finish:
- Send fresh tasks inside an isolated workspace: Coding, browser, and agentic tasks run inside a clean sandbox or software container using source repositories and task specs created or updated after training cutoffs.
- Meter every token in and out: As the AI writes chain of thought notes (reasoning tokens), edits files, inspects errors, and generates reports, the runner logs the exact input and output tokens billed over the provider's API.
- Verify the result automatically: Code fixes must pass automated unit tests; math and calculus answers are parsed with regex rules and verified for algebraic equivalence using SymPy (arXiv:2406.19314); browser tasks check the final web-app state; and design tasks run structural checks on the generated deck or PDF.
-
Adjust the bill for failed attempts: LiveBench computes
Cost per successful taskas(total dollars spent / total tasks / score) x 100(livebench.ai). Dividing by the success rate prevents a weak, cheap model that fails two-thirds of its jobs from looking cheaper per finished deliverable than a model that succeeds on the first try.
4. Two Catches Behind Published LiveBench Numbers
Before picking a model off the Overall column alone, two details in the underlying numbers are worth checking:
-
List Prices per Million Tokens Hide Verbosity Swings: Two models with nearly identical API rates on a model card can differ 3x to 5x in real cost per task when one spends heavy test-time compute generating long internal reasoning traces or retrying failed commands. Promotional windows also move a model's position on the cost chart: Gemini 4 Carbon costs $2.02 per task under Google's 50% introductory rate through November 30, 2026, and doubles to roughly $4.04 per task once standard
$4 / $20per-million token pricing takes effect on December 1 (Google API pricing). - Small Task Counts in Long Suites Create Wider Tie Zones: Reasoning (202 items) and Data Analysis (150 items) average over hundreds of fast checks, whereas Long-Horizon Agentic has just 18 multi-hour tasks and Research has 22. Missing 1 single agentic job drops that category by more than 5 points. In addition, text-only endpoints score 0.0 on Browser Use because they cannot inspect screenshots (pulling down otherwise capable models), and LiveBench 2.0 scores cannot be compared directly to pre-October 2026 LiveBench 1.0 numbers (DeepInfra November 2026 coding audit).
5. Where Models Stand Today (and What Score You Need)
Pairing LiveBench 2.0 category scores with measured task costs on livebench.ai and Modello separates models into three clear tiers:
- Multi-Hour Agentic Flagships ($2 to $10 per Task): Claude Fable 5.2 ranks 1st Overall at 84.9 (and leads Long-Horizon Agentic by more than 7 points at 82.9, plus Research at 94.5) at $9.85 per successful task. GPT-6 Astra sits a virtual tie behind Overall at 84.5 ($5.32/task, 91.3 Browser Use), and Gemini 4 Carbon scores 83.5 Overall (while leading LiveBench 2.0 Coding at 91.0 and Data Analysis at 86.5) for $2.02/task on intro rates ($4.04 standard).
- Sub-$1 API Workhorses and Open-Weight Leaders: GPT-6.1 Sol reaches 81.4 Overall (and #1 on Browser Use at 93.3) for $0.82/task, while GPT-6.1 Terra hits 78.8 Overall (89.2 Coding) for $0.67/task. Among open-weight models, DeepSeek V4.1 Flash leads at 70.3 Overall (80.4 Coding) for $0.20/task and MiMo V2.6 Pro scores 66.1 at $0.18/task, whereas a 24 GB local model like Qwen3.8-27B holds up well on contained Coding (69.3) and Reasoning (80.0) while dropping to 15.9 on multi-hour Agentic runs (54.0 Overall).
-
A Simple Rule of Thumb: For daily coding, table analysis, slide drafting, and web automation, filter for models scoring 78+ with a
Max Cost Ceilingof $1 to $2.50 per task in Set Your Bar (such asGPT-6.1 Terra,GPT-6.1 Sol, orGemini 4 Carbon). Reserve$5+to$10per-task runs likeClaude Fable 5.2for long, unattended multi-file agent jobs where higher completion rates on the 18-task Agentic suite justify the extra token spend. Also see our companion guides to SWE-bench Pro and ARC-AGI-2.