← Blog

LiveBench 2.0 Explained: How Fresh Work and Real Task Costs Test AI

Two problems quietly ruin most AI leaderboards. First, once a fixed test sits on the public internet for a few months, newer AI models read the questions during training and score high from memory. Second, almost every chart reports an accuracy score while hiding whether a model spent 20 cents or 10 dollars of computing time to finish the job.

LiveBench 2.0 tackles both problems at the same time. It is a regularly refreshed benchmark that tests AI models on newly created tasks across coding, web browsers, slide decks, research reports, data tables, and logic puzzles, grades their outputs automatically without biased AI judges on objective tasks, and publishes the exact dollar cost per successful task right beside every score.

1. Why Static Tests and AI Judges Kept Getting Gamed

Researchers from NYU, the University of Maryland, USC, UIUC, and Abacus.AI (including Colin White, Samuel Dooley, Tom Goldstein, and Yann LeCun) introduced the original LiveBench in a June 2024 paper (arXiv:2406.19314, accepted at ICLR 2025) to fix two recurring flaws in older evaluations:

  • Public Question Leakage: Static tests posted on GitHub or academic sites quickly run into data contamination once web crawlers sweep those pages into future training data. To block memorization, LiveBench built tasks from freshly published math competitions (AMC12 and AIME), recent arXiv abstracts, new movies, and newly released datasets on Hugging Face, refreshing questions over time.
  • Biased AI Judges: Many open-ended chat boards rely on LLM-as-a-judge scoring, where a model like GPT-4 reads two answers and picks a winner. In Section 4 of the LiveBench paper, the authors showed that AI judges suffer from verbosity bias (rewarding padded bullet lists), favor their own model family's phrasing, and even marked a valid, compiling C++ program wrong because its algorithm looked unfamiliar to the judge.

To avoid judge mistakes, LiveBench 1.0 restricted its 6 original categories (Math, Coding, Reasoning, Language, Instruction Following, and Data Analysis) to questions with one verified ground truth answer, checked by deterministic grading scripts. Over two years, however, models got so good at isolated 1-turn questions that by June 2026, 18 major models bunched into a narrow 12-point band between 71.6 and 83.4 (LiveBench 2.0 release analysis).

2. What Changed in LiveBench 2.0: Grading Finished Work

In October 2026 (with the current leaderboard release dated November 2026 on livebench.ai), the team froze version 1.0 and launched LiveBench 2.0. Instead of short standalone quizzes, LiveBench 2.0 grades 471 items across 7 categories centered on finished work deliverables:

  • Long-Horizon Agentic (18 tasks, 2x weight): Multi-hour long-horizon tasks where an AI agent extends an application across several rounds of changing requirements or operates an existing codebase over simulated weeks, graded deterministically.
  • Coding (29 tasks, 1.5x weight): Real software bug fixes taken from merged pull requests on a frozen repository, where breaking any existing feature scores 0.
  • Documents & Design (25 tasks, 1.5x weight): Building formatted slide decks, PDFs, and structured documents, graded automatically on layout and file-structure checks of the finished file.
  • Browser Use (25 tasks, 1x weight): Multi-step browser use workflows inside live web apps. Because the agent must read screen captures to click and navigate, any model that is not multimodal automatically scores 0.
  • Research (22 tasks, 1x weight): Open-ended research briefs that produce a full written report, checked against an explicit factual and structural rubric.
  • Reasoning (202 questions, 1x weight): Short logic, spatial geometry, and deduction puzzles (such as Zebra line-up clues, combinatorial cuts on polygons, and Web of Lies truth-teller chains) with one exact answer.
  • Data Analysis (150 questions, 1x weight): Practical table operations across CSV and structured records, including mapping columns across tables, tracking consecutive growth runs, and classifying column types (such as identifying meteorite codes as recclass in livebench/data_analysis).

Instead of counting every category equally, LiveBench 2.0 computes its Overall score as a weighted mean: Long-Horizon Agentic counts 2x, while Coding and Documents & Design count 1.5x each, meaning those three multi-step build suites account for 56% of a model's total score (livebench.ai). Switching from short questions to multi-step work immediately separated the field: the same 18 models that sat compressed between 71.6 and 83.4 on v1 spread out across more than 41 points (from 43.9 to 85.7) on LiveBench 2.0 (vakati-tech.ai).

LiveBench 2.0 breakdown of 7 suites, 471 graded items, category weights where Agentic, Coding, and Design make up 56 percent of Overall, and the 41.8-point score spread compared with frozen LiveBench v1
LiveBench 2.0 weights finished deliverables heavily (56% across Agentic, Coding, and Design), widening the gap between short-answer chat models and reliable multi-step agents from 11.8 points to 41.8 points. Source: LiveBench 2.0 Leaderboard and LiveBench 2.0 Category Weights Analysis.

3. How One Run Grades Accuracy and Dollar Cost

Every model on LiveBench 2.0 is tested in a single pass@1 inference run at its highest supported effort setting. Here is what happens from start to finish:

  1. Send fresh tasks inside an isolated workspace: Coding, browser, and agentic tasks run inside a clean sandbox or software container using source repositories and task specs created or updated after training cutoffs.
  2. Meter every token in and out: As the AI writes chain of thought notes (reasoning tokens), edits files, inspects errors, and generates reports, the runner logs the exact input and output tokens billed over the provider's API.
  3. Verify the result automatically: Code fixes must pass automated unit tests; math and calculus answers are parsed with regex rules and verified for algebraic equivalence using SymPy (arXiv:2406.19314); browser tasks check the final web-app state; and design tasks run structural checks on the generated deck or PDF.
  4. Adjust the bill for failed attempts: LiveBench computes Cost per successful task as (total dollars spent / total tasks / score) x 100 (livebench.ai). Dividing by the success rate prevents a weak, cheap model that fails two-thirds of its jobs from looking cheaper per finished deliverable than a model that succeeds on the first try.
Log-scale plot of LiveBench 2.0 Overall scores against measured dollar cost per successful task, tracing the value frontier from $0.18 MiMo V2.6 Pro and $0.20 DeepSeek V4.1 Flash up to $2.02 Gemini 4 Carbon, $5.32 GPT-6 Astra, and $9.85 Claude Fable 5.2
Plotting LiveBench 2.0 quality against measured task cost traces the value frontier: models on the blue curve deliver the highest score at each price tier, while Gemini 4 Carbon's 50% intro rate ($2.02/task) shifts to $4.04/task on Dec 1, 2026. Source: LiveBench 2.0 Insights and Modello Catalog.

4. Two Catches Behind Published LiveBench Numbers

Before picking a model off the Overall column alone, two details in the underlying numbers are worth checking:

  • List Prices per Million Tokens Hide Verbosity Swings: Two models with nearly identical API rates on a model card can differ 3x to 5x in real cost per task when one spends heavy test-time compute generating long internal reasoning traces or retrying failed commands. Promotional windows also move a model's position on the cost chart: Gemini 4 Carbon costs $2.02 per task under Google's 50% introductory rate through November 30, 2026, and doubles to roughly $4.04 per task once standard $4 / $20 per-million token pricing takes effect on December 1 (Google API pricing).
  • Small Task Counts in Long Suites Create Wider Tie Zones: Reasoning (202 items) and Data Analysis (150 items) average over hundreds of fast checks, whereas Long-Horizon Agentic has just 18 multi-hour tasks and Research has 22. Missing 1 single agentic job drops that category by more than 5 points. In addition, text-only endpoints score 0.0 on Browser Use because they cannot inspect screenshots (pulling down otherwise capable models), and LiveBench 2.0 scores cannot be compared directly to pre-October 2026 LiveBench 1.0 numbers (DeepInfra November 2026 coding audit).

5. Where Models Stand Today (and What Score You Need)

Pairing LiveBench 2.0 category scores with measured task costs on livebench.ai and Modello separates models into three clear tiers:

  • Multi-Hour Agentic Flagships ($2 to $10 per Task): Claude Fable 5.2 ranks 1st Overall at 84.9 (and leads Long-Horizon Agentic by more than 7 points at 82.9, plus Research at 94.5) at $9.85 per successful task. GPT-6 Astra sits a virtual tie behind Overall at 84.5 ($5.32/task, 91.3 Browser Use), and Gemini 4 Carbon scores 83.5 Overall (while leading LiveBench 2.0 Coding at 91.0 and Data Analysis at 86.5) for $2.02/task on intro rates ($4.04 standard).
  • Sub-$1 API Workhorses and Open-Weight Leaders: GPT-6.1 Sol reaches 81.4 Overall (and #1 on Browser Use at 93.3) for $0.82/task, while GPT-6.1 Terra hits 78.8 Overall (89.2 Coding) for $0.67/task. Among open-weight models, DeepSeek V4.1 Flash leads at 70.3 Overall (80.4 Coding) for $0.20/task and MiMo V2.6 Pro scores 66.1 at $0.18/task, whereas a 24 GB local model like Qwen3.8-27B holds up well on contained Coding (69.3) and Reasoning (80.0) while dropping to 15.9 on multi-hour Agentic runs (54.0 Overall).
  • A Simple Rule of Thumb: For daily coding, table analysis, slide drafting, and web automation, filter for models scoring 78+ with a Max Cost Ceiling of $1 to $2.50 per task in Set Your Bar (such as GPT-6.1 Terra, GPT-6.1 Sol, or Gemini 4 Carbon). Reserve $5+ to $10 per-task runs like Claude Fable 5.2 for long, unattended multi-file agent jobs where higher completion rates on the 18-task Agentic suite justify the extra token spend. Also see our companion guides to SWE-bench Pro and ARC-AGI-2.