Accuracy Without Cost Is Half the Story: A Plain-Language Guide to AI Benchmarks
Every few weeks, an AI company posts a bar chart showing its newest model ahead by 1.4 points on a string of capital letters: MMLU, GPQA, SWE-bench, ARC-AGI, HLE. Two days later, a rival company posts a different chart with a slightly different prompt setup and claims first place back.
For anyone choosing an AI model for daily coding, writing, math, or business tasks, those charts skip the three questions that decide what you actually use: what job did the benchmark test, did the model already see the answers in its training data, and what does it cost to finish the job?
1. Why Headline AI Scores Are Hard to Read at Face Value
A benchmark is a fixed set of test tasks used to measure how accurately an AI model performs a specific type of work. Over time, public tests lose their usefulness for three practical reasons:
- Training Data Overlap: AI models learn by reading billions of public web pages during training. Once a benchmark sits on GitHub or public forums for months, newer models have often seen the exact questions or close variations before taking the test.
- Broken Answer Keys on Older Tests: For years, MMLU (Massive Multitask Language Understanding) was the default multiple-choice scoreboard for AI. When researchers checked the test line by line in MMLU-Redux, they found that roughly 6.5% of the questions (nearly 1 in 15) had missing text, typos, or wrong answer keys. Today, almost every major model scores between 90% and 93% on MMLU, where the remaining differences are mostly test errors.
- Small Score Gaps That Feel Identical in Daily Use: On 0-to-100 leaderboards such as BenchLM v5.8, a 1-point or 2-point lead rarely produces a noticeable difference on everyday prompts. When two models sit within 2.5 points of each other, response speed, context window, and price matter much more than the tiebreaker decimal.
2. Three Traps Hidden Behind Leaderboard Charts
Even on newer evaluations, a single accuracy percentage leaves out important context about how that score was produced:
- Container Shortcuts and Version-History Leaks: Coding agents run inside software containers where they execute command-line programs to search and edit code. A clean-room audit of coding benchmarks (arXiv:2610.24206) found agents querying live GitHub pull-request pages in 32 of 642 audited runs to look up the human developer's finished fix, or reading saved commit history inside Git. Blocking outside GitHub connections dropped one model from 351 solved bugs down to 237.
- Formatting Bias and Confident Guessing: On blind human-voting boards like Arena AI, hurried voters frequently click longer answers with neat headings and bullet points even when a calculation or citation is wrong. On PhD-level exams like Humanity's Last Exam, models still report roughly 80% confidence on the questions they miss.
- Hidden Reasoning Token Bills: Published API rates per million tokens look small on paper, yet a model can generate thousands of internal reasoning tokens while triple-checking a single puzzle. It might beat a rival by 2%, while quietly turning a $0.82 job into a $9.85 bill on LiveBench 2.0. Accuracy without cost is only half the story.
3. How Modello Put the Numbers Together
Modello pulls public results from independent evaluations and model cards, cleans up the fine print, and places accuracy next to real-world cost:
-
First-Try Scores (pass@1): Model profiles and tables display standard single-attempt results from BenchLM v5.8 across overall quality, multi-file coding, and math/logic reasoning. Models without a published score in a category appear as
Untested, and score differences of 2.5 points or less are labeled as a practical tie. - Token Rates Next to Measured Task Cost: Alongside input and output token prices per 1M tokens, each model lists its measured dollar cost to finish one complete task on LiveBench 2.0, counting prompt tokens and internal reasoning tokens together.
- Promotional Windows and Standard Rates: Models on temporary launch discounts or third-party host promotions display both today's active price and the scheduled standard price, with a toggle to re-sort tables using standard pricing.
- Subscription Plans and Local Hardware Fit: Access tags indicate whether a model has a free web tier, fits inside a $20/month chat plan, requires metered API credits, or is an open-weight model (as opposed to a closed model) small enough to run as a local model offline within 24 GB of RAM or VRAM.
- Active Catalog and Archive Vault: The main directory tracks active models across major labs, while older releases surpassed by newer versions move into a searchable Archive Vault alongside monthly historical snapshots.
4. What the 10 Core Benchmarks Actually Test
Instead of relying on one composite number, each benchmark in Modello's studio tests a specific capability and exposes a specific catch:
- SWE-bench Pro (Multi-File Bug Fixing): Hands the model a real bug ticket inside a large software repository and checks its code patch (averaging 107.4 changed lines across 4.1 files across Python, TypeScript, Go, and C# in arXiv:2509.16941 and arXiv:2606.11976) inside an offline container. Every fail2pass and pass2pass test must pass; breaking even one existing feature gives 0%. Human developers score around 92%; the top AI today reaches 89.8% on BenchLM Coding (93.0% raw SWE-bench Pro). (Full SWE-bench Pro guide →)
- ARC-AGI-2 (Unseen Visual Pattern Puzzles): Shows colored pixel grids that follow a visual transformation rule the model has not seen before, with zero text hints. Human testers average 66%; top reasoning models reach 95.0%, though difficult puzzles can consume $1 to $5+ of extra thinking time per puzzle.
- LiveBench 2.0 (Fresh Monthly Work and Task Cost): Refreshes tasks regularly from recent math, coding, data analysis, and language problems so models cannot rely on memorized web pages, while recording the dollar bill per finished task.
- Terminal-Bench 4.0 (Live Command-Line Jobs): Places an AI inside a Linux terminal to configure servers, compile code, repair broken environments, and migrate files. Human server engineers clear roughly 85%; the top-scoring AI clears 67.2%, meaning roughly 1 in 3 live server jobs still fails.
- Humanity's Last Exam (2,500 PhD Holdout Questions): Collected from domain specialists across mathematics, chemistry, physics, classics, and engineering to test where frontier knowledge runs out. Early 2025 models scored 2.7%, while current frontier reasoning models reach 74.3%.
- AA-Omniscience v1.1 (Factuality vs. Hallucination): Awards +1 point for a correct factual answer, 0 points for admitting "I don't know", and -1 point for guessing wrong. Because many models still guess instead of abstaining, smaller models often finish below zero while the highest-scoring model reaches +67 net (vs. ~80 net for a careful human checking sources).
- MMMU-Pro (Reading Screenshots, Charts, and Diagrams): Places the question and dataset directly inside screenshots, financial tables, circuit diagrams, and scientific figures with zero text input, testing whether vision models read dense visual layouts accurately (top AI: 89.0% vs. ~88% human specialist).
- AA-LCR v1.1 (Long-Document Recall): Tests whether a model advertising a "1M token" context window can actually connect facts scattered across pages 12, 47, and 89 of a long filing without losing track of middle pages (top AI: 88.7% vs. ~94% careful human with search).
- Arena AI (Blind Human Chat Preference): Pairs two anonymous models on the same user prompt and lets the user vote on which reply reads better, reported both raw and with style controls for length and formatting.
- MMLU (Retired) (Legacy 2020 Trivia Quiz): Kept in the studio as a historical baseline showing how multiple-choice tests saturate once training overlap and answer-key defects accumulate.
5. Reading the Tradeoffs Before You Pick a Model
When accuracy and cost sit side by side, picking a model stops being a search for one universal "#1 AI" and becomes a simple threshold check:
- Pick the Score Floor Your Task Needs: Fixing a multi-file bug in Cursor or Claude Code benefits from a 75%+ coding score, whereas summarizing notes or classifying bulk text works reliably on a 60+ overall model streaming at 180+ tokens per second.
- Pair That Floor With a Cost Ceiling: Once you filter to models that clear your quality bar in Set Your Bar, per-task cost and plan access separate everyday workhorses (free-tier models, offline 24 GB local models, or sub-$1/task APIs) from $9+/task flagship runs reserved for your hardest problems.
- Treat Small Score Gaps as Practical Ties: When two models land within 2.5 points on the benchmark that matches your work, choosing the faster or cheaper option lowers your bill without a noticeable drop in output quality.