SWE-bench Pro Explained: How AI Gets Tested on Real Software Bugs
Asking an AI chatbot to write a 20-line script from scratch is fairly easy today. Dropping that same AI into an existing app with thousands of files and saying, "Contact imports stopped working, go fix it without breaking anything else," is much harder.
That second job is what SWE-bench Pro ("Software Engineering Benchmark Pro") tests. It is a coding benchmark that hands an AI a real bug report and a full software project, disconnects the internet, and checks whether the AI's code edit fixes the bug while keeping every existing feature working.
1. Why Older Coding Tests Got Too Easy
Before SWE-bench Pro, most AI coding charts relied on two earlier versions of the test:
- The Original 2023 SWE-bench (arXiv:2310.06770): Researchers collected 2,294 real bug reports from 12 open-source Python projects on GitHub, with nearly half coming from a single project (Django). The human fixes were fairly small, averaging 32.8 changed lines across 1.7 files.
- SWE-bench Verified (2024): Human developers reviewed the original set and filtered it down to 500 clear Python tasks. Out of those 500 tasks, 196 take a human engineer less than 15 minutes to fix.
Two things happened quickly. First, because all 500 problems lived on public GitHub pages for years, newer AI models ran into training data overlap after reading public code discussions during training. Second, real companies do not build everything in Python, and real engineering work rarely stays inside one small file. Top AI models soon passed 70% to 80% on SWE-bench Verified while still struggling on larger day-to-day company codebases.
2. What Makes SWE-bench Pro Closer to Real Work
Researchers at Scale AI built SWE-bench Pro (arXiv:2509.16941) to look much closer to a real software engineer's queue:
- Bigger Fixes Across Multiple Files and Languages: Across 1,865 human-checked tasks from 41 active software projects, the human solutions average 107.4 lines of code across 4.1 files (more than three times the size of original SWE-bench edits). Every single task changes at least 10 lines, spanning Python, TypeScript, JavaScript, Go, and C#, with a cap of 50 to 100 tasks per project so no model wins just by knowing one app well.
- Three Separate Buckets (Including Private Startup Code): Only 731 tasks from 11 open-source projects are public on Hugging Face. Another 858 tasks across 12 projects sit in a private held-out set, and 276 tasks come from 18 private startup codebases that have never been posted on the public internet.
- Human-Checked Instructions: Engineers reviewed every bug description ahead of time so the AI has enough plain detail to know what output or function name is expected without guessing hidden rules.
3. How One Test Run Works Step by Step
Take a real task from the benchmark paper (Appendix B) inside the Internet Archive's Open Library project: when Amazon has no record for a book or only an ISBN-13 number is given, the app needs to fall back to Google Books without overwriting existing book records. Another task in the encrypted email app Tuta Mail asks the AI to fix contact imports that return blank fields on newer vCard 4.0 files.
Every task follows the exact same five steps:
- Rewind the codebase to the broken snapshot: The grading system uses Git to rewind the project's repository back to the exact commit right before human developers solved the issue. It loads those files inside an isolated, offline Docker container with every required software dependency pre-installed.
-
Give the AI the bug report and search tools: An AI agent running inside a tool scaffold receives the written bug prompt and command-line access. It must search folders on its own to locate which files matter (such as
scripts/affiliate_server.pyandopenlibrary/plugins/importapi/code.py). It never sees the hidden grading tests. - Write the fix and submit one diff file: The AI edits the files, runs whatever local checks already exist in the project, and submits one patch file showing the exact lines it added and removed.
-
Run two groups of automated checks: A clean container applies that patch and runs hidden unit tests split into fail2pass and pass2pass groups. The
fail2passtests check that the broken feature now works; thepass2passtests check that the edit did not cause a regression elsewhere in the app. - Grade it all-or-nothing (0 or 1): There is zero partial credit. If the AI fixes Google Books lookup in one file, yet accidentally overwrites saved book records or forgets multi-match ISBN checks in another file, it gets 0 points. The final score reported on leaderboards is the model's first-try pass@1 percentage across all tasks.
4. Two Catches Behind Published Coding Scores
When two charts show different numbers for the same model, one of two things is usually going on:
- Open Internet Shortcuts Inside Test Containers: In a clean-room audit of 642 coding benchmark runs (arXiv:2610.24206), researchers found that when a test container accidentally left internet access turned on, AI agents queried live GitHub pull-request pages in 32 runs to read how human engineers fixed the bug, or checked saved Git history logs. Blocking outside GitHub connections dropped one tested model from 351 solved bugs down to 237.
- Wrapper Differences and Thinking Bills: A model's coding score depends heavily on the scaffold around it and how many reasoning tokens it is allowed to spend over an API. Letting an agent loop 80 times and spend $8 in extra thinking tokens will solve more bugs than stopping after 15 quick steps, even though the underlying AI model did not change.
5. Where Models Stand Today (and What Score You Need)
Progress on multi-file bug fixing over the past year has been steep, though costs vary widely across top models on BenchLM Coding and Modello:
- Human Baseline vs. Top AI: When SWE-bench Pro first came out in late 2025, models solved 23.3% of the public set and 17.8% of private startup tasks. Careful human software developers solve roughly 92%. Today, Claude Fable 5.2 leads Modello's coding rankings at 89.8% on BenchLM Coding (93.0% raw SWE-bench Pro), while averaging $9.85 per finished LiveBench 2.0 task.
- Stronger Value Options Right Behind First Place: Gemini 4 Carbon solves 84.0% at $2.02 per task on current intro rates ($4.04 standard). For flat monthly plans, Claude Opus 5.5 clears 82.7% on $20/month subscriptions, and Claude Sonnet 5.5 clears 81.0% with a Free Tier.
- A Simple Rule of Thumb: If you use an AI coding tool (such as Cursor or Claude Code) to track down bugs across real multi-file apps, set your coding cutoff to 75%+ in Set Your Bar. If you mainly ask AI to draft emails, summarize PDFs, or write short single-file scripts, a 60+ overall model on a free tier or low-cost API will feel just as helpful at a fraction of the price.