← Blog

Glossary

A

Agent (AI Agent)

#

Software that uses an AI model in a loop to run tools, execute commands, read errors, and edit files until a task finishes.

A standard chat prompt produces one reply and stops. An agent connects the model to external tools such as a code editor, web browser, or terminal so it can search files, run tests, inspect failure logs, and submit a multi-step result.

API (Application Programming Interface)

#

A direct software connection that lets a program or coding tool send text to an AI model and receive its reply, billed per token.

Instead of typing inside a consumer chat website for a flat monthly fee, developers and coding tools (like Cursor or Claude Code) can connect to a model's API and pay the provider based on the exact number of input and output tokens processed.

B

Benchmark

#

A fixed set of test tasks and grading rules used to measure how accurately different AI models perform a specific type of work.

Every model takes the exact same set of tasks under controlled rules so scores can be compared side by side. Separate benchmarks test separate skills, such as multi-file bug fixing (SWE-bench Pro), visual puzzles (ARC-AGI-2), or terminal administration (Terminal-Bench 4.0).

Bug Report (Issue)

#

A written note filed in a software project describing something in the app that is broken, giving wrong output, or crashing.

A typical bug report explains what a user or developer tried to do, what went wrong, and what should have happened instead. Coding benchmarks hand the AI the real bug report text as its starting instruction.

C

Claude Code

#

Anthropic's AI coding agent that reads a project's files, edits code across folders, and runs terminal commands and tests.

Developers run Claude Code from a terminal, desktop app, code editor, or web session to track down bugs and ship multi-file changes. In benchmark testing, it acts as an agent scaffold around Claude models.

Closed Model (Proprietary Model)

#

An AI model whose internal numeric files are kept private on the creator's servers and accessed only through a web app or API.

Models such as Claude Fable 5.2, GPT-6 Astra, and Gemini 4 Carbon run exclusively on Anthropic, OpenAI, or Google cloud hardware. Users send prompts over the internet rather than downloading the model files onto their own computer.

Command Line (Terminal)

#

A text interface where a user or AI types operating-system commands directly instead of clicking buttons or menus.

Developers and server engineers use the command line to install packages, run automated test suites, search folders, and configure Linux servers. Benchmarks like Terminal-Bench 4.0 grade whether an AI can execute multi-step terminal commands without breaking the system.

Commit (Git Commit)

#

A saved snapshot of a project's files at one exact moment in time, identified by a short code like af7a0be.

Every time a developer saves a set of changes in Git, it creates a commit. Coding benchmarks rewind a codebase back to the exact commit right before a bug was fixed so the AI starts from the same broken code the human developer faced.

Container (Docker Container)

#

An isolated, disposable virtual workspace on a computer that holds a specific codebase, its software dependencies, and its test suite.

Coding benchmarks run each AI attempt inside a fresh container with internet access turned off. If the AI breaks system files during a test, the container is deleted afterward, and clean-room verifiers apply the final patch inside a separate unmodified container.

Context Window

#

The maximum amount of text and images (measured in tokens) an AI model can hold in active working memory during a single conversation.

Everything you paste, every file read from a codebase, and every reply in the current chat shares the context window. A 128,000-token context window holds roughly 1 full-length book (~96,000 words); a 1,000,000-token context window holds roughly 8 full books.

C# (C-Sharp)

#

A typed, compiled programming language developed by Microsoft and widely used for enterprise backend services, desktop apps, and games.

SWE-bench Pro includes real multi-file bug-fixing tasks in C# alongside Python, TypeScript, JavaScript, and Go so coding leaderboards test languages used across business software.

Cursor

#

A desktop code editor with built-in AI agents that search a full project, edit multiple files, and run terminal commands.

Cursor lets developers pick among models from Anthropic, OpenAI, Google, and other labs to write code or fix bugs across an existing repository.

D

Dependency (Software Dependency)

#

An outside code package that an app relies on to handle common jobs like reading files, checking dates, or talking to a database.

Instead of writing every basic feature from scratch, software projects install shared software dependencies. Inside an offline benchmark container, all required dependencies are pre-installed ahead of time so the AI can run the app without internet access.

Diff File (Code Diff)

#

A plain-text comparison file that shows the exact lines removed (-) and added (+) when source code is edited.

Git generates a diff file automatically whenever files change in a repository. Reviewing a diff lets developers or grading scripts inspect every altered line across multiple files in one place.

Django

#

A widely used open-source Python framework for building websites, databases, and backend web apps.

Nearly half of the tasks in the original 2023 SWE-bench came from Django's public GitHub repository. Newer benchmarks cap how many tasks can come from Django or any single project so scores reflect broader coding ability.

Docker

#

Software that packages an app, its operating-system tools, and all its code libraries into isolated containers that run the same way on any machine.

Coding benchmarks use Docker so each AI attempt runs inside a clean, reproducible Linux environment with all project dependencies ready and outbound internet turned off.

F

fail2pass Test

#

An automated check in a coding benchmark that fails on the broken starting code and must pass after the AI applies its fix.

A fail2pass test directly checks the bug or missing feature described in the bug report. Turning every fail2pass test from FAIL to PASS proves the reported issue was actually solved.

G

Git

#

Version-control software that records every saved change (commit) made to files in a codebase over time.

Git lets developers inspect exact line-by-line edits, revert mistakes, and check out how a project looked on any past date. Coding benchmarks use Git to reset a codebase to the exact commit right before a real bug was fixed.

GitHub

#

A website that hosts Git code repositories online alongside bug reports, discussions, and proposed code changes.

Millions of open-source software projects store their source code and issue trackers on GitHub. Modern coding benchmarks block outbound network connections to GitHub during testing so an AI cannot look up how human developers later solved the issue.

Go (Golang)

#

A fast, compiled programming language created at Google and widely used for cloud servers, command-line tools, and networking software.

SWE-bench Pro includes real bug-fixing tasks in Go repositories alongside Python, TypeScript, JavaScript, and C# so coding scores measure more than Python alone.

H

Hallucination

#

When an AI model states a false fact, invented citation, or wrong number as if it were verified fact.

Language models generate text by predicting probable token sequences and do not automatically pause when a fact is missing from memory. Benchmarks like AA-Omniscience penalize wrong guesses (-1 point) and reward saying "I don't know" (0 points) to measure hallucination rates.

Held-Out Set (Private Test Split)

#

A batch of test questions kept private by benchmark creators so AI companies cannot train their models on the answers.

When every test task sits on a public website, newer AI models eventually read the answers during training. Keeping a held-out set offline lets researchers verify whether high public scores still hold up on unseen tasks.

I

Input Tokens vs. Output Tokens

#

Input tokens are the text you send into the AI (prompts and files); output tokens are the text the AI generates back (replies and reasoning).

API providers charge separate prices per 1 million input tokens and per 1 million output tokens. Output tokens usually cost 3x to 8x more per token than input tokens because generating new tokens requires more server compute than reading existing text.

ISBN (International Standard Book Number)

#

A unique 10-digit or 13-digit identification number assigned to every published book edition (such as ISBN-13).

Libraries and bookstores use ISBN codes to look up exact book titles, authors, and publishers across catalog APIs like Amazon and Google Books.

J

JavaScript

#

The standard programming language that makes web pages interactive in every browser, and also runs backend servers through Node.js.

Millions of web apps use JavaScript for menus, forms, live updates, and server APIs. Multi-language coding benchmarks test whether AI models can fix bugs across JavaScript and TypeScript apps, not just Python scripts.

L

Local Model (Running Offline)

#

An open-weight AI model downloaded and run directly on your own laptop or desktop hardware with zero internet connection or per-token fee.

Tools like Ollama or LM Studio run quantized open-weight models (such as Sage-A1.5 or Qwen3.8-27B) directly on your computer's memory. Prompts stay private on your device and cost $0 per token, bounded by how much RAM or VRAM your machine has.

M

Model Card

#

The technical specification sheet published by an AI lab when releasing a model, listing its context limit, training cutoff, pricing, and benchmark scores.

A model card documents what inputs a model accepts (text, images, audio), its license rules, known failure cases, and how the creator configured its evaluation runs.

O

Open-Source Software

#

Software whose source code is published openly online so anyone can read it, run it, report bugs, or suggest fixes.

Projects like Python, Linux, and the Internet Archive's Open Library are open-source. Because their code and bug trackers are public, coding benchmarks combine open-source projects with private company codebases to test AI models.

Open-Weight Model

#

An AI model whose trained numeric weight files are publicly downloadable so anyone can inspect, host, or run the model on their own hardware.

Small open-weight models (around 24 GB) fit on a personal Mac or PC, while very large open-weight models (600 GB to 2,300 GB, such as Kimi K3.1 Pro or DeepSeek V4.1) require multi-GPU cloud servers or third-party API hosts.

P

pass@1 (First-Try Score)

#

The percentage of benchmark tasks an AI model solves correctly on its first submitted attempt, without multiple retries.

Some release charts report pass@5 or majority-vote scores where a model gets several tries at the same problem. Tracking pass@1 keeps comparisons fair and reflects what a user experiences when asking a question once.

pass2pass Test

#

An automated check for existing app features that already passed before the edit and must still pass after the AI's fix.

pass2pass tests verify that fixing one bug did not break unrelated parts of the software. On SWE-bench Pro, failing even a single pass2pass test results in 0 points for the task.

Patch (Patch File)

#

A saved diff file (often named patch.diff) that can be applied automatically to another copy of a codebase to replay the exact same code edits.

In SWE-bench Pro, the AI submits one patch file at the end of its run. The grading system applies that patch inside a fresh, untouched container to run the hidden test suite. Reference patches in SWE-bench Pro average 107.4 changed lines across 4.1 files.

Prompt

#

The instruction, question, files, or images given to an AI model at the start of a task.

In everyday chat, a prompt is the message you type. In automated benchmarks, the prompt is a standardized task description (such as a bug report or exam question) sent identically to every tested model.

Pull Request (PR)

#

A proposed set of code changes submitted on GitHub or GitLab for human review and automated testing before merging into the main codebase.

Each pull request records the problem discussion, the exact code diff, and test results. Many coding benchmark tasks are built from real historical pull requests by taking the codebase right before the pull request was merged.

Python

#

A popular, easy-to-read programming language used heavily for web backends, data analysis, automation scripts, and AI software.

Earlier coding benchmarks like the original SWE-bench and SWE-bench Verified tested Python projects exclusively. Newer benchmarks like SWE-bench Pro test Python alongside TypeScript, JavaScript, Go, and C#.

R

RAM, Unified Memory, and VRAM

#

The high-speed computer memory that must hold an entire local AI model at once while it generates replies.

Dedicated graphics cards use VRAM (video memory), while Apple Silicon Macs use Unified Memory shared between CPU and GPU. To run a local model smoothly offline, the model's file size plus working context must fit inside your available VRAM or Unified Memory (for example, 24 GB or 32 GB).

Reasoning Tokens (Thinking Tokens)

#

Extra output tokens generated internally by an AI model to break down steps and check its work before writing the final visible answer.

Even when hidden behind a collapsible summary in chat, reasoning tokens are billed at the model's output token rate on an API. A model that writes 15,000 thinking tokens on a hard math or coding task can cost several dollars for a single reply.

Regression (Breaking Old Features)

#

When a new code change fixes one bug or adds a feature, and accidentally breaks something else in the app that used to work.

In real software teams, a fix that breaks login, search, or checkout cannot ship. SWE-bench Pro checks for regressions using pass2pass unit tests and awards 0 points if even one existing feature breaks.

Repository (Codebase / Repo)

#

The complete folder structure of a software project, containing all source code files, configuration files, tests, and Git version history.

Real-world software repositories often contain thousands of interconnected files across multiple programming languages, requiring an AI coding tool to search across directories rather than editing one file in isolation.

S

Scaffold (Agent Harness)

#

The wrapper program around an AI model that controls how it searches folders, opens files, runs tests, and edits code.

An AI model on its own only takes text in and writes text out. A scaffold (such as SWE-agent, Claude Code, or Cursor) supplies the file-editing tools and step limits, which means the same AI model can score higher or lower depending on the scaffold used.

Snapshot (Code Snapshot)

#

An exact saved copy of all files in a software project (or model catalog) frozen at one specific point in time.

Coding benchmarks load a code snapshot from right before a bug was fixed so every AI model starts from the exact same broken files.

T

Token

#

The basic chunk of text (a short word, part of a longer word, or punctuation mark) that AI models read, generate, and bill by.

In English text, 1 token equals roughly 0.75 words (100 tokens is about 75 words). One million tokens equals roughly 750,000 words, or about 8 full-length books.

Training Data and Training Data Overlap

#

Training data is the collection of text, code, and images used to build an AI model; overlap (contamination) happens when benchmark test questions appear inside that training data.

If an AI model read a test's questions and answers during training, its high score reflects recall of seen text rather than problem-solving on new tasks. Modern benchmarks counter this by using private holdout questions (ARC-AGI-2, HLE) or refreshing tasks monthly (LiveBench 2.0).

TypeScript

#

An extended version of JavaScript that adds explicit data types (such as text, number, or user object) so code editors catch mistakes before the app runs.

Large modern web and desktop apps (such as Tuta Mail) are often written in TypeScript to keep multi-file codebases easier to maintain across big teams.

U

Unit Tests

#

Small automated checks written in code that feed inputs into specific functions of an app and verify the outputs match what is expected.

Large apps contain thousands of unit tests that run in seconds and report PASS or FAIL. Coding benchmarks run unit tests to grade AI patches automatically instead of relying on a human judge's opinion.