Benchmarks / Agentic Coding
SWE-bench
agent + modelResolving real GitHub issues from popular Python (and multilingual/multimodal) repositories. Results are agent+model systems — the scaffold matters.
maintained by SWE-bench · www.swebench.com · 400 observations on record
Versions
6 versions
| Version | Tasks |
|---|---|
| SWE-bench Verified current swe-bench-verified | 500 |
| SWE-bench bash-only swe-bench-bash-only | — |
| SWE-bench (full test split) swe-bench-full | 2,294 |
| SWE-bench Lite swe-bench-lite | 300 |
| SWE-bench Multilingual swe-bench-multilingual | — |
| SWE-bench Multimodal swe-bench-multimodal | — |
Current leaderboard
SWE-bench Verified · Resolved rate · top 25 of 143 systems
| # | System | Resolved rate % |
|---|---|---|
| 1 | agent + model Epoch Inspect harness + Claude Opus 4.7max effort Anthropic | 83.5% |
| 2 | agent + model Epoch Inspect harness + GPT 5.5 Pre Releasexhigh effort OpenAI | 80.6% |
| 3 | agent + model Epoch Inspect harness + Gemini 3.5 Flashhigh effort Google DeepMind | 79.3% |
| 4 | agent + model live-SWE-agent + Claude Opus 4.5 20251101 Anthropic | 79.2% |
| 5 | agent + model Sonar Foundation Agent + Claude Opus 4.5 Anthropic | 79.2% |
| 6 | agent + model Epoch Inspect harness + Claude Opus 4.6 Anthropic | 78.7% |
| 7 | agent + model Epoch Inspect harness + GLM 5.2max effort Z.ai | 78.7% |
| 8 | agent + model Epoch Inspect harness + Deepseek V4 Promax effort DeepSeek | 77.6% |
| 9 | agent + model live-SWE-agent + Gemini 3 Pro Preview Google DeepMind | 77.4% |
| 10 | agent + model Epoch Inspect harness + Qwen3 7max effort Alibaba | 77.3% |
| 11 | agent + model Epoch Inspect harness + GPT 5.4 (2026-03-05)high effort OpenAI | 76.9% |
| 12 | agent + model mini-SWE-agent + Claude 4.5 Opus Unknown | 76.8% |
| 13 | agent + model EPAM AI/Run Developer Agent v20250719 + Claude Sonnet 4 20250514 Anthropic | 76.8% |
| 14 | agent + model Epoch Inspect harness + Qwen3 6 Max Preview Alibaba | 76.7% |
| 15 | agent + model Epoch Inspect harness + Claude Opus 4.5 20251101 Anthropic | 76.7% |
| 16 | agent + model Epoch Inspect harness + Kimi K2 6 Moonshot AI | 76.7% |
| 17 | agent + model mini-SWE-agent + Minimax M2 5 MiniMax | 75.8% |
| 18 | agent + model mini-SWE-agent + Gemini 3 Flash Preview Google DeepMind | 75.8% |
| 19 | agent + model Epoch Inspect harness + Gemini 3.1 Pro Preview Customtools Google DeepMind | 75.6% |
| 20 | agent + model mini-SWE-agent + Claude Opus 4.6 Anthropic | 75.6% |
| 21 | agent + model Epoch Inspect harness + Gemini 3 Flash Preview Google DeepMind | 75.4% |
| 22 | agent + model Epoch Inspect harness + Claude Sonnet 4.6 Anthropic | 75.2% |
| 23 | agent + model Harness AI + Claude Sonnet 4 20250514 Anthropic | 74.8% |
| 24 | agent + model Sonar Foundation Agent + Claude Sonnet 4.5 Anthropic | 74.8% |
| 25 | agent + model Epoch Inspect harness + GPT 5.3 Codexhigh effort OpenAI | 74.8% |
One row per evaluated system — reasoning-effort variants rank separately. Where a system has attr variants for this version (eval splits, style control), only the canonical variant is ranked: split=semi_private, style_control=true, or the unsplit run.
Source agreement
where evaluating organizations agree — and don't
SWE-bench VerifiedEpoch AI Benchmarking Hub vs SWE-bench Leaderboard
overlap 13 systemsrank correlation 0.81mean |Δ| 2.49max |Δ| 5.45
most contested: Claude 3.7 Sonnet 20250219 (-5.45) · GPT 5 Mini (2025-08-07) (+4.87) · Gemini 3 Pro Preview (-4.47)
Possible reasons: different agent harnesses ran the tasks; verification differs (independent vs unverified); run setting 'attempts' differs (— vs 1).