Sources / swebench

SWE-bench Leaderboard

failingdefault: community

www.swebench.com · trust rank 80 · adapter swebench

Observations

365

append-only score facts

Raw snapshots

immutable fetched payloads

Last success

—

end of last succeeded/partial run

Freshness

—

hours since last success

License & attribution

The swe-bench/experiments leaderboard data has no LICENSE file — reused as facts with attribution and a link back (the main SWE-bench repo is MIT).

What we ingest

Embedded leaderboard JSON from each split page (Verified, Lite, bash-only, full, Multimodal, Multilingual) — every entry is an agent+model system. Maintainer-checked entries land as community-verified; unchecked ones as unverified. bash-only standardizes on mini-swe-agent, our canonical harness for composite purposes.

Ingestion runs

last 10 of the run history

Started	Duration	Status	Key stats
2026-09-12 08:00Z	2.8 s	failed FetchError: GET https://www.swebench.com/multilingual-leaderboard.html → 404	—
2026-09-11 08:00Z	2.8 s	failed FetchError: GET https://www.swebench.com/multilingual-leaderboard.html → 404	—
2026-09-10 08:00Z	2.9 s	failed FetchError: GET https://www.swebench.com/multilingual-leaderboard.html → 404	—
2026-09-09 08:00Z	2.9 s	failed FetchError: GET https://www.swebench.com/multilingual-leaderboard.html → 404	—
2026-09-08 08:00Z	2.3 s	failed FetchError: GET https://www.swebench.com/multilingual-leaderboard.html → 404	—
2026-09-07 08:00Z	2.3 s	failed FetchError: GET https://www.swebench.com/multilingual-leaderboard.html → 404	—
2026-09-06 08:00Z	3.0 s	failed FetchError: GET https://www.swebench.com/multilingual-leaderboard.html → 404	—
2026-09-05 08:00Z	2.6 s	failed FetchError: GET https://www.swebench.com/multilingual-leaderboard.html → 404	—
2026-09-04 08:00Z	2.3 s	failed FetchError: GET https://www.swebench.com/multilingual-leaderboard.html → 404	—
2026-09-03 08:00Z	2.3 s	failed FetchError: GET https://www.swebench.com/multilingual-leaderboard.html → 404	—

partial = run finished but some rows are blocked on identity review — nothing is published until every referenced model/agent/benchmark is resolved.

Raw snapshots

latest 5

Fetched	URL	SHA-256	Last re-fetch
2026-08-09 08:00Z	https://www.swebench.com/multimodal.html	d11417ba0878…	2026-08-09 08:00Z
2026-08-08 08:00Z	https://www.swebench.com/multimodal.html	e64e6bb680ad…	2026-08-08 08:00Z
2026-08-07 08:00Z	https://www.swebench.com/multimodal.html	99264cc9d815…	2026-08-07 08:00Z
2026-08-06 08:00Z	https://www.swebench.com/multimodal.html	db26fc5f5e7a…	2026-08-06 08:00Z
2026-08-05 08:00Z	https://www.swebench.com/multimodal.html	f6f350a1663d…	2026-08-05 08:00Z

Snapshots are immutable raw payloads — every published number traces back to one. Re-fetching an identical payload bumps the re-fetch time instead of storing a duplicate. Scores from this source appear on benchmark profiles and model pages with this source linked.