Sources / swebench

SWE-bench Leaderboard

healthydefault: community

www.swebench.com · trust rank 80 · adapter swebench

Observations

365

append-only score facts

Raw snapshots

immutable fetched payloads

Last success

2026-07-14 00:15Z

end of last succeeded/partial run

Freshness

1.4 h

hours since last success

License & attribution

The swe-bench/experiments leaderboard data has no LICENSE file — reused as facts with attribution and a link back (the main SWE-bench repo is MIT).

What we ingest

Embedded leaderboard JSON from each split page (Verified, Lite, bash-only, full, Multimodal, Multilingual) — every entry is an agent+model system. Maintainer-checked entries land as community-verified; unchecked ones as unverified. bash-only standardizes on mini-swe-agent, our canonical harness for composite purposes.

Ingestion runs

last 3 of the run history

Started	Duration	Status	Key stats
2026-07-14 00:15Z	4.7 s	succeeded	docs 6 · staged 365 · promoted 365 · blocked 0 · scoreAdded 0 · unchanged 5
2026-07-14 00:09Z	5.1 s	succeeded	docs 6 · staged 365 · promoted 365 · blocked 0 · scoreAdded 0 · unchanged 5
2026-07-13 20:03Z	4.1 s	partial	docs 6 · staged 365 · promoted 0 · blocked 365 · scoreAdded 0 · unchanged 0

partial = run finished but some rows are blocked on identity review — nothing is published until every referenced model/agent/benchmark is resolved.

Raw snapshots

latest 5

Fetched	URL	SHA-256	Last re-fetch
2026-07-14 00:15Z	https://www.swebench.com/multimodal.html	7d6799e3ea3a…	2026-07-14 00:15Z
2026-07-14 00:09Z	https://www.swebench.com/multimodal.html	6e8e834c5754…	2026-07-14 00:09Z
2026-07-13 20:03Z	https://www.swebench.com/original.html	dc0b44886a4e…	2026-07-14 00:15Z
2026-07-13 20:03Z	https://www.swebench.com/multilingual-leaderboard.html	01e08f05371b…	2026-07-14 00:15Z
2026-07-13 20:03Z	https://www.swebench.com/multimodal.html	b7dafdd1b407…	2026-07-13 20:03Z

Snapshots are immutable raw payloads — every published number traces back to one. Re-fetching an identical payload bumps the re-fetch time instead of storing a duplicate. Scores from this source appear on benchmark profiles and model pages with this source linked.