active composite: BenchAtlas Index v0.3 · bai-v0-3 · since 2026-07-14
Methodology
How the BenchAtlas Index is computed, and the rules that keep comparisons honest. The methodology text below is stored with the composite version itself — what you read is what the pipeline ran.
The index, in its own words
BenchAtlas Index v0.3
v0.3 = v0.2 (percentiles + coverage shrinkage) plus measured benchmark quality weighting. Within a category, families no longer average equally: each family's weight is a quality score in (0,1] computed from properties we can defend and publish —
quality = headroom × discrimination × contamination-resistance × reliability
- headroom: median of the top-5 pool scores vs the scale ceiling. A benchmark whose frontier sits at 98–100% (AIME, MATH-500, GPQA-Diamond today) is saturated: residual differences among top models are mostly noise and dilute real signal. Full weight at ≥30 points of headroom, floored at 0.15. - discrimination: standard deviation among the top-15 pool scores — does the test separate the frontier at all? Full weight at ≥3 points of spread, floored at 0.3. - contamination resistance (curated metadata): live/rotating task sets 1.0 (LiveBench, LiveCodeBench windows, arenas) > private holdouts 0.9 (ARC semi-private, FrontierMath, HLE, professional suites) > static public sets 0.7 (GPQA, MMLU-Pro, past AIME exams — these leak into training corpora). - reliability: task-count floor (n<50 → 0.7, n<150 → 0.85) — a 30-problem contest is noisy.
Headroom and discrimination are recomputed from live pools at every analytics run and published per family on its benchmark page — the Index's weights are auditable measurements, not hand-picked numbers.
Category weights are demand-anchored (full evidence: docs/design/scoring.md): the Index's audience is model-choosers — API and enterprise buyers — whose measured usage is dominated by coding and agents (46% of Anthropic API traffic is computer & mathematical; >50% of OpenRouter tokens are programming; 55% of 2025 enterprise LLM spend is coding). Weights: agentic coding/tool-use 0.40, coding 0.15, knowledge & science 0.15, reasoning & math 0.10, professional domains 0.10, long-context & instruction 0.05, multimodal 0.05.
Human-preference arena ratings are excluded from the capability composite — crowdsourced Elo is a biased, gameable sensor (undisclosed best-of-N variant testing, vote campaigns of ~1,000 votes can reorder rankings, style confounds substance) and pool-relative besides. Arena standings remain a first-class panel throughout the site, clearly separated from capability.
Coverage gate, shrinkage (k=3) and all previous limitations unchanged from v0.2.
Parameters
| Minimum pool systems scored on a version before it enters percentile ranking | 5 |
| shrinkage_k shrinkage_k | 3 |
| Coverage gate · families benchmark families required before a composite score exists | 4 |
| High confidence · families families needed for a high-confidence label | 8 |
| Coverage gate · categories categories required before a composite score exists | 3 |
| High confidence · categories categories needed for a high-confidence label | 5 |
Category weights
| Category | Weight |
|---|---|
| Agentic Coding | 0.40 |
| Coding | 0.15 |
| Knowledge & Science | 0.15 |
| Professional Domains | 0.10 |
| Reasoning & Math | 0.10 |
| Long Context & Instruction | 0.05 |
| Multimodal | 0.05 |
A system missing a category has its remaining weights renormalized — missing data never counts as zero. Benchmark families in categories without a weight are display-only and never enter the Index.
Comparability rules
| # | When two scores differ in… | Verdict |
|---|---|---|
| 1 | Different metrics — different metrics measure different things. | not comparable |
| 2 | Different benchmark families — only comparable via a normalized index, never raw. | not comparable |
| 3 | Agent+model system vs bare model — a harness result never compares with a bare-model result. | not comparable |
| 4 | Pool-relative ratings (Elo) from different sources — Elo is only meaningful within a single arena. | not comparable |
| 5 | Same-arena Elo captured more than 30 days apart — the arena pool has drifted. | caveat |
| 6 | Different benchmark versions — different versions are different task sets. | caveat |
| 7 | A comparability-relevant run setting differs: style_control, subset, split, attempts, implementation, index_version, livebench_version, category. | caveat |
| 8 | One result vendor-reported, the other externally evaluated — a verification boundary. | caveat |
| 9 | Two agent+model systems with different agents — comparing systems, not the underlying models. | comparable as systems |
| 10 | Same family, version, metric, run settings and verification class. | comparable |
Every side-by-side row on the compare page carries one of these verdicts, computed at query time from the two observations’ stored dimensions — not-comparable pairs are never rendered as a bar pair.
Verification vocabulary
| independent | A third party ran the evaluation itself (Epoch AI’s Inspect runs, Artificial Analysis, arena vote data). The strongest class — wins dedup ties. |
| community | Checked by a benchmark maintainer or community process (e.g. SWE-bench “checked” entries) without a fully independent re-run. |
| vendor-reported | The model developer’s own number. Used only where nothing external exists, and always labelled — never silently mixed with external results. |
| unverified | Self-reported submissions with no verification trail. Lowest rank in dedup. |
Dedup preference wherever one best observation is needed: verification class, then newer evaluation date, then source trust rank, then first-observed time. Every score keeps its raw value and links to the snapshot it came from.