BenchAtlas

Rankings / Anthropic

Claude 3.5 Sonnet 20241022

released 2024-10-22

BenchAtlas Index

as of 2026-09-11
36.3
base configuration
rank #332
6 families · 3 categories · medium
29.7
base configuration
rank #380
6 families · 3 categories · medium

Benchmark evidence

45 results
coding
LiveCodeBench v5older version
problems=279 · window_start=2024-08-01
37.2 %independentLiveCodeBench Leaderboard
LiveCodeBench v6
problems=454 · window_start=2024-08-01
36.4 %independentLiveCodeBench Leaderboard
external indices
ECI133.6 pointsindependentEpoch AI Benchmarking Hub
human preference
Coding (style control)older version
arena=text · category=coding · style_control=true
1435.4
1429.8–1440.9
communityLMArena Leaderboard Dataset
Industry Software And It Services (style control)older version
arena=text · category=industry_software_and_it_services · style_control=true
1407.8
1403.2–1412.5
communityLMArena Leaderboard Dataset
Hard Prompts English (style control)older version
arena=text · category=hard_prompts_english · style_control=true
1403.4
1398.0–1408.9
communityLMArena Leaderboard Dataset
Hard Prompts (style control)older version
arena=text · category=hard_prompts · style_control=true
1398.4
1394.0–1402.7
communityLMArena Leaderboard Dataset
Longer Query (style control)older version
arena=text · category=longer_query · style_control=true
1395.5
1389.8–1401.2
communityLMArena Leaderboard Dataset
French (style control)older version
arena=text · category=french · style_control=true
1392.9
1371.8–1414.1
communityLMArena Leaderboard Dataset
Multi Turn (style control)older version
arena=text · category=multi_turn · style_control=true
1389.5
1383.8–1395.3
communityLMArena Leaderboard Dataset
Polish (style control)older version
arena=text · category=polish · style_control=true
1385.0
1371.0–1399.0
communityLMArena Leaderboard Dataset
Industry Medicine And Healthcare (style control)older version
arena=text · category=industry_medicine_and_healthcare · style_control=true
1384.6
1374.6–1394.6
communityLMArena Leaderboard Dataset
Industry Legal And Government (style control)older version
arena=text · category=industry_legal_and_government · style_control=true
1383.5
1374.4–1392.5
communityLMArena Leaderboard Dataset
English (style control)older version
arena=text · category=english · style_control=true
1382.6
1378.6–1386.5
communityLMArena Leaderboard Dataset
German (style control)older version
arena=text · category=german · style_control=true
1381.3
1367.7–1394.8
communityLMArena Leaderboard Dataset
Industry Life And Physical And Social Science (style control)older version
arena=text · category=industry_life_and_physical_and_social_science · style_control=true
1376.5
1370.7–1382.3
communityLMArena Leaderboard Dataset
Industry Writing And Literature And Language (style control)older version
arena=text · category=industry_writing_and_literature_and_language · style_control=true
1376.5
1371.5–1381.4
communityLMArena Leaderboard Dataset
Russian (style control)older version
arena=text · category=russian · style_control=true
1375.0
1368.2–1381.8
communityLMArena Leaderboard Dataset
Overall (style control)
arena=text · category=overall · style_control=true
1373.7
1370.4–1376.9
communityLMArena Leaderboard Dataset
Instruction Following (style control)older version
arena=text · category=instruction_following · style_control=true
1373.4
1368.9–1377.8
communityLMArena Leaderboard Dataset
Expert (style control)older version
arena=text · category=expert · style_control=true
1371.6
1362.3–1381.0
communityLMArena Leaderboard Dataset
Creative Writing (style control)older version
arena=text · category=creative_writing · style_control=true
1370.1
1363.8–1376.3
communityLMArena Leaderboard Dataset
Spanish (style control)older version
arena=text · category=spanish · style_control=true
1368.9
1349.4–1388.3
communityLMArena Leaderboard Dataset
Chinese (style control)older version
arena=text · category=chinese · style_control=true
1367.8
1359.5–1376.1
communityLMArena Leaderboard Dataset
Industry Entertainment And Sports And Media (style control)older version
arena=text · category=industry_entertainment_and_sports_and_media · style_control=true
1364.5
1358.9–1370.2
communityLMArena Leaderboard Dataset
Industry Mathematical (style control)older version
arena=text · category=industry_mathematical · style_control=true
1359.5
1352.5–1366.5
communityLMArena Leaderboard Dataset
Non English (style control)older version
arena=text · category=non_english · style_control=true
1358.5
1354.6–1362.4
communityLMArena Leaderboard Dataset
Industry Business And Management And Financial Operations (style control)older version
arena=text · category=industry_business_and_management_and_financial_operations · style_control=true
1353.9
1347.5–1360.3
communityLMArena Leaderboard Dataset
Math (style control)older version
arena=text · category=math · style_control=true
1351.2
1344.5–1357.9
communityLMArena Leaderboard Dataset
Exclude Ties (style control)older version
arena=text · category=exclude_ties · style_control=true
1345.4
1341.1–1349.8
communityLMArena Leaderboard Dataset
Japanese (style control)older version
arena=text · category=japanese · style_control=true
1309.0
1294.8–1323.3
communityLMArena Leaderboard Dataset
Korean (style control)older version
arena=text · category=korean · style_control=true
1305.1
1286.5–1323.6
communityLMArena Leaderboard Dataset
knowledge science
GPQA Diamond
implementation=vals-ai
59.3 %independentVals AI
GPQA Diamond55.3 %independentEpoch AI Benchmarking Hub
2025-01-27
MMLU-Pro
implementation=vals-ai
78.4 %independentVals AI
multimodal
MMMU
implementation=vals-ai
68.8 %independentVals AI
professional
CorpFin53.6 %independentVals AI
MedQA83.2 %independentVals AI
TaxEval70.2 %independentVals AI
reasoning math
AIME
implementation=vals-ai
10.0 %independentVals AI
FrontierMath2.1 %independentEpoch AI Benchmarking Hub
2025-03-06
FrontierMath Tier 4older version0.0 %independentEpoch AI Benchmarking Hub
2025-07-01
MATH-500
implementation=vals-ai
72.4 %independentVals AI
MATH Level 557.0 %independentEpoch AI Benchmarking Hub
2025-01-27
OTIS Mock AIME 2024–20258.5 %independentEpoch AI Benchmarking Hub
2025-02-25

Agent + model results

systems, not bare-model scores
agent + model AgentScope + Claude 3.5 Sonnet 20241022SWE-bench Verified63.4 %unverifiedSWE-bench Leaderboard
agent + model EPAM AI/Run Developer Agent v20250219 + Claude 3.5 Sonnet 20241022SWE-bench Verified62.8 %unverifiedSWE-bench Leaderboard
agent + model CodeStory Midwit Agent + Claude 3.5 Sonnet 20241022SWE-bench Verified62.2 %unverifiedSWE-bench Leaderboard
agent + model Learn-by-interact + Claude 3.5 Sonnet 20241022SWE-bench Verified60.2 %unverifiedSWE-bench Leaderboard
agent + model EPAM AI/Run Developer Agent v20241212 + Claude 3.5 Sonnet 20241022SWE-bench Verified55.4 %unverifiedSWE-bench Leaderboard
agent + model devlo + Claude 3.5 Sonnet 20241022SWE-bench Verified54.2 %unverifiedSWE-bench Leaderboard
agent + model Engine Labs (2024-11-25) + Claude 3.5 Sonnet 20241022SWE-bench Verified51.8 %unverifiedSWE-bench Leaderboard
agent + model AutoCodeRover-v2.1 + Claude 3.5 Sonnet 20241022SWE-bench Verified51.6 %unverifiedSWE-bench Leaderboard
agent + model Isea + Claude 3.5 Sonnet 20241022SWE-bench Lite51.3 %unverifiedSWE-bench Leaderboard
agent + model Agentless-1.5 + Claude 3.5 Sonnet 20241022SWE-bench Verified50.8 %unverifiedSWE-bench Leaderboard
agent + model devlo + Claude 3.5 Sonnet 20241022SWE-bench Lite47.3 %unverifiedSWE-bench Leaderboard
agent + model AutoCodeRover-v2.0 + Claude 3.5 Sonnet 20241022SWE-bench Verified46.2 %unverifiedSWE-bench Leaderboard
agent + model KGCompass + Claude 3.5 Sonnet 20241022SWE-bench Lite46.0 %unverifiedSWE-bench Leaderboard
agent + model Kodu-v1 + Claude 3.5 Sonnet 20241022SWE-bench Lite44.7 %unverifiedSWE-bench Leaderboard
agent + model Lingxi + Claude 3.5 Sonnet 20241022SWE-bench Lite42.7 %unverifiedSWE-bench Leaderboard
agent + model OpenHands + Claude 3.5 Sonnet 20241022SWE-bench Lite41.7 %communitySWE-bench Leaderboard
agent + model PatchKitty-0.9 + Claude 3.5 Sonnet 20241022SWE-bench Lite41.3 %unverifiedSWE-bench Leaderboard
agent + model OrcaLoca + Claude 3.5 Sonnet 20241022SWE-bench Lite41.0 %unverifiedSWE-bench Leaderboard
agent + model Agentless-1.5 + Claude 3.5 Sonnet 20241022SWE-bench Lite40.7 %unverifiedSWE-bench Leaderboard
agent + model EPAM AI/Run Developer Agent v20241029 + Claude 3.5 Sonnet 20241022SWE-bench Verified39.6 %unverifiedSWE-bench Leaderboard
agent + model Moatless Tools + Claude 3.5 Sonnet 20241022SWE-bench Lite39.0 %communitySWE-bench Leaderboard
agent + model Patched.Codes Patchwork + Claude 3.5 Sonnet 20241022SWE-bench Lite37.0 %unverifiedSWE-bench Leaderboard
agent + model SWE-agent + Claude 3.5 Sonnet 20241022SWE-bench Verified33.6 %communitySWE-bench Leaderboard
agent + model Kortix AI + Claude 3.5 Sonnet 20241022SWE-bench Lite30.0 %communitySWE-bench Leaderboard
agent + model OpenHands + Claude 3.5 Sonnet 20241022SWE-bench Lite26.7 %communitySWE-bench Leaderboard
agent + model Moatless Tools + Claude 3.5 Sonnet 20241022SWE-bench Lite26.7 %communitySWE-bench Leaderboard
agent + model Agentless Lite + Claude 3.5 Sonnet 20241022SWE-bench Multimodal25.3 %communitySWE-bench Leaderboard
agent + model AutoCodeRover-v2.0 + Claude 3.5 Sonnet 20241022SWE-bench (full test split)24.9 %unverifiedSWE-bench Leaderboard
agent + model SWE-agent + Claude 3.5 Sonnet 20241022SWE-bench Lite23.0 %communitySWE-bench Leaderboard
agent + model SWE-agent + Claude 3.5 Sonnet 20241022SWE-bench (full test split)18.1 %communitySWE-bench Leaderboard
agent + model SWE-agent Multimodal + Claude 3.5 Sonnet 20241022SWE-bench Multimodal11.4 %communitySWE-bench Leaderboard
agent + model Agentless + Claude 3.5 Sonnet 20241022SWE-bench Multimodal6.2 %communitySWE-bench Leaderboard
agent + model RAG + Claude 3.5 Sonnet 20241022SWE-bench Multimodal5.0 %communitySWE-bench Leaderboard

These scores measure the whole agent system (scaffold, tools, budgets) — they are never merged into the bare model’s numbers.

Related variants

Claude 3.5 Sonnet 20240620