getsentry/sentry

Developer-first error tracking and performance monitoring

Run by mupt-ai · Released 2026-10-17 · 78 tasks · 15 settings

Model Tradeoffs

Which coding agent works best on getsentry/sentry? Most accurate: Claude Sonnet 5.5, 82.1% at $0.59 per task. Best for less: GPT-6.1 Sol, 80.8% at $0.50. 15 model settings scored on 78 tasks from its merged pull requests.

All Settings

ModelHarnessReasoningAccessAccuracyCost / TaskFrontier
Claude Sonnet 5.5Claude CodeHighAPI Key82.1%$0.59Yes
Claude Opus 5.5Claude CodeHighAPI Key82.1%$1.22No
GPT-6.1 SolCodexHighAPI Key80.8%$0.50Yes
DeepSeek V4.1 Flash 1017PiHighVercel AI Gateway79.5%$0.069Yes
GPT-6 AstraCodexMaxAPI Key78.2%$4.37No
Claude Opus 5.5Claude CodeLowAPI Key76.9%$0.53No
Kimi K3PiHighVercel AI Gateway76.9%$1.09No
GPT-6.1 SolCodexLowAPI Key74.4%$0.27No
GLM 5.3PiHighVercel AI Gateway73.1%$2.80No
Mistral Large 4PiHighVercel AI Gateway69.2%$2.07No
Claude Haiku 5.5Claude CodeHighAPI Key67.9%$0.27No
Claude Haiku 5.5Claude CodeLowAPI Key66.7%$0.069Yes
DeepSeek V4.1 FlashPiHighVercel AI Gateway57.7%$0.081No
GPT-6 LunaCodexHighAPI Key51.3%$0.025Yes
GPT-6 LunaCodexLowAPI Key38.5%$0.026No

Results by Task

Whether each setting passed each task, with how its attempt was graded and its transcript.

Tasks

All 78 tasks, each with its instruction, tests, and solution to browse and download.

All Repositories on SelfBench