getsentry/ sentry
Developer-first error tracking and performance monitoring
Run by mupt-ai · Released 2026-10-17 · 78 tasks · 15 settings
Model Tradeoffs
Which coding agent works best on getsentry/sentry? Most accurate: Claude Sonnet 5.5, 82.1% at $0.59 per task. Best for less: GPT-6.1 Sol, 80.8% at $0.50. 15 model settings scored on 78 tasks from its merged pull requests.
All Settings
| Model | Harness | Reasoning | Access | Accuracy | Cost / Task | Frontier |
|---|---|---|---|---|---|---|
| Claude Sonnet 5.5 | Claude Code | High | API Key | 82.1% | $0.59 | Yes |
| Claude Opus 5.5 | Claude Code | High | API Key | 82.1% | $1.22 | No |
| GPT-6.1 Sol | Codex | High | API Key | 80.8% | $0.50 | Yes |
| DeepSeek V4.1 Flash 1017 | Pi | High | Vercel AI Gateway | 79.5% | $0.069 | Yes |
| GPT-6 Astra | Codex | Max | API Key | 78.2% | $4.37 | No |
| Claude Opus 5.5 | Claude Code | Low | API Key | 76.9% | $0.53 | No |
| Kimi K3 | Pi | High | Vercel AI Gateway | 76.9% | $1.09 | No |
| GPT-6.1 Sol | Codex | Low | API Key | 74.4% | $0.27 | No |
| GLM 5.3 | Pi | High | Vercel AI Gateway | 73.1% | $2.80 | No |
| Mistral Large 4 | Pi | High | Vercel AI Gateway | 69.2% | $2.07 | No |
| Claude Haiku 5.5 | Claude Code | High | API Key | 67.9% | $0.27 | No |
| Claude Haiku 5.5 | Claude Code | Low | API Key | 66.7% | $0.069 | Yes |
| DeepSeek V4.1 Flash | Pi | High | Vercel AI Gateway | 57.7% | $0.081 | No |
| GPT-6 Luna | Codex | High | API Key | 51.3% | $0.025 | Yes |
| GPT-6 Luna | Codex | Low | API Key | 38.5% | $0.026 | No |
Results by Task
Whether each setting passed each task, with how its attempt was graded and its transcript.
Tasks
All 78 tasks, each with its instruction, tests, and solution to browse and download.