PostHog/
PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Run by mupt-ai · Released 2026-10-17 · 35 tasks · 14 settings
Model Tradeoffs
Which coding agent works best on PostHog/posthog? Most accurate: Claude Opus 5.5, 68.6% at $2.33 per task. Best for less: DeepSeek V4.1 Flash 1017, 65.7% at $0.078. 14 model settings scored on 35 tasks from its merged pull requests.
All Settings
| Model | Harness | Reasoning | Access | Accuracy | Cost / Task | Frontier |
|---|---|---|---|---|---|---|
| Claude Opus 5.5 | Claude Code | High | API Key | 68.6% | $2.33 | Yes |
| DeepSeek V4.1 Flash 1017 | Pi | High | Vercel AI Gateway | 65.7% | $0.078 | Yes |
| GPT-6 Astra | Codex | High | API Key | 65.7% | $3.18 | No |
| GPT-6.1 Sol | Codex | High | API Key | 62.9% | $0.64 | No |
| Claude Opus 5.5 | Claude Code | Low | API Key | 60% | $0.83 | No |
| GPT-6 Sol | Codex | High | API Key | 54.3% | $1.26 | No |
| GLM 5.3 | Pi | High | Vercel AI Gateway | 51.4% | $3.51 | No |
| Mistral Large 4 | Pi | High | Vercel AI Gateway | 48.6% | $1.41 | No |
| Claude Haiku 5.5 | Claude Code | Low | API Key | 45.7% | $0.15 | No |
| Kimi K3 | Pi | High | Vercel AI Gateway | 45.7% | $1.99 | No |
| GPT-6 Sol | Codex | Low | API Key | 42.9% | $0.38 | No |
| GPT-6 Luna | Codex | High | API Key | 37.1% | $0.037 | Yes |
| Claude Haiku 5.5 | Claude Code | High | API Key | 37.1% | $0.56 | No |
| GPT-6 Luna | Codex | Low | API Key | 25.7% | $0.034 | Yes |
Results by Task
Whether each setting passed each task, with how its attempt was graded and its transcript.
Tasks
All 35 tasks, each with its instruction, tests, and solution to browse and download.