Which coding agent solves the harder DeepSWE tasks?
On the 76 medium and hard DeepSWE tasks, CodeAF solved 36, ahead of mini-swe-agent's 30 and Codex CLI's 29, with all ten harnesses on one model.
On the 76 DeepSWE tasks classed medium or hard, CodeAF in senior-dev mode solved 36, the most of ten harnesses in AgentField's comparison, ahead of mini-swe-agent (30) and Codex CLI (29). Every harness ran the same model, deepseek/deepseek-v4-flash-0731, for one pass. CodeAF was also first in each band on its own: 24 of 38 medium tasks and 12 of 38 hard ones. On the 37 easy tasks it tied mini-swe-agent at 26.
The harder the task, the more the agent around the model matters: on the medium band CodeAF's lead over the next harness was four tasks, against a tie on the easy band.
At a glance
| Harness | Easy (37) | Medium (38) | Hard (38) | Medium and hard (76) |
|---|---|---|---|---|
| CodeAF (senior-dev) | 26 | 24 | 12 | 36 |
| mini-swe-agent | 26 | 20 | 10 | 30 |
| Codex CLI | 22 | 18 | 11 | 29 |
| Pi | 19 | 15 | 8 | 23 |
| Oh My Pi | 12 | 9 | 10 | 19 |
| OpenCode | 13 | 10 | 7 | 17 |
| Kilo Code | 16 | 9 | 5 | 14 |
| Claude Code | 10 | 2 | 4 | 6 |
| DeepSeek Harness | 9 | 4 | 3 | 7 |
| Muse Code | 2 | 1 | 0 | 1 |
Counts are tasks the official DeepSWE verifier passed. The band is the difficulty class each task carries in the per-task records (band: easy, medium or hard).
Method
The comparison behind these rows is on the main DeepSWE results page: the full 113-task set, the official verifiers at 0b9fabb, 3 hours per task, one pass per harness, one model through OpenRouter. This page groups the per-task records by their band column and counts passes. The CodeAF rows are stored under senior-dev.
Results
CodeAF solved 36 of 76 medium and hard tasks (47%), against 30 for mini-swe-agent (39%) and 29 for Codex CLI (38%). On the hard band alone the gaps are small: 12 for CodeAF, 11 for Codex CLI, 10 each for mini-swe-agent and Oh My Pi.
The 3-hour budget is what senior-dev mode is built for: it plans, works and checks its own result on one long task without a person in the loop. Its mean agent time was 54 minutes a task, with about the same mean on easy (52), medium (57) and hard (54) tasks. A coding agent that runs for hours explains how that mode works.
Where the other tools did better
- Easy tasks were a tie. mini-swe-agent solved the same 26 of 37 easy tasks as CodeAF.
- Speed. mini-swe-agent (44 minutes) and Codex CLI (46 minutes) took less agent time per task on average than CodeAF (54).
- Oh My Pi did unusually well on hard tasks. It solved 10 hard tasks, nearly as many as its 12 easy ones, and level with mini-swe-agent on that band.
Limits of this result
- It is AgentField's comparison, one pass each. A hard-band gap of one or two tasks is not a settled ranking.
- CodeAF used the provider's default sampling; the other nine sent temperature 1.0 and top-p 0.95. CodeAF ran a day later, on 2026-09-12 UTC.
- The bands come from the task records, not from AgentField's judgement, and Epoch AI's review of DeepSWE v1.1 found "issues in at least 23 out of 113 tasks".
Questions people ask
Which AI coding agent is best at hard, long-horizon tasks?
In AgentField's DeepSWE comparison, with every harness on DeepSeek V4 Flash, CodeAF in senior-dev mode solved the most medium and hard tasks: 36 of 76, against 30 for mini-swe-agent and 29 for Codex CLI. On the hard band alone it led by one task, so treat the hard-only order as close.
Does a better harness matter more on harder tasks?
In this comparison it did. On the easy band the top two harnesses tied at 26 of 37. On the medium band CodeAF solved 24 of 38 and the next harness 20. The model was the same for all of them, so the difference is the agent around it.
How long did CodeAF take per task?
CodeAF in senior-dev mode averaged 54 minutes of agent time per task, inside a 3-hour budget, and about the same on easy, medium and hard tasks. mini-swe-agent averaged 44 minutes and Codex CLI 46.
Sources
- Per-task records and the campaign README, checked 2026-11-21.
- Epoch AI review of DeepSWE, checked 2026-11-21.
Last updated 2026-11-21. Every count recomputed from the per-task CSVs.