CodeAF head to head with nine coding agents on DeepSWE
Task by task on 113 DeepSWE tasks, CodeAF solved more tasks the other harness missed than the reverse, against all nine harnesses on one model.
Compared task by task on the 113 DeepSWE tasks, with every harness on the same model (deepseek/deepseek-v4-flash-0731), CodeAF in senior-dev mode solved more tasks that the other harness failed than the other way round, against each of the nine harnesses in AgentField's comparison. The closest pairing is mini-swe-agent: 29 tasks only CodeAF solved, 23 only mini-swe-agent solved. The widest is Muse Code: 60 against 1.
A total score hides how two agents differ. Two harnesses can both solve about half the tasks and still solve different halves. Pairing them task by task shows which one you would rather have had on the tasks where they disagree.
At a glance
| Opponent | Only CodeAF solved | Only the opponent solved | Both solved | Neither |
|---|---|---|---|---|
| mini-swe-agent | 29 | 23 | 33 | 28 |
| Codex CLI | 25 | 14 | 37 | 37 |
| Pi | 33 | 13 | 29 | 38 |
| Oh My Pi | 44 | 13 | 18 | 38 |
| OpenCode | 40 | 8 | 22 | 43 |
| Kilo Code | 41 | 9 | 21 | 42 |
| DeepSeek Harness | 53 | 7 | 9 | 44 |
| Claude Code | 51 | 5 | 11 | 46 |
| Muse Code | 60 | 1 | 2 | 50 |
Each row adds up to 113. "Solved" means the official DeepSWE verifier passed the task. A task with no verifier result counts as not solved, as on the main results page.
Method
This page reads the per-task records of the comparison on the main DeepSWE results page: ten harnesses, the full 113-task set, one model for all of them through OpenRouter, the official verifiers at 0b9fabb, 3 hours per task, one pass each. The records are one CSV per harness, 113 rows each, in the campaign folder. In those files the CodeAF row is named senior-dev, the mode that ran.
Any row of the table can be recounted from those files. Join them, then compare the reward column of two harnesses on each task:
awk 'FNR>1 || NR==1' tasks/*.csv > all-arms.csvResults
Against the three strongest other harnesses, CodeAF's margin on disagreements was 6 tasks over mini-swe-agent, 11 over Codex CLI and 20 over Pi. Against the rest it was 31 or more.
The "Neither" column matters too. 15 tasks were solved by no harness at all, so even the best pairing leaves a block of tasks neither agent could do on this model.
Where the other tools did better
- mini-swe-agent solved 23 tasks CodeAF did not. That is the most any harness took from CodeAF, and mini-swe-agent also solved the most tasks no other harness solved: 7, against CodeAF's 5.
- Codex CLI solved 14 tasks CodeAF did not, so the two are complementary on a real share of the set.
- Kept tests. mini-swe-agent kept slightly more of each repository's existing tests passing (99.66% against CodeAF's 99.42%).
Limits of this result
- It is AgentField's own comparison, one pass per harness, not a public leaderboard. A gap of a few tasks is within run-to-run noise.
- CodeAF used the provider's default sampling; the other nine sent temperature 1.0 and top-p 0.95. CodeAF also ran a day later, on 2026-09-12 UTC.
- Every harness drove DeepSeek V4 Flash. The Claude Code row is Claude Code driving that model, not Claude Code with Claude models.
- Epoch AI's review of DeepSWE v1.1 found "issues in at least 23 out of 113 tasks". Every harness met the same tasks and verifiers, but some disagreements may come from flawed tasks.
Questions people ask
Does CodeAF beat mini-swe-agent on DeepSWE?
In AgentField's comparison on DeepSeek V4 Flash, CodeAF in senior-dev mode solved 62 of 113 tasks and mini-swe-agent 56. On the tasks where they disagreed, CodeAF solved 29 that mini-swe-agent missed and mini-swe-agent solved 23 that CodeAF missed. With one pass each, that gap is small.
Does CodeAF beat Codex CLI on DeepSWE?
On DeepSeek V4 Flash, CodeAF solved 62 tasks and Codex CLI 0.154.0 solved 51. Task by task, 25 were solved only by CodeAF and 14 only by Codex CLI; both solved 37. Codex CLI here drove DeepSeek, not an OpenAI model.
How does CodeAF compare with Claude Code task by task?
With both on DeepSeek V4 Flash, CodeAF solved 51 tasks that Claude Code 2.1.267 failed, and Claude Code solved 5 that CodeAF failed. That measures the harness on DeepSeek; it does not say how Claude Code performs with Claude models.
Can I check these numbers myself?
Yes. The per-task CSVs for all ten harnesses are public in the CodeAF repository, linked under Method. Each row has the task, its band and language, the verifier reward, the fail-to-pass and pass-to-pass test counts, wall time and exit status.
Sources
- Per-task records, one CSV per harness, checked 2026-11-21.
- Campaign README and
arms.csv, checked 2026-11-21. - Epoch AI review of DeepSWE, checked 2026-11-21.
Last updated 2026-11-21. Every count recomputed from the per-task CSVs; versions are those on the main results page.