Last updated · Santosh Kumar Radha

CodeAF head to head with nine coding agents on DeepSWE

Task by task on 113 DeepSWE tasks, CodeAF solved more tasks the other harness missed than the reverse, against all nine harnesses on one model.

Compared task by task on the 113 DeepSWE tasks, with every harness on the same model (deepseek/deepseek-v4-flash-0731), CodeAF in senior-dev mode solved more tasks that the other harness failed than the other way round, against each of the nine harnesses in AgentField's comparison. The closest pairing is mini-swe-agent: 29 tasks only CodeAF solved, 23 only mini-swe-agent solved. The widest is Muse Code: 60 against 1.

A total score hides how two agents differ. Two harnesses can both solve about half the tasks and still solve different halves. Pairing them task by task shows which one you would rather have had on the tasks where they disagree.

At a glance

OpponentOnly CodeAF solvedOnly the opponent solvedBoth solvedNeither
mini-swe-agent29233328
Codex CLI25143737
Pi33132938
Oh My Pi44131838
OpenCode4082243
Kilo Code4192142
DeepSeek Harness537944
Claude Code5151146
Muse Code601250

Each row adds up to 113. "Solved" means the official DeepSWE verifier passed the task. A task with no verifier result counts as not solved, as on the main results page.

Method

This page reads the per-task records of the comparison on the main DeepSWE results page: ten harnesses, the full 113-task set, one model for all of them through OpenRouter, the official verifiers at 0b9fabb, 3 hours per task, one pass each. The records are one CSV per harness, 113 rows each, in the campaign folder. In those files the CodeAF row is named senior-dev, the mode that ran.

Any row of the table can be recounted from those files. Join them, then compare the reward column of two harnesses on each task:

sh
awk 'FNR>1 || NR==1' tasks/*.csv > all-arms.csv

Results

Against the three strongest other harnesses, CodeAF's margin on disagreements was 6 tasks over mini-swe-agent, 11 over Codex CLI and 20 over Pi. Against the rest it was 31 or more.

The "Neither" column matters too. 15 tasks were solved by no harness at all, so even the best pairing leaves a block of tasks neither agent could do on this model.

Where the other tools did better

  • mini-swe-agent solved 23 tasks CodeAF did not. That is the most any harness took from CodeAF, and mini-swe-agent also solved the most tasks no other harness solved: 7, against CodeAF's 5.
  • Codex CLI solved 14 tasks CodeAF did not, so the two are complementary on a real share of the set.
  • Kept tests. mini-swe-agent kept slightly more of each repository's existing tests passing (99.66% against CodeAF's 99.42%).

Limits of this result

  • It is AgentField's own comparison, one pass per harness, not a public leaderboard. A gap of a few tasks is within run-to-run noise.
  • CodeAF used the provider's default sampling; the other nine sent temperature 1.0 and top-p 0.95. CodeAF also ran a day later, on 2026-09-12 UTC.
  • Every harness drove DeepSeek V4 Flash. The Claude Code row is Claude Code driving that model, not Claude Code with Claude models.
  • Epoch AI's review of DeepSWE v1.1 found "issues in at least 23 out of 113 tasks". Every harness met the same tasks and verifiers, but some disagreements may come from flawed tasks.

Questions people ask

Does CodeAF beat mini-swe-agent on DeepSWE?

In AgentField's comparison on DeepSeek V4 Flash, CodeAF in senior-dev mode solved 62 of 113 tasks and mini-swe-agent 56. On the tasks where they disagreed, CodeAF solved 29 that mini-swe-agent missed and mini-swe-agent solved 23 that CodeAF missed. With one pass each, that gap is small.

Does CodeAF beat Codex CLI on DeepSWE?

On DeepSeek V4 Flash, CodeAF solved 62 tasks and Codex CLI 0.154.0 solved 51. Task by task, 25 were solved only by CodeAF and 14 only by Codex CLI; both solved 37. Codex CLI here drove DeepSeek, not an OpenAI model.

How does CodeAF compare with Claude Code task by task?

With both on DeepSeek V4 Flash, CodeAF solved 51 tasks that Claude Code 2.1.267 failed, and Claude Code solved 5 that CodeAF failed. That measures the harness on DeepSeek; it does not say how Claude Code performs with Claude models.

Can I check these numbers myself?

Yes. The per-task CSVs for all ten harnesses are public in the CodeAF repository, linked under Method. Each row has the task, its band and language, the verifier reward, the fail-to-pass and pass-to-pass test counts, wall time and exit status.

Sources

Last updated 2026-11-21. Every count recomputed from the per-task CSVs; versions are those on the main results page.