DeepSWE results: ten coding agents on one model (2026)
CodeAF solved 62 of 113 DeepSWE tasks (54.9%) at the lowest cost per solved issue of ten harnesses on DeepSeek V4 Flash. Method, versions, caveats.
CodeAF, the open-source coding agent by AgentField, solved 62 of 113 DeepSWE tasks (54.9%) in senior-dev mode, the most of ten harnesses in our own comparison, where every harness ran the same model (deepseek/deepseek-v4-flash-0731 through OpenRouter) for one pass each. It also had the lowest cost per solved issue: 39.9 cents, at 22 cents a task. This is a comparison we ran and published ourselves, with every number in arms.csv, not a rank on a public leaderboard.
At a glance
| Result | CodeAF (senior-dev mode) solved 62 of 113 (54.9%), first of ten harnesses in this comparison |
| Model, for every harness | deepseek/deepseek-v4-flash-0731 through OpenRouter |
| Cost | 22 cents a task, 39.9 cents per solved issue (the lowest of the ten) |
| Next best | mini-swe-agent, 56 of 113 (49.6%), 1.9x CodeAF's cost per solved issue |
| Attempts | one pass of the full set per harness |
| Sampling | CodeAF used the provider default; the other nine sent temperature 1.0 and top-p 0.95 |
| Who ran it | AgentField, the makers of CodeAF. Not a public leaderboard entry |
| Later runs | 77.9% on DeepSeek V4.1 Flash and 69.0% on Kimi K3: CodeAF alone, not a second comparison |
The full table
Every row ran the full DeepSWE set on the same model. "Cost per solved issue" is each harness's spend divided by tasks solved, shown as a multiple of CodeAF's 39.9 cents. "Mean time" is mean agent wall time per task.
| Harness | Solved (of 113) | Cost per task | Cost per solved issue | Mean time |
|---|---|---|---|---|
| CodeAF (senior-dev) | 62, 54.9% | 22 cents | 1x | 54 min |
| mini-swe-agent | 56, 49.6% | 38 cents | 1.9x | 44 min |
| Codex CLI | 51, 45.1% | 37 cents | 2.1x | 46 min |
| Pi | 42, 37.2% | 35 cents | 2.4x | 52 min |
| Oh My Pi (omp) | 31, 27.4% | 50 cents | 4.5x | 49 min |
| OpenCode | 30, 26.5% | 50 cents | 4.8x | 48 min |
| Kilo Code | 30, 26.5% | 48 cents | 4.6x | 54 min |
| Claude Code | 16, 14.2% | 19 cents | 3.4x | 32 min |
| DeepSeek Harness | 16, 14.2% | 150 cents | 26.6x | 94 min |
| Muse Code | 3, 2.7% | 12 cents | 11.3x | 16 min |
30 of 113 is 26.5%. The method page prints 26.6% for OpenCode and Kilo Code, rounded from the stored reward rate of 0.2655; the task count is the same. In arms.csv the CodeAF row is named senior-dev, because the binary had an earlier name on the day of the run. At a shell, the same mode is codeaf senior-dev; in a conversation it is /senior-dev (senior-dev docs).
Results by slice
The per-task records, one CSV per harness with 113 rows each, show where the lead came from:
- Head to head: against each of the nine other harnesses, CodeAF solved more of the tasks where the two disagreed. Closest: mini-swe-agent, 29 to 23. See head to head.
- Harder tasks: on the 76 medium and hard tasks CodeAF solved 36, against 30 for mini-swe-agent and 29 for Codex CLI. On the easy tasks it tied mini-swe-agent. See by difficulty.
- Go and Python: CodeAF solved 21 of 34 Go tasks (next: 17) and 20 of 34 Python tasks (next: 18). On TypeScript, mini-swe-agent solved one more. See by language.
- Cost: 39.9 cents per solved task, against 76.9 for the next cheapest. See cost per solved issue.
How the comparison was run
| Benchmark | full DeepSWE set, 113 tasks |
| Model | deepseek/deepseek-v4-flash-0731 through OpenRouter |
| Verifiers | official DeepSWE at 0b9fabb |
| Budget | 3 hours per task |
| Isolation | four shards per harness, a dedicated OpenRouter key per harness |
| Ran | nine harnesses on 2026-09-10 to 11 UTC; CodeAF on 2026-09-12 UTC |
| Attempts | one pass of the set per harness; that pass is the table |
| Sampling | CodeAF: the provider's default. The others: temperature 1.0, top-p 0.95 |
How the columns are derived, from the method page:
- Solved counts tasks the verifier passed. A task with no verifier outcome counts as unsolved. Five tasks produced none: Codex CLI 1, Pi 1, Oh My Pi 2, OpenCode 1.
- Cost per task is billed OpenRouter spend divided by 113, so tasks without a verifier result stay in the denominator. Oh My Pi, OpenCode, Kilo Code, DeepSeek Harness, Claude Code and Muse Code did not record per-attempt cost; their figure is that harness's total spend.
- Cost per solved issue is spend divided by tasks solved.
Tested versions
| Harness | Version tested | Release date (UTC) |
|---|---|---|
| CodeAF | first stable release containing senior-dev mode: v0.5.0 (integration commit dd0fcc654, #1488) | 2026-09-30 |
| mini-swe-agent | 2.4.6 | 2026-07-23 |
| Codex CLI | 0.154.0 | 2026-09-09 |
| Pi | 0.85.1 | 2026-09-05 |
| Oh My Pi | 18.1.17 | 2026-09-10 |
| OpenCode | 1.18.30 | 2026-09-09 |
| Kilo Code | 7.6.2 | 2026-09-10 |
| Claude Code | 2.1.267 | 2026-09-09 |
DeepSeek Harness (@deepseek-ai/dsh) | 0.1.5-rc.1 | 2026-09-10 |
| Muse Code | 1.1.1-R2514.1 | not independently established |
The CodeAF comparison ran before the integration commit landed. The v0.5.0 row says where the measured agent became public; it is not a new run on v0.5.0. versions.csv holds full source revisions and release timestamps.
Limits of this result
Read these before quoting the table. Each is stated on the method page.
- It is our comparison, not a public leaderboard. AgentField chose the harnesses, ran them and published the result. It ranks harnesses on one model, nothing more.
- Sampling differed. CodeAF used the provider's default sampling; the other nine sent temperature 1.0 and top-p 0.95.
- One pass each. With 113 tasks and no repeat passes, a gap of a few tasks (62 against mini-swe-agent's 56) is small. Do not read the order of close rows as settled.
- CodeAF ran a day later, on 2026-09-12 UTC, after the other nine (2026-09-10 to 11).
- Release timing. Most competing versions were released within a week of the campaign; mini-swe-agent 2.4.6 was about seven weeks old but still the latest published release. The comparison does not establish how newer versions perform.
- OpenCode was 1.x. 1.18.30 was the latest production 1.x release at launch; 1.18.31 followed on September 14. This row does not measure OpenCode 2.
- Claude Code was pinned before a patch release. 2.1.268 was published on September 10 at 20:30:54 UTC, before 112 of the arm's 113 tasks started. Its notes include a fix for HTTP 400 responses on third-party Anthropic-compatible endpoints, and the campaign used such an endpoint. Whether that fix would change the result has not been established.
- DeepSeek Harness had a newer release candidate. 0.1.5-rc.2 was published before the arm ran; the frozen package lock selected rc.1.
- The model was DeepSeek V4 Flash for everyone. The Claude Code row measures Claude Code driving DeepSeek V4 Flash, not Claude Code with Claude models.
DeepSWE versus SWE-bench Verified
DeepSWE, published by Datacurve, is "a benchmark for measuring frontier coding agents on original, long-horizon software engineering tasks": 113 tasks across TypeScript, Go, Python, JavaScript and Rust, each graded by a verifier that "accepts any solution whose observable behavior is correct" (DeepSWE README). SWE-bench Verified is "a human-validated subset of 500 SWE-bench instances."
The axis differs too. DeepSWE's own leaderboard ranks models, with "all models run on mini-swe-agent for consistency." Our comparison holds the model fixed and changes the harness, so it measures what the agent around the model adds. Epoch AI's review of DeepSWE v1.1 (2026-09-07) found "issues in at least 23 out of 113 tasks" and rated the benchmark "Flawed". Every harness here was graded by the same verifiers on the same tasks, but that noise is one more reason not to over-read small gaps.
Later runs of the same mode (not a comparison)
On the same 113 tasks, later runs of CodeAF's senior-dev mode solved 88 (77.9%, exact 95% CI 69.1% to 85.1%) with DeepSeek V4.1 Flash, and 78 (69.0%, 59.6% to 77.4%) with Kimi K3. In the method page's words, these figures are "this mode with those models. They are not a second comparison against the other harnesses." No other harness was run on those models.
How to reproduce
The scripts that run and grade DeepSWE tasks are in bench/deepswe/ in the CodeAF repository. One task through CodeAF:
bench/deepswe/run.sh <task-id> <model-id> <seed-tag>It needs a linux/amd64 codeaf binary in CODEAF_BIN, a provider key, and the DeepSWE task corpus, which Datacurve distributes as a gated dataset. The compare-* scripts beside it install a pinned comparator, freeze its container image and run it. The per-task records of this campaign are public, so every table on this page can be recounted from them. A reproduction is still a fresh run, and its numbers can differ from ours.
Questions people ask
Is CodeAF number one on DeepSWE?
CodeAF was first of ten harnesses in AgentField's own DeepSWE comparison, solving 62 of 113 tasks (54.9%) with every harness on DeepSeek V4 Flash. It is not a rank on a public leaderboard; DeepSWE's own leaderboard ranks models running on mini-swe-agent. Say "first of ten harnesses on one model", not "#1 on DeepSWE".
Which AI coding agent scores highest on SWE-bench style benchmarks right now?
It depends on the benchmark and on whether it ranks models or harnesses. DeepSWE's own leaderboard ranks models, all running inside one harness (mini-swe-agent). When the model is held fixed (DeepSeek V4 Flash) and the harness changes, CodeAF solved the most DeepSWE tasks of the ten harnesses in AgentField's comparison, 62 of 113.
What is the most cost-efficient coding agent per solved issue?
In AgentField's DeepSWE comparison on DeepSeek V4 Flash, CodeAF had the lowest cost per solved issue, 39.9 cents, ahead of mini-swe-agent (1.9x), Codex CLI (2.1x) and Claude Code (3.4x). Claude Code and Muse Code cost less per task (19 and 12 cents against CodeAF's 22) but solved fewer tasks. Cost per solved issue depends on the model as much as the harness.
Does this mean CodeAF beats Claude Code?
It means that on DeepSeek V4 Flash, Claude Code 2.1.267 solved 16 tasks and CodeAF solved 62. Both ran DeepSeek, not Claude; Claude Code reached it through a third-party Anthropic-compatible endpoint, and a patch released during the run (2.1.268) fixed HTTP 400 responses on such endpoints. It measures the harness on that model, not Claude Code with Claude models.
Is 77.9% part of the comparison?
No. 77.9% (88 of 113, exact 95% CI 69.1% to 85.1%) is a later run of CodeAF's senior-dev mode with DeepSeek V4.1 Flash, and 69.0% is the same mode with Kimi K3. No other harness ran on those models, so they are not a second comparison.
Can I reproduce these results?
Partly. The run and grading scripts are in bench/deepswe/ in the CodeAF repository, and the per-task records of all ten harnesses are public, so every count here can be checked. The DeepSWE corpus is a gated dataset, and a reproduction is a fresh run on the same rig, not a re-grade of ours.
Sources
- DeepSWE method page, with
arms.csvandversions.csvbeside it: every number on this page, checked 2026-11-21. - Per-task records, one CSV per harness, checked 2026-11-21.
bench/deepswe/in the CodeAF repository, checked 2026-11-21.- senior-dev docs, checked 2026-11-21.
- DeepSWE: deepswe.datacurve.ai, README, dataset, checked 2026-11-21.
- Epoch AI review of DeepSWE, checked 2026-11-21.
- SWE-bench Verified, checked 2026-11-21.
- Release pages of each harness, linked in the versions table, checked 2026-11-21.
For how CodeAF compares beyond this benchmark, see CodeAF vs Claude Code and CodeAF vs Codex CLI. For the 3-hour, unattended way senior-dev works, see a coding agent that runs for hours, and for the terminal side, terminal coding agents.
Last updated 2026-11-21. Every figure checked against arms.csv and versions.csv at the method page; harness versions are those of the 2026-09-10 to 12 campaign.