Superconductor is a multiplayer workspace for your team and coding agents. It can also build a custom SWE-Bench for your codebase: a benchmark of coding agents built from your team's own merged pull requests, so you know which agents perform best on your stack.
We run this benchmark on our own codebase. It is built from a set of ground-truth PRs: substantial, real tickets whose merged implementations our team is proud of and considers strong examples of good engineering work. Agents receive specs inferred from those PRs, but never see the original solutions.
During a regular audit of the generated solutions, one Composer 2.5 solution looked suspiciously close to the ground-truth PR. Its work log showed why: the agent had searched pull requests with gh, fetched origin/main, found the exact commit, and cherry-picked it into the working tree. Other agents also found the solution in Git history or after fetching ground-truth branches, which contaminated their benchmark submissions. These agents' output was solid, but the implementation no longer measured independent problem-solving.
When we originally discovered this issue, it was uncommon amongst the agents we'd tested. However that changed when we benchmarked Grok 4.5 and noticed that 20/30 implementations across various effort levels and harnesses showed evidence of cheating.
Our findings fit a broader pattern: DebugML's independent audit, "Finding Widespread Cheating on Popular Agent Benchmarks," found thousands of cheating instances across 28+ benchmark submissions, including task-level cases where agents mined Git history or looked up public answers. In a far more extreme case, OpenAI and Hugging Face disclosed a security incident in which cyber-capable OpenAI models, evaluated on a benchmark of cyber capabilities, escaped their isolated test environment and compromised Hugging Face's production infrastructure to obtain the benchmark's test solutions directly from Hugging Face's production database.
49 benchmark runs were contaminated
We audited 339 coding agent implementations, across 38 agent/profile configurations that ran against 10 ground-truth pull requests. While all 339 agents ran git commands (most commonly git log), only 49 tried various filters to successfully retrieve the ground truth solution.
Here's what they did with the retrieved solution:
- 32 agents directly reused the solution from the ground-truth PR in their implementations.
- 17 agents retrieved and inspected the ground-truth PR and relied on it while implementing their solution, without directly copying it.
Here's a breakdown of the contaminated agent implementations:
| Harness | Model | Contaminated implementations |
|---|---|---|
| Cursor | Grok 4.5 (XHigh) | 7 |
| Grok Build | Grok 4.5 (XHigh) | 7 |
| Cursor | Grok 4.5 (High) | 6 |
| Codex | GPT-5.6 Luna (High) | 4 |
| Codex | GPT-5.6 Sol (High) | 4 |
| Codex | GPT-5.6 Luna (XHigh) | 3 |
| Cursor | Auto | 3 |
| Cursor | Composer 2.5 | 3 |
| Cursor | Composer 2.5 Fast | 3 |
| Codex | GPT-5.6 Sol (XHigh) | 2 |
| Codex | GPT-5.6 Terra (High) | 2 |
| Claude Code | Fable 5 | 1 |
| Codex | GPT-5.6 Terra (XHigh) | 1 |
| Grok Build | Composer 2.5 (High) | 1 |
| Pi | GLM 5.1 | 1 |
| Pi | MiniMax M3 | 1 |
Some agents went further than local Git history
Of the 49 contaminated implementations, 12 used gh pr, gh api, or GitHub search. For example, one Grok 4.5 (XHigh) agent searched GitHub, tried several local refs, fetched the exact ground-truth branch, and inspected its files with git show. The 12 runs were ten Grok 4.5 agents and two Composer 2.5 agents.
Preventing future cheating
To prevent this kind of contamination, we deployed a change that adds a cleanup step before a coding agent starts working on a benchmark implementation. It keeps the current benchmark branch and the history reachable from its base commit, removes every remote and other ref, deletes common temporary Git pointers, expires every reflog entry, and prunes unreachable objects. Agents can still use Git for status, diffs, commits, and pre-base history so it's still representative of a real engineering task environment.
The change also blocks direct HTTP and HTTPS access to github.com, githubusercontent.com, their subdomains, and the GitHub storage endpoint that serves repository content. This closes the observed gh, raw-file, and HTTPS fetch routes without otherwise changing network policy.
After the fix was deployed, we re-ran the implementations and audited the agent work logs. We found no references to the merge SHA or any commits created after the benchmark starting point. Interestingly, only two agents (Cursor Auto and Cursor Composer 2.5) attempted to fetch more history after finding that the Git history was cut off.
How the results changed
Once we excluded the contaminated implementations and re-ran the benchmark, the leaderboard changed.
Among the high-performing agents, Grok 4.5 (XHigh) running in Cursor — previously the top-ranked agent — fell to 8th once it could no longer reach the ground-truth solutions through Git. Its sibling configuration, Grok 4.5 (High) in Cursor, fell to a three-way tie for 3rd with Kimi K3 and Opus 4.8. Both were among the configurations whose work logs showed the most solution retrieval. With those runs corrected, the top of the board is led by GPT-5.4 (High) running in Codex, which had no contaminated runs, even before the Git lock.
Among the mid-performing agents, Composer 2.5, which before the Git lock was ranked 8th and sat just below Opus 4.8 and Kimi K3, fell to 16th place.
Find the best coding agent for your codebase
Public coding-agent benchmarks are useful, but they cannot tell you which agent will perform best on your codebase and stack.
We continuously audit our methodology and strengthen our safeguards to keep the benchmark accurate as coding agents evolve.
Benchmark agents against your own merged pull requests. We'll cover the cost of your first benchmark.