GPT-5.6 lost our coding benchmark. We switched to it anyway.
We ran the new GPT-5.6 Sol, Terra, and Luna models on Superconductor's Custom SWE-Bench, which evaluates coding agents on our own production Ruby on Rails codebase. All three GPT-5.6 models are available now in Superconductor.
The benchmark is built from a set of ground-truth PRs: substantial, real tickets whose merged implementations our team is proud of and considers strong examples of good engineering work. Agents receive specs inferred from those PRs, but never see the original solutions. So the per-ticket timings below represent real development work, not toy exercises.
TLDR:
- Fable 5 is still the clear quality winner on our repo.
- Opus 4.8 beats GPT-5.6 Sol at every effort level we tested.
- GPT-5.6 dominates on speed and cost. The new models finish tickets about 5x faster than Opus and Fable.
- We switched our daily driver to GPT-5.6 Sol High. The fastest model our team can actually use all day is more useful than the benchmark winner we keep running out of.
Fable 5 still writes the best code
On absolute quality, the result is not particularly close:
- Fable 5: about 83%
- Opus 4.8: about 80%
- GPT-5.6 Sol High: about 77%
- GPT-5.6 Terra High: about 70%
- GPT-5.6 Luna High: about 66%
Even Sol, the strongest GPT-5.6 variant, does not catch Opus 4.8 on our repo. More reasoning effort did not change that ordering.
The GPT-5.6 family does own the cheap side of the chart. Sol, Terra, and Luna sit on the cost Pareto frontier: depending on how much quality you need, each gives you a new efficient option.
Composer 2.5 Fast remains a standout surprise. It lands near Sol on quality and cost. Despite the name, though, it is still slower.
The speed difference is not subtle
Fable 5 and Opus 4.8 take roughly 23 to 25 minutes per ticket in this run. The GPT-5.6 models finish in roughly 5 to 6 minutes.
That changes how the model feels in actual use. Five minutes is short enough to stay in the loop. Twenty-five minutes is long enough to start doing something else, forget what you asked for, and come back with different context.
Grok 4.5 XHigh is also on the speed Pareto frontier, landing near Opus 4.8 quality in about nine minutes. We still do not know what it costs, unfortunately, so it cannot join the cost comparison yet.
Why GPT-5.6 still loses on quality
The score is not just benchmark noise. Looking at the failures, GPT-5.6 sometimes writes code that is simply invalid. This happened even with Sol.
Its taste is also not on par with Fable 5 or Opus 4.8. The code can solve the narrow problem while missing the shape of the surrounding system: the abstraction is off, the implementation is awkward, or the change is harder to maintain than it needed to be.
This is exactly why we benchmark against real merged pull requests. A generic coding benchmark can tell you whether a model completes a task. A benchmark built from your own repo can tell you whether it writes code your team would actually want to merge.
OpenAI and Anthropic are playing different games
One meta-observation from the chart: Anthropic tends to release a new flagship when it is clearly better. Our benchmark shows a clean quality progression from Opus 4.6 to 4.7 to 4.8 to Fable 5.
OpenAI ships more variants, more often. That gives developers more points on the quality, speed, and cost curve, but it also adds noise. A newer model is not necessarily a better model for your codebase. Older versions sometimes win.
We do not think one strategy is obviously correct. The Anthropic progression makes model selection easy. The OpenAI menu makes it possible to pick a more precise tradeoff.
Why we switched anyway
We switched our daily driver to GPT-5.6 Sol High.
Fable 5 still produces the best code on our benchmark, and we would pick it when quality is the only constraint. But we are constantly running out of Fable 5 usage. GPT-5.6 Sol High is fast, strong enough, and available when our team needs it.
On tickets where the tradeoff is not obvious, we use Implementation Recommendations. We run multiple agents on the same task, then Superconductor evaluates the implementations and recommends the strongest one. This is especially useful when a ticket is worth throwing several agents at and the right quality, speed, and cost tradeoff is hard to predict upfront.
The benchmark winner is not automatically the best daily driver. The useful question is which model gives you the right combination of:
- quality
- cost
- availability
- fit for your codebase and coding style
But don't take our result, or any single benchmark result, at face value. Build your own benchmark and see how GPT-5.6, Fable, Opus, and the rest perform on your codebase. We are offering a limited number of free custom benchmarks for teams. Let us know if your team is interested.