← Back to the post

Claude Opus 5.5 vs Claude Sonnet 5: baseline coding comparison
Prepared 2026-10-12
Models claude-opus-5-5 and claude-sonnet-5, both through Claude Code 2.1.282
Runs Sonnet 5: 59 runs (L01–L12 on 2026-10-08, L13–L20 on 2026-10-12). Opus 5.5: 60 runs on 2026-10-09
Setup No skills or plugins, no web access, each run in a fresh isolated container

1. Summary

Overall. On the 12 easier tasks the models are close: both are correct, and Opus is faster but only somewhat cheaper. On the 8 harder tasks Opus is clearly ahead on every measure except code quality, where the two are level. Opus passed every hidden test in all 60 of its runs; Sonnet did so in 52 of 59.

On the 12 easier tasks (L01–L12):

  • Correctness. Both models produced working code on every task. Opus passed every hidden test in all 36 of its runs. Sonnet did so in 33 of 35. Its other two runs (both on L01) each failed 2 of 270 tests.
  • Cost and speed. Opus used fewer tokens and finished faster on every one of the 12 tasks. Summed over the per-task medians, Opus used 67% fewer tokens (1.35 M against 4.16 M) and 59% less time (15 min 6 s against 36 min 55 s).
  • Dollar cost at API list prices. Opus was about 24% cheaper in total ($2.92 against $3.83, summed over the per-task median cost of a run). That’s much less than its 67% token saving, for two reasons. Opus charges twice Sonnet’s rate for input, cache writes and output. And most of the tokens it saves are cache reads, which cost the same $0.20 per million on both models. Task by task, Opus was cheaper on 5 of the 12 and more expensive on 7. The two longest tasks, L07 and L10, account for more than the whole saving: Sonnet cost 2.7× and 2.5× as much on them.
  • Subscription usage. The two models used about the same share of the five-hour usage allowance per run: 2.2 points per run for Opus and 2.3 for Sonnet. Opus’s lower token count did not make it cheaper to run on a subscription.
  • Consistency of solution size. The two are about equal. The average spread in lines of code between repeat runs was 7.8% for Opus and 7.6% for Sonnet.
  • Consistency of growth class. Sonnet was more consistent. The growth class is how a solution’s work grows with input size, such as n or n². Across repeat runs, Sonnet landed in the same class on 7 of 9 measured tasks and Opus on 5 of 9. The cases where Opus varied are mixed: it was sometimes better than Sonnet and once worse (§3.3).
  • Readability. Both models met the readability floor equally often: 16 of 35 Sonnet runs and 16 of 36 Opus runs.

On the 8 harder tasks (L13–L20, §4):

  • Correctness separates the models. Opus passed every hidden test in all 24 runs. Sonnet did so in 19 of 24. On L13 no Sonnet run passed every test (best: 66% as stored, 70% on regrading, §4.1). On L16 one of three did; the other two were stopped at the 20-minute time limit with 55% and 47% passing.
  • Cost and speed. Opus used fewer tokens and finished faster on all 8 tasks. Summed over the per-task medians, Opus used 86% fewer tokens (1.18 M against 8.64 M) and 79% less time (16 min 54 s against 81 min 25 s).
  • Dollar cost at API list prices. Opus was 58% cheaper ($2.64 against $6.32, summed over per-task medians). It was cheaper on 7 of the 8 tasks; on L16 the two are within 2%.
  • Subscription usage. Sonnet used about 3× as much of the five-hour allowance per run: 7.3 points against Opus’s 2.3.
  • Consistency of solution size. Opus was slightly more consistent: an average spread of 13.4% against Sonnet’s 15.8%. Sonnet wrote the larger solution on all 8 tasks by median.
  • Growth class. Identical. On all 6 measured tasks, both models’ runs had the same average growth class.
  • Readability. No run from either model met the readability floor on these tasks.

These figures come from 2–3 runs per task per model. As more runs from both models are added, the averages will firm up, and some figures in this report may change slightly.

2. How the comparison was run

Tasks. Eurystheus is a benchmark harness. It gives a model a fixed ladder of coding tasks, called labours, in five tiers of difficulty:

  • L01–L04 are tier 1, L05–L08 tier 2 and L09–L12 tier 3. These are, in order, everyday fixes and parsers, algorithms and refactors, and performance work and interpreters.
  • L13–L16 are tier 4: a collaborative-text data structure, a bytecode VM, a stiff ODE solver and robust Delaunay triangulation.
  • L17–L20 are tier 5: mathematical tasks.

Each run gets a freshly generated variant of its task, from a random seed. Repeat runs are therefore similar problems, not identical ones, and some of the variation reported here comes from the tasks themselves.

Conditions. Each run took place in its own Docker container with:

  • a blank Claude Code configuration (no personal skills, plugins or connectors);
  • only the Anthropic API reachable on the network;
  • permission mode auto.

The hidden tests, reference solutions and grading run in a separate container after the agent’s container has exited, so the model never sees them.

What was measured, per run:

Measure Meaning
Pass rate Share of the hidden tests that passed (8–2,001 tests per task, depending on the task)
Lines added Lines of code the solution added to the starting files
Growth class How the solution’s operation count grows with input size, such as n, n log n or n², fitted from exact bytecode operation counts at several input sizes. This is measured for 9 of the 12 shared tasks and 6 of the 8 harder ones.
Readability floor Pass or fail. Every function must have cyclomatic complexity ≤ 15, length ≤ 80 lines and nesting ≤ 4, and no single-letter names except agreed mathematical symbols
Tokens Input + output + cache-read + cache-write tokens, as Claude Code reports them. Cache reads make up most of this total
Output tokens Tokens the model generated
Time Wall-clock time of the agent run
Usage Change in the account’s five-hour usage figure, as Claude Code’s live rate-limit events report it
Cost What the run’s tokens would cost at Anthropic’s published API list prices (§3.2)

Trials. 3 runs per task per model, except Sonnet L12, which has 2 because its session stopped at the usage limit after two runs. A run is stopped if it reaches 1,200 s (20 minutes), and graded on whatever it had written by then.

3. Head-to-head results, L01–L12

3.1 Correctness

Sonnet 5 Opus 5.5
Runs 35 36
Runs that finished normally 35 36
Runs passing every hidden test 33 36
Lowest pass rate in any run 99.26% (L01, 268 of 270) 100%

The tasks don’t separate the two models on correctness: both are effectively at the ceiling.

3.2 Cost and speed

Per-task medians:

Labour Tokens, Sonnet Tokens, Opus Output tokens, Sonnet Output tokens, Opus Time, Sonnet Time, Opus Turns, Sonnet Turns, Opus Cost, Sonnet Cost, Opus
L01 91k 53k 2.6k 2.7k 25 s 23 s 3 3 $0.083 $0.123
L02 115k 48k 0.5k 0.7k 10 s 9 s 5 3 $0.035 $0.034
L03 95k 52k 3.9k 2.3k 41 s 21 s 3 3 $0.088 $0.101
L04 260k 75k 4.3k 3.1k 56 s 33 s 12 4 $0.138 $0.144
L05 99k 55k 4.5k 3.7k 57 s 38 s 3 3 $0.102 $0.138
L06 286k 100k 13.2k 5.2k 130 s 65 s 7 5 $0.269 $0.200
L07 734k 106k 30.5k 6.7k 341 s 69 s 12 6 $0.622 $0.232
L08 231k 92k 6.2k 3.7k 64 s 33 s 7 5 $0.145 $0.158
L09 223k 113k 5.7k 5.5k 71 s 68 s 7 6 $0.125 $0.172
L10 941k 157k 54.9k 11.4k 642 s 115 s 14 6 $0.959 $0.382
L11 540k 293k 32.8k 12.5k 364 s 232 s 10 9 $0.562 $0.528
L12 547k 210k 41.6k 23.1k 412 s 201 s 9 6 $0.699 $0.706
Sum of medians 4.16 M 1.35 M 201k 81k 2,215 s 906 s $3.83 $2.92

  • Tokens: Opus used fewer on all 12 tasks. The largest gaps are on the longest tasks: L10 took 6.0× as many tokens with Sonnet, and L07 7.0×.

  • Output tokens: Opus generated fewer on 10 of 12 tasks. On L01 and L02, the two smallest, it generated slightly more.

  • Time: Opus was faster on all 12 tasks by median. The gap is small on L01, L02 and L09 (within 3 s) and large on L07 (5.0×) and L10 (5.6×).

  • Turns: Opus used the same number of turns or fewer on every task.

  • Cost: the cost columns show the median cost of one run, priced at Anthropic’s API list prices on 2026-10-11 (platform.claude.com/docs/en/about-claude/pricing):

    Per million tokens Input Cache write (1 hour) Cache read Output
    Opus 5.5 $4.00 $8.00 $0.20 $20.00
    Sonnet 5 $2.00 $4.00 $0.20 $10.00
    • How the figures were checked. Every cost is calculated at the 1-hour cache-write rate, which Claude Code uses. Claude Code also records a list-price figure for each run that finishes, which gives a check. Of the 119 runs in this report, 114 recorded that figure, and the calculated cost matches it to the cent for 113 of them. The exception is one Sonnet L16 run, where Claude Code recorded $1.075 against $1.180 calculated, probably because part of that run’s cache was written at the cheaper 5-minute rate. It doesn’t change any median. The 5 runs stopped at the time limit (§4) recorded no figure, and their cost is calculated the same way.
    • Where each model is cheaper. Opus is cheaper on L02, L06, L07, L10 and L11. Sonnet is cheaper on L01, L03, L04, L05, L08, L09 and L12. On L02, L04 and L12 the two are within 5%.
    • What these figures are and aren’t. They are what the same work would cost through the API at list price. A Claude Pro or Max subscription, which these runs used, charges a flat fee, not per token.

Subscription usage per run:

Session Runs Usage figure at start → end Points per run
Sonnet, L01–L12 35 6% → 88% 2.34
Opus, L01–L12 35 10% → 87% 2.20
Opus, L13–L20 24 4% → 60% 2.33

The usage figure is reported in whole percent. It also counts any other use of the same account during the session, so these are approximate. Within that precision, the two models cost the same per run in subscription usage on these tasks, even though Opus used a third of the tokens. That suggests Opus uses up the allowance faster per token. This report doesn’t measure that directly. On the harder tasks, Sonnet’s much larger token use outweighs this, and it used about 3× as much allowance per run (§4.2).

3.3 Consistency across repeat runs

Correctness: Opus passed every test in all 36 runs. Sonnet’s runs were consistent except L01, where 2 of 3 runs each missed 2 of 270 tests.

Solution size (lines added, one value per run):

Labour Sonnet Opus
L01 89, 92, 107 97, 93, 91
L02 1, 1, 1 1, 1, 1
L03 89, 69, 73 82, 72, 77
L04 36, 36, 35 27, 34, 39
L05 60, 66, 72 102, 81, 84
L06 147, 159, 136 124, 127, 114
L07 111, 168, 168 141, 142, 155
L08 155, 137, 143 186, 180, 175
L09 16, 13, 18 60, 37, 31
L10 253, 210, 246 240, 235, 242
L11 430, 554, 497 546, 385, 437
L12 793, 829 1096, 1115, 1020
Average spread (coefficient of variation) 7.6% 7.8%
  • Spread: the average spread between repeat runs is about the same for both models.
  • Most variable: Sonnet on L07 (111–168 lines) and Opus on L09 (31–60 lines).
  • Size: neither model consistently writes larger solutions. Opus’s median is larger on 6 tasks, smaller on 5 and equal on 1. The clear exceptions are L12, where Opus wrote 1,020–1,115 lines against Sonnet’s 793–829, and L09, where it wrote 31–60 against 13–18.

Growth class (one value per run; the 9 tasks where it’s measured):

Labour Sonnet Opus
L01 n log n ×3 n log n ×3
L03 n² ×3 n², n log n, n²
L05 n ×3 n ×3
L07 n² log n, n², n² n², n³, n²
L08 n² ×3 n², n, n
L09 n log n ×3 n log n ×3
L10 n², n log n, n n, n, n²
L11 n ×3 n ×3
L12 n ×2 n ×3
Same class in every run 7 of 9 5 of 9

Opus is less consistent here, but its variation goes both ways:

  • Better than Sonnet: on L08, two Opus runs reached n, a better class than Sonnet reached in any run. On L03, one Opus run reached n log n.
  • Worse than Sonnet: on L07, one Opus run landed at n³, the worst class either model produced on any task.
  • Both varied: on L10, both models varied; Opus reached n in two runs and Sonnet in one.

This report doesn’t say whether a change of class reflects a genuinely different algorithm or a borderline fit between two classes. The solutions have not been inspected individually.

Did either model consistently write faster-scaling code? No, not consistently, though Opus has a slight edge. To average growth classes, each class gets its rank on the harness’s own scale, from best to worst: 1, log n, n^½, n^¾, n = 4, n log n = 5, n² = 6, n² log n = 7, n³ = 8, 2ⁿ = 9. Lower ranks mean the code’s work grows more slowly as inputs get bigger.

Sonnet 5 Opus 5.5
Average rank (mean per task, averaged over the 9 tasks) 5.04 4.85
Tasks with the lower (better) average rank 1 (L07) 3 (L03, L08, L10)
Tasks with the same average rank 5 5
  • The overall gap is small. Opus averages 0.19 of a rank lower, about a fifth of one class, with both models just above n log n.
  • Head to head, most comparisons tie. Pairing every Sonnet run with every Opus run on the same task gives 78 comparisons. 58 are ties (74%), Opus is in the better class in 15 (19%), and Sonnet in 5 (6%).
  • The edge comes from a few tasks. Most of it is L08, where two of the three Opus runs reached n and every Sonnet run stayed at n². It’s partly offset by Opus’s single n³ run on L07.
  • Neither model was ever worse on every run. On no task was every run of one model in a worse class than every run of the other.
  • What this does and doesn’t show. With 2–3 runs per task, this difference is not statistically significant. Growth class measures how the number of operations grows with input size, not measured run time.
  • Across all 20 tasks. On the harder tasks the two models’ average ranks are identical (§4.3). Over all 15 measured tasks, the average rank is 4.78 for Sonnet and 4.67 for Opus.

3.4 Readability floor

Labour Sonnet: runs passing Opus: runs passing Sonnet: violations per run Opus: violations per run
L01 0/3 0/3 3, 2, 2 3, 3, 3
L02 3/3 3/3 0, 0, 0 0, 0, 0
L03 0/3 0/3 3, 3, 2 3, 2, 2
L04 3/3 3/3 0, 0, 0 0, 0, 0
L05 2/3 1/3 1, 0, 0 0, 1, 1
L06 3/3 3/3 0, 0, 0 0, 0, 0
L07 1/3 1/3 2, 1, 0 0, 2, 3
L08 0/3 0/3 4, 5, 6 8, 4, 4
L09 3/3 3/3 0, 0, 0 0, 0, 0
L10 1/3 2/3 1, 0, 4 0, 0, 1
L11 0/3 0/3 14, 14, 13 16, 12, 13
L12 0/2 0/3 12, 13 52, 57, 52
Total 16/35 16/36
  • Same tasks pass and fail for both models. Both models passed in every run on the same 4 tasks (L02, L04, L06, L09) and failed in every run on the same 5 (L01, L03, L08, L11, L12). Both were mixed on the remaining 3. The one large difference is L12, where Opus had about 4× as many violations as Sonnet, in line with its larger solutions there.
  • Small violation counts on easier tasks: on L01 and L03, both models fail the floor on just 2–3 violations. Single-letter names, such as a and b in comparison functions, account for all 16 of the L01 violations and 8 of the 15 on L03. The rest of L03’s are complexity, nesting and length.

4. Head-to-head results, L13–L20

Both models’ L13–L20 runs used the same container images and the same harness commit (43710b0).

4.1 Correctness

Labour Hidden tests Sonnet: pass rate per run Opus: pass rate per run
L13 300 0%, 0%*‡ (70% regraded), 66% 100% ×3
L14 161 100% ×3 100% ×3
L15 60 100% ×3 (one stopped*) 100% ×3
L16 201 100%, 55%, 47% 100% ×3 (one stopped*)
L17 12 100% ×3 100% ×3
L18 8 100% ×3 100% ×3
L19 24 100% ×3 100% ×3
L20 40 100% ×3 100% ×3
Sonnet 5 Opus 5.5
Runs 24 24
Runs that finished normally 20 23
Runs stopped at the 1,200 s time limit 4 1
Runs passing every hidden test 19 24

* A run marked * reached the 1,200 s time limit and was stopped. It is graded on whatever solution it had written by then. Two stopped runs, Sonnet’s on L15 and Opus’s on L16, had already written solutions that pass every test.

‡ This run’s stored grade is 0%, caused by a defect in the harness, not the solution (limitation 4). Regraded with the defect corrected, it passes 210 of 300 tests (70%). The stored evidence is unchanged.

  • L13 (a collaborative-text data structure) is where the models differ most. Every Opus run passed all 300 tests, in about 97 s each. No Sonnet run did:
    • one finished at 0% because its solution lacks a text() method that the task’s specification requires, so every test errors;
    • one was stopped at the time limit, and scores 70% on regrading;
    • one finished with 198 of 300 tests passing (66%).
  • L16 (robust Delaunay triangulation): one Sonnet run passed every test. The other two were stopped at the time limit, with 111 and 94 of 201 tests passing.
  • This report doesn’t analyse why the remaining tests failed.

4.2 Cost and speed

Per-task medians:

Labour Tokens, Sonnet Tokens, Opus Output tokens, Sonnet Output tokens, Opus Time, Sonnet Time, Opus Turns, Sonnet Turns, Opus Cost, Sonnet Cost, Opus
L13 825k† 109k 66.6k† 9.4k 884 s 97 s 9 5 $1.109† $0.279
L14 729k 200k 56.3k 17.2k 575 s 150 s 11 7 $0.934 $0.548
L15 3.63 M† 142k 57.7k† 10.2k 1,164 s 221 s 42 6 $1.685† $0.309
L16 1.54 M† 412k n/a† 29.6k† 1,200 s 382 s 14 10 $0.906† $0.922†
L17 447k 56k 15.9k 3.9k 196 s 41 s 9 3 $0.355 $0.143
L18 344k 75k 30.0k 4.1k 350 s 42 s 7 4 $0.480 $0.154
L19 411k 107k 19.4k 4.0k 214 s 43 s 10 6 $0.348 $0.134
L20 713k 76k 24.8k 3.7k 303 s 39 s 15 4 $0.501 $0.153
Sum of medians 8.64 M 1.18 M 271k (excl. L16) 52k (excl. L16) 4,885 s 1,014 s $6.32 $2.64

† The median includes a run stopped at the time limit. A stopped run never sends Claude Code’s final summary, so the harness counts its tokens from the messages streamed before the stop. That count is a lower bound, and output tokens in particular are badly undercounted (37–224 output tokens per stopped run). Medians marked † may therefore be low. Sonnet’s L16 output median isn’t shown because two of its three runs were stopped, which makes it meaningless.

  • Tokens: Opus used fewer on all 8 tasks. The largest gap is L15, at 25.5× as many tokens with Sonnet: every Sonnet run took 38–42 turns, against 3–23 for Opus.
  • Time: Opus was faster on all 8 tasks by median, by 3.1× (L16, where Sonnet’s median is the time limit) to 9.1× (L13).
  • Turns: Opus used fewer turns on all 8 tasks.
  • Cost: Opus was cheaper on 7 of 8 tasks, by 1.7× (L14) to 5.5× (L15). On L16 the two are within 2%, but two of Sonnet’s three L16 costs are lower bounds. Rates are as in §3.2.
  • Total spent. Adding every run rather than medians, the 24 runs cost $18.58 for Sonnet and $8.58 for Opus.

Subscription usage per run:

Session Runs Usage figure at start → end Points per run
Sonnet, L13–L15 7 15% → 87% 10.3
Sonnet, L15–L19 12 0% → 86% 7.2
Sonnet, L19–L20 5 0% → 17% 3.4
Sonnet, all L13–L20 24 7.3
Opus, L13–L20 24 4% → 60% 2.3

Sonnet’s L13–L20 runs needed three five-hour windows; Opus’s fit in one. The heaviest single run was a Sonnet L15 run, which used 20 points (67% → 87%) on its own.

4.3 Consistency across repeat runs

Solution size (lines added, one value per run):

Labour Sonnet Opus
L13 362, 277, 282 300, 224, 246
L14 979, 946, 936 732, 1002, 903
L15 278, 175, 256 279, 210, 202
L16 348, 504, 228 372, 277, 289
L17 81, 83, 77 54, 57, 69
L18 270, 137, 248 122, 58, 114
L19 104, 113, 164 62, 66, 54
L20 153, 157, 193 104, 99, 91
Average spread (coefficient of variation) 15.8% 13.4%
  • Spread: Opus was slightly more consistent. Both spread more than on L01–L12, where both were under 8%. Over all 20 tasks, the average spread is 10.9% for Sonnet and 10.0% for Opus.
  • Size: Sonnet wrote the larger solution on all 8 tasks by median, most clearly on L18 (248 lines against 114) and L19 (113 against 62). This is the reverse of L12, Opus’s largest gap the other way.

Growth class (one value per run; the 6 tasks where it’s measured):

Labour Sonnet Opus
L13 n log n (1 run measured§) n log n ×3
L14 n ×3 n ×3
L16 n log n, n², n log n n², n log n, n log n
L17 n^¾ ×3 n^¾ ×3
L18 log n ×3 log n ×3
L19 n³ ×3 n³ ×3
Same class in every run 4 of 5 (L13 excluded) 5 of 6

§ A growth class is measured only for a solution that runs. The two Sonnet L13 runs graded 0% have none.

  • Identical averages. On every one of the 6 tasks, the two models’ average growth rank is the same. On L16 each model had one n² run and two n log n runs. Neither model wrote faster-scaling code on these tasks.

4.4 Readability floor

This many failures is indicative of bad grading rather than actually bad code. We will have to look at the code itself written and see if it’s actually this illegible or if our testing is bad.

Labour Sonnet: violations per run Opus: violations per run
L13 14, 6, 8 11, 10, 5
L14 18, 18, 20 25, 33, 29
L15 9, 2, 6 7, 6, 6
L16 18, 12, 6 6, 5, 4
L17 2, 3, 2 2, 2, 2
L18 6, 4, 7 4, 1, 2
L19 7, 2, 2 3, 3, 2
L20 6, 7, 5 4, 2, 3
Total 190 177
  • No run met the floor. All 48 runs failed it.
  • Violation counts are similar. Opus had fewer violations in total on 6 of 8 tasks. It had more on L15 (19 against 17) and on L14, where its runs had 25–33 violations each against Sonnet’s 18–20.
  • Over all 20 tasks, each model met the floor in 16 runs: 16 of 59 for Sonnet and 16 of 60 for Opus.

4.5 Scaling with task difficulty and fix length

The five tiers are the benchmark’s own difficulty ladder (§2). Fix length is the number of non-blank lines the task’s reference solution adds to the starting code: a measure of how much code the task needs, independent of either model. Each figure below is a mean over the tier’s 4 tasks of each task’s median run. Ratios are Sonnet’s tier mean divided by Opus’s.

Tier Task types Fix length, lines (mean; range) Tokens, Sonnet Tokens, Opus Token ratio Time, Sonnet Time, Opus Cost, Sonnet Cost, Opus Cost ratio Runs passing every test, Sonnet Runs passing every test, Opus
1 Everyday fixes and parsers (L01–L04) 29; 3–46 140k 57k 2.5× 33 s 21 s $0.086 $0.100 0.86× 10/12 12/12
2 Algorithms and refactors (L05–L08) 78; 59–116 338k 88k 3.8× 148 s 51 s $0.285 $0.182 1.56× 12/12 12/12
3 Performance work and interpreters (L09–L12) 175; 20–329 563k 193k 2.9× 373 s 154 s $0.586 $0.447 1.31× 11/11§ 12/12
4 Data structures, VMs, numerics, geometry (L13–L16) 199; 79–464 1.68 M† 216k 7.8× 956 s 212 s $1.159† $0.514† 2.25× 7/12 12/12
5 Mathematical (L17–L20) 131; 120–140 479k 78k 6.1× 266 s 41 s $0.421 $0.146 2.88× 12/12 12/12

† Includes runs stopped at the time limit, whose token counts are lower bounds (§4.2). § Counts Sonnet’s two L12 runs as passing, after their regrade (limitation 3).

  • Sonnet’s token use grows much faster with difficulty. From tier 1 to tier 4, Opus’s tokens per task grew 3.8× (57k to 216k) and Sonnet’s 12× (140k to 1.68 M). Median turns grew from 3.2 to 7 for Opus and from 5.8 to 19 for Sonnet. The time ratio widens the same way, from 1.6× in tier 1 to 6.5× in tier 5.
  • Correctness falls off only at tier 4. Below that, both models are at or near the ceiling.

Per task, by fix length:

Labour Task Fix length, lines Cost ratio (Sonnet ÷ Opus) Token ratio
L02 Off-by-one in pagination 3 1.02× 2.4×
L09 Make a log analyser ≥ 50× faster 20 0.73× 2.0×
L04 Tags for a bundled todo CLI 22 0.96× 3.5×
L03 RFC 4180 CSV parser 44 0.87× 1.8×
L01 Semver parse and compare 46 0.68× 1.7×
L05 Thread-safe LRU cache with TTL 59 0.74× 1.8×
L06 Behaviour-preserving refactor 65 1.35× 2.9×
L07 Myers diff to unified format 71 2.68× 7.0×
L15 Adaptive solver for stiff ODEs 79 5.46× 25.5×
L16 Robust Delaunay triangulation 114 0.98׆ 3.7׆
L08 JSON Schema subset validator 116 0.92× 2.5×
L17 Prime power sums 120 2.48× 8.0×
L20 40 digits of a slow series 123 3.28× 9.4×
L10 Async job scheduler 130 2.51× 6.0×
L13 Sequence CRDT (RGA) 138 3.98׆ 7.6׆
L18 Constrained grid colourings 140 3.11× 4.6×
L19 Fading choirs 140 2.60× 3.8×
L11 Regex engine with linear time 220 1.06× 1.8×
L12 Mini SQL over CSV 329 0.99× 2.6×
L14 Bytecode compiler and VM with GC 464 1.71× 3.7×

  • Short fixes are where Sonnet pays off. On every task whose fix is 60 lines or fewer (6 tasks, 3–59 lines), Sonnet was cheaper or level: 0.68× to 1.02× Opus’s cost. Above 60 lines, Sonnet cost more than 1.1× as much on 10 of 14 tasks.

  • Past about 60 lines, length stops predicting the gap. The three longest fixes (220–464 lines) are close to level, at 0.99× to 1.71×. The worst ratios are on medium-length fixes: 2.5× to 3.3× on the mathematical tasks (120–140 lines), and 5.5× on the stiff ODE solver (79 lines).

  • Which measure predicts the gap best. Rank correlation with the cost ratio across all 20 tasks:

    Measure Rank correlation with cost ratio
    Difficulty tier 0.67
    Fix length (reference solution) 0.52
    Lines Sonnet added (median) 0.49
    Lines Opus added (median) 0.29
    Tokens Opus used (median) 0.27

    With 20 tasks, a correlation of 0.52 is only borderline meaningful. The 60-line threshold is the firmer finding, because it holds on every task.

  • What makes the long-fix tasks expensive. Above 60 lines, Sonnet cost more than 2.4× as much as Opus on 8 tasks: L07, L10, L13, L15 and L17–L20. Most of these turn on finding a specific algorithm or a piece of mathematics, rather than writing a lot of code. Sonnet spent much longer iterating on them; on L15, for example, it took 38–42 turns against Opus’s 3–23. L11 is an exception to that reading: a linear-time regex engine also needs a specific algorithm, yet the two models cost about the same on it.

Where Sonnet makes sense. In this benchmark, only narrowly.

  • On a Pro or Max subscription, Opus is the better choice for every task tier. It used the same allowance per run as Sonnet on L01–L12 (2.2 against 2.3 points) and a third as much on L13–L20. It was also faster on all 20 tasks and passed every test in all 60 runs.
  • Billed per token through the API, Sonnet is cheaper only on short fixes, of about 60 lines or fewer.
    • How much it saves: on those 6 tasks, Sonnet’s median cost summed to $0.571 against Opus’s $0.712, about 20% less, or roughly 2 cents a task.
    • What it gives up there: it was slower on all 6, and 2 of its 18 runs on them missed tests.
    • What happens when it loses: the extra cost is large, as on L15 ($1.69 against $0.31).
  • A mixed workload favours Opus. It’s rarely clear in advance which tasks will turn out hard. Over all 20 tasks, the per-task medians sum to $5.56 for Opus and $10.15 for Sonnet, so Opus is 45% cheaper.
  • When Sonnet is still reasonable: a high volume of short, well-specified fixes, where a test suite catches mistakes cheaply and speed doesn’t matter.
  • This is a no-skill baseline with Sonnet 5. A skill that guides Sonnet’s approach, or a newer Sonnet, could narrow the gap. That is what Eurystheus is built to measure next.

5. Limitations

  1. Small samples. There are 2–3 runs per task per model, so this report gives ranges, not significance tests. A single run can move a median, and as more runs from both models are added, the averages will firm up and some figures here may change slightly.
  2. Different task variants. Each run gets a newly generated variant of its task, so differences between runs mix the model’s own variability with differences between task variants.
  3. The Sonnet L12 results were regraded. The original grading of Sonnet’s two L12 runs had a harness defect: the hidden tests’ setup step didn’t run, so every test errored for every solution, including the reference. That defect has been fixed. Sonnet’s stored L12 solutions were then regraded with the corrected grader, and both passed all 500 tests. All Opus runs were graded with the corrected grader from the start. The defect affected only L12, the only task whose tests use that setup step.
  4. One Sonnet L13 run was regraded. Each task’s hidden tests load the solution in a way that breaks one valid Python pattern: postponed annotations (from __future__ import annotations) combined with data classes. A solution written that way fails to load, so every test errors. One stored run is affected, a Sonnet L13 run stopped at the time limit. It is graded 0% in the vault, and passes 210 of 300 tests (70%) when regraded with the loader corrected. Every other run of both models was checked, and none is affected. The regrade was done offline and didn’t change the stored evidence.
  5. The container image and harness changed between sessions. This affects only L01–L12: both models’ L13–L20 runs used identical images and harness code. For L01–L12, the agent container image was rebuilt from the same Dockerfile between the Sonnet and Opus sessions, so its digest differs. 59 of the 60 Opus runs used the rebuilt image; the first Opus run (L01) used the same image as Sonnet’s L01–L12 runs. The Dockerfile pins the base image by digest and Claude Code at 2.1.282. Operating-system packages installed at build time, such as git and Node.js, may differ in patch version. The harness code also moved forward between the sessions (from commit 9e0ed94 to 5f22561, then 43710b0). The three changes were the grader fix in limitation 3, an option to choose the model, and reuse of stored runs. None of them changes the task, the prompt or the agent’s container.
  6. Different days. Sonnet ran L01–L12 on 2026-10-08 and L13–L20 on 2026-10-12; Opus ran all 20 on 2026-10-09. Any change in the service between those days is uncontrolled.
  7. Token counts for stopped runs are lower bounds. A run stopped at the time limit doesn’t send Claude Code’s final usage summary, so its tokens are counted from the messages streamed before the stop, which undercounts output tokens most. This affects 5 runs: four Sonnet runs (L13, L15 and two on L16) and one Opus run (L16). Their tokens and costs, and the medians marked † in §4.2, may be low. Since four of the five are Sonnet’s, the true gap in Opus’s favour is, if anything, wider than shown.
  8. Approximate usage figures. The usage figure is reported in whole percent, and it also counts any other use of the same account during a session.
  9. Correctness is at the ceiling on L01–L12. Without any skill, both models solve the easier tasks, so correctness can’t tell the models apart there. It does on L13 and L16.

6. Data sources

Every run’s manifest, run record, measurements, solution archive and diff are kept, unchanged, in the private evidence vault. A separate check recomputed every figure in this report from that evidence.

Model Session Labours Runs
Sonnet 5 suite 0cf632a5da6e4d0d99ff641f78093a96 L01–L12 35
Opus 5.5 suite b0dc8a487ab1… (single test run) L01 1
Opus 5.5 suite bf7fb7773530… L01–L12 35
Opus 5.5 suite c554bf2ea70f… L13–L20 24
Sonnet 5 suite 8e07ccb07d6d… L13–L15 7
Sonnet 5 suite d2966090345e… L15–L19 12
Sonnet 5 suite d7ec988340e0… L19–L20 5
  • Regrading: the Sonnet L12 regrade (runs b61312d4… and ea926772…) and the Sonnet L13 regrade (run 3b68b565…) were done offline and didn’t change the stored evidence.
  • Excluded runs: 12 earlier Sonnet runs from 2026-09-28 are left out. They were recorded by an older harness version, which carries no runner_version stamp and refused the agent’s file edits, so they contain no real work.