The AI Leaderboard Is a Farce

001

In the summer of 2026, a screenshot made the rounds in AI circles.

It showed DeepSWE’s multipass results. GLM‑5.3 Flash and Claude Fable 5 were 6.3 percentage points apart at pass@1 — 63.4% vs. 69.7%. That looked like a normal generational gap.

But at pass@3 and pass@4, something strange happened.

Both scores converged to about 84‑85%, nearly identical. Two curves that should have stayed apart merged once multiple attempts were allowed.

Statistically, that shouldn’t happen.

If pass@1 differs by six full points, it suggests a real gap in how the two models reason about the same problem. That gap doesn’t disappear; it should widen with more sampling. Instead, it flattened.

The only explanation that holds: some models rely on memory retrieval at pass@1. Retrieval is probabilistic, so first‑try success is unreliable. But give them three or four shots, and they’ll likely land on a problem they’ve already seen in training.

That’s not reasoning. That’s pulling a slot machine.

002

Fidian’s CEO did something blunt.

He took the public Terminal‑Bench 2.1 and replaced its content — carefully. The task descriptions, target scenarios, and complexity stayed the same. But the underlying codebases, file paths, dependencies, and specific contexts were swapped with private versions never seen before. He ran every major model again on this private variant, called TB‑fn.

The results were stark.

GPT‑5.6 Sol and Claude Opus 5 showed almost no change. On some subtasks, their scores actually ticked up. They handled new, unseen scenarios without trouble.

But GLM‑5.3 fell about 7 points from its near‑top position. Grok 4.5, Kimi K3, and GLM‑5.2 dropped between 6 and 11 points each.

Same models, same capability dimension. Only the questions were new — and the scores collapsed.

That’s not a decline in ability. That’s ability that was never there.

003

We ran our own stress tests.

Subjects: DeepSeek and Meta’s Muse Spark. Method: we mutated DeepSWE’s public tasks. The mutations were modest — swapping a codebase’s underlying framework, renaming core functions, adjusting exception‑handling order. Task difficulty didn’t increase; it stayed level.

The results were sobering.

Once the specific codebase deviated from the original, both models’ success rates decayed linearly. More mutations → steeper drop. By the third mutation, they could barely complete a single repair flow.

They had memorised the “standard answer” for that problem, but they hadn’t learned how to solve problems.

As a control, we ran Claude Opus 5 through the same mutated tests. It didn’t drop — it improved. We double‑checked our pipeline. No errors.

The reason: the mutated codebases stripped out surface noise and specific patterns from the original test set. Those patterns are exactly the index signals that memory‑based models use to trigger “I’ve seen this before.” Opus 5 doesn’t need those signals. It solves by understanding the context, re‑reading error logs, and tracing call chains step by step.

004

The divide is deeper than it looks.

Think of two types of students.

One grinds past exam papers. Change the numbers and conditions, and they freeze. On the exact questions they’ve practised, they score full marks. The other doesn’t grind much but has derived every theorem from first principles at least three times. Give them any variation, and they can re‑derive from the ground up.

Today’s public benchmarks are that repeatedly photocopied question bank.

Every vendor has the same book. Some feed the entire question bank into training, letting the model see shadows of these problems and answers across trillions of tokens. Others scrub those data out, keeping only corpus that doesn’t overlap with the benchmark.

The former look good on leaderboards; the latter don’t.

But the real world doesn’t hand out past papers. Every bug in production is new. Every codebase has its own messy history. Every requirement carries unspoken omissions from the product manager.

You can polish a known problem to perfection, but in production, users won’t forgive you just because you’ve memorised something similar.

005

If this were three years ago, it would be an academic debate.

But it’s not. Large models are now embedded in real development workflows. They face codebases that run to hundreds of thousands of lines, dependency chains spanning five or six open‑source projects, and compilation errors nested three layers deep in microservice calls.

In these long‑horizon, real‑world code scenarios, the difference between memorisation and understanding becomes brutally clear.

Another experiment: we gave models a historical issue from a real GitHub repository, asking them to locate and fix it. We didn’t allow any repository that appears in public benchmarks — all were obscure, poorly documented, with messy commit histories.

Under these conditions, Opus 5 could complete a full repair cycle in about 35 minutes, proactively asking about parts of the compile log it didn’t understand. Most models that closely trailed it on public leaderboards fell into a loop within the first ten minutes — repeatedly outputting similar wrong fixes, hitting the same wall every time.

A model without generalisation, in a long‑horizon task, acts like a robotic vacuum without sensors — bumping into the same wall over and over, hoping to pass through next time.

006

So what do we do?

Fidian’s experiment already points the way: move from static public benchmarks to dynamic private ones. This isn’t a technical tweak; it’s a fundamental overhaul of evaluation.

The next generation of benchmarks must meet three conditions.

First, privacy. The content must never appear in any public channel, and it must not be scrapable by crawlers or open‑source corpora. De‑duplication isn’t enough — models are more sensitive than text‑level overlap. They can treat indentation style, variable‑naming preferences, and even comment tone as index signals.

Second, dynamic generation. Each test round should generate new mutated instances from the same underlying template. The logical framework stays, but codebases, dependency versions, and exception paths are randomly replaced. Vendors can’t optimise against a fixed set, because the set keeps changing.

Third, introduce a mutation‑robustness coefficient. A benchmark should report not just the pass rate on standard problems, but a ratio: private‑variant score / public‑original score. The closer that ratio is to 1, the more reliable the model’s generalisation. If it’s significantly below 1, that’s a red flag — you’ve memorised the test, and your score is fake.

007

This isn’t just about tweaking methodology.

There’s a deeper judgement: the first half of the AI race was about memory and data scale; the second half is about generalisation and reasoning. We’re standing at the boundary where the first half ends and the second begins.

Those scores inflated by “past‑paper tactics” on public leaderboards are losing credibility fast. In private technical circles, more engineering leads no longer treat public benchmark rankings as the primary basis for model selection. They build their own private test suites, using real code from their own business scenarios.

That’s a healthy shift.

008

One detail from Fidian’s experiment is worth repeating.

In their public report, they noted that GPT‑5.6 Sol and Claude Opus 5 not only avoided degradation on TB‑fn’s unseen tasks — they actually showed higher stability on several subtasks than on the public benchmark.

Why? The public benchmark contained many low‑quality, poorly formatted, and ambiguously worded samples — noise. The private variant had clearer task descriptions and more precise problem boundaries. For models that solve by understanding, a well‑stated hard problem is easier than a poorly stated easy one.

That tells us something:

A truly good model doesn’t need past papers. Give it a clear problem, and it does even better.

And those models that go mute without the original questions? They need to wake up.

009

Our focus has always been on long‑horizon, real‑world code scenarios.

Not because it’s easy — quite the opposite. It forces us to build more complex private task templates, design more diverse mutation strategies, and accumulate repositories that will never be made public.

But it’s the only path worth taking.

Because engineers in the real world never ask a model, “Have you seen this before?” They say one thing:

“Hey, can you check why this build is failing?”

And that question is always the first time.