AI Benchmarking Methodology — How We Test AI Models & Detect Drift

OUTPUT TO DATE

Counted from the database when this page was last generated, not written by hand — the previous figures had drifted thousands of rows behind and stated two different tool-session counts in the same paragraph. Since the first benchmark on 8 August 2025:
→ 182,695 scored benchmark runs
→ 63,719 tool-calling sessions
→ 5,033 deep-reasoning sessions
→ 1,482 drift incidents and change points recorded

WHAT MAKES A TASK DISCRIMINATE

Thirty-six repo candidates have been built to ship seven. The ones that failed taught the rule: a mechanical slip — a wrong comparison, a swapped argument, an off-by-one — gets fixed by every model, every time, and so is any bug whose rule is stated in the report. Pagination, cache keys, penny rounding, rate-limiter refill, timezone handling across a clock change, CIDR matching, partial updates with explicit nulls, locale number parsing, currency minor units and single-use expiring tokens were all solved by the whole fleet.

What separates models is a bug whose correct repair requires a judgement about intended behaviour that the report supports but does not spell out, paired with a cheaper fix that satisfies the reported symptom and is wrong. That is what the hidden tests are there to catch, and it is a fairness judgement every time: a task ships only if every hidden assertion traces to the bug report and at least a fifth of the fleet fails it across three independent runs. One of our own assertions failed that test in September 2026 — it graded a correct fix as wrong for not refetching a cache — and was corrected.

WHEN A MODEL DECLINES A TASK

Some models refuse to answer some prompts. We have measured it on entirely benign ones — a script that buckets sales figures by date, a price-cache fixture, and in one case a function that checks whether a number is prime.

A refusal is a provider’s content decision, not a fact about the model’s ability, so we do not score it as a zero. The task drops out and the model is scored over what it attempted. That is the fair treatment but it is not a neutral one: declined tasks are disproportionately the hard ones, so such a score covers an easier corpus than its rivals. Those rows show their coverage under the score (“5/7 tasks”), name the declined tasks on the model page, and are never called tied with a model measured on all of them. We do not impute a value for work that was never done.

WHY 7 TRIALS?

→ AI models are stochastic (same prompt, different outputs)
→ Single measurements are unreliable
→ 7 trials = optimal balance of cost vs statistical power
→ The seven trials of a task collapse to one outcome by median, so a single unlucky sample cannot move a task; the interval on the board comes from run-to-run repeatability (below), not from the trials

EXAMPLE CALCULATION:

A model’s seven per-task scores in one sweep:
92 | 94 | 90 | 93 | 91 | 92 | 93  (7 tasks)

Mean = 92.1
Std Dev = 1.35
Std Error = 1.35 / sqrt(7) = 0.51
t-value = 2.447 (df=6, 95% CI)
Margin = 2.447 x 0.51 = 1.24

Final: 92.1 +/- 1.2
95% CI: [90.9, 93.4]

HOW A RANK IS ASSIGNED

A model’s rank is 1 + the number of models measurably better than it: those whose score leads it by more than 1.96 × √(SE&sub1;² + SE&sub2;²), a two-sample test at 95%. Two models inside each other’s noise share a rank, shown as “=N”, instead of being separated by a place the measurement cannot support. The standard error is the run-to-run repeatability of the number on the board: each suite’s last five measured runs on its current configuration, combined with the composite’s weights (SE² = Σ (wᵢ/W)² SEᵢ²); the interval shown is ±1.96 SE around the displayed score. When a suite has fewer than two runs on its current configuration — the day after any configuration change — its typical run-to-run spread is used instead (coding 2, tool use 1.5, reasoning 8 points). Until 13 September 2026 the interval was computed over the last five rows of anysuite, so it measured how far apart the three suites sit rather than how repeatable the score is — a model scoring 90 carried an interval of 93.7–97.1.

PAGE-HINKLEY TEST (DECREASE DETECTION):

Scores are on a 0–1 scale where higher is better, so the statistic
accumulates when a score lands below the running mean:

1. meanₜ = running mean of observations since the last reset
2. mₜ = mₜ₋₁ + (meanₜ − xₜ − delta)
3. Mₜ = min(m₁ … mₜ)
4. PHₜ = mₜ − Mₜ
5. If PHₜ > lambda: ALERT, then reset the detector fully

Parameters (lib/page-hinkley.ts):
→ Tolerance (delta): 0.01 — one point of the 0–100 score
→ Threshold (lambda): 0.30 — thirty points of accumulated shortfall
→ Cold start: 10 observations before it may fire
→ Rolling baseline for alerting: 28 days

WHAT THE DETECTOR CAN AND CANNOT SEE — MEASURED

Scope, stated plainly: the Page-Hinkley detector runs on each suite’s own daily series, never on a blend of them — coding (six sweeps a day, daily median), tool use (nine sessions a day) and reasoning (four multi-turn tasks a day) each carry their own statistic, and a model’s status takes the largest. Until 13 September 2026 only the coding suite was watched, and this paragraph said the other two were too noisy: tool use was moving 26 points day to day. That number was wrong about the instrument. Measured on days when the sessions actually completed, on one configuration, the tool-use score moves1.4 points a day across all 24 models — the 26 was a credit outage, its modelled filler and two scoring changes, all inside one window. The reasoning suite was genuinely noisier (6–8 points), and the cause was structural: it ran one task a day, rotating through four, so consecutive days compared different tasks. It now runs all four every day and the daily figure is their mean; from the per-session spread (8 points) that puts its day-to-day noise near 4, inside the range the detector was validated on. Each suite’s statistic restarts on its own configuration change and needs ten days of history before it can fire.

A drift detector is only worth trusting if two numbers are known: how often it fires when nothing changed, and how reliably it fires when something did. Both are measured against the exact production code path by injecting a sustained drop of known size into a stationary series (400 repetitions per cell) and by running the detector on a series with no change at all. “Detected” means it fired within 30 days; the delay is the median number of days to the first alert.

How to read it. Twenty of the 24 ranked models sit at a day-to-day noise of 1.5–3 points on the current score; the noisiest few sit near 5. So for most of the fleet a 5-point sustained regression is caught essentially every time within about a week, a 3-point one nine times in ten within about two weeks, and a spurious alert on an unchanged model happens somewhere between never and once every twelve years. On the noisiest models the price of that sensitivity is one false alert per roughly six months.

What it will miss. A sustained drop of about 2 points or less. A one-day dip of any size, by design — the detector runs on daily medians and asks about sustained change, so a single bad day cannot fire it and neither can a single bad run.

Why the tolerance moved from 0.02 to 0.01. It was set as a quarter of the day-to-day noise when that noise was 8.4 points on an earlier, steeper score curve. The score is now a plain weighted mean with a fraction of that noise, and a 2-point tolerance had become most of it — a 3-point sustained regression was being caught only one time in five. Halving it made that nine in ten at no measurable cost on the quiet majority.

Checked against real history too. Each model’s actual daily history under one benchmark version, block-resampled to destroy any genuine change points while keeping its own noise, gives a fleet false-alarm rate of about one per model every 86 days — but that history predates the current score curve and is dominated by two models that were far noisier under it, so it bounds the worst case rather than describing today.

Reproduce it: node dist/jobs/validate-drift-detector.js in the API repository. Deterministic seed; the table above is its output.

WHY A FULL RESET AFTER AN ALERT

A partial reset that kept the old mean would keep firing on every subsequent observation until the running mean caught up to the new level, producing a sawtooth of duplicate alerts. Resetting completely lets the detector re-learn the post-change level and stay quiet until the next real change.

ALERT SEVERITY LEVELS

NORMAL — every suite’s drift statistic is below its warning line
WARNING — a suite’s Page-Hinkley statistic is more than halfway to its alarm threshold, or recent scores are unusually spread
ALERT — a suite’s statistic crossed the alarm threshold, or the model is measurably below its own 28-day baseline on its current configuration

TEST YOUR KEYS

Run benchmarks with your own API keys to verify we're not making up numbers

Scores update every 4 hours. Rankings shift based on continuous performance monitoring.

A FREE KEY IS REQUIRED

The data API lives at /api/v1 and authenticates with a bearer token. Creating a key takes about thirty seconds and costs nothing — create one here, or read the full API reference.

Current rankings with confidence intervals
Rate Limit: 1 request/minute free · 60 on Pro

Historical time-series data (7 days)
Rate Limit: 1 request/minute free · 60 on Pro

Detailed model breakdown by task
Rate Limit: 1 request/minute free · 60 on Pro

Models currently degrading, with magnitude
Rate Limit: 1 request/minute free · 60 on Pro

RATE LIMITING AND PROTECTION

All public APIs protected with automatic rate limiting:
→ Per-key quotas: 10 requests/day free, 10,000 on Pro
→ X-RateLimit-Limit / -Remaining / -Reset on every response
→ Returns 429 status code when exceeded
→ Daily quota resets at 00:00 UTC
→ Higher tiers available for larger workloads

EXPLORE THE RANKINGS

See how the models actually perform, across 182,695 scored benchmark runs
Updated every 4 hours with statistical confidence intervals