Q:
How do you score AI models?
A:The coding suite scores nine axes and combines them as a plain weighted mean: Correctness (55%), Stability (10%), Edge Cases (10%), Debugging (10%), Code Quality (5%), Efficiency (5%), Format (3%), Safety (2%) and Complexity (0% — measured and shown, but it moved nobody's rank, so it no longer pretends to). Correctness on the repo-debugging tasks is graded by running the project's own test suite, including tests the model never sees. Each model runs every task 7 times. There is no exponent, gate or penalty curve anywhere in the formula — every one that used to be there was removed in September 2026 after two identical sweeps showed the curve, not the models, was producing 47-point swings. Other suites score differently: the deep-reasoning suite scores nine axes of its own — six shared with coding (correctness, code quality, stability, edge cases, debugging, safety) plus three continuity axes (memory retention, plan coherence, context window); complexity, efficiency, format and hallucination rate were retracted from it in September 2026 because they measured response length or nothing at all. The tool-calling suite uses its own 7 axes (task completion, tool selection, parameter accuracy, efficiency, error handling, context awareness, safety compliance).
Q:
Why do you run 7 trials instead of just 1?
A:AI models are stochastic (probabilistic), meaning the same prompt can produce different outputs. A single measurement could be a lucky or unlucky result. Running 7 trials lets us: (1) capture natural variance, (2) calculate confidence intervals, (3) use the median to avoid outlier bias, and (4) estimate true performance more accurately. It's a balance between statistical rigor and computational cost.
Q:
What is drift detection and how does it work?
A:Two detectors, and we are explicit about what each can see. For sustained change we use the Page-Hinkley test, a cumulative-sum (CUSUM-family) change detector, on each suite's own daily series — coding, tool use and reasoning each carry their own statistic, never a blend of them, and a model's status takes the largest. It accumulates how far each day falls below the running mean, less a small tolerance, so daily noise cancels out while a real sustained decline builds up until it crosses the alarm threshold; it restarts whenever a suite's configuration changes and needs ten days of history before it can fire. For fast change we run an hourly canary — two fixed probes, two trials each — and test the last 6 hours and the last 24 hours against the prior week on the same configuration with Welch's t-test; an incident needs a fall of at least 12 points at p < 0.01, and closes itself when the gap does. Every constant, and the measured false-alarm and detection rates behind each, is on the methodology page.
Q:
What tasks do you use for benchmarking?
A:The coding suite hands a model a small working project and a bug report written as a user complaint — no file is named, so it has to find the defect itself — and grades the fix by running the project's own test suite, including tests the model never sees. All seven coding tasks run every sweep: six repo debugging tasks and one hard single-function task. Everything is executed, not pattern-matched. We retire what stops measuring: eight single-function tasks went in early September 2026 after hitting 99-100% pass rates across all 24 models, and on 14 September two trivial floor checks and one repo task went the same way, while one new repo task was added after it failed 79% of the fleet in three validation rounds. A task everybody passes ranks nobody. New tasks now have to clear a stated bar before they count: every hidden assertion must trace to the bug report, and at least a fifth of the fleet must fail across three independent runs. The tool-calling suite runs nine further tasks in real Docker sandboxes, nine sessions per model per day, and the deep-reasoning suite runs four multi-turn scenarios — all four every day since 13 September 2026, so consecutive days compare the same work. The hidden tests are the one thing we keep back — without them, a fix that silences the reported symptom and leaves the defect in place scores full marks.
Q:
How accurate are your benchmarks?
A:Every number on the board carries a standard error measured from its own run-to-run repeatability: each suite's last five measured runs on its current configuration, combined with the composite's weights. On the coding suite alone, two identical sweeps of the whole fleet differ by about 2 points on average, and the median model does not move at all; the composite's standard error is typically 2–3 points on a 100-point scale, and the interval shown is ±1.96 standard errors. So "86 ± 4.5" means the same model re-measured tomorrow would land in that range 95% of the time. The day after any configuration change there are not yet five runs to measure, and the suite's typical spread is used instead — the methodology page says which. This is far more rigorous than single-shot benchmarks that show no uncertainty, and it is why adjacent places on the leaderboard are often ties.
Q:
Why use median instead of mean?
A:Median is robust to outliers. If one trial produces an anomalous result (model hallucination, API timeout, random brilliance), it won't skew the entire score. The median represents typical performance better than the mean when dealing with small sample sizes and potential outliers.
