Four suites, on a fixed schedule
Coding runs every four hours on real repository defects, graded by the project’s own test suite including tests the model never sees. Multi-turn reasoning and tool-calling run daily. A small probe runs hourly to catch a sudden change between full runs.
Repeated, then taken as a median
Each coding task runs seven times and the median is scored, because one sample cannot tell a model getting worse from a model having a bad afternoon. That repetition is most of what the benchmark costs to run, and it is why an alert is worth believing.
Gaps are shown, not filled
If a provider declines a task or a session does not finish, that task drops out rather than being scored zero. The row says how much of the set it covers, and a score measured over fewer tasks is never called tied with one measured over all of them.
Free is a real tier
Every category ranking over the last 24 hours, 7 days and 30 days costs nothing with a free account, and model pages and the full methodology need no account at all. A benchmark nobody can check is not worth reading, so the evidence is not the paid part. The newest run as it lands, depth and workflow are.
What happens at a limit
Nothing breaks silently, and nothing is ever billed after the fact. When a month’s Smart Router allowance is used up, routing pauses with a clear message: top up with prepaid credits — from $5, 2,000 requests per $1, and they never expire — or move up a plan: on Developer and Teams a request costs less than a credit. Data API calls beyond the daily ceiling are refused with a clear error rather than throttled into a timeout.
Why annual is cheaper
There is no discount code and no countdown. An annual plan is billed once instead of twelve times, which costs us less in payment fees and in churn, and we pass that back as two free months. You can still cancel; we do not hold you to the year.
What we do not charge for
On every plan, provider inference. You connect your own OpenAI, Anthropic, Google, DeepSeek, Kimi or GLM keys, and those providers bill you directly at their rates. We charge for the software and the measurement, never a markup on tokens. Custom continuous benchmarking works the other way round and says so above: we run it on our accounts and the inference is in the price.
Existing subscribers
If you already subscribe, you keep your current price and everything you already had for at least twelve months. Nothing about these plans reduces what you have today. Your plan only changes if you choose to change it, or if you cancel and come back later.
Does routing save money?
In our own benchmark the cheapest model matching the top score cost about 90% less per request. Measured on our own coding benchmark across 16 models, September 2026. Your workload will differ — that is what the comparison is for.
