How it works

One task, every model, and what each answer cost. It is an exhibit, not a leaderboard.

Each prompt goes to every model on the roster. The answers sit side by side, in the same order on every page, with the price and the time of each answer underneath it. Nothing is scored. What you make of the difference is up to you.

The exact wording of each prompt is withheld, so it cannot end up in training data and be answered from memory. What you can read is a plain description of the task and its public constraints, which is enough to check whether an answer did what was asked.

Each prompt version has a SHA-256 fingerprint of its exact text, recorded when the version is created, before any model runs it. The database refuses any edit to a version after that, so the fingerprint shown with an answer proves the prompt was not changed afterwards.

Models are always listed by vendor list price per million output tokens, cheapest first. That is a price, not a ranking.

A tie goes to the lower input price. The order is the same on every page, and it only moves when a vendor changes its prices. The full list with prices is on the models page.

The cost under an answer is the tokens that answer actually used, input and output (reasoning included), multiplied by the vendor’s rate recorded at the moment of the run. The rate is stored with the answer, so the figure never changes when a vendor reprices later. It is the cost of that one answer, not an average.

The range at the top of each prompt is different: it runs from the lowest to the highest median cost per answer, one median per model over its finished runs. The large “×” figure divides the highest of those medians by the lowest. A single run can cost more or less than either end.

Every model is asked every prompt three times. A few runs are missing, and they are counted as not run (our error). The same model given the same prompt twice does not give the same answer, so a single answer would pass off one draw from a random process as the model’s answer. Three runs are not a statistic. They are enough to see whether the answers vary.

Every run has a cap on output tokens and a time limit, set per category. The token cap counts reasoning tokens too, so a model that thinks for a long time can spend the cap before it writes anything.

GLM 5.2, Gemini 3.1 Pro and Kimi K3 run with their own limits in every category, in place of the table: 32,000 tokens and 13 min. Their reasoning alone overran the defaults.

Every answer stays on the page, including the ones that failed. A failed answer carries one of three labels, depending on whose failure it was.

Model failed

The model answered, and the answer failed: it ran out of room in the answer itself, refused, broke the format or drew nothing.

Hit our limit

The model was still working when it reached a limit we set: our time limit, or our token cap spent mostly on reasoning.

Not run (our error)

Our run failed before the model answered, so nothing was generated. These do not count against the model.

All three stay visible. Only “Model failed” counts against a model on its page.

Each model carries a tier: Ultra, High, Medium or Low. It marks where the vendor positions the model in its own line-up, from flagship down to small and cheap. It is not our grade of its answers.

There is no score, vote, rank or leaderboard here. A score needs a judge, and every judge (a person, a panel, another model) brings its own taste to it. A vote measures who turned up. The site shows the answers, what they cost and how long they took, and leaves the comparison to you.