Method published · first run pending

A Litmiq leaderboard, with the method first

On this page

The short answer

  • No scores yet. The first Litmiq leaderboard run is pending. This page publishes the method so the rules are public before any number is.
  • When a run is completed it will be dated, the suite and model versions will be listed, and the raw method will stay on this page.
  • Unknowns are not zeros. A pending cell does not mean a score of 0. See also the Labs method and the reliability benchmark.

Status

Status as of 19 Nov 2026

First published run: pending. No API spend has been made for this leaderboard. No model scores are listed below because none have been produced.

Method

Each official run will use a fixed public suite, a named list of models, a named judge, and fixed decoding parameters. The suite will be a Litmiq JSON file under /assets/suites/, or a successor path linked from this page. We will not quietly swap cases after scores exist. If the suite has to change, that becomes a new run with a new date.

Runs will be done in the Litmiq browser app with a provider API key that does not go to a Litmiq server, or from a recorded session that we can replay. We will list which of those two we used.

Tasks

The first run, when it happens, is planned to use four everyday tasks, each already published as an illustrative Litmiq template. The cases are fictional; they are testing the prompt, not a real customer:

We may add a coding or tool-calling task later. That addition would be a new run, not a silent edit of an old table.

Models we intend to run

The first run is intended to include current generally available chat models from OpenAI and Anthropic, named with the exact model IDs used on the day of the run. We will not list a model as tested until it has actually been run. A reserved row is not a result.

Planned first Litmiq leaderboard. Every score cell is pending.
ModelSupportExtractionSQLInstructions
OpenAI model, exact ID to be recorded on the run datePendingPendingPendingPending
Anthropic model, exact ID to be recorded on the run datePendingPendingPendingPending

Scoring

Each case has an expected behavior written as a checkable rule. A named judge model will score 0–10 against that rule, with a short reason. We will publish the judge model ID, temperature and the rubric. Judge scores are opinions of that model, not ground truth. We will not invent human ratings or an overall “winner” if the run is incomplete.

Reproducibility

For each official run we will publish: the date, the suite URL, the model IDs, the judge ID and parameters, and a note on what was not run. Labs notes will go on Litmiq Labs. Model providers who want a launch-day run can read the provider offer.

FAQ

Are there scores on this page?

No. The first run is pending. Litmiq will not publish scores until a dated, reproducible run is completed. Empty cells are not zeros.

Why publish the method before the run?

So the rules are fixed before anyone sees a number, and so the page can still be cited. A leaderboard that appears with scores and no method is not useful.

Will you use paid APIs?

Yes, when a run happens, with a bring-your-own-key or a recorded spend we can disclose. No API spend has been made for this leaderboard yet.

Can I submit a run?

Not yet as a self-serve upload. When the first official run is published, the suite JSON and the method on this page will be the source.

Turn the method into a test

Run one of the planned task templates on your own prompt, with your own key.

  • Free, no signup
  • Runs in your browser
  • Your OpenAI key, sent only to OpenAI