Litmiq Labs · 19 Nov 2026

How Litmiq Labs will publish a run

On this page

The short answer

  • This is the standing method for Litmiq Labs. It is published before any findings note.
  • A findings note will be dated and will name the suite, models, judge and parameters, or mark what is unknown.
  • The first run is pending. No API spend has been made for it as of 19 Nov 2026.

What a note must include

  • The date of the run, in America/Los_Angeles, and the date it was published if those differ.
  • A link to the suite (a Litmiq JSON file or an equivalent, versioned document).
  • Exact model IDs, not marketing names alone.
  • The judge model ID, the rubric, and decoding parameters (at least temperature and any cap on tokens).
  • Which cases were actually run. An incomplete run is labelled incomplete; missing scores are not written as zero.
  • A short limits section: what the run does not show.

The leaderboard method is the public table that will follow the same rules for official comparative runs.

What we will not invent

  • Scores, rankings, or “winners” for models we did not run.
  • No user counts, testimonials, star ratings, or revenue figures.
  • A human evaluation we did not perform.
  • A cost we did not record. If spend is not disclosed, we will say it is not disclosed, not that it was free.

First run

Status: pending. Planned tasks are listed on the leaderboard page. When the run is completed, a Labs note will appear in the feed and the leaderboard table will be updated in the same release, not before.

How to cite

Until a findings note exists, cite this method page as: Litmiq, “How Litmiq Labs will publish a run,” 19 November 2026, https://litmiq.com/labs/methodology.

FAQ

Has the first Labs run happened?

No. As of 19 Nov 2026 the first run is pending and no API spend has been made for it.

Will you backfill a date?

No. A findings note will carry the date the run was actually completed, not the date this method page was published.

What licence will the data use?

When a run produces a public dataset, we intend to release it under CC BY 4.0, the same family as the existing reliability benchmark, unless a later note says otherwise.

See the leaderboard method

The public table uses the same no-invented-scores rule.

  • Free, no signup
  • Runs in your browser
  • Your OpenAI key, sent only to OpenAI