Litmiq Labs · 19 Nov 2026
How Litmiq Labs will publish a run
On this page
The short answer
- This is the standing method for Litmiq Labs. It is published before any findings note.
- A findings note will be dated and will name the suite, models, judge and parameters, or mark what is unknown.
- The first run is pending. No API spend has been made for it as of 19 Nov 2026.
What a note must include
- The date of the run, in America/Los_Angeles, and the date it was published if those differ.
- A link to the suite (a Litmiq JSON file or an equivalent, versioned document).
- Exact model IDs, not marketing names alone.
- The judge model ID, the rubric, and decoding parameters (at least temperature and any cap on tokens).
- Which cases were actually run. An incomplete run is labelled incomplete; missing scores are not written as zero.
- A short limits section: what the run does not show.
The leaderboard method is the public table that will follow the same rules for official comparative runs.
What we will not invent
- Scores, rankings, or “winners” for models we did not run.
- No user counts, testimonials, star ratings, or revenue figures.
- A human evaluation we did not perform.
- A cost we did not record. If spend is not disclosed, we will say it is not disclosed, not that it was free.
First run
Status: pending. Planned tasks are listed on the leaderboard page. When the run is completed, a Labs note will appear in the feed and the leaderboard table will be updated in the same release, not before.
How to cite
Until a findings note exists, cite this method page as: Litmiq, “How Litmiq Labs will publish a run,” 19 November 2026, https://litmiq.com/labs/methodology.
FAQ
Has the first Labs run happened?
No. As of 19 Nov 2026 the first run is pending and no API spend has been made for it.
Will you backfill a date?
No. A findings note will carry the date the run was actually completed, not the date this method page was published.
What licence will the data use?
When a run produces a public dataset, we intend to release it under CC BY 4.0, the same family as the existing reliability benchmark, unless a later note says otherwise.
See the leaderboard method
The public table uses the same no-invented-scores rule.
- Free, no signup
- Runs in your browser
- Your OpenAI key, sent only to OpenAI