Dataset · CC BY 4.0 · v1.0
LLM Reliability Benchmark dataset, version 1.0
On this page
The short answer
- Version 1.0 is the dataset first published on 12 November 2026 with the LLM Reliability Benchmark. Nothing newer has been released.
- Download CSV and JSON from
/assets/benchmark/v1.0/. Unversioned paths under/assets/benchmark/are aliases of the same files. - Licence: CC BY 4.0. Credit “Litmiq LLM Reliability Benchmark 2026” and link to the benchmark page.
Download v1.0
These are byte-for-byte copies of the files already linked from the benchmark page, placed in a versioned folder so citations can pin v1.0. The machine-readable pointer is /assets/benchmark/latest.json.
| File | Contents |
|---|---|
| results.csv / results.json | Every graded output (run, model, test, pass, tokens, latency, output text). |
| leaderboard.csv | One row per model with category scores. |
| summary.json | Aggregates, per-test pass counts and related metadata. |
| suite-v1.0.json | The 40-case test suite, including expected behavior and pass rules. |
| grade.py | The deterministic grader (Python standard library only). |
| run_benchmark.py | The runner used for the published measurement. |
| CITATION.txt, CITATION.bib, LICENSE.txt, CHANGELOG.md | Citation, licence and version history. |
Direct folder: changelog · citation · licence · results.csv · results.json.
How to cite
BibTeX is in CITATION.bib. The same text already appears in the “Cite this” section of the benchmark page.
Changelog
v1.0 — initial public release
12 open models, 40 deterministic tests, measured on Cloudflare Workers AI. No later version has been published as of 13 November 2026.
The full notes are in CHANGELOG.md.
FAQ
Is this a new measurement?
No. Version 1.0 is the dataset already published with the 12 November 2026 benchmark. This page versions those files and states how to cite them.
Will there be a v1.1?
Only when we actually re-run or expand the suite. This page will not invent a newer version.
What licence is it under?
CC BY 4.0. Credit “Litmiq LLM Reliability Benchmark 2026” and link to the benchmark page.
Read the benchmark write-up
The methods, leaderboard and limits live on the report page.
- Free, no signup
- Runs in your browser
- Your OpenAI key, sent only to OpenAI