Dataset · CC BY 4.0 · v1.0

LLM Reliability Benchmark dataset, version 1.0

On this page

The short answer

  • Version 1.0 is the dataset first published on 12 November 2026 with the LLM Reliability Benchmark. Nothing newer has been released.
  • Download CSV and JSON from /assets/benchmark/v1.0/. Unversioned paths under /assets/benchmark/ are aliases of the same files.
  • Licence: CC BY 4.0. Credit “Litmiq LLM Reliability Benchmark 2026” and link to the benchmark page.

Download v1.0

These are byte-for-byte copies of the files already linked from the benchmark page, placed in a versioned folder so citations can pin v1.0. The machine-readable pointer is /assets/benchmark/latest.json.

Files in v1.0
FileContents
results.csv / results.jsonEvery graded output (run, model, test, pass, tokens, latency, output text).
leaderboard.csvOne row per model with category scores.
summary.jsonAggregates, per-test pass counts and related metadata.
suite-v1.0.jsonThe 40-case test suite, including expected behavior and pass rules.
grade.pyThe deterministic grader (Python standard library only).
run_benchmark.pyThe runner used for the published measurement.
CITATION.txt, CITATION.bib, LICENSE.txt, CHANGELOG.mdCitation, licence and version history.

Direct folder: changelog · citation · licence · results.csv · results.json.

How to cite

BibTeX is in CITATION.bib. The same text already appears in the “Cite this” section of the benchmark page.

Changelog

  1. v1.0 — initial public release

    12 open models, 40 deterministic tests, measured on Cloudflare Workers AI. No later version has been published as of 13 November 2026.

The full notes are in CHANGELOG.md.

FAQ

Is this a new measurement?

No. Version 1.0 is the dataset already published with the 12 November 2026 benchmark. This page versions those files and states how to cite them.

Will there be a v1.1?

Only when we actually re-run or expand the suite. This page will not invent a newer version.

What licence is it under?

CC BY 4.0. Credit “Litmiq LLM Reliability Benchmark 2026” and link to the benchmark page.

Read the benchmark write-up

The methods, leaderboard and limits live on the report page.

  • Free, no signup
  • Runs in your browser
  • Your OpenAI key, sent only to OpenAI

Follow Litmiq on LinkedIn