Deadline guide · updated 13 Nov 2026
OpenAI Evals is shutting down: export your evals and migrate them
Scheduled shutdown: 30 Nov 2026
On this page
The short answer
- OpenAI Evals became read-only on 31 Oct 2026 and is scheduled to shut down on 30 Nov 2026 (dashboard, API and graders for eval workflows). OpenAI recommends migrating to Promptfoo. Source
- Export first. There is no export button, but the Evals API still returns your evals, runs and output items. Our free script saves all of them, read-only.
- Convert in your browser. Paste the data into the converter and get a
promptfooconfig.yaml, a CSV, or a Litmiq suite you can run in one click. - Re-check scores. Similarity and model-graded scores won't match OpenAI's exactly. Re-run a sample and re-tune thresholds before you trust them.
Key numbers
- OpenAI's Evals dashboard, API and graders are read-only and shut down on 30 Nov 2026; OpenAI recommends Promptfoo as the replacement. (OpenAI deprecations page, 13 Nov 2026)
- OpenAI shuts down gpt-5, gpt-5-mini, gpt-5-nano, gpt-5-pro, o3 and o3-pro on 11 Dec 2026 (replacements: gpt-6.1-sol, gpt-6.1-terra, gpt-6-luna). (OpenAI deprecations page, 13 Nov 2026)
Timeline: what changes and when
| Date | What happens | Source |
|---|---|---|
| 3 Jun 2026 | OpenAI publishes the deprecation notice for the Evals dashboard, API and graders for eval workflows, and for /v1/prompts reusable prompt objects. | OpenAI deprecations |
| 31 Oct 2026 | Evals become read-only for existing users. | OpenAI Evals guide |
| 30 Nov 2026 | The Evals platform (dashboard, API and graders for eval workflows) is scheduled to shut down; OpenAI recommends migrating to Promptfoo. Reusable prompt objects are deprecated the same day: move prompt text into your app code. | OpenAI deprecations · Evals guide |
| 11 Dec 2026 | Separate but close: gpt-5, gpt-5-mini, gpt-5-nano, gpt-5-pro, o3 and o3-pro shut down. If your evals pinned one of these, re-baseline on the replacement. | OpenAI deprecations · model migration guide |
Dates as published by OpenAI on the pages linked, read on 13 Nov 2026. If OpenAI changes them, those pages are authoritative.
Your options
- Promptfoo (OpenAI's recommendation). Open-source CLI with YAML configs, deterministic assertions (
equals,contains,regex,is-json), similarity metrics, model-graded rubrics and CI. OpenAI announced on 9 Mar 2026 that it would acquire Promptfoo (OpenAI). Best if your evals are code-based or run in CI. - Litmiq. Free, in your browser, no install or signup. Paste a system prompt, run your cases on several OpenAI models side by side, and an LLM judge scores each output with reasons. Best for quick checks of a prompt or a model change. No code graders or deterministic assertions.
- Both. Keep exact-match and CI checks in Promptfoo; use Litmiq to eyeball a prompt or model change before you commit it. The converter below outputs both formats from the same export.
- Keep some parts in OpenAI. OpenAI's cookbook says dataset review and prompt iteration remain in OpenAI (cookbook).
Comparing tools in more depth: Litmiq vs Promptfoo, Litmiq vs OpenAI Playground, prompt testing tools compared.
Step 1: Export your evals (free script)
This Python script saves every eval in an OpenAI project: the eval definition (item schema and graders), every run (model, prompt template, result counts), every output item (each input, the model's output and each grader's result) and the JSONL data files the runs used. Standard library only, GET requests only, and your key only goes to api.openai.com. It uses the list, runs and output-items endpoints in the Evals API reference.
export OPENAI_API_KEY=sk-... # a key for the project whose evals you want
python3 export-openai-evals.py
# -> openai-evals-export/<eval_id>/eval.json, runs/<run_id>/run.json, output_items.jsonl, files/*.jsonl
Download export-openai-evals.py
Show the full script (74 lines)
#!/usr/bin/env python3
"""Export your OpenAI Evals before the scheduled 30 Nov 2026 shutdown.
Saves every eval in the project your key belongs to, every run, every run's output items
and the JSONL data files the runs used, into ./openai-evals-export/. Python 3.8+, standard
library only. It only sends GET requests (read-only). Your key goes only to api.openai.com.
export OPENAI_API_KEY=... # a key for the project whose evals you want
python3 export-openai-evals.py
Run it once per OpenAI project. Source: https://litmiq.com/migrate-from-openai-evals (MIT license).
"""
import json, os, pathlib, sys, urllib.error, urllib.parse, urllib.request
API = os.environ.get("OPENAI_BASE_URL", "https://api.openai.com/v1").rstrip("/")
KEY = os.environ.get("OPENAI_API_KEY") or sys.exit("Set OPENAI_API_KEY first.")
OUT = pathlib.Path(sys.argv[1] if len(sys.argv) > 1 else "openai-evals-export")
def get(path, params=None, raw=False):
url = API + path + ("?" + urllib.parse.urlencode(params) if params else "")
req = urllib.request.Request(url, headers={"Authorization": "Bearer " + KEY})
with urllib.request.urlopen(req, timeout=120) as r:
body = r.read()
return body if raw else json.loads(body)
def pages(path):
"""Yield every object from a paginated list endpoint (has_more / last_id)."""
params = {}
while True:
page = get(path, params)
yield from page.get("data", [])
if not page.get("has_more") or not page.get("last_id"):
return
params["after"] = page["last_id"]
def save(p, obj):
p.parent.mkdir(parents=True, exist_ok=True)
p.write_text(json.dumps(obj, indent=2, ensure_ascii=False), encoding="utf-8")
n_evals = n_runs = n_items = n_files = 0
for ev in pages("/evals"):
n_evals += 1
d = OUT / ev["id"]
save(d / "eval.json", ev) # name, data_source_config (item schema) and testing_criteria (graders)
for run in pages(f"/evals/{ev['id']}/runs"):
n_runs += 1
rd = d / "runs" / run["id"]
save(rd / "run.json", run) # model, prompt template (input_messages), data source, result counts
rd.mkdir(parents=True, exist_ok=True)
with open(rd / "output_items.jsonl", "w", encoding="utf-8") as f:
for item in pages(f"/evals/{ev['id']}/runs/{run['id']}/output_items"):
n_items += 1
f.write(json.dumps(item, ensure_ascii=False) + "\n")
src = (run.get("data_source") or {}).get("source") or {}
if src.get("type") == "file_id":
fp = d / "files" / (src["id"] + ".jsonl")
if not fp.exists():
try:
fp.parent.mkdir(parents=True, exist_ok=True)
fp.write_bytes(get(f"/files/{src['id']}/content", raw=True))
n_files += 1
except urllib.error.HTTPError as e:
print(f" could not download file {src['id']} (HTTP {e.code}); its rows are still in output_items.jsonl")
elif src.get("type") == "file_content":
with open(rd / "file_content.jsonl", "w", encoding="utf-8") as f:
for row in src.get("content", []):
f.write(json.dumps(row, ensure_ascii=False) + "\n")
print(f"exported {ev.get('name') or ev['id']}")
print(f"Done: {n_evals} evals, {n_runs} runs, {n_items} output items, {n_files} data files in {OUT.resolve()}")
- One project at a time. API keys are scoped to a project, so run it once per project.
- Do it before 30 Nov. After the shutdown, don't expect the API to return anything.
- Keep the raw export. It's your record of past results, even if you never re-run them.
Step 2: Convert to Litmiq or Promptfoo (in your browser)
Paste a data file: {"item": {…}} JSONL, a JSON array, a CSV, an output_items.jsonl from the export, or the legacy open-source Evals format {"input": [messages], "ideal": …}. Optionally paste eval.json and run.json to bring the graders and the prompt template across. Nothing leaves this page.
Already converted to Promptfoo? Open the promptfooconfig.yaml in a browser with the Promptfoo importer, or wire it into pull requests with the CI/CD guide.
Grader mapping: OpenAI Evals → Promptfoo → Litmiq
OpenAI's grader types are documented in its graders guide, Promptfoo's in its assertions reference. The converter applies this table automatically.
| OpenAI grader | What it checks | Promptfoo assertion | Litmiq |
|---|---|---|---|
string_check eq | Exact match, case-sensitive | equals | Expected behavior: “Output is exactly X”, scored by the judge (not a deterministic check) |
string_check ne | Not an exact match | not-equals | Expected behavior: “Output is not X” |
string_check like | Contains, case-sensitive | contains | Expected behavior: “Output contains X” |
string_check ilike | Contains, any case | icontains | Expected behavior: “Output contains X (any case)” |
text_similarity bleu / gleu / meteor | Similarity to a reference, with pass_threshold | bleu / gleu / meteor with threshold | Reference answer in expected behavior; the judge scores closeness |
text_similarity rouge_1…rouge_5, rouge_l | ROUGE overlap | rouge-n / rouge-l with threshold | Same as above |
text_similarity cosine | Embedding similarity | similar with threshold | Same as above |
text_similarity fuzzy_match | Fuzzy string match (rapidfuzz) | No identical metric: use similar or levenshtein and re-tune the threshold | Same as above |
label_model | A model picks a label; passes if it is in passing_labels | llm-rubric (or model-graded-closedqa) | Put the grader instructions in expected behavior or scoring criteria |
score_model | A model returns a score compared with pass_threshold | llm-rubric with threshold (scores are 0–1) | Litmiq’s judge scores 0–10 on four metrics; describe the bar in the criteria |
python | Your Python function returns a score | python assertion, rewritten to promptfoo’s function signature | Not supported: Litmiq has no code graders. Use Promptfoo for these |
| Several graders on one eval | All must pass | Several assert entries per test | One expected-behavior line per grader |
Template variables: OpenAI templates use {{ item.field }} for dataset columns and {{ sample.output_text }} for the model output. In Promptfoo, dataset columns become vars and assertions check the output directly. The converter renders item.* into each test.
Step 3a: Recreate in Litmiq (no install)
- Open it. Click Open in Litmiq in the converter. The system prompt, scoring criteria and every case load into the app.
- Pick models. Select the model your eval used (or its replacement: see the model migration guide) and one alternative.
- Start small. Each case costs one completion per model plus one judge call on your OpenAI key. Run 20–40 cases first.
- Read the reasons. The judge explains every score. Compare failures with your last OpenAI run's output items.
Step 3b: Recreate in Promptfoo
Download promptfooconfig.yaml from the converter and check the TODO comments (Python graders need porting by hand to a python assertion), then validate and run it. The commands are from OpenAI's migration cookbook:
npx promptfoo@latest validate config -c promptfooconfig.yaml
npx promptfoo@latest eval -c promptfooconfig.yaml --no-cache
npx promptfoo@latest view
A minimal hand-written version of OpenAI's own example:
# promptfooconfig.yaml (the IT-ticket example from OpenAI's Evals guide)
description: IT ticket categorization
prompts:
- label: categorize
raw: '[{"role":"system","content":"You are an expert in categorizing IT support tickets. Given the support ticket below, categorize the request into one of ''Hardware'', ''Software'', or ''Other''. Respond with only one of those words."},{"role":"user","content":{{ ticket_text | dump }}}]'
providers:
- openai:chat:gpt-6.1-terra
tests:
- vars: { ticket_text: "My monitor won't turn on!" }
assert: [{ type: equals, value: Hardware }]
- vars: { ticket_text: "I'm in vim and I can't quit!" }
assert: [{ type: equals, value: Software }]
- vars: { ticket_text: "Best restaurants in Cleveland?" }
assert: [{ type: equals, value: Other }]
Step 4: Verify the migration
- Same inputs, compare pass rates. Run the migrated suite on the same model and compare with the last OpenAI run's
result_countsinrun.json. - Exact-match graders should agree almost perfectly in Promptfoo. If they don't, check whitespace, case and how the prompt template rendered.
- Similarity and model graders will drift. OpenAI's cookbook warns that similarity-based scoring may not produce identical numbers and that LLM-as-a-judge graders should be validated. Label 30–50 outputs yourself and measure agreement: see calibrating an LLM judge.
- Keep one regression set. Save the migrated suite and re-run it on every prompt or model change: prompt regression testing.
Migration checklist
- List every OpenAI project that has evals.
- Run the export script once per project and keep the folder.
- Note which model each eval pinned. If it's on the 11 Dec 2026 shutdown list, plan a model migration test.
- Convert each eval (Promptfoo YAML, Litmiq, or both).
- Port Python graders by hand.
- Move any reusable prompt objects into your code before 30 Nov 2026.
- Re-run, compare with the old results, re-tune thresholds.
- Add the suite to CI (Promptfoo) or your release checklist (prompt testing checklist).
Limits of this guide and the tools
- The converter maps graders to their nearest equivalent. It's a starting point: review every assertion.
- Litmiq runs single-turn tests on OpenAI and Anthropic models with an LLM judge. It has no deterministic, Python or similarity graders, and no CI integration today.
- The export script relies on the Evals API endpoints documented on 13 Nov 2026. If OpenAI changes or removes them early, it will stop working.
- Litmiq is not affiliated with OpenAI or Promptfoo.
FAQ
When does OpenAI Evals shut down?
OpenAI's deprecations page lists the Evals dashboard, API and graders for eval workflows with a 30 November 2026 date, and the Evals guide says Evals became read-only for existing users on 31 October 2026 and the platform is scheduled to shut down on 30 November 2026. Sources read on 13 Nov 2026.
Is there an official export button for OpenAI Evals?
OpenAI's cookbook on moving to Promptfoo recreates evals by hand and says the approach does not require an OpenAI Evals export feature. The Evals API still lets you read your evals, runs and output items, which is what the export script on this page does, with GET requests only.
What is the best OpenAI Evals alternative?
OpenAI recommends Promptfoo, an open-source CLI that OpenAI announced it would acquire in March 2026. Promptfoo fits code-based, CI-driven evals with exact-match and similarity assertions. Litmiq fits quick, browser-based checks of one system prompt on several OpenAI models with an LLM judge and no install. Many teams can use both.
Will my scores be the same after migrating?
Not necessarily. OpenAI's cookbook notes that similarity-based scoring may not produce identical numerical results and that LLM-as-a-judge graders should be validated after migration. Re-run a sample, compare it with your last OpenAI run, and re-tune thresholds before you trust new results.
Does Litmiq support exact-match graders?
No. Litmiq scores every output with an LLM judge (accuracy, tone, helpfulness and conciseness, 0-10). The converter turns exact-match graders into expected-behavior text the judge checks, which is close but not deterministic. For evals that depend on exact or code-based checks, use the Promptfoo YAML the converter generates.
Is my data uploaded when I use the converter?
No. The converter runs in your browser and nothing is sent to Litmiq. If you click Open in Litmiq, the suite is saved in your browser's localStorage and opened in the app; the app only sends data to OpenAI, with your key, when you run an eval.
What about OpenAI Datasets and stored prompts?
OpenAI's cookbook says platform features such as dataset review and prompt iteration remain in OpenAI. Reusable prompt objects (/v1/prompts) are deprecated on 30 November 2026, and OpenAI advises moving the prompt text into your application code.
Sources (read 13 Nov 2026)
- OpenAI API deprecations: Evals dashboard, API and graders for eval workflows, 30 Nov 2026, notice 3 Jun 2026, migrate to Promptfoo;
/v1/promptsdeprecated 30 Nov 2026; model shutdowns. - OpenAI Evals guide: “Evals will become read-only for existing users on October 31, 2026, and the platform is scheduled to shut down on November 30, 2026.” Also the IT-ticket example used above.
- OpenAI Cookbook: Moving from OpenAI Evals to Promptfoo: manual recreation, Promptfoo commands, the warning about similarity and LLM-judge scores.
- OpenAI Evals API reference: list evals, runs and output items; pagination with
after,has_moreandlast_id. - OpenAI graders guide: string_check operations and text_similarity metrics.
- Promptfoo assertions and metrics, llm-rubric and python assertions.
- OpenAI to acquire Promptfoo (9 Mar 2026).
Spotted a change or an error? Email hello@litmiq.com.
Get model deprecation alerts
We email you when OpenAI, Anthropic or Google publish a new shutdown date for a model on our tracker. Sent by hand from hello@litmiq.com, a few times a month at most. Unsubscribe by replying.
We store only your email, through our form provider. See the privacy policy.
Run your migrated eval in Litmiq
Load OpenAI's example (already converted), or paste your own export into the converter above. Free, in your browser.
- Free, no signup
- Runs in your browser
- Your OpenAI key, sent only to OpenAI