Deadline guide · updated 13 Nov 2026

OpenAI Evals is shutting down: export your evals and migrate them

Scheduled shutdown: 30 Nov 2026

On this page

The short answer

  • OpenAI Evals became read-only on 31 Oct 2026 and is scheduled to shut down on 30 Nov 2026 (dashboard, API and graders for eval workflows). OpenAI recommends migrating to Promptfoo. Source
  • Export first. There is no export button, but the Evals API still returns your evals, runs and output items. Our free script saves all of them, read-only.
  • Convert in your browser. Paste the data into the converter and get a promptfooconfig.yaml, a CSV, or a Litmiq suite you can run in one click.
  • Re-check scores. Similarity and model-graded scores won't match OpenAI's exactly. Re-run a sample and re-tune thresholds before you trust them.

Key numbers

  • OpenAI's Evals dashboard, API and graders are read-only and shut down on 30 Nov 2026; OpenAI recommends Promptfoo as the replacement. (OpenAI deprecations page, 13 Nov 2026)
  • OpenAI shuts down gpt-5, gpt-5-mini, gpt-5-nano, gpt-5-pro, o3 and o3-pro on 11 Dec 2026 (replacements: gpt-6.1-sol, gpt-6.1-terra, gpt-6-luna). (OpenAI deprecations page, 13 Nov 2026)

Timeline: what changes and when

OpenAI Evals shutdown timeline (sources read 13 Nov 2026)
DateWhat happensSource
3 Jun 2026OpenAI publishes the deprecation notice for the Evals dashboard, API and graders for eval workflows, and for /v1/prompts reusable prompt objects.OpenAI deprecations
31 Oct 2026Evals become read-only for existing users.OpenAI Evals guide
30 Nov 2026The Evals platform (dashboard, API and graders for eval workflows) is scheduled to shut down; OpenAI recommends migrating to Promptfoo. Reusable prompt objects are deprecated the same day: move prompt text into your app code.OpenAI deprecations · Evals guide
11 Dec 2026Separate but close: gpt-5, gpt-5-mini, gpt-5-nano, gpt-5-pro, o3 and o3-pro shut down. If your evals pinned one of these, re-baseline on the replacement.OpenAI deprecations · model migration guide

Dates as published by OpenAI on the pages linked, read on 13 Nov 2026. If OpenAI changes them, those pages are authoritative.

Your options

Comparing tools in more depth: Litmiq vs Promptfoo, Litmiq vs OpenAI Playground, prompt testing tools compared.

Step 1: Export your evals (free script)

This Python script saves every eval in an OpenAI project: the eval definition (item schema and graders), every run (model, prompt template, result counts), every output item (each input, the model's output and each grader's result) and the JSONL data files the runs used. Standard library only, GET requests only, and your key only goes to api.openai.com. It uses the list, runs and output-items endpoints in the Evals API reference.

export OPENAI_API_KEY=sk-...   # a key for the project whose evals you want
python3 export-openai-evals.py
# -> openai-evals-export/<eval_id>/eval.json, runs/<run_id>/run.json, output_items.jsonl, files/*.jsonl

Download export-openai-evals.py

Show the full script (74 lines)
#!/usr/bin/env python3
"""Export your OpenAI Evals before the scheduled 30 Nov 2026 shutdown.

Saves every eval in the project your key belongs to, every run, every run's output items
and the JSONL data files the runs used, into ./openai-evals-export/. Python 3.8+, standard
library only. It only sends GET requests (read-only). Your key goes only to api.openai.com.

  export OPENAI_API_KEY=...      # a key for the project whose evals you want
  python3 export-openai-evals.py

Run it once per OpenAI project. Source: https://litmiq.com/migrate-from-openai-evals (MIT license).
"""
import json, os, pathlib, sys, urllib.error, urllib.parse, urllib.request

API = os.environ.get("OPENAI_BASE_URL", "https://api.openai.com/v1").rstrip("/")
KEY = os.environ.get("OPENAI_API_KEY") or sys.exit("Set OPENAI_API_KEY first.")
OUT = pathlib.Path(sys.argv[1] if len(sys.argv) > 1 else "openai-evals-export")


def get(path, params=None, raw=False):
    url = API + path + ("?" + urllib.parse.urlencode(params) if params else "")
    req = urllib.request.Request(url, headers={"Authorization": "Bearer " + KEY})
    with urllib.request.urlopen(req, timeout=120) as r:
        body = r.read()
    return body if raw else json.loads(body)


def pages(path):
    """Yield every object from a paginated list endpoint (has_more / last_id)."""
    params = {}
    while True:
        page = get(path, params)
        yield from page.get("data", [])
        if not page.get("has_more") or not page.get("last_id"):
            return
        params["after"] = page["last_id"]


def save(p, obj):
    p.parent.mkdir(parents=True, exist_ok=True)
    p.write_text(json.dumps(obj, indent=2, ensure_ascii=False), encoding="utf-8")


n_evals = n_runs = n_items = n_files = 0
for ev in pages("/evals"):
    n_evals += 1
    d = OUT / ev["id"]
    save(d / "eval.json", ev)  # name, data_source_config (item schema) and testing_criteria (graders)
    for run in pages(f"/evals/{ev['id']}/runs"):
        n_runs += 1
        rd = d / "runs" / run["id"]
        save(rd / "run.json", run)  # model, prompt template (input_messages), data source, result counts
        rd.mkdir(parents=True, exist_ok=True)
        with open(rd / "output_items.jsonl", "w", encoding="utf-8") as f:
            for item in pages(f"/evals/{ev['id']}/runs/{run['id']}/output_items"):
                n_items += 1
                f.write(json.dumps(item, ensure_ascii=False) + "\n")
        src = (run.get("data_source") or {}).get("source") or {}
        if src.get("type") == "file_id":
            fp = d / "files" / (src["id"] + ".jsonl")
            if not fp.exists():
                try:
                    fp.parent.mkdir(parents=True, exist_ok=True)
                    fp.write_bytes(get(f"/files/{src['id']}/content", raw=True))
                    n_files += 1
                except urllib.error.HTTPError as e:
                    print(f"  could not download file {src['id']} (HTTP {e.code}); its rows are still in output_items.jsonl")
        elif src.get("type") == "file_content":
            with open(rd / "file_content.jsonl", "w", encoding="utf-8") as f:
                for row in src.get("content", []):
                    f.write(json.dumps(row, ensure_ascii=False) + "\n")
    print(f"exported {ev.get('name') or ev['id']}")

print(f"Done: {n_evals} evals, {n_runs} runs, {n_items} output items, {n_files} data files in {OUT.resolve()}")

Step 2: Convert to Litmiq or Promptfoo (in your browser)

Paste a data file: {"item": {…}} JSONL, a JSON array, a CSV, an output_items.jsonl from the export, or the legacy open-source Evals format {"input": [messages], "ideal": …}. Optionally paste eval.json and run.json to bring the graders and the prompt template across. Nothing leaves this page.

Paste or upload your eval data to start. Nothing leaves this page.

Already converted to Promptfoo? Open the promptfooconfig.yaml in a browser with the Promptfoo importer, or wire it into pull requests with the CI/CD guide.

Grader mapping: OpenAI Evals → Promptfoo → Litmiq

OpenAI's grader types are documented in its graders guide, Promptfoo's in its assertions reference. The converter applies this table automatically.

OpenAI Evals graders and their nearest Promptfoo assertion and Litmiq setup
OpenAI graderWhat it checksPromptfoo assertionLitmiq
string_check eqExact match, case-sensitiveequalsExpected behavior: “Output is exactly X”, scored by the judge (not a deterministic check)
string_check neNot an exact matchnot-equalsExpected behavior: “Output is not X”
string_check likeContains, case-sensitivecontainsExpected behavior: “Output contains X”
string_check ilikeContains, any caseicontainsExpected behavior: “Output contains X (any case)”
text_similarity bleu / gleu / meteorSimilarity to a reference, with pass_thresholdbleu / gleu / meteor with thresholdReference answer in expected behavior; the judge scores closeness
text_similarity rouge_1…rouge_5, rouge_lROUGE overlaprouge-n / rouge-l with thresholdSame as above
text_similarity cosineEmbedding similaritysimilar with thresholdSame as above
text_similarity fuzzy_matchFuzzy string match (rapidfuzz)No identical metric: use similar or levenshtein and re-tune the thresholdSame as above
label_modelA model picks a label; passes if it is in passing_labelsllm-rubric (or model-graded-closedqa)Put the grader instructions in expected behavior or scoring criteria
score_modelA model returns a score compared with pass_thresholdllm-rubric with threshold (scores are 0–1)Litmiq’s judge scores 0–10 on four metrics; describe the bar in the criteria
pythonYour Python function returns a scorepython assertion, rewritten to promptfoo’s function signatureNot supported: Litmiq has no code graders. Use Promptfoo for these
Several graders on one evalAll must passSeveral assert entries per testOne expected-behavior line per grader

Template variables: OpenAI templates use {{ item.field }} for dataset columns and {{ sample.output_text }} for the model output. In Promptfoo, dataset columns become vars and assertions check the output directly. The converter renders item.* into each test.

Step 3a: Recreate in Litmiq (no install)

  1. Open it. Click Open in Litmiq in the converter. The system prompt, scoring criteria and every case load into the app.
  2. Pick models. Select the model your eval used (or its replacement: see the model migration guide) and one alternative.
  3. Start small. Each case costs one completion per model plus one judge call on your OpenAI key. Run 20–40 cases first.
  4. Read the reasons. The judge explains every score. Compare failures with your last OpenAI run's output items.

Step 3b: Recreate in Promptfoo

Download promptfooconfig.yaml from the converter and check the TODO comments (Python graders need porting by hand to a python assertion), then validate and run it. The commands are from OpenAI's migration cookbook:

npx promptfoo@latest validate config -c promptfooconfig.yaml
npx promptfoo@latest eval -c promptfooconfig.yaml --no-cache
npx promptfoo@latest view

A minimal hand-written version of OpenAI's own example:

# promptfooconfig.yaml (the IT-ticket example from OpenAI's Evals guide)
description: IT ticket categorization
prompts:
  - label: categorize
    raw: '[{"role":"system","content":"You are an expert in categorizing IT support tickets. Given the support ticket below, categorize the request into one of ''Hardware'', ''Software'', or ''Other''. Respond with only one of those words."},{"role":"user","content":{{ ticket_text | dump }}}]'
providers:
  - openai:chat:gpt-6.1-terra
tests:
  - vars: { ticket_text: "My monitor won't turn on!" }
    assert: [{ type: equals, value: Hardware }]
  - vars: { ticket_text: "I'm in vim and I can't quit!" }
    assert: [{ type: equals, value: Software }]
  - vars: { ticket_text: "Best restaurants in Cleveland?" }
    assert: [{ type: equals, value: Other }]

Step 4: Verify the migration

Migration checklist

  1. List every OpenAI project that has evals.
  2. Run the export script once per project and keep the folder.
  3. Note which model each eval pinned. If it's on the 11 Dec 2026 shutdown list, plan a model migration test.
  4. Convert each eval (Promptfoo YAML, Litmiq, or both).
  5. Port Python graders by hand.
  6. Move any reusable prompt objects into your code before 30 Nov 2026.
  7. Re-run, compare with the old results, re-tune thresholds.
  8. Add the suite to CI (Promptfoo) or your release checklist (prompt testing checklist).

Limits of this guide and the tools

FAQ

When does OpenAI Evals shut down?

OpenAI's deprecations page lists the Evals dashboard, API and graders for eval workflows with a 30 November 2026 date, and the Evals guide says Evals became read-only for existing users on 31 October 2026 and the platform is scheduled to shut down on 30 November 2026. Sources read on 13 Nov 2026.

Is there an official export button for OpenAI Evals?

OpenAI's cookbook on moving to Promptfoo recreates evals by hand and says the approach does not require an OpenAI Evals export feature. The Evals API still lets you read your evals, runs and output items, which is what the export script on this page does, with GET requests only.

What is the best OpenAI Evals alternative?

OpenAI recommends Promptfoo, an open-source CLI that OpenAI announced it would acquire in March 2026. Promptfoo fits code-based, CI-driven evals with exact-match and similarity assertions. Litmiq fits quick, browser-based checks of one system prompt on several OpenAI models with an LLM judge and no install. Many teams can use both.

Will my scores be the same after migrating?

Not necessarily. OpenAI's cookbook notes that similarity-based scoring may not produce identical numerical results and that LLM-as-a-judge graders should be validated after migration. Re-run a sample, compare it with your last OpenAI run, and re-tune thresholds before you trust new results.

Does Litmiq support exact-match graders?

No. Litmiq scores every output with an LLM judge (accuracy, tone, helpfulness and conciseness, 0-10). The converter turns exact-match graders into expected-behavior text the judge checks, which is close but not deterministic. For evals that depend on exact or code-based checks, use the Promptfoo YAML the converter generates.

Is my data uploaded when I use the converter?

No. The converter runs in your browser and nothing is sent to Litmiq. If you click Open in Litmiq, the suite is saved in your browser's localStorage and opened in the app; the app only sends data to OpenAI, with your key, when you run an eval.

What about OpenAI Datasets and stored prompts?

OpenAI's cookbook says platform features such as dataset review and prompt iteration remain in OpenAI. Reusable prompt objects (/v1/prompts) are deprecated on 30 November 2026, and OpenAI advises moving the prompt text into your application code.

Sources (read 13 Nov 2026)

Spotted a change or an error? Email hello@litmiq.com.

Get model deprecation alerts

We email you when OpenAI, Anthropic or Google publish a new shutdown date for a model on our tracker. Sent by hand from hello@litmiq.com, a few times a month at most. Unsubscribe by replying.

We store only your email, through our form provider. See the privacy policy.

Run your migrated eval in Litmiq

Load OpenAI's example (already converted), or paste your own export into the converter above. Free, in your browser.

  • Free, no signup
  • Runs in your browser
  • Your OpenAI key, sent only to OpenAI

Follow Litmiq on LinkedIn