Azure OpenAI evals
On this page
LangSmith is LangChain's hosted platform for tracing, monitoring and evaluating LLM applications and agents; Litmiq is a free browser app for quick, side-by-side prompt tests with LLM-judge scores. They overlap on one thing, running a prompt against test cases and scoring the outputs, and differ on almost everything else: accounts, hosting, scale, production monitoring and price. This page compares them using LangSmith's own documentation and pricing page as checked on 11 November 2026 and rechecked on 19 November 2026.
The short answer
Azure OpenAI (in Microsoft Foundry) has its own evals API: you create an evaluation (a data source schema plus testing criteria), then create a run to execute it. Microsoft Learn documents this at POST {endpoint}/openai/v1/evals and POST {endpoint}/openai/v1/evals/{eval_id}/runs. The v1 /openai/v1/evals endpoint is no longer in preview and no longer requires an aoai-evals: preview header (API version lifecycle page, read 19 November 2026).
Those evals are tied to your Azure / Foundry project. If you want a suite you can keep when you also call OpenAI, Anthropic or another provider, store the cases yourself — in a file, a CI framework, or a browser tester — and treat Azure's evals API as one runner, not the only copy of the truth.
What Azure evals include
Microsoft's Azure OpenAI graders page (read 19 November 2026) lists four grader types you can run locally or remotely:
label_model— an LLM classifies output into labels; pass/fail frompassing_labels.score_model— an LLM assigns a numeric score, optionally against a pass threshold.string_check— deterministic string or pattern comparison.text_similarity— a similarity metric (for example BLEU, ROUGE, cosine) against a reference.
Creating an evaluation does not run it. You add a compatible data source when you create a run. Batch evaluations are documented with a maximum of 100,000 rows and 2 MB per row (evaluation regions and limits page, read 19 November 2026).
How to keep tests portable
- Write cases as ordinary rows: input, optional expected output or context, and the hard rules you care about.
- Keep that file in git. YAML (Promptfoo), test code (DeepEval), or a JSON suite you can paste into Litmiq all work. The format matters less than having one source of cases.
- Use Azure graders or the Azure evals API when you are scoring an Azure deployment. Re-run the same cases against another provider with a CLI or with Litmiq, rather than leaving the only copy of the suite inside one project.
- Prefer hard, portable checks (valid JSON, required phrases, “I don't know” when context is empty) over a judge prompt that only exists in one vendor's UI.
Litmiq can hold the portable, single-turn slice: paste the prompt and cases, run them with your own key, export the JSON. It does not call the Azure evals API for you, and it does not replace Foundry's project-level runners.
Limits of this guide
API shapes, grader names and limits come from Microsoft Learn pages read on 19 November 2026 and may have changed. We did not run a new Azure eval in this article, and we do not publish Azure list prices here. We are not affiliated with Microsoft. Confirm the live docs before you integrate.
FAQ
What are Azure OpenAI evals?
They are evaluations you define in Azure OpenAI / Microsoft Foundry: a data source schema plus testing criteria. You create the evaluation, then create a run to execute it, via the /openai/v1/evals API documented on Microsoft Learn (read 19 November 2026).
Are Azure OpenAI evals still in preview?
Microsoft's v1 API version lifecycle page, read 19 November 2026, says /openai/v1/evals is no longer in preview and no longer requires an aoai-evals: preview header. Confirm on the live page before you integrate.
What graders does Azure document?
The Azure OpenAI graders page, read 19 November 2026, lists label_model, score_model, string_check and text_similarity, runnable locally or remotely.
How do I keep tests portable?
Store the cases in git in a format you can also run elsewhere — Promptfoo, DeepEval, or a JSON suite you can paste into Litmiq — and treat Azure's evals API as one runner, not the only copy of the suite.
Does Litmiq call the Azure evals API?
No. Litmiq can hold a portable, single-turn suite in the browser with your own key. It does not create Foundry evals or replace Azure's project-level runners.
Product names such as Promptfoo, LangSmith, LangChain, OpenAI and DeepEval are trademarks of their respective owners. They are used on this page only to identify those products. Litmiq is not affiliated with, endorsed by, or sponsored by those owners.
Sources
- Microsoft Learn, Azure OpenAI Evals REST reference (read 19 Nov 2026).
- Microsoft Learn, Create an evaluation (read 19 Nov 2026).
- Microsoft Learn, Azure OpenAI Graders (read 19 Nov 2026).
- Microsoft Learn, Azure OpenAI v1 API version lifecycle (read 19 Nov 2026):
/openai/v1/evalsis no longer in preview. - Microsoft Learn, Rate limits, region support, and enterprise features for evaluation (read 19 Nov 2026): 100,000 rows per batch, 2 MB per row.
Try it in Litmiq
Paste your system prompt, add a few test cases and compare OpenAI models side by side. Runs in your browser with your own OpenAI key.
- Free, no signup
- Runs in your browser
- Your OpenAI key, sent only to OpenAI
Litmiq Pro
Saved suites, CI on a schedule and regression alerts for $19/month. The app stays free.
See pricing →Related guides
Compare & reference
- Prompt testing tools compared
- LangSmith alternative
- LangSmith pricing
- Litmiq vs Promptfoo
- Litmiq vs OpenAI Playground
Share this guide