Azure OpenAI evals

On this page

LangSmith is LangChain's hosted platform for tracing, monitoring and evaluating LLM applications and agents; Litmiq is a free browser app for quick, side-by-side prompt tests with LLM-judge scores. They overlap on one thing, running a prompt against test cases and scoring the outputs, and differ on almost everything else: accounts, hosting, scale, production monitoring and price. This page compares them using LangSmith's own documentation and pricing page as checked on 11 November 2026 and rechecked on 19 November 2026.

The short answer

Azure OpenAI (in Microsoft Foundry) has its own evals API: you create an evaluation (a data source schema plus testing criteria), then create a run to execute it. Microsoft Learn documents this at POST {endpoint}/openai/v1/evals and POST {endpoint}/openai/v1/evals/{eval_id}/runs. The v1 /openai/v1/evals endpoint is no longer in preview and no longer requires an aoai-evals: preview header (API version lifecycle page, read 19 November 2026).

Those evals are tied to your Azure / Foundry project. If you want a suite you can keep when you also call OpenAI, Anthropic or another provider, store the cases yourself — in a file, a CI framework, or a browser tester — and treat Azure's evals API as one runner, not the only copy of the truth.

What Azure evals include

Microsoft's Azure OpenAI graders page (read 19 November 2026) lists four grader types you can run locally or remotely:

  • label_model — an LLM classifies output into labels; pass/fail from passing_labels.
  • score_model — an LLM assigns a numeric score, optionally against a pass threshold.
  • string_check — deterministic string or pattern comparison.
  • text_similarity — a similarity metric (for example BLEU, ROUGE, cosine) against a reference.

Creating an evaluation does not run it. You add a compatible data source when you create a run. Batch evaluations are documented with a maximum of 100,000 rows and 2 MB per row (evaluation regions and limits page, read 19 November 2026).

How to keep tests portable

  1. Write cases as ordinary rows: input, optional expected output or context, and the hard rules you care about.
  2. Keep that file in git. YAML (Promptfoo), test code (DeepEval), or a JSON suite you can paste into Litmiq all work. The format matters less than having one source of cases.
  3. Use Azure graders or the Azure evals API when you are scoring an Azure deployment. Re-run the same cases against another provider with a CLI or with Litmiq, rather than leaving the only copy of the suite inside one project.
  4. Prefer hard, portable checks (valid JSON, required phrases, “I don't know” when context is empty) over a judge prompt that only exists in one vendor's UI.

Litmiq can hold the portable, single-turn slice: paste the prompt and cases, run them with your own key, export the JSON. It does not call the Azure evals API for you, and it does not replace Foundry's project-level runners.

Limits of this guide

API shapes, grader names and limits come from Microsoft Learn pages read on 19 November 2026 and may have changed. We did not run a new Azure eval in this article, and we do not publish Azure list prices here. We are not affiliated with Microsoft. Confirm the live docs before you integrate.

FAQ

What are Azure OpenAI evals?

They are evaluations you define in Azure OpenAI / Microsoft Foundry: a data source schema plus testing criteria. You create the evaluation, then create a run to execute it, via the /openai/v1/evals API documented on Microsoft Learn (read 19 November 2026).

Are Azure OpenAI evals still in preview?

Microsoft's v1 API version lifecycle page, read 19 November 2026, says /openai/v1/evals is no longer in preview and no longer requires an aoai-evals: preview header. Confirm on the live page before you integrate.

What graders does Azure document?

The Azure OpenAI graders page, read 19 November 2026, lists label_model, score_model, string_check and text_similarity, runnable locally or remotely.

How do I keep tests portable?

Store the cases in git in a format you can also run elsewhere — Promptfoo, DeepEval, or a JSON suite you can paste into Litmiq — and treat Azure's evals API as one runner, not the only copy of the suite.

Does Litmiq call the Azure evals API?

No. Litmiq can hold a portable, single-turn suite in the browser with your own key. It does not create Foundry evals or replace Azure's project-level runners.

Product names such as Promptfoo, LangSmith, LangChain, OpenAI and DeepEval are trademarks of their respective owners. They are used on this page only to identify those products. Litmiq is not affiliated with, endorsed by, or sponsored by those owners.

Sources

Try it in Litmiq

Paste your system prompt, add a few test cases and compare OpenAI models side by side. Runs in your browser with your own OpenAI key.

  • Free, no signup
  • Runs in your browser
  • Your OpenAI key, sent only to OpenAI

Follow Litmiq on LinkedIn