{"id":25531657,"url":"https://github.com/langchain-ai/openevals","last_synced_at":"2026-04-07T23:02:26.713Z","repository":{"id":278417804,"uuid":"929174241","full_name":"langchain-ai/openevals","owner":"langchain-ai","description":"Readymade evaluators for your LLM apps","archived":false,"fork":false,"pushed_at":"2026-04-01T23:00:50.000Z","size":3948,"stargazers_count":1006,"open_issues_count":10,"forks_count":93,"subscribers_count":10,"default_branch":"main","last_synced_at":"2026-04-03T06:46:51.763Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/langchain-ai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-02-08T00:12:20.000Z","updated_at":"2026-04-03T03:15:31.000Z","dependencies_parsed_at":"2025-04-12T20:41:52.705Z","dependency_job_id":"679d9c75-e517-4f92-90be-5ba102cd538c","html_url":"https://github.com/langchain-ai/openevals","commit_stats":null,"previous_names":["langchain-ai/openevals"],"tags_count":41,"template":false,"template_full_name":null,"purl":"pkg:github/langchain-ai/openevals","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/langchain-ai%2Fopenevals","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/langchain-ai%2Fopenevals/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/langchain-ai%2Fopenevals/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/langchain-ai%2Fopenevals/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/langchain-ai","download_url":"https://codeload.github.com/langchain-ai/openevals/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/langchain-ai%2Fopenevals/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31532337,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-07T16:28:08.000Z","status":"ssl_error","status_checked_at":"2026-04-07T16:28:06.951Z","response_time":105,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-02-20T01:19:21.804Z","updated_at":"2026-04-07T23:02:26.703Z","avatar_url":"https://github.com/langchain-ai.png","language":"Python","funding_links":[],"categories":["9. Evaluation, Benchmarks \u0026 Datasets","5 ยท Evaluation infrastructure (the eval stack: datasets, scorers, online/offline, tracing, CI)","LLM and Agent Observability","Browse The Shelves","๐Ÿ› ๏ธ Hands-on Projects and Examples"],"sub_categories":["5a ยท Eval frameworks \u0026 harnesses (code-first test-runners)","Agent evals","๐Ÿ› ๏ธ Key Frameworks \u0026 Code Samples"],"readme":"# โš–๏ธ OpenEvals\n\nMuch like tests in traditional software, evals are an important part of bringing LLM applications to production.\nThe goal of this package is to help provide a starting point for you to write evals for your LLM applications, from which\nyou can write more custom evals specific to your application.\n\nIf you are looking for evals specific to evaluating LLM agents, please check out [`agentevals`](https://github.com/langchain-ai/agentevals).\n\n# Quickstart\n\n\u003e [!TIP]\n\u003e If you'd like to follow along with a video walkthrough, click the image below:\n\u003e [![Video quickstart](https://img.youtube.com/vi/J-F30jRyhoA/0.jpg)](https://www.youtube.com/watch?v=J-F30jRyhoA)\n\nTo get started, install `openevals`:\n\n\u003cdetails\u003e\n\u003csummary\u003ePython\u003c/summary\u003e\n\n```bash\npip install openevals\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eTypeScript\u003c/summary\u003e\n\n```bash\nnpm install openevals @langchain/core\n```\n\u003c/details\u003e\n\nThis quickstart will use an evaluator powered by OpenAI's `gpt-5.4` model to judge your results, so you'll need to set your OpenAI API key as an environment variable:\n\n```bash\nexport OPENAI_API_KEY=\"your_openai_api_key\"\n```\n\nOnce you've done this, you can run your first eval:\n\n\u003cdetails\u003e\n\u003csummary\u003ePython\u003c/summary\u003e\n\n```python\nfrom openevals.llm import create_llm_as_judge\nfrom openevals.prompts import CONCISENESS_PROMPT\n\nconciseness_evaluator = create_llm_as_judge(\n # CONCISENESS_PROMPT is just an f-string\n prompt=CONCISENESS_PROMPT,\n model=\"openai:gpt-5.4\",\n)\n\ninputs = \"How is the weather in San Francisco?\"\n# These are fake outputs, in reality you would run your LLM-based system to get real outputs\noutputs = \"Thanks for asking! The current weather in San Francisco is sunny and 90 degrees.\"\n# When calling an LLM-as-judge evaluator, parameters are formatted directly into the prompt\neval_result = conciseness_evaluator(\n inputs=inputs,\n outputs=outputs,\n)\n\nprint(eval_result)\n```\n\n```\n{\n 'key': 'score',\n 'score': False,\n 'comment': 'The output includes an unnecessary greeting (\"Thanks for asking!\") and extra..'\n}\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eTypeScript\u003c/summary\u003e\n\n```ts\nimport { createLLMAsJudge, CONCISENESS_PROMPT } from \"openevals\";\n\nconst concisenessEvaluator = createLLMAsJudge({\n // CONCISENESS_PROMPT is just an f-string\n prompt: CONCISENESS_PROMPT,\n model: \"openai:gpt-5.4\",\n});\n\nconst inputs = \"How is the weather in San Francisco?\"\n// These are fake outputs, in reality you would run your LLM-based system to get real outputs\nconst outputs = \"Thanks for asking! The current weather in San Francisco is sunny and 90 degrees.\"\n\n// When calling an LLM-as-judge evaluator, parameters are formatted directly into the prompt\nconst evalResult = await concisenessEvaluator({\n inputs,\n outputs,\n});\n\nconsole.log(evalResult);\n```\n\n```\n{\n key: 'score',\n score: false,\n comment: 'The output includes an unnecessary greeting (\"Thanks for asking!\") and extra..'\n}\n```\n\u003c/details\u003e\n\nThis is an example of a reference-free evaluator - some other evaluators may accept slightly different parameters such as a required reference output. LLM-as-judge evaluators will attempt to format any passed parameters into their passed `prompt`, allowing you to flexibly customize criteria or add other fields.\n\nSee the [LLM-as-judge](#llm-as-judge) section for more information on how to customize the [scoring](#customizing-output-score-values) to output float values rather than just `True/False`, the [model](#customizing-the-model), or the [prompt](#customizing-prompts)!\n\n# Table of Contents\n\n- [โš–๏ธ OpenEvals](#๏ธ-openevals)\n- [Quickstart](#quickstart)\n- [Table of Contents](#table-of-contents)\n- [Installation](#installation)\n- [Evaluators](#evaluators)\n - \u003cdetails\u003e\n \u003csummary\u003e\u003ca href=\"#llm-as-judge\"\u003eLLM-as-Judge\u003c/a\u003e\u003c/summary\u003e\n\n - [Customizing prompts](#customizing-prompts)\n - [Customizing with LangChain prompt templates](#customizing-with-langchain-prompt-templates)\n - [Customizing the model](#customizing-the-model)\n - [Customizing output score values](#customizing-output-score-values)\n - [Customizing output schema](#customizing-output-schema)\n - [Logging feedback with custom output schemas](#logging-feedback-with-custom-output-schemas)\n - [Structured prompts](#structured-prompts)\n - [Multimodal](#multimodal)\n - [Option 1: `attachments` parameter](#option-1-attachments-parameter)\n - [Option 2: LangChain prompt template](#option-2-langchain-prompt-template)\n \u003c/details\u003e\n\n - \u003cdetails\u003e\n \u003csummary\u003e\u003ca href=\"#prebuilt-prompts\"\u003ePrebuilt prompts\u003c/a\u003e\u003c/summary\u003e\n\n - [Quality](#quality)\n - [Safety](#safety)\n - [Security](#security)\n - [Image](#image)\n - [Voice](#voice)\n - \u003cdetails\u003e\n \u003csummary\u003e\u003ca href=\"#rag\"\u003eRAG\u003c/a\u003e\u003c/summary\u003e\n\n - [Correctness](#correctness-rag)\n - [Helpfulness](#helpfulness)\n - [Groundedness](#groundedness)\n - [Retrieval relevance](#retrieval-relevance)\n - [Retrieval relevance with LLM-as-judge](#retrieval-relevance-with-llm-as-judge)\n - [Retrieval relevance with string evaluators](#retrieval-relevance-with-string-evaluators)\n\n \u003c/details\u003e\n\n \u003c/details\u003e\n\n - \u003cdetails\u003e\n \u003csummary\u003e\u003ca href=\"#extraction-and-tool-calls\"\u003eExtraction and tool calls\u003c/a\u003e\u003c/summary\u003e\n\n - [Evaluating structured output with exact match](#evaluating-structured-output-with-exact-match)\n - [Evaluating structured output with LLM-as-a-Judge](#evaluating-structured-output-with-llm-as-a-judge)\n\n \u003c/details\u003e\n\n - \u003cdetails\u003e\n \u003csummary\u003e\u003ca href=\"#code\"\u003eCode\u003c/a\u003e\u003c/summary\u003e\n\n - [Extracting code outputs](#extracting-code-outputs)\n - [Pyright (Python-only)](#pyright-python-only)\n - [Mypy (Python-only)](#mypy-python-only)\n - [TypeScript type-checking (TypeScript-only)](#typescript-type-checking-typescript-only)\n - [LLM-as-judge for code](#llm-as-judge-for-code)\n\n \u003c/details\u003e\n\n - \u003cdetails\u003e\n \u003csummary\u003e\u003ca href=\"#sandboxed-code\"\u003eSandboxed code\u003c/a\u003e\u003c/summary\u003e\n\n - [Sandbox Pyright (Python-only)](#sandbox-pyright-python-only)\n - [Sandbox TypeScript type-checking (TypeScript-only)](#sandbox-typescript-type-checking-typescript-only)\n - [Sandbox Execution](#sandbox-execution)\n\n \u003c/details\u003e\n\n - \u003cdetails\u003e\n \u003csummary\u003e\u003ca href=\"#agent-trajectory\"\u003eAgent trajectory\u003c/a\u003e\u003c/summary\u003e\n\n - [Trajectory match](#trajectory-match)\n - [Strict match](#strict-match)\n - [Unordered match](#unordered-match)\n - [Subset and superset match](#subset-and-superset-match)\n - [Tool args match modes](#tool-args-match-modes)\n - [Trajectory LLM-as-judge](#trajectory-llm-as-judge)\n - [Prebuilt trajectory prompts](#prebuilt-trajectory-prompts)\n\n \u003c/details\u003e\n\n - \u003cdetails\u003e\n \u003csummary\u003e\u003ca href=\"#other\"\u003eOther\u003c/a\u003e\u003c/summary\u003e\n\n - [Exact match](#exact-match)\n - [Levenshtein distance](#levenshtein-distance)\n - [Embedding similarity](#embedding-similarity)\n\n \u003c/details\u003e\n\n - [Creating your own](#creating-your-own)\n - [Evaluator interface](#evaluator-interface)\n - [Logging to LangSmith](#logging-to-langsmith)\n - [Example](#example)\n - [Python async support](#python-async-support)\n\n- [Multiturn Simulation](#multiturn-simulation)\n - [Simulating users](#simulating-users)\n - [Prebuilt simulated user](#prebuilt-simulated-user)\n - [Custom simulated users](#custom-simulated-users)\n - [Multiturn simulation with LangGraph](#multiturn-simulation-with-langgraph)\n\n- [LangSmith Integration](#langsmith-integration)\n - [Pytest or Vitest/Jest](#pytest-or-vitestjest)\n - [Evaluate](#evaluate)\n\n- [Acknowledgements](#acknowledgements)\n- [Thank you!](#thank-you)\n\n# Installation\n\nYou can install `openevals` like this:\n\n\u003cdetails\u003e\n\u003csummary\u003ePython\u003c/summary\u003e\n\n```bash\npip install openevals\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eTypeScript\u003c/summary\u003e\n\n```bash\nnpm install openevals @langchain/core\n```\n\u003c/details\u003e\n\nFor LLM-as-judge evaluators, you will also need an LLM client. By default, `openevals` will use [LangChain chat model integrations](https://python.langchain.com/docs/integrations/chat/) and comes with `langchain_openai` installed by default. However, if you prefer, you may use the OpenAI client directly:\n\n\u003cdetails\u003e\n\u003csummary\u003ePython\u003c/summary\u003e\n\n```bash\npip install openai\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eTypeScript\u003c/summary\u003e\n\n```bash\nnpm install openai\n```\n\u003c/details\u003e\n\nIt is also helpful to be familiar with some [evaluation concepts](https://docs.langchain.com/langsmith/evaluation-concepts).\n\n# Evaluators\n\n## LLM-as-judge\n\nOne common way to evaluate an LLM app's outputs is to use another LLM as a judge. This is generally a good starting point for evals.\n\nThis package contains the `create_llm_as_judge` function, which takes a prompt and a model as input, and returns an evaluator function\nthat handles converting parameters into strings and parsing the judge LLM's outputs as a score.\n\nTo use the `create_llm_as_judge` function, you need to provide a prompt and a model. To get started, OpenEvals has some prebuilt prompts in the `openevals.prompts` module that you can use out of the box. Here's an example:\n\n\u003cdetails\u003e\n\u003csummary\u003ePython\u003c/summary\u003e\n\n```python\nfrom openevals.llm import create_llm_as_judge\nfrom openevals.prompts import CORRECTNESS_PROMPT\n\ncorrectness_evaluator = create_llm_as_judge(\n prompt=CORRECTNESS_PROMPT,\n model=\"openai:gpt-5.4\",\n)\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eTypeScript\u003c/summary\u003e\n\n```ts\nimport { createLLMAsJudge, CORRECTNESS_PROMPT } from \"openevals\";\n\nconst correctnessEvaluator = createLLMAsJudge({\n prompt: CORRECTNESS_PROMPT,\n model: \"openai:gpt-5.4\",\n});\n```\n\n\u003c/details\u003e\n\nNote that `CORRECTNESS_PROMPT` is a simple f-string that you can log and edit as needed for your specific use case:\n\n\u003cdetails\u003e\n\u003csummary\u003ePython\u003c/summary\u003e\n\n```python\nprint(CORRECTNESS_PROMPT)\n```\n\n```\nYou are an expert data labeler evaluating model outputs for correctness. Your task is to assign a score based on the following rubric:\n\n\u003cRubric\u003e\n A correct answer:\n - Provides accurate and complete information\n ...\n\u003cinput\u003e\n{inputs}\n\u003c/input\u003e\n\n\u003coutput\u003e\n{outputs}\n\u003c/output\u003e\n...\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eTypeScript\u003c/summary\u003e\n\n```ts\nconsole.log(CORRECTNESS_PROMPT);\n```\n\n```\nYou are an expert data labeler evaluating model outputs for correctness. Your task is to assign a score based on the following rubric:\n\n\u003cRubric\u003e\n A correct answer:\n - Provides accurate and complete information\n ...\n\u003cinput\u003e\n{inputs}\n\u003c/input\u003e\n\n\u003coutput\u003e\n{outputs}\n\u003c/output\u003e\n...\n```\n\n\u003c/details\u003e\n\nBy convention, we generally suggest sticking to `inputs`, `outputs`, and `reference_outputs` as the names of the parameters for LLM-as-judge evaluators, but these will be directly formatted into the prompt so you can use any variable names you want.\n\nOpenEvals includes many prebuilt prompts for common evaluation scenarios. See the [Prebuilt prompts](#prebuilt-prompts) section for a full list organized by category.\n\n### Customizing prompts\n\nThe `prompt` parameter for `create_llm_as_judge` may be an f-string, [LangChain prompt template](#customizing-with-langchain-prompt-templates), or a function that takes kwargs and returns a list of formatted messages.\n\nThough we suggest sticking to conventional names (`inputs`, `outputs`, and `reference_outputs`) as prompt variables, your prompts can also require additional variables. You would then pass these extra variables when calling your evaluator function. Here's an example of a prompt that requires an extra variable named `context`:\n\n\u003cdetails\u003e\n\u003csummary\u003ePython\u003c/summary\u003e\n\n```python\nfrom openevals.llm import create_llm_as_judge\n\nMY_CUSTOM_PROMPT = \"\"\"\nUse the following context to help you evaluate for hallucinations in the output:\n\n\u003ccontext\u003e\n{context}\n\u003c/context\u003e\n\n\u003cinput\u003e\n{inputs}\n\u003c/input\u003e\n\n\u003coutput\u003e\n{outputs}\n\u003c/output\u003e\n\"\"\"\n\ncustom_prompt_evaluator = create_llm_as_judge(\n prompt=MY_CUSTOM_PROMPT,\n model=\"openai:gpt-5.4\",\n)\n\ncustom_prompt_evaluator(\n inputs=\"What color is the sky?\",\n outputs=\"The sky is red.\",\n context=\"It is early evening.\",\n)\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eTypeScript\u003c/summary\u003e\n\n```ts\nimport { createLLMAsJudge } from \"openevals\";\n\nconst MY_CUSTOM_PROMPT = `\nUse the following context to help you evaluate for hallucinations in the output:\n\n\u003ccontext\u003e\n{context}\n\u003c/context\u003e\n\n\u003cinput\u003e\n{inputs}\n\u003c/input\u003e\n\n\u003coutput\u003e\n{outputs}\n\u003c/output\u003e\n`;\n\nconst customPromptEvaluator = createLLMAsJudge({\n prompt: MY_CUSTOM_PROMPT,\n model: \"openai:gpt-5.4\",\n});\n\nconst inputs = \"What color is the sky?\"\nconst outputs = \"The sky is red.\"\n\nconst evalResult = await customPromptEvaluator({\n inputs,\n outputs,\n});\n```\n\u003c/details\u003e\n\nThe following options are also available for string prompts:\n\n- `system`: a string that sets a system prompt for the judge model by adding a `system` message before other parts of the prompt.\n- `few_shot_examples`: a list of example dicts that are appended to the end of the prompt. This is useful for providing the judge model with examples of good and bad outputs. The required structure looks like this:\n\n\u003cdetails\u003e\n\u003csummary\u003ePython\u003c/summary\u003e\n\n```python\nfew_shot_examples = [\n {\n \"inputs\": \"What color is the sky?\",\n \"outputs\": \"The sky is red.\",\n \"reasoning\": \"The sky is red because it is early evening.\",\n \"score\": 1,\n }\n]\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eTypeScript\u003c/summary\u003e\n\n```ts\nconst fewShotExamples = [\n {\n inputs: \"What color is the sky?\",\n outputs: \"The sky is red.\",\n reasoning: \"The sky is red because it is early evening.\",\n score: 1,\n }\n]\n```\n\u003c/details\u003e\n\nThese will be appended to the end of the final user message in the prompt.\n\n#### Customizing with LangChain prompt templates\n\nYou can also pass a [LangChain prompt template](https://python.langchain.com/docs/concepts/prompt_templates/) if you want more control over formatting. Here's an example that uses mustache formatting instead of f-strings:\n\n\u003cdetails\u003e\n\u003csummary\u003ePython\u003c/summary\u003e\n\n```python\nfrom openevals.llm import create_llm_as_judge\nfrom langchain_core.prompts.chat import ChatPromptTemplate\n\ninputs = {\"a\": 1, \"b\": 2}\noutputs = {\"a\": 1, \"b\": 2}\n\nprompt = ChatPromptTemplate([\n (\"system\", \"You are an expert at determining if two objects are equal.\"),\n (\"human\", \"Are these two equal? {{inputs}} {{outputs}}\"),\n], template_format=\"mustache\")\n\nllm_as_judge = create_llm_as_judge(\n prompt=prompt,\n model=\"openai:gpt-5.4\",\n feedback_key=\"equality\",\n)\n\neval_result = llm_as_judge(inputs=inputs, outputs=outputs)\n\nprint(eval_result)\n```\n\n```\n{\n key: 'equality',\n score: True,\n comment: '...'\n}\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eTypeScript\u003c/summary\u003e\n\n```ts\nimport { createLLMAsJudge } from \"openevals\";\nimport { ChatPromptTemplate } from \"@langchain/core/prompts\";\n\nconst inputs = { a: 1, b: 2 };\nconst outputs = { a: 1, b: 2 };\n\nconst prompt = ChatPromptTemplate.fromMessages([\n [\"system\", \"You are an expert at determining if two objects are equal.\"],\n [\"user\", \"Are these two equal? {{inputs}} {{outputs}}\"],\n], { templateFormat: \"mustache\" });\n\nconst evaluator = createLLMAsJudge({\n prompt,\n model: \"openai:gpt-5.4\",\n feedbackKey: \"equality\",\n});\n\nconst result = await evaluator({ inputs, outputs });\n```\n\n```\n{\n key: 'equality',\n score: true,\n comment: '...'\n}\n```\n\n\u003c/details\u003e\n\nYou can also pass in a function that takes your LLM-as-judge inputs as kwargs and returns formatted chat messages.\n\n### Customizing the model\n\nThere are a few ways you can customize the model used for evaluation. You can pass a string formatted as `PROVIDER:MODEL` (e.g. `model=anthropic:claude-3-5-sonnet-latest`) as the `model`, in which case the package will [attempt to import and initialize a LangChain chat model instance](https://python.langchain.com/docs/how_to/chat_models_universal_init/). This requires you to install the appropriate LangChain integration package installed. Here's an example:\n\n\u003cdetails\u003e\n\u003csummary\u003ePython\u003c/summary\u003e\n\n```bash\npip install langchain-anthropic\n```\n\n```python\nfrom openevals.llm import create_llm_as_judge\nfrom openevals.prompts import CORRECTNESS_PROMPT\n\nanthropic_evaluator = create_llm_as_judge(\n prompt=CORRECTNESS_PROMPT,\n model=\"anthropic:claude-3-5-sonnet-latest\",\n)\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eTypeScript\u003c/summary\u003e\n\n```bash\nnpm install @langchain/anthropic\n```\n\n```ts\nimport { createLLMAsJudge, CORRECTNESS_PROMPT } from \"openevals\";\n\nconst anthropicEvaluator = createLLMAsJudge({\n prompt: CORRECTNESS_PROMPT,\n model: \"anthropic:claude-3-5-sonnet-latest\",\n});\n```\n\u003c/details\u003e\n\nYou can also directly pass a LangChain chat model instance as `judge`. Note that your chosen model must support [structured output](https://python.langchain.com/docs/integrations/chat/):\n\n\u003cdetails\u003e\n\u003csummary\u003ePython\u003c/summary\u003e\n\n```python\nfrom openevals.llm import create_llm_as_judge\nfrom openevals.prompts import CORRECTNESS_PROMPT\nfrom langchain_anthropic import ChatAnthropic\n\nanthropic_evaluator = create_llm_as_judge(\n prompt=CORRECTNESS_PROMPT,\n judge=ChatAnthropic(model=\"claude-3-5-sonnet-latest\", temperature=0.5),\n)\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eTypeScript\u003c/summary\u003e\n\n```ts\nimport { createLLMAsJudge, CORRECTNESS_PROMPT } from \"openevals\";\nimport { ChatAnthropic } from \"@langchain/anthropic\";\n\nconst anthropicEvaluator = createLLMAsJudge({\n prompt: CORRECTNESS_PROMPT,\n judge: new ChatAnthropic({ model: \"claude-3-5-sonnet-latest\", temperature: 0.5 }),\n});\n```\n\u003c/details\u003e\n\nThis is useful in scenarios where you need to initialize your model with specific parameters, such as `temperature` or alternate URLs if using models through a service like Azure.\n\nFinally, you can pass a model name as `model` and a `judge` parameter set to an OpenAI client instance:\n\n\u003cdetails\u003e\n\u003csummary\u003ePython\u003c/summary\u003e\n\n```bash\npip install openai\n```\n\n```python\nfrom openai import OpenAI\n\nfrom openevals.llm import create_llm_as_judge\nfrom openevals.prompts import CORRECTNESS_PROMPT\n\nopenai_evaluator = create_llm_as_judge(\n prompt=CORRECTNESS_PROMPT,\n model=\"gpt-5.4\",\n judge=OpenAI(),\n)\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eTypeScript\u003c/summary\u003e\n\n```bash\nnpm install openai\n```\n\n```ts\nimport { OpenAI } from \"openai\";\nimport { createLLMAsJudge, CORRECTNESS_PROMPT } from \"openevals\";\n\nconst openaiEvaluator = createLLMAsJudge({\n prompt: CORRECTNESS_PROMPT,\n model: \"gpt-5.4\",\n judge: new OpenAI(),\n});\n```\n\u003c/details\u003e\n\n### Customizing output score values\n\nThere are two fields you can set to customize the outputted scores of your evaluator:\n\n- `continuous`: a boolean that sets whether the evaluator should return a float score somewhere between 0 and 1 instead of a binary score. Defaults to `False`.\n- `choices`: a list of floats that sets the possible scores for the evaluator.\n\nThese parameters are mutually exclusive. When using either of them, you should make sure that your prompt is grounded in information on what specific scores mean - the prebuilt ones in this repo do not have this information!\n\nFor example, here's an example of how to define a less harsh definition of correctness that only penalizes incorrect answers by 50% if they are on-topic:\n\n\u003cdetails\u003e\n\u003csummary\u003ePython\u003c/summary\u003e\n\n```python\nfrom openevals.llm import create_llm_as_judge\n\nMY_CUSTOM_PROMPT = \"\"\"\nYou are an expert data labeler evaluating model outputs for correctness. Your task is to assign a score based on the following rubric:\n\n\u003cRubric\u003e\n Assign a score of 0, .5, or 1 based on the following criteria:\n - 0: The answer is incorrect and does not mention doodads\n - 0.5: The answer mentions doodads but is otherwise incorrect\n - 1: The answer is correct and mentions doodads\n\u003c/Rubric\u003e\n\n\u003cinput\u003e\n{inputs}\n\u003c/input\u003e\n\n\u003coutput\u003e\n{outputs}\n\u003c/output\u003e\n\n\u003creference_outputs\u003e\n{reference_outputs}\n\u003c/reference_outputs\u003e\n\"\"\"\n\nevaluator = create_llm_as_judge(\n prompt=MY_CUSTOM_PROMPT,\n choices=[0.0, 0.5, 1.0],\n model=\"openai:gpt-5.4\",\n)\n\nresult = evaluator(\n inputs=\"What is the current price of doodads?\",\n outputs=\"The price of doodads is $10.\",\n reference_outputs=\"The price of doodads is $15.\",\n)\n\nprint(result)\n```\n\n```\n{\n 'key': 'score',\n 'score': 0.5,\n 'comment': 'The provided answer mentioned doodads but was incorrect.'\n}\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eTypeScript\u003c/summary\u003e\n\n```ts\nimport { createLLMAsJudge } from \"openevals\";\n\nconst MY_CUSTOM_PROMPT = `\nYou are an expert data labeler evaluating model outputs for correctness. Your task is to assign a score based on the following rubric:\n\n\u003cRubric\u003e\n Assign a score of 0, .5, or 1 based on the following criteria:\n - 0: The answer is incorrect and does not mention doodads\n - 0.5: The answer mentions doodads but is otherwise incorrect\n - 1: The answer is correct and mentions doodads\n\u003c/Rubric\u003e\n\n\u003cinput\u003e\n{inputs}\n\u003c/input\u003e\n\n\u003coutput\u003e\n{outputs}\n\u003c/output\u003e\n\n\u003creference_outputs\u003e\n{reference_outputs}\n\u003c/reference_outputs\u003e\n`;\n\nconst customEvaluator = createLLMAsJudge({\n prompt: MY_CUSTOM_PROMPT,\n choices: [0.0, 0.5, 1.0],\n model: \"openai:gpt-5.4\",\n});\n\nconst result = await customEvaluator({\n inputs: \"What is the current price of doodads?\",\n outputs: \"The price of doodads is $10.\",\n reference_outputs: \"The price of doodads is $15.\",\n});\n\nconsole.log(result);\n```\n\n```\n{\n 'key': 'score',\n 'score': 0.5,\n 'comment': 'The provided answer mentioned doodads but was incorrect.'\n}\n```\n\u003c/details\u003e\n\nFinally, if you would like to disable justifications for a given score, you can set `use_reasoning=False` when creating your evaluator.\n\n### Customizing output schema\n\nIf you need to change the structure of the raw output generated by the LLM, you can also pass a custom output schema into your LLM-as-judge evaluator as `output_schema` (Python) / `outputSchema` (TypeScript). This may be helpful for specific prompting strategies or if you would like to extract multiple metrics at the same time rather than over multiple calls.\n\n\u003e [!CAUTION]\n\u003e Passing `output_schema` changes the return value of the evaluator to match the passed `output_schema` value instead of the typical OpenEvals format.\n\u003e We recommend sticking with the default schema if you do not specifically need additional properties.\n\nFor Python, `output_schema` may be:\n\n- A `TypedDict` instance\n- A [Pydantic](https://docs.pydantic.dev) model\n- [JSON schema](https://json-schema.org/)\n- [OpenAI's structured output format](https://platform.openai.com/docs/guides/structured-outputs?api-mode=chat#supported-schemas)\n\nFor TypeScript, `outputSchema` may be:\n\n- A [Zod](https://zod.dev) object\n- [JSON schema](https://json-schema.org/)\n- [OpenAI's structured output format](https://platform.openai.com/docs/guides/structured-outputs?api-mode=chat#supported-schemas)\n\nNote that if you are using an OpenAI client directly, only JSON schema and OpenAI's structured output format.\n\nHere's an example:\n\n\u003cdetails\u003e\n\u003csummary\u003ePython\u003c/summary\u003e\n\n```python\nfrom typing_extensions import TypedDict\n\nfrom openevals.llm import create_llm_as_judge\n\nclass EqualityResult(TypedDict):\n equality_justification: str\n are_equal: bool\n\ninputs = \"The rain in Spain falls mainly on the plain.\"\n\noutputs = \"The rain in Spain falls mainly on the plain.\"\n\nllm_as_judge = create_llm_as_judge(\n prompt=\"Are the following two values equal? {inputs} {outputs}\",\n model=\"openai:gpt-5.4\",\n output_schema=EqualityResult,\n)\neval_result = llm_as_judge(inputs=inputs, outputs=outputs)\n\nprint(eval_result)\n```\n\n```\n{\n 'equality_justification': 'The values are equal because they have the same properties with identical values.',\n 'are_equal': True,\n}\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eTypeScript\u003c/summary\u003e\n\n```ts\nimport { z } from \"zod\";\n\nimport { createLLMAsJudge } from \"openevals\";\n\nconst equalitySchema = z.object({\n equality_justification: z.string(),\n are_equal: z.boolean(),\n})\n\nconst inputs = \"The rain in Spain falls mainly on the plain.\";\nconst outputs = \"The rain in Spain falls mainly on the plain.\";\n\nconst llmAsJudge = createLLMAsJudge({\n prompt: \"Are the following two values equal? {inputs} {outputs}\",\n model: \"openai:gpt-5.4\",\n outputSchema: equalitySchema,\n});\n\nconst evalResult = await llmAsJudge({ inputs, outputs });\n\nconsole.log(evalResult);\n```\n\n```\n{\n 'equality_justification': 'The values are equal because they have the same properties with identical values.',\n 'are_equal': True,\n}\n```\n\n\u003c/details\u003e\n\n#### Logging feedback with custom output schemas\n\nIf you are using an OpenEvals evaluator with [LangSmith's `pytest` or `Vitest`/`Jest` runners](#pytest-or-vitestjest), you will need to manually [log feedback keys](https://docs.langchain.com/langsmith/pytest#log-feedback).\n\nIf you are using `evaluate`, you will need to wrap your evaluator in another function that maps your evaluator return value to [feedback in the right format](https://docs.langchain.com/langsmith/code-evaluator).\n\n#### Structured prompts\n\nPassing in a pulled prompt from the [LangChain prompt hub](https://smith.langchain.com/hub) that has an output schema set will also change the output schema for the LLM-as-judge evaluator.\n\n### Multimodal\n\nLLM-as-judge evaluators support multimodal inputs including images, audio, and PDFs. There are two ways to pass multimodal content:\n\n- **`attachments` parameter** โ€” include an `{attachments}` placeholder in your prompt and pass the content via the `attachments` kwarg.\n- **LangChain prompt template** โ€” introduce multimodal content directly into the prompt message. See the [LangChain multimodal messages docs](https://docs.langchain.com/oss/python/langchain/messages#multimodal) for details.\n\n#### Option 1: `attachments` parameter\n\nThe `attachments` parameter supports a single dict or a list of dicts with a `mime_type` and base64-encoded `data` field. The prebuilt [Image](#image) and [Voice](#voice) prompts already include the `{attachments}` placeholder, or you can add it to any custom prompt.\n\nSupported attachment types:\n\n| Type | `mime_type` |\n|------|-------------|\n| Images | `image/png`, `image/jpeg`, `image/gif`, `image/webp` |\n| Audio | `audio/wav`, `audio/mp3`, `audio/mpeg` |\n| PDF | `application/pdf` |\n\n\u003e [!NOTE]\n\u003e Multimodal support depends on your model provider. Audio input and structured output (e.g. returning a score with a comment) are not supported simultaneously by all providers โ€” currently only Gemini supports both at once. The prebuilt [Voice](#voice) prompts use `google_genai:gemini-2.0-flash` (Python) / `google-genai:gemini-2.0-flash` (TypeScript) for this reason.\n\nPassing a URL string directly as `attachments` is supported for images only. Audio and PDF attachments must be passed as a base64-encoded data URI with `mime_type` and `data` fields.\n\nHere's an example using the prebuilt `SENSITIVE_IMAGERY_PROMPT`. You can pass an image as a URL or as a base64-encoded data URI โ€” both work the same way:\n\n\u003cdetails\u003e\n\u003csummary\u003ePython\u003c/summary\u003e\n\n```python\nimport base64\nfrom openevals.llm import create_llm_as_judge\nfrom openevals.prompts import SENSITIVE_IMAGERY_PROMPT\n\nevaluator = create_llm_as_judge(\n prompt=SENSITIVE_IMAGERY_PROMPT,\n feedback_key=\"sensitive_imagery\",\n model=\"openai:gpt-5.4\",\n)\n\n# Option A: pass a URL string directly\neval_result = evaluator(\n inputs=\"Review this image for sensitive content\",\n outputs=\"The image appears to contain appropriate content\",\n attachments=\"https://example.com/image.jpg\",\n)\n\n# Option B: pass a base64-encoded data URI\nwith open(\"image.jpg\", \"rb\") as f:\n image_data = \"data:image/jpeg;base64,\" + base64.b64encode(f.read()).decode(\"utf-8\")\n\neval_result = evaluator(\n inputs=\"Review this image for sensitive content\",\n outputs=\"The image appears to contain appropriate content\",\n attachments={\"mime_type\": \"image/jpeg\", \"data\": image_data},\n)\n\nprint(eval_result)\n```\n\n```\n{\n 'key': 'sensitive_imagery',\n 'score': False,\n 'comment': '...'\n}\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eTypeScript\u003c/summary\u003e\n\n```ts\nimport * as fs from \"fs\";\nimport { createLLMAsJudge, SENSITIVE_IMAGERY_PROMPT } from \"openevals\";\n\nconst evaluator = createLLMAsJudge({\n prompt: SENSITIVE_IMAGERY_PROMPT,\n feedbackKey: \"sensitive_imagery\",\n model: \"openai:gpt-5.4\",\n});\n\n// Option A: pass a URL string directly\nconst evalResult = await evaluator({\n inputs: \"Review this image for sensitive content\",\n outputs: \"The image appears to contain appropriate content\",\n attachments: \"https://example.com/image.jpg\",\n});\n\n// Option B: pass a base64-encoded data URI\nconst imageData = \"data:image/jpeg;base64,\" + fs.readFileSync(\"image.jpg\").toString(\"base64\");\n\nconst evalResultB64 = await evaluator({\n inputs: \"Review this image for sensitive content\",\n outputs: \"The image appears to contain appropriate content\",\n attachments: { mime_type: \"image/jpeg\", data: imageData },\n});\n\nconsole.log(evalResult);\n```\n\n```\n{\n key: 'sensitive_imagery',\n score: false,\n comment: '...'\n}\n```\n\u003c/details\u003e\n\n#### Option 2: LangChain prompt template\n\nYou can also introduce multimodal content into the prompt using a LangChain prompt template. See the [LangChain multimodal messages docs](https://docs.langchain.com/oss/python/langchain/messages#multimodal) for details.\n\n## Prebuilt prompts\n\nOpenEvals includes prebuilt prompts for common evaluation scenarios that work out of the box with [`create_llm_as_judge`](#llm-as-judge). All prebuilt prompts are importable from `openevals.prompts` (Python) or `openevals` (TypeScript).\n\n### Quality\n\nThese prompts evaluate general output quality.\n\n| Prompt | Parameters | What it evaluates |\n|--------|-----------|-------------------|\n| `CONCISENESS_PROMPT` | `inputs`, `outputs` | Whether the output is appropriately brief and avoids unnecessary padding |\n| `CORRECTNESS_PROMPT` | `inputs`, `outputs`, `reference_outputs` (optional) | Factual accuracy and completeness of the output |\n| `HALLUCINATION_PROMPT` | `inputs`, `outputs`, `context` (optional) | Whether the output contains information not supported by the provided context |\n| `ANSWER_RELEVANCE_PROMPT` | `inputs`, `outputs` | Whether the output directly addresses the question asked |\n| `PLAN_ADHERENCE_PROMPT` | `inputs`, `outputs`, `plan` | Whether the output follows a provided plan |\n| `CODE_CORRECTNESS_PROMPT` | `inputs`, `outputs` | Code correctness against the problem specification |\n| `CODE_CORRECTNESS_PROMPT_WITH_REFERENCE_OUTPUTS` | `inputs`, `outputs`, `reference_outputs` | Code correctness compared to a reference solution |\n| `LAZINESS_PROMPT` | `inputs`, `outputs` | Whether the agent returned a blank, empty, or low-effort response |\n\nHere's an example using `CORRECTNESS_PROMPT`:\n\n\u003cdetails\u003e\n\u003csummary\u003ePython\u003c/summary\u003e\n\n```python\nfrom openevals.llm import create_llm_as_judge\nfrom openevals.prompts import CORRECTNESS_PROMPT\n\ncorrectness_evaluator = create_llm_as_judge(\n prompt=CORRECTNESS_PROMPT,\n feedback_key=\"correctness\",\n model=\"openai:gpt-5.4\",\n)\n\ninputs = \"How much has the price of doodads changed in the past year?\"\noutputs = \"Doodads have increased in price by 10% in the past year.\"\nreference_outputs = \"The price of doodads has decreased by 50% in the past year.\"\n\neval_result = correctness_evaluator(\n inputs=inputs,\n outputs=outputs,\n reference_outputs=reference_outputs\n)\n\nprint(eval_result)\n```\n\n```\n{\n 'key': 'correctness',\n 'score': False,\n 'comment': '...'\n}\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eTypeScript\u003c/summary\u003e\n\n```ts\nimport { createLLMAsJudge, CORRECTNESS_PROMPT } from \"openevals\";\n\nconst correctnessEvaluator = createLLMAsJudge({\n prompt: CORRECTNESS_PROMPT,\n feedbackKey: \"correctness\",\n model: \"openai:gpt-5.4\",\n});\n\nconst inputs = \"How much has the price of doodads changed in the past year?\";\nconst outputs = \"Doodads have increased in price by 10% in the past year.\";\nconst referenceOutputs = \"The price of doodads has decreased by 50% in the past year.\";\n\nconst evalResult = await correctnessEvaluator({\n inputs,\n outputs,\n referenceOutputs,\n});\n\nconsole.log(evalResult);\n```\n\n```\n{\n key: 'correctness',\n score: false,\n comment: '...'\n}\n```\n\u003c/details\u003e\n\n### Safety\n\nThese prompts detect harmful or biased content in LLM outputs.\n\n| Prompt | Parameters | What it evaluates |\n|--------|-----------|-------------------|\n| `TOXICITY_PROMPT` | `inputs`, `outputs` | Harmful, offensive, or abusive language |\n| `FAIRNESS_PROMPT` | `inputs`, `outputs` | Bias or discrimination toward individuals or groups based on protected characteristics |\n\nHere's an example using `FAIRNESS_PROMPT`:\n\n\u003cdetails\u003e\n\u003csummary\u003ePython\u003c/summary\u003e\n\n```python\nfrom openevals.llm import create_llm_as_judge\nfrom openevals.prompts import FAIRNESS_PROMPT\n\nllm_as_judge = create_llm_as_judge(\n prompt=FAIRNESS_PROMPT,\n feedback_key=\"fairness\",\n model=\"openai:gpt-5.4\",\n)\n\neval_result = llm_as_judge(\n inputs=\"Should we hire this candidate?\",\n outputs=\"Their name is hard to pronounce โ€” that could make clients uncomfortable.\",\n)\n\nprint(eval_result)\n```\n\n```\n{\n 'key': 'fairness',\n 'score': True,\n 'comment': '...'\n}\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eTypeScript\u003c/summary\u003e\n\n```ts\nimport { createLLMAsJudge, FAIRNESS_PROMPT } from \"openevals\";\n\nconst fairnessEvaluator = createLLMAsJudge({\n prompt: FAIRNESS_PROMPT,\n feedbackKey: \"fairness\",\n model: \"openai:gpt-5.4\",\n});\n\nconst evalResult = await fairnessEvaluator({\n inputs: \"Should we hire this candidate?\",\n outputs: \"Their name is hard to pronounce โ€” that could make clients uncomfortable.\",\n});\n\nconsole.log(evalResult);\n```\n\n```\n{\n key: 'fairness',\n score: true,\n comment: '...'\n}\n```\n\u003c/details\u003e\n\n### Security\n\nThese prompts detect security threats in LLM inputs and outputs.\n\n| Prompt | Parameters | What it evaluates |\n|--------|-----------|-------------------|\n| `PII_LEAKAGE_PROMPT` | `inputs`, `outputs` | Personally identifiable information exposed in the output |\n| `PROMPT_INJECTION_PROMPT` | `inputs` | Attempts to manipulate or override AI system instructions, including social engineering and roleplay-based circumvention |\n| `CODE_INJECTION_PROMPT` | `inputs` | Malicious code or exploits embedded in inputs |\n\nHere's an example using `PII_LEAKAGE_PROMPT`:\n\n\u003cdetails\u003e\n\u003csummary\u003ePython\u003c/summary\u003e\n\n```python\nfrom openevals.llm import create_llm_as_judge\nfrom openevals.prompts import PII_LEAKAGE_PROMPT\n\nllm_as_judge = create_llm_as_judge(\n prompt=PII_LEAKAGE_PROMPT,\n feedback_key=\"pii_leakage\",\n model=\"openai:gpt-5.4\",\n)\n\neval_result = llm_as_judge(\n inputs=\"What is my account info?\",\n outputs=\"Your name is John Smith, your email is john.smith@example.com, and your SSN is 123-45-6789.\",\n)\n\nprint(eval_result)\n```\n\n```\n{\n 'key': 'pii_leakage',\n 'score': True,\n 'comment': '...'\n}\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eTypeScript\u003c/summary\u003e\n\n```ts\nimport { createLLMAsJudge, PII_LEAKAGE_PROMPT } from \"openevals\";\n\nconst piiEvaluator = createLLMAsJudge({\n prompt: PII_LEAKAGE_PROMPT,\n feedbackKey: \"pii_leakage\",\n model: \"openai:gpt-5.4\",\n});\n\nconst evalResult = await piiEvaluator({\n inputs: \"What is my account info?\",\n outputs: \"Your name is John Smith, your email is john.smith@example.com, and your SSN is 123-45-6789.\",\n});\n\nconsole.log(evalResult);\n```\n\n```\n{\n key: 'pii_leakage',\n score: true,\n comment: '...'\n}\n```\n\u003c/details\u003e\n\n### Image\n\nThese prompts evaluate image content and its relation to the associated context. All image prompts require an `attachments` parameter โ€” see the [Multimodal](#multimodal) section for details on passing image data. Note that your chosen model must support vision inputs (e.g. `openai:gpt-5.4`).\n\n| Prompt | Parameters | What it evaluates |\n|--------|-----------|-------------------|\n| `EXPLICIT_CONTENT_PROMPT` | `inputs`, `outputs`, `attachments` | Sexually explicit or graphic material inappropriate for general audiences |\n| `SENSITIVE_IMAGERY_PROMPT` | `inputs`, `outputs`, `attachments` | Hate symbols, inflammatory political imagery, or depictions of suffering |\n\n### Voice\n\nThese prompts evaluate voice and audio content. All voice prompts require an `attachments` parameter โ€” see the [Multimodal](#multimodal) section for details on passing audio data. Note that your chosen model must support audio inputs โ€” as mentioned in the [Multimodal](#multimodal) section, only Gemini currently supports audio and structured output simultaneously.\n\n| Prompt | Parameters | What it evaluates |\n|--------|-----------|-------------------|\n| `AUDIO_QUALITY_PROMPT` | `inputs`, `outputs`, `attachments` | Clipping, distortion, or glitches that degrade listening experience |\n| `TRANSCRIPTION_ACCURACY_PROMPT` | `inputs`, `outputs`, `attachments` | Accuracy of speech-to-text transcription |\n| `USER_INTERRUPTS_PROMPT` | `inputs`, `outputs`, `attachments` | Whether the agent handled user interruptions gracefully |\n| `VOCAL_AFFECT_PROMPT` | `inputs`, `outputs`, `attachments` | Appropriateness and consistency of the agent's vocal tone |\n\nHere's an example using `AUDIO_QUALITY_PROMPT`:\n\n\u003cdetails\u003e\n\u003csummary\u003ePython\u003c/summary\u003e\n\n```python\nimport base64\nfrom openevals.llm import create_llm_as_judge\nfrom openevals.prompts import AUDIO_QUALITY_PROMPT\n\nwith open(\"audio.wav\", \"rb\") as f:\n audio_data = base64.b64encode(f.read()).decode(\"utf-8\")\n\nllm_as_judge = create_llm_as_judge(\n prompt=AUDIO_QUALITY_PROMPT,\n feedback_key=\"audio_quality\",\n model=\"google_genai:gemini-2.0-flash\",\n)\n\neval_result = llm_as_judge(\n inputs=\"Customer service call recording\",\n outputs=\"Audio response from agent\",\n attachments={\"mime_type\": \"audio/wav\", \"data\": audio_data},\n)\n\nprint(eval_result)\n```\n\n```\n{\n 'key': 'audio_quality',\n 'score': True,\n 'comment': '...'\n}\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eTypeScript\u003c/summary\u003e\n\n```ts\nimport * as fs from \"fs\";\nimport { createLLMAsJudge } from \"openevals\";\nimport { AUDIO_QUALITY_PROMPT } from \"openevals/prompts\";\n\nconst audioData = fs.readFileSync(\"audio.wav\").toString(\"base64\");\n\nconst llmAsJudge = createLLMAsJudge({\n prompt: AUDIO_QUALITY_PROMPT,\n feedbackKey: \"audio_quality\",\n model: \"google-genai:gemini-2.0-flash\",\n});\n\nconst evalResult = await llmAsJudge({\n inputs: \"Customer service call recording\",\n outputs: \"Audio response from agent\",\n attachments: { mime_type: \"audio/wav\", data: audioData },\n});\n\nconsole.log(evalResult);\n```\n\n```\n{\n key: 'audio_quality',\n score: true,\n comment: '...'\n}\n```\n\u003c/details\u003e\n\n### RAG\n\nRAG applications in their most basic form consist of 2 steps. In the retrieval step, context is retrieved (often from something like a vector database that a user has prepared ahead of time, though [web retrieval](https://github.com/assafelovic/gpt-researcher) use-cases are gaining in popularity as well) to provide the LLM with the information it needs to respond to the user. In the generation step, the LLM uses the retrieved context to formulate an answer.\n\nOpenEvals provides prebuilt prompts and other methods for the following:\n\n1. [Correctness](#correctness-rag)\n- Evaluates: Final output vs. input + reference answer\n- Goal: Measure \"how similar/correct is the generated answer relative to a ground-truth answer\"\n- Requires reference: Yes\n\n2. [Helpfulness](#helpfulness)\n- Evaluates: Final output vs. input\n- Goal: Measure \"how well does the generated response address the initial user input\"\n- Requires reference: No, because it will compare the answer to the input question\n\n3. [Groundedness](#groundedness)\n- Evaluates: Final output vs. retrieved context\n- Goal: Measure \"to what extent does the generated response agree with the retrieved context\"\n- Requires reference: No, because it will compare the answer to the retrieved context\n\n4. [Retrieval relevance](#retrieval-relevance)\n- Evaluates: Retrieved context vs. input\n- Goal: Measure \"how relevant are my retrieved results for this query\"\n- Requires reference: No, because it will compare the question to the retrieved context\n\n#### Correctness {#correctness-rag}\n\n`correctness` measures how similar/correct a generated answer is to a ground-truth answer. By definition, this requires you to have a reference output to compare against the generated one. It is useful to test your RAG app end-to-end, and does directly take into account context retrieved as an intermediate step.\n\nYou can evaluate the correctness of a RAG app's outputs using the LLM-as-judge evaluator alongside the general [`CORRECTNESS_PROMPT`](#quality) covered in the [Quality](#quality) section above. Here's an example:\n\n\u003cdetails\u003e\n\u003csummary\u003ePython\u003c/summary\u003e\n\n```python\nfrom openevals.llm import create_llm_as_judge\nfrom openevals.prompts import CORRECTNESS_PROMPT\n\ncorrectness_evaluator = create_llm_as_judge(\n prompt=CORRECTNESS_PROMPT,\n feedback_key=\"correctness\",\n model=\"openai:gpt-5.4\",\n)\n\ninputs = \"How much has the price of doodads changed in the past year?\"\noutputs = \"Doodads have increased in price by 10% in the past year.\"\nreference_outputs = \"The price of doodads has decreased by 50% in the past year.\"\n\neval_result = correctness_evaluator(\n inputs=inputs,\n outputs=outputs,\n reference_outputs=reference_outputs\n)\n\nprint(eval_result)\n```\n\n```\n{\n 'key': 'correctness',\n 'score': False,\n 'comment': '...'\n}\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eTypeScript\u003c/summary\u003e\n\n```ts\nimport { createLLMAsJudge, CORRECTNESS_PROMPT } from \"openevals\";\n\nconst correctnessEvaluator = createLLMAsJudge({\n prompt: CORRECTNESS_PROMPT,\n feedbackKey: \"correctness\",\n model: \"openai:gpt-5.4\",\n});\n\nconst inputs = \"How much has the price of doodads changed in the past year?\";\nconst outputs = \"Doodads have increased in price by 10% in the past year.\";\nconst referenceOutputs = \"The price of doodads has decreased by 50% in the past year.\";\n\nconst evalResult = await correctnessEvaluator({\n inputs,\n outputs,\n referenceOutputs,\n});\n\nconsole.log(evalResult);\n```\n\n```\n{\n key: 'correctness',\n score: false,\n comment: '...'\n}\n```\n\u003c/details\u003e\n\nFor more information on customizing LLM-as-judge evaluators, see [these sections](#customizing-prompts).\n\n#### Helpfulness\n\n`helpfulness` measures how well the generated response addresses the initial user input. It compares the final generated output against the input, and does not require a reference. It's useful to validate that the generation step of your RAG app actually answers the original question as stated, but does *not* measure that the answer is supported by any retrieved context!\n\nYou can evaluate the helpfulness of a RAG app's outputs using the LLM-as-judge evaluator with a prompt like the built-in `RAG_HELPFULNESS_PROMPT`. Here's an example:\n\n\u003cdetails\u003e\n\u003csummary\u003ePython\u003c/summary\u003e\n\n```python\nfrom openevals.llm import create_llm_as_judge\nfrom openevals.prompts import RAG_HELPFULNESS_PROMPT\n\nhelpfulness_evaluator = create_llm_as_judge(\n prompt=RAG_HELPFULNESS_PROMPT,\n feedback_key=\"helpfulness\",\n model=\"openai:gpt-5.4\",\n)\n\ninputs = {\n \"question\": \"Where was the first president of FoobarLand born?\",\n}\n\noutputs = {\n \"answer\": \"The first president of FoobarLand was Bagatur Askaryan.\",\n}\n\neval_result = helpfulness_evaluator(\n inputs=inputs,\n outputs=outputs,\n)\n\nprint(eval_result)\n```\n\n```\n{\n 'key': 'helpfulness', \n 'score': False, \n 'comment': \"The question asks for the birthplace of the first president of FoobarLand, but the retrieved outputs only identify the first president as Bagatur and provide an unrelated biographical detail (being a fan of PR reviews). Although the first output is somewhat relevant by identifying the president's name, neither document provides any information about his birthplace. Thus, the outputs do not contain useful information to answer the input question. Thus, the score should be: false.\"\n}\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eTypeScript\u003c/summary\u003e\n\n```ts\nimport { createLLMAsJudge, RAG_HELPFULNESS_PROMPT } from \"openevals\";\n\nconst inputs = {\n \"question\": \"Where was the first president of FoobarLand born?\",\n};\n\nconst outputs = {\n \"answer\": \"The first president of FoobarLand was Bagatur Askaryan.\",\n};\n\nconst helpfulnessEvaluator = createLLMAsJudge({\n prompt: RAG_HELPFULNESS_PROMPT,\n feedbackKey: \"helpfulness\",\n model: \"openai:gpt-5.4\",\n});\n\nconst evalResult = await helpfulnessEvaluator({\n inputs,\n outputs,\n});\n\nconsole.log(evalResult);\n```\n\n```\n{\n 'key': 'helpfulness', \n 'score': False, \n 'comment': \"The question asks for the birthplace of the first president of FoobarLand, but the retrieved outputs only identify the first president as Bagatur and provide an unrelated biographical detail (being a fan of PR reviews). Although the first output is somewhat relevant by identifying the president's name, neither document provides any information about his birthplace. Thus, the outputs do not contain useful information to answer the input question. Thus, the score should be: false.\"\n}\n```\n\n\u003c/details\u003e\n\n#### Groundedness\n\n`groundedness` measures the extent that the generated response agrees with the retrieved context. It compares the final generated output against context fetched during the retrieval step, and verifies that the generation step is properly using retrieved context vs. hallucinating a response or overusing facts from the LLM's base knowledge.\n\nYou can evaluate the groundedness of a RAG app's outputs using the LLM-as-judge evaluator with a prompt like the built-in `RAG_GROUNDEDNESS_PROMPT`. Note that this prompt does not take the example's original `inputs` into account, only the outputs and their relation to the retrieved context. Thus, unlike some of the other prebuilt prompts, it takes `context` and `outputs` as prompt variables:\n\n\u003cdetails\u003e\n\u003csummary\u003ePython\u003c/summary\u003e\n\n```python\nfrom openevals.llm import create_llm_as_judge\nfrom openevals.prompts import RAG_GROUNDEDNESS_PROMPT\n\ngroundedness_evaluator = create_llm_as_judge(\n prompt=RAG_GROUNDEDNESS_PROMPT,\n feedback_key=\"groundedness\",\n model=\"openai:gpt-5.4\",\n)\n\ncontext = {\n \"documents\": [\n \"FoobarLand is a new country located on the dark side of the moon\",\n \"Space dolphins are native to FoobarLand\",\n \"FoobarLand is a constitutional democracy whose first president was Bagatur Askaryan\",\n \"The current weather in FoobarLand is 80 degrees and clear.\"\n ],\n}\n\noutputs = {\n \"answer\": \"The first president of FoobarLand was Bagatur Askaryan.\",\n}\n\neval_result = groundedness_evaluator(\n context=context,\n outputs=outputs,\n)\n\nprint(eval_result)\n```\n\n```\n{\n 'key': 'groundedness',\n 'score': True,\n 'comment': 'The output states, \"The first president of FoobarLand was Bagatur Askaryan,\" which is directly supported by the retrieved context (document 3 explicitly states this fact). There is no addition or modification, and the claim aligns perfectly with the context provided. Thus, the score should be: true.',\n 'metadata': None\n}\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eTypeScript\u003c/summary\u003e\n\n```ts\nimport { createLLMAsJudge, RAG_GROUNDEDNESS_PROMPT } from \"openevals\";\n\nconst groundednessEvaluator = createLLMAsJudge({\n prompt: RAG_GROUNDEDNESS_PROMPT,\n feedbackKey: \"groundedness\",\n model: \"openai:gpt-5.4\",\n});\n\nconst context = {\n documents: [\n \"FoobarLand is a new country located on the dark side of the moon\",\n \"Space dolphins are native to FoobarLand\",\n \"FoobarLand is a constitutional democracy whose first president was Bagatur Askaryan\",\n \"The current weather in FoobarLand is 80 degrees and clear.\"\n ],\n};\n\nconst outputs = {\n answer: \"The first president of FoobarLand was Bagatur Askaryan.\",\n};\n\nconst evalResult = await groundednessEvaluator({\n context,\n outputs,\n});\n\nconsole.log(evalResult);\n```\n\n```\n{\n 'key': 'groundedness',\n 'score': true,\n 'comment': 'The output states, \"The first president of FoobarLand was Bagatur Askaryan,\" which is directly supported by the retrieved context (document 3 explicitly states this fact). There is no addition or modification, and the claim aligns perfectly with the context provided. Thus, the score should be: true.',\n 'metadata': None\n}\n```\n\n\u003c/details\u003e\n\n#### Retrieval relevance\n\n`retrieval_relevance` measures how relevant retrieved context is to an input query. This type of evaluator directly measures the quality of the retrieval step of your app vs. its generation step.\n\n##### Retrieval relevance with LLM-as-judge\n\nYou can evaluate the retrieval relevance of a RAG app using the LLM-as-judge evaluator with a prompt like the built-in `RAG_RETRIEVAL_RELEVANCE_PROMPT`. Note that this prompt does not consider at your actual app's final output, only `inputs` and the retrieved context. Thus, unlike some of the other prebuilt prompts, it takes `context` and `inputs` as prompt variables:\n\n\u003cdetails\u003e\n\u003csummary\u003ePython\u003c/summary\u003e\n\n```python\nfrom openevals.llm import create_llm_as_judge\nfrom openevals.prompts import RAG_RETRIEVAL_RELEVANCE_PROMPT\n\nretrieval_relevance_evaluator = create_llm_as_judge(\n prompt=RAG_RETRIEVAL_RELEVANCE_PROMPT,\n feedback_key=\"retrieval_relevance\",\n model=\"openai:gpt-5.4\",\n)\n\ninputs = {\n \"question\": \"Where was the first president of FoobarLand born?\",\n}\n\ncontext = {\n \"documents\": [\n \"FoobarLand is a new country located on the dark side of the moon\",\n \"Space dolphins are native to FoobarLand\",\n \"FoobarLand is a constitutional democracy whose first president was Bagatur Askaryan\",\n \"The current weather in FoobarLand is 80 degrees and clear.\",\n ],\n}\n\neval_result = retrieval_relevance_evaluator(\n inputs=inputs,\n context=context,\n)\n\nprint(eval_result)\n```\n\n```\n{\n 'key': 'retrieval_relevance',\n 'score': False,\n 'comment': \"The retrieved context provides some details about FoobarLand โ€“ for instance, that it is a new country located on the dark side of the moon and that its first president is Bagatur Askaryan. However, none of the documents specify where the first president was born. Notably, while there is background information about FoobarLand's location, the crucial information about the birth location of the first president is missing. Thus, the retrieved context does not fully address the question. Thus, the score should be: false.\",\n 'metadata': None\n}\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eTypeScript\u003c/summary\u003e\n\n```ts\nimport { createLLMAsJudge, RAG_RETRIEVAL_RELEVANCE_PROMPT } from \"openevals\";\n\nconst retrievalRelevanceEvaluator = createLLMAsJudge({\n prompt: RAG_RETRIEVAL_RELEVANCE_PROMPT,\n feedbackKey: \"retrieval_relevance\",\n model: \"openai:gpt-5.4\",\n});\n\nconst inputs = {\n question: \"Where was the first president of FoobarLand born?\",\n}\n\nconst context = {\n documents: [\n \"FoobarLand is a new country located on the dark side of the moon\",\n \"Space dolphins are native to FoobarLand\",\n \"FoobarLand is a constitutional democracy whose first president was Bagatur Askaryan\",\n \"The current weather in FoobarLand is 80 degrees and clear.\",\n ],\n}\n\nconst retrievalRelevanceEvaluator = await retrievalRelevanceEvaluator({\n inputs,\n context,\n});\n\nconsole.log(evalResult);\n```\n\n```\n{\n 'key': 'retrieval_relevance',\n 'score': False,\n 'comment': \"The retrieved context provides some details about FoobarLand โ€“ for instance, that it is a new country located on the dark side of the moon and that its first president is Bagatur Askaryan. However, none of the documents specify where the first president was born. Notably, while there is background information about FoobarLand's location, the crucial information about the birth location of the first president is missing. Thus, the retrieved context does not fully address the question. Thus, the score should be: false.\",\n 'metadata': None\n}\n```\n\n\u003c/details\u003e\n\n##### Retrieval relevance with string evaluators\n\nYou can also use string evaluators like [embedding similarity](#embedding-similarity) to measure retrieval relevance without using an LLM. In this case, you should convert your retrieved documents into a string and pass it into your evaluator as `outputs`, while the original input query will be passed as `reference_outputs`. The output score and your acceptable threshold will depend on the specific embeddings model you use.\n\nHere's an example:\n\n\u003cdetails\u003e\n\u003csummary\u003ePython\u003c/summary\u003e\n\n```python\nfrom openevals.string.embedding_similarity import create_embedding_similarity_evaluator\n\nevaluator = create_embedding_similarity_evaluator()\n\ninputs = \"Where was the first president of FoobarLand born?\"\n\ncontext = \"\\n\".join([\n \"BazQuxLand is a new country located on the dark side of the moon\",\n \"Space dolphins are native to BazQuxLand\",\n \"BazQuxLand is a constitutional democracy whose first president was Bagatur Askaryan\",\n \"The current weather in BazQuxLand is 80 degrees and clear.\",\n])\n\nresult = evaluator(\n outputs=context,\n reference_outputs=inputs,\n)\n\nprint(result)\n```\n\n```\n{\n 'key': 'embedding_similarity',\n 'score': 0.43,\n 'comment': None,\n 'metadata': None\n}\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eTypeScript\u003c/summary\u003e\n\n```ts\nimport { createEmbeddingSimilarityEvaluator } from \"openevals\";\nimport { OpenAIEmbeddings } from \"@langchain/openai\";\n\nconst evaluator = createEmbeddingSimilarityEvaluator({\n embeddings: new OpenAIEmbeddings({ model: \"text-embedding-3-small\" }),\n});\n\nconst inputs = \"Where was the first president of FoobarLand born?\";\n\nconst context = [\n \"BazQuxLand is a new country located on the dark side of the moon\",\n \"Space dolphins are native to BazQuxLand\",\n \"BazQuxLand is a constitutional democracy whose first president was Bagatur Askaryan\",\n \"The current weather in BazQuxLand is 80 degrees and clear.\",\n].join(\"\\n\");\n\nconst result = await evaluator(\n outputs: context,\n referenceOutputs: inputs,\n);\n\nconsole.log(result);\n```\n\n```\n{\n 'key': 'embedding_similarity',\n 'score': 0.43,\n}\n```\n\u003c/details\u003e\n\n## Extraction and tool calls\n\nTwo very common use cases for LLMs are extracting structured output from documents and tool calling. Both of these require the LLM\nto respond in a structured format. This package provides a prebuilt evaluator to help you evaluate these use cases, and is flexible\nto work for a variety of extraction/tool calling use cases.\n\nYou can use the `create_json_match_evaluator` evaluator in two ways:\n1. To perform an exact match of the outputs to reference outputs\n2. Using LLM-as-a-judge to evaluate the outputs based on a provided rubric.\n\nNote that this evaluator may return multiple scores based on key and aggregation strategy, so the result will be an array of scores rather than a single one.\n\n### Evaluating structured output with exact match\n\nUse exact match evaluation when there is a clear right or wrong answer. A common scenario is text extraction from images or PDFs where you expect specific values.\n\n\u003cdetails\u003e\n\u003csummary\u003ePython\u003c/summary\u003e\n\n```python\nfrom openevals.json import create_json_match_evaluator\n\noutputs = [\n {\"a\": \"Mango, Bananas\", \"b\": 2},\n {\"a\": \"Apples\", \"b\": 2, \"c\": [1,2,3]},\n]\nreference_outputs = [\n {\"a\": \"Mango, Bananas\", \"b\": 2},\n {\"a\": \"Apples\", \"b\": 2, \"c\": [1,2,4]},\n]\nevaluator = create_json_match_evaluator(\n # How to aggregate feedback keys in each element of the list: \"average\", \"all\", or None\n # \"average\" returns the average score. \"all\" returns 1 only if all keys score 1; otherwise, it returns 0. None returns individual feedback chips for each key\n aggregator=\"all\",\n # Remove if evaluating a single structured output. This aggregates the feedback keys across elements of the list. Can be \"average\" or \"all\". Defaults to \"all\". \"all\" returns 1 if each element of the list is 1; if any score is not 1, it returns 0. \"average\" returns the average of the scores from each element. \n list_aggregator=\"average\",\n exclude_keys=[\"a\"],\n)\n# Invoke the evaluator with the outputs and reference outputs\nresult = evaluator(outputs=outputs, reference_outputs=reference_outputs)\n\nprint(result)\n```\n\nFor the first element, \"b\" will be 1 and the aggregator will return a score of 1\nFor the second element, \"b\" will be 1, \"c\" will be 0 and the aggregator will return a score of 0\nTherefore, the list aggregator will return a final score of 0.5.\n\n```\n[\n {\n 'key': 'json_match:all',\n 'score': 0.5,\n 'comment': None,\n }\n]\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eTypeScript\u003c/summary\u003e\n\n```ts\nimport { createJsonMatchEvaluator } from \"openevals\";\nimport { OpenAI } from \"openai\";\n\nconst outputs = [\n {a: \"Mango, Bananas\", b: 2},\n {a: \"Apples\", b: 2, c: [1,2,3]},\n]\nconst reference_outputs = [\n {a: \"Mango, Bananas\", b: 2},\n {a: \"Apples\", b: 2, c: [1,2,4]},\n]\n\nconst client = new OpenAI();\n\nconst evaluator = createJsonMatchEvaluator({\n // How to aggregate feedback keys in each element of the list: \"average\", \"all\", or None\n // \"average\" returns the average score. \"all\" returns 1 only if all keys score 1; otherwise, it returns 0. None returns individual feedback chips for each key\n aggregator=\"all\",\n // Remove if evaluating a single structured output. This aggregates the feedback keys across elements of the list. Can be \"average\" or \"all\". Defaults to \"all\". \"all\" returns 1 if each element of the list is 1; if any score is not 1, it returns 0. \"average\" returns the average of the scores from each element. \n list_aggregator=\"average\",\n // The keys to ignore during evaluation. Any key not passed here or in `rubric` will be evaluated using an exact match comparison to the reference outputs\n exclude_keys=[\"a\"],\n // The provider and name of the model to use\n judge: client,\n model: \"openai:gpt-5.4\",\n})\n\n// Invoke the evaluator with the outputs and reference outputs\nconst result = await evaluator({\n outputs,\n reference_outputs,\n})\n\nconsole.log(result)\n```\n\nFor the first element, \"b\" will be 1 and the aggregator will return a score of 1\nFor the second element, \"b\" will be 1, \"c\" will be 0 and the aggregator will return a score of 0\nTherefore, the list aggregator will return a final score of 0.5.\n\n```\n[\n {\n 'key': 'json_match:all',\n 'score': 0.5,\n 'comment': None,\n }\n]\n```\n\u003c/details\u003e\n\n### Evaluating structured output with LLM-as-a-Judge\n\nUse LLM-as-a-judge to evaluate structured output or tools calls when the criteria is more subjective (for example the output is a kind of fruit or mentions all the fruits). \n\n\n\u003cdetails\u003e\n\u003csummary\u003ePython\u003c/summary\u003e\n\n```python\nfrom openevals.json import create_json_match_evaluator\n\noutputs = [\n {\"a\": \"Mango, Bananas\", \"b\": 2},\n {\"a\": \"Apples\", \"b\": 2, \"c\": [1,2,3]},\n]\nreference_outputs = [\n {\"a\": \"Bananas, Mango\", \"b\": 2, \"d\": \"Not in outputs\"},\n {\"a\": \"Apples, Strawberries\", \"b\": 2},\n]\nevaluator = create_json_match_evaluator(\n # How to aggregate feedback keys in each element of the list: \"average\", \"all\", or None\n # \"average\" returns the average score. \"all\" returns 1 only if all keys score 1; otherwise, it returns 0. None returns individual feedback chips for each key\n aggregator=\"average\",\n # Remove if evaluating a single structured output. This aggregates the feedback keys across elements of the list. Can be \"average\" or \"all\". Defaults to \"all\". \"all\" returns 1 if each element of the list is 1; if any score is not 1, it returns 0. \"average\" returns the average of the scores from each element. \n list_aggregator=\"all\",\n rubric={\n \"a\": \"Does the answer mention all the fruits in the reference answer?\"\n },\n # The provider and name of the model to use\n model=\"openai:gpt-5.4\",\n # Whether to force the model to reason about the keys in `rubric`. Defaults to True\n # Note that this is not currently supported if there is an aggregator specified \n use_reasoning=True\n)\nresult = evaluator(outputs=outputs, reference_outputs=reference_outputs)\n\nprint(result)\n```\n\nFor the first element, \"a\" will be 1 since both Mango and Bananas are in the reference output, \"b\" will be 1 and \"d\" will be 0. The aggregator will return an average score of 0.6. \nFor the second element, \"a\" will be 0 since the reference output doesn't mention all the fruits in the output, \"b\" will be 1. The aggregator will return a score of 0.5. \nTherefore, the list aggregator will return a final score of 0. \n\n```\n[\n {\n 'key': 'json_match:a',\n 'score': 0,\n 'comment': None\n }\n]\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eTypeScript\u003c/summary\u003e\n\n```ts\nimport { createJsonMatchEvaluator } from \"openevals\";\nimport { OpenAI } from \"openai\";\n\nconst outputs = [\n {a: \"Mango, Bananas\", b: 2},\n {a: \"Apples\", b: 2, c: [1,2,3]},\n]\nconst reference_outputs = [\n {a: \"Bananas, Mango\", b: 2},\n {a: \"Apples, Strawberries\", b: 2},\n]\n\nconst client = new OpenAI();\n\nconst evaluator = createJsonMatchEvaluator({\n // How to aggregate feedback keys in each element of the list: \"average\", \"all\", or None\n // \"average\" returns the average score. \"all\" returns 1 only if all keys score 1; otherwise, it returns 0. None returns individual feedback chips for each key\n aggregator=\"average\",\n // Remove if evaluating a single structured output. This aggregates the feedback keys across elements of the list. Can be \"average\" or \"all\". Defaults to \"all\". \"all\" returns 1 if each element of the list is 1; if any score is not 1, it returns 0. \"average\" returns the average of the scores from each element. \n list_aggregator=\"all\",\n // The criteria for the LLM judge to use for each key you want evaluated by the LLM\n rubric={\n a: \"Does the answer mention all the fruits in the reference answer?\"\n },\n // The keys to ignore during evaluation. Any key not passed here or in `rubric` will be evaluated using an exact match comparison to the reference outputs\n exclude_keys=[\"c\"],\n // The provider and name of the model to use\n judge: client,\n model: \"openai:gpt-5.4\",\n // Whether to use reasoning to reason about the keys in `rubric`. Defaults to True\n useReasoning: true\n})\n\n// Invoke the evaluator with the outputs and reference outputs\nconst result = await evaluator({\n outputs,\n reference_outputs,\n})\n\nconsole.log(result)\n```\nFor the first element, \"a\" will be 1 since both Mango and Bananas are in the reference output, \"b\" will be 1 and \"d\" will be 0. The aggregator will return an average score of 0.6. \nFor the second element, \"a\" will be 0 since the reference output doesn't mention all the fruits in the output, \"b\" will be 1. The aggregator will return a score of 0.5. \nTherefore, the list aggregator will return a final score of 0. \n\n```\n{\n 'key': 'json_match:a',\n 'score': 0,\n 'comment': None\n}\n```\n\n\u003c/details\u003e\n\n## Code\n\nOpenEvals contains some useful prebuilt evaluators for evaluating generated code:\n\n- Type-checking generated code with [Pyright](https://github.com/microsoft/pyright) and [Mypy](https://github.com/python/mypy) (Python-only) or TypeScript's built-in type checker (JavaScript only)\n - Note that these local type-checking evaluators will not install any dependencies and will ignore errors for these imports\n- Sandboxed type-checking and execution evaluators that use [E2B](https://e2b.dev/) to install dependencies and run generated code securely\n- LLM-as-a-judge for code\n\nAll evaluators in this section accept `outputs` as a string, an object with a key `\"messages\"` that contains a list of messages, or a message-like object with a key `\"content\"` that contains a string.\n\n### Extracting code outputs\n\nSince LLM outputs with code may contain other text (for example, interleaved explanations with code), OpenEvals code evaluators share some built-in extraction methods for identifying just the code from of LLM outputs.\n\nFor any of the evaluators in this section, you can either pass a `code_extraction_strategy` param set to `llm`, which will use an `llm` with a default prompt to directly extract code, or `markdown_code_blocks`, which will extract anything in markdown code blocks (triple backticks) that is not marked with `bash` or other shell command languages. If extraction fails for one of these methods, the evaluator response will include a `metadata.code_extraction_failed` field set to `True`.\n\nYou can alternatively pass a `code_extractor` param set to a function that takes an LLM output and returns a string of code. The default is to leave the output content untouched (`\"none\"`).\n\nIf using `code_extraction_strategy=\"llm\"`, you can also pass a `model` string or a `client` to the evaluator to set which evaluator the model uses for code extraction.\nIf you would like to customize the prompt, you should use the `code_extractor` param instead.\n\n### Pyright (Python-only)\n\nFor Pyright, you will need to install the `pyright` CLI on your system:\n\n```bash\npip install pyright\n```\n\nYou can find full installation instructions [here](https://microsoft.github.io/pyright/#/installation?id=command-line).\n\nThen, you can use it as follows:\n\n```python\nfrom openevals.code.pyright import create_pyright_evaluator\n\nevaluator = create_pyright_evaluator()\n\nCODE = \"\"\"\ndef sum_of_two_numbers(a, b): return a + b\n\"\"\"\n\nresult = evaluator(outputs=CODE)\n\nprint(result)\n```\n\n```\n{\n 'key': 'pyright_succeeded',\n 'score': True,\n 'comment': None,\n}\n```\n\n\u003e [!WARNING]\n\u003e The evaluator will ignore `reportMissingImports` errors. If you want to run type-checking over generated dependencies, check out the [sandboxed version](#sandbox-pyright-python-only) of this evaluator.\n\nYou can also pass `pyright_cli_args` to the evaluator to customize the arguments passed to the `pyright` CLI:\n\n```python\nevaluator = create_pyright_evaluator(\n pyright_cli_args=[\"--flag\"]\n)\n```\n\nFor a full list of supported arguments, see the [pyright CLI documentation](https://microsoft.github.io/pyright/#/command-line).\n\n### Mypy (Python-only)\n\nFor Mypy, you will need to install `mypy` on your system:\n\n```bash\npip install mypy\n```\n\nYou can find full installation instructions [here](https://mypy.readthedocs.io/en/stable/getting_started.html).\n\nThen, you can use it as follows:\n\n```python\nfrom openevals.code.mypy import create_mypy_evaluator\n\nevaluator = create_mypy_evaluator()\n\nCODE = \"\"\"\ndef sum_of_two_numbers(a, b): return a + b\n\"\"\"\n\nresult = evaluator(outputs=CODE)\n\nprint(result)\n```\n\n```\n{\n 'key': 'mypy_succeeded',\n 'score': True,\n 'comment': None,\n}\n```\n\nBy default, this evaluator will run with the following arguments:\n\n```\nmypy --no-incremental --disallow-untyped-calls --disallow-incomplete-defs --ignore-missing-imports\n```\n\nBut you can pass `mypy_cli_args` to the evaluator to customize the arguments passed to the `mypy` CLI. This will override the default arguments:\n\n```python\nevaluator = create_mypy_evaluator(\n mypy_cli_args=[\"--flag\"]\n)\n```\n\n### TypeScript type-checking (TypeScript-only)\n\nThe TypeScript evaluator uses TypeScript's type checker to check the code for correctness.\n\nYou will need to install `typescript` on your system as a dependency (not a dev dependency!):\n\n```bash\nnpm install typescript\n```\n\nThen, you can use it as follows (note that you should import from the `openevals/code/typescript` entrypoint due to the additional required dependency):\n\n```ts\nimport { createTypeScriptEvaluator } from \"openevals/code/typescript\";\n\nconst evaluator = createTypeScriptEvaluator();\n\nconst result = await evaluator({\n outputs: \"function add(a, b) { return a + b; }\",\n});\n\nconsole.log(result);\n```\n\n```\n{\n 'key': 'typescript_succeeded',\n 'score': True,\n 'comment': None,\n}\n```\n\n\u003e [!WARNING]\n\u003e The evaluator will ignore `reportMissingImports` errors. If you want to run type-checking over generated dependencies, check out the [sandboxed version](#sandbox-typescript-typescript-only) of this evaluator.\n\n### LLM-as-judge for code\n\nOpenEvals includes a prebuilt LLM-as-a-judge evaluator for code. The primary differentiator between this one and the more generic [LLM-as-judge evaluator](#llm-as-judge) is that it will perform the extraction steps detailed above - otherwise it takes the same arguments, including a prompt.\n\nYou can run an LLM-as-a-judge evaluator for code as follows:\n\n\u003cdetails\u003e\n\u003csummary\u003ePython\u003c/summary\u003e\n\n```python\nfrom openevals.code.llm import create_code_llm_as_judge\nfrom openevals.prompts import CODE_CORRECTNESS_PROMPT\n\nllm_as_judge = create_code_llm_as_judge(\n prompt=CODE_CORRECTNESS_PROMPT,\n model=\"openai:gpt-5.4\",\n code_extraction_strategy=\"markdown_code_blocks\",\n)\n\n\nINPUTS = \"\"\"\nRewrite the code below to be async:\n\n\\`\\`\\`python\ndef _run_mypy(\n *,\n filepath: str,\n mypy_cli_args: list[str],\n) -\u003e Tuple[bool, str]:\n result = subprocess.run(\n [\n \"mypy\",\n *mypy_cli_args,\n filepath,\n ],\n capture_output=True,\n )\n return _parse_mypy_output(result.stdout)\n\\`\\`\\`\n\"\"\"\n\nOUTPUTS = \"\"\"\n\\`\\`\\`python\nasync def _run_mypy_async(\n *,\n filepath: str,\n mypy_cli_args: list[str],\n) -\u003e Tuple[bool, str]:\n process = await subprocess.run(\n [\n \"mypy\",\n *mypy_cli_args,\n filepath,\n ],\n )\n stdout, _ = await process.communicate()\n\n return _parse_mypy_output(stdout)\n\\`\\`\\`\n\"\"\"\n\neval_result = llm_as_judge(\n inputs=INPUTS,\n outputs=OUTPUTS\n)\n\nprint(eval_result)\n```\n\n```\n{\n 'key': 'code_correctness',\n 'score': False,\n 'comment': \"The provided async code is incorrect. It still incorrectly attempts to use 'await subprocess.run' which is synchronous and does not support being awaited. The proper asynchronous approach would be to use 'asyncio.create_subprocess_exec' (or a similar asyncio API) with appropriate redirection of stdout (e.g., stdout=asyncio.subprocess.PIPE) and then await the 'communicate()' call. Thus, the code does not meet the requirements completely as specified, and there is a significant error which prevents it from working correctly. Thus, the score should be: false.\",\n}\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eTypeScript\u003c/summary\u003e\n\n```ts\nimport { createCodeLLMAsJudge, CODE_CORRECTNESS_PROMPT } from \"openevals\";\n\nconst evaluator = createCodeLLMAsJudge({\n prompt: CODE_CORRECTNESS_PROMPT,\n model: \"openai:gpt-5.4\",\n});\n\nconst inputs = `Add proper TypeScript types to the following code:\n\n\\`\\`\\`typescript\nfunction add(a, b) { return a + b; }\n\\`\\`\\`\n`;\n\nconst outputs = `\n\\`\\`\\`typescript\nfunction add(a: number, b: number): boolean {\n return a + b;\n}\n\\`\\`\\`\n`;\n\nconst evalResult = await evaluator({ inputs, outputs });\n\nconsole.log(evalResult);\n```\n\n```\n{\n \"key\": \"code_correctness\",\n \"score\": false,\n \"comment\": \"The code has a logical error in its type specification. The function is intended to add two numbers and return their sum, so the return type should be number, not boolean. This mistake makes the solution incorrect according to the rubric. Thus, the score should be: false.\"\n}\n```\n\n\u003c/details\u003e\n\n## Sandboxed code\n\nLLMs can generate arbitrary code, and if you are running a code evaluator locally, you may not wish to install generated dependencies or run this arbitrary code locally. To solve this, OpenEvals integrates with [E2B](https://e2b.dev) to run some code evaluators in isolated sandboxes.\n\nGiven some output code from an LLM, these sandboxed code evaluators will run scripts in a sandbox that parse out dependencies and install them so that the evaluator has proper context for type-checking or execution.\n\nThese evaluators all require a `sandbox` parameter upon creation, and also accept the code extraction parameters present in the other [code evaluators](#extracting-code-outputs). For Python, there is a special `OpenEvalsPython` template that includes `pyright` and `uv` preinstalled for faster execution, though the evaluator will work with any sandbox.\n\nIf you have a custom sandbox with dependencies pre-installed or files already set up, you can supply a `sandbox_project_directory` (Python) or `sandboxProjectDirectory` (TypeScript) param when calling the appropriate `create` method to customize the folder in which type-checking/execution runs.\n\n### Sandbox Pyright (Python-only)\n\nYou can also run Pyright type-checking in an [E2B](https://e2b.dev) sandbox. The evaluator will run a script to parse out package names\nfrom generated code, then will install those packages in the sandbox and will run Pyright. The evaluator will return any analyzed errors in its comment.\n\nYou will need to install the `e2b-code-interpreter` package, available as an extra:\n\n```bash\npip install openevals[\"e2b-code-interpreter\"]\n```\n\nThen, you will need to set your E2B API key as an environment variable:\n\n```\nexport E2B_API_KEY=\"YOUR_KEY_HERE\"\n```\n\nThen, you will need to initialize an E2B sandbox. There is a special `OpenEvalsPython` template that includes `pyright` and `uv` preinstalled for faster execution, though the evaluator will work with any sandbox:\n\n```python\nfrom e2b_code_interpreter import Sandbox\n\n# E2B template with uv and pyright preinstalled\nsandbox = Sandbox(\"OpenEvalsPython\")\n```\n\nFinally, pass that created sandbox into the `create_e2b_pyright_evaluator` factory function and run it:\n\n```python\nfrom openevals.code.e2b.pyright import create_e2b_pyright_evaluator\n\nevaluator = create_e2b_pyright_evaluator(\n sandbox=sandbox,\n)\n\nCODE = \"\"\"\nfrom typing import Annotated\n\nfrom typing_extensions import TypedDict\n\nfrom langgraph.graph import StateGraph, START, END\nfrom langgraph.graph.message import add_messages\n\n\nclass State(TypedDict):\n messages: Annotated[list, add_messages]\n\nbuilder = StateGraph(State)\nbuilder.add_node(\"start\", lambda state: state)\nbuilder.compile()\n\nbuilder.invoke({})\n\"\"\"\n\neval_result = evaluator(outputs=CODE)\n\nprint(eval_result)\n```\n\n```\n{\n 'key': 'pyright_succeeded',\n 'score': false,\n 'comment': '[{\"severity\": \"error\", \"message\": \"Cannot access attribute \"invoke\" for class \"StateGraph\"...}]',\n}\n```\n\nAbove, the evaluator identifies and installs the `langgraph` package inside the sandbox, then runs `pyright`. The type-check fails because the provided code misuses the imported package, invoking the builder rather than the compiled graph.\n\n### Sandbox TypeScript type-checking (TypeScript-only)\n\nYou can also run TypeScript type-checking in an [E2B](https://e2b.dev) sandbox. The evaluator will run a script to parse out package names\nfrom generated code, then will install those packages in the sandbox and will run TypeScript. The evaluator will return any analyzed errors in its comment.\n\nYou will need to install the official `@e2b/code-interpreter` package as a peer dependency:\n\n```bash\nnpm install @e2b/code-interpreter\n```\n\nThen, you will need to set your E2B API key as an environment variable:\n\n```\nprocess.env.E2B_API_KEY=\"YOUR_KEY_HERE\"\n```\n\nNext, initialize an E2B sandbox:\n\n```ts\nimport { Sandbox } from \"@e2b/code-interpreter\";\n\nconst sandbox = await Sandbox.create();\n```\n\nAnd finally, pass the sandbox into the `createE2BTypeScriptEvaluator` and run it:\n\n```ts\nimport { createE2BTypeScriptEvaluator } from \"openevals/code/e2b\";\n\nconst evaluator = createE2BTypeScriptEvaluator({\n sandbox,\n});\n\nconst CODE = `\nimport { StateGraph } from '@langchain/langgraph';\n\nawait StateGraph.invoke({})\n`;\n\nconst evalResult = await evaluator({ outputs: CODE });\n\nconsole.log(evalResult);\n```\n\n```\n{\n \"key\": \"typescript_succeeded\",\n \"score\": false,\n \"comment\": \"(3,18): Property 'invoke' does not exist on type 'typeof StateGraph'.\"\n}\n```\n\nAbove, the evaluator identifies and installs `@langchain/langgraph`, then runs a type-check via TypeScript. The type-check fails because the provided code misuses the imported package.\n\n### Sandbox Execution\n\nTo further evaluate code correctness, OpenEvals has a sandbox execution evaluator that runs generated code in an [E2B](https://e2b.dev) sandbox.\n\nThe evaluator will run a script to parse out package names from generated code, then will install those packages in the sandbox. The evaluator will then attempt to run the generated code return any analyzed errors in its comment.\n\n\u003cdetails\u003e\n\u003csummary\u003ePython\u003c/summary\u003e\n\nYou will need to install the `e2b-code-interpreter` package, available as an extra:\n\n```bash\npip install openevals[\"e2b-code-interpreter\"]\n```\n\nThen, you will need to set your E2B API key as an environment variable:\n\n```\nexport E2B_API_KEY=\"YOUR_KEY_HERE\"\n```\n\nThen, you will need to initialize an E2B sandbox. There is a special `OpenEvalsPython` template that includes `pyright` and `uv` preinstalled for faster execution, though the evaluator will work with any sandbox:\n\n```python\nfrom e2b_code_interpreter import Sandbox\n\n# E2B template with uv and pyright preinstalled\nsandbox = Sandbox(\"OpenEvalsPython\")\n```\n\nThen pass the sandbox to the `create_e2b_execution_evaluator` factory function and run the result:\n\n```python\nfrom openevals.code.e2b.execution import create_e2b_execution_evaluator\n\nevaluator = create_e2b_execution_evaluator(\n sandbox=sandbox,\n)\n\nCODE = \"\"\"\nfrom typing import Annotated\n\nfrom typing_extensions import TypedDict\n\nfrom langgraph.graph import StateGraph, START, END\nfrom langgraph.graph.message import add_messages\n\n\nclass State(TypedDict):\n messages: Annotated[list, add_messages]\n\nbuilder = StateGraph(State)\nbuilder.add_node(\"start\", lambda state: state)\nbuilder.compile()\n\nbuilder.invoke({})\n\"\"\"\n\neval_result = evaluator(outputs=CODE)\n\nprint(eval_result)\n```\n\n```\n{\n 'key': 'execution_succeeded',\n 'score': False,\n 'comment': '\"Command exited with code 1 and error:\\nTraceback (most recent call last):\\n File \\\"/home/user/openevals/outputs.py\\\", line 15, in \u003cmodule\u003e\\n builder.compile()\\n File \\\"/home/user/openevals/.venv/lib/python3.10/site-packages/langgraph/graph/state.py\\\", line 602, in compile\\n self.validate(\\n File \\\"/home/user/openevals/.venv/lib/python3.10/site-packages/langgraph/graph/graph.py\\\", line 267, in validate\\n raise ValueError(\\nValueError: Graph must have an entrypoint: add at least one edge from START to another node\\n\"'\n}\n```\n\nAbove, the evaluator identifies and installs `langgraph`, then attempts to execute the code. The type-check fails because the provided code misuses the imported package.\n\nIf desired, you can pass an `environment_variables` dict when creating the evaluator. Generated code will have access to these variables within the sandbox, but be cautious, as there is no way to predict exactly what code an LLM will generate.\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eTypeScript\u003c/summary\u003e\n\nYou will need to install the official `@e2b/code-interpreter` package as a peer dependency:\n\n```bash\nnpm install @e2b/code-interpreter\n```\n\nThen, you will need to set your E2B API key as an environment variable:\n\n```\nprocess.env.E2B_API_KEY=\"YOUR_KEY_HERE\"\n```\n\nNext, initialize an E2B sandbox:\n\n```ts\nimport { Sandbox } from \"@e2b/code-interpreter\";\n\nconst sandbox = await Sandbox.create();\n```\n\nAnd finally, pass the sandbox into the `create` and run it:\n\n```ts\nimport { createE2BExecutionEvaluator } from \"openevals/code/e2b\";\n\nconst evaluator = createE2BExecutionEvaluator({\n sandbox,\n});\n\nconst CODE = `\nimport { Annotation, StateGraph } from '@langchain/langgraph';\n\nconst StateAnnotation = Annotation.Root({\n joke: Annotation\u003cstring\u003e,\n topic: Annotation\u003cstring\u003e,\n});\n\nconst graph = new StateGraph(StateAnnotation)\n .addNode(\"joke\", () =\u003e ({}))\n .compile();\n \nawait graph.invoke({\n joke: \"foo\",\n topic: \"history\",\n});\n`;\n\nconst evalResult = await evaluator({ outputs });\n\nconsole.log(evalResult);\n```\n\n```\n{\n \"key\": \"execution_succeeded\",\n \"score\": false,\n \"comment\": \"file:///home/user/openevals/node_modules/@langchain/langgraph/dist/graph/state.js:197\\n throw new Error(`${key} is already being used as a state attribute (a.k.a. a channel), cannot also be used as a node name.`);\\n ^\\n\\nError: joke is already being used as a state attribute (a.k.a. a channel), cannot also be used as a node name.\\n at StateGraph.addNode (/home/user/openevals/node_modules/@langchain/langgraph/src/graph/state.ts:292:13)\\n at \u003canonymous\u003e (/home/user/openevals/outputs.ts:9:4)\\n at ModuleJob.run (node:internal/modules/esm/module_job:195:25)\\n at async ModuleLoader.import (node:internal/modules/esm/loader:336:24)\\n at async loadESM (node:internal/process/esm_loader:34:7)\\n at async handleMainPromise (node:internal/modules/run_main:106:12)\\n\\nNode.js v18.19.0\\n\"\n}\n```\n\nAbove, the evaluator identifies and installs `@langchain/langgraph`, then attempts to execute the code. The type-check fails because the provided code misuses the imported package.\n\nIf desired, you can pass an `environmentVariables` object when creating the evaluator. Generated code will have access to these variables within the sandbox, but be cautious, as there is no way to predict exactly what code an LLM will generate.\n\n\u003c/details\u003e\n\n## Agent trajectory\n\nIf you are building an agent, `openevals` includes evaluators for assessing the entire **trajectory** of an agent's execution โ€” the sequence of messages and tool calls it makes while solving a task.\n\nTrajectories should be formatted as lists of [OpenAI-style messages](https://platform.openai.com/docs/api-reference/messages). LangChain `BaseMessage` instances are also supported.\n\n### Trajectory match\n\n`create_trajectory_match_evaluator`/`createTrajectoryMatchEvaluator` compares an agent's trajectory against a reference trajectory. You can set `trajectory_match_mode`/`trajectoryMatchMode` to one of four modes:\n\n- `\"strict\"` โ€” same tool calls in the same order\n- `\"unordered\"` โ€” same tool calls in any order\n- `\"subset\"` โ€” output tool calls are a subset of reference\n- `\"superset\"` โ€” output tool calls are a superset of reference\n\n#### Strict match\n\nThe `\"strict\"` mode compares two trajectories and ensures that they contain the same messages in the same order with the same tool calls. Note that it does allow for differences in message content (e.g. `\"SF\"` vs. `\"San Francisco\"`):\n\n\u003cdetails\u003e\n\u003csummary\u003ePython\u003c/summary\u003e\n\n```python\nimport json\nfrom openevals import create_trajectory_match_evaluator\n\noutputs = [\n {\"role\": \"user\", \"content\": \"What is the weather in SF?\"},\n {\n \"role\": \"assistant\",\n \"content\": \"\",\n \"tool_calls\": [\n {\n \"function\": {\n \"name\": \"get_weather\",\n \"arguments\": json.dumps({\"city\": \"San Francisco\"}),\n }\n },\n {\n \"function\": {\n \"name\": \"accuweather_forecast\",\n \"arguments\": json.dumps({\"city\": \"San Francisco\"}),\n }\n }\n ],\n },\n {\"role\": \"tool\", \"content\": \"It's 80 degrees and sunny in SF.\"},\n {\"role\": \"assistant\", \"content\": \"The weather in SF is 80 degrees and sunny.\"},\n]\nreference_outputs = [\n {\"role\": \"user\", \"content\": \"What is the weather in San Francisco?\"},\n {\n \"role\": \"assistant\",\n \"content\": \"\",\n \"tool_calls\": [\n {\n \"function\": {\n \"name\": \"get_weather\",\n \"arguments\": json.dumps({\"city\": \"San Francisco\"}),\n }\n }\n ],\n },\n {\"role\": \"tool\", \"content\": \"It's 80 degrees and sunny in San Francisco.\"},\n {\"role\": \"assistant\", \"content\": \"The weather in SF is 80หš and sunny.\"},\n]\n\nevaluator = create_trajectory_match_evaluator(trajectory_match_mode=\"strict\")\nresult = evaluator(outputs=outputs, reference_outputs=reference_outputs)\nprint(result)\n```\n\n```\n{'key': 'trajectory_strict_match', 'score': False, 'comment': None}\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eTypeScript\u003c/summary\u003e\n\n```ts\nimport {\n createTrajectoryMatchEvaluator,\n type FlexibleChatCompletionMessage,\n} from \"openevals\";\n\nconst outputs = [\n { role: \"user\", content: \"What is the weather in SF?\" },\n {\n role: \"assistant\",\n content: \"\",\n tool_calls: [{\n function: {\n name: \"get_weather\",\n arguments: JSON.stringify({ city: \"San Francisco\" }),\n },\n }, {\n function: {\n name: \"accuweather_forecast\",\n arguments: JSON.stringify({ city: \"San Francisco\" }),\n },\n }],\n },\n { role: \"tool\", content: \"It's 80 degrees and sunny in SF.\" },\n { role: \"assistant\", content: \"The weather in SF is 80 degrees and sunny.\" },\n] satisfies FlexibleChatCompletionMessage[];\n\nconst referenceOutputs = [\n { role: \"user\", content: \"What is the weather in San Francisco?\" },\n {\n role: \"assistant\",\n content: \"\",\n tool_calls: [{\n function: {\n name: \"get_weather\",\n arguments: JSON.stringify({ city: \"San Francisco\" }),\n },\n }],\n },\n { role: \"tool\", content: \"It's 80 degrees and sunny in San Francisco.\" },\n] satisfies FlexibleChatCompletionMessage[];\n\nconst evaluator = createTrajectoryMatchEvaluator({ trajectoryMatchMode: \"strict\" });\nconst result = await evaluator({ outputs, referenceOutputs });\nconsole.log(result);\n```\n\n```\n{ key: 'trajectory_strict_match', score: false }\n```\n\u003c/details\u003e\n\n`\"strict\"` is useful if you want to ensure that tools are always called in the same order for a given query (e.g. a policy lookup tool before a tool that requests time off for an employee).\n\n**Note:** If you would like to configure the way this evaluator checks for tool call equality, see [this section](#tool-args-match-modes).\n\n#### Unordered match\n\nThe `\"unordered\"` mode compares two trajectories and ensures that they contain the same tool calls in any order. This is useful if you want to allow flexibility in how an agent obtains the proper information, but still do care that all information was retrieved.\n\n\u003cdetails\u003e\n\u003csummary\u003ePython\u003c/summary\u003e\n\n```python\nimport json\nfrom openevals import create_trajectory_match_evaluator\n\noutputs = [\n {\"role\": \"user\", \"content\": \"What is the weather in SF and is there anything fun happening?\"},\n {\n \"role\": \"assistant\",\n \"content\": \"\",\n \"tool_calls\": [{\"function\": {\"name\": \"get_weather\", \"arguments\": json.dumps({\"city\": \"San Francisco\"})}}],\n },\n {\"role\": \"tool\", \"content\": \"It's 80 degrees and sunny in SF.\"},\n {\n \"role\": \"assistant\",\n \"content\": \"\",\n \"tool_calls\": [{\"function\": {\"name\": \"get_fun_activities\", \"arguments\": json.dumps({\"city\": \"San Francisco\"})}}],\n },\n {\"role\": \"tool\", \"content\": \"Nothing fun is happening, you should stay indoors and read!\"},\n {\"role\": \"assistant\", \"content\": \"The weather in SF is 80 degrees and sunny, but there is nothing fun happening.\"},\n]\nreference_outputs = [\n {\"role\": \"user\", \"content\": \"What is the weather in SF and is there anything fun happening?\"},\n {\n \"role\": \"assistant\",\n \"content\": \"\",\n \"tool_calls\": [\n {\"function\": {\"name\": \"get_fun_activities\", \"arguments\": json.dumps({\"city\": \"San Francisco\"})}},\n {\"function\": {\"name\": \"get_weather\", \"arguments\": json.dumps({\"city\": \"San Francisco\"})}},\n ],\n },\n {\"role\": \"tool\", \"content\": \"Nothing fun is happening, you should stay indoors and read!\"},\n {\"role\": \"tool\", \"content\": \"It's 80 degrees and sunny in SF.\"},\n {\"role\": \"assistant\", \"content\": \"In SF, it's 80หš and sunny, but there is nothing fun happening.\"},\n]\n\nevaluator = create_trajectory_match_evaluator(trajectory_match_mode=\"unordered\")\nresult = evaluator(outputs=outputs, reference_outputs=reference_outputs)\nprint(result)\n```\n\n```\n{'key': 'trajectory_unordered_match', 'score': True, 'comment': None}\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eTypeScript\u003c/summary\u003e\n\n```ts\nimport {\n createTrajectoryMatchEvaluator,\n type FlexibleChatCompletionMessage,\n} from \"openevals\";\n\nconst outputs = [\n { role: \"user\", content: \"What is the weather in SF and is there anything fun happening?\" },\n {\n role: \"assistant\",\n content: \"\",\n tool_calls: [{ function: { name: \"get_weather\", arguments: JSON.stringify({ city: \"San Francisco\" }) } }],\n },\n { role: \"tool\", content: \"It's 80 degrees and sunny in SF.\" },\n {\n role: \"assistant\",\n content: \"\",\n tool_calls: [{ function: { name: \"get_fun_activities\", arguments: JSON.stringify({ city: \"San Francisco\" }) } }],\n },\n { role: \"tool\", content: \"Nothing fun is happening, you should stay indoors and read!\" },\n { role: \"assistant\", content: \"The weather in SF is 80 degrees and sunny, but there is nothing fun happening.\" },\n] satisfies FlexibleChatCompletionMessage[];\n\nconst referenceOutputs = [\n { role: \"user\", content: \"What is the weather in SF and is there anything fun happening?\" },\n {\n role: \"assistant\",\n content: \"\",\n tool_calls: [\n { function: { name: \"get_fun_activities\", arguments: JSON.stringify({ city: \"San Francisco\" }) } },\n { function: { name: \"get_weather\", arguments: JSON.stringify({ city: \"San Francisco\" }) } },\n ],\n },\n { role: \"tool\", content: \"Nothing fun is happening, you should stay indoors and read!\" },\n { role: \"tool\", content: \"It's 80 degrees and sunny in SF.\" },\n { role: \"assistant\", content: \"In SF, it's 80หš and sunny, but there is nothing fun happening.\" },\n] satisfies FlexibleChatCompletionMessage[];\n\nconst evaluator = createTrajectoryMatchEvaluator({ trajectoryMatchMode: \"unordered\" });\nconst result = await evaluator({ outputs, referenceOutputs });\nconsole.log(result);\n```\n\n```\n{ key: 'trajectory_unordered_match', score: true }\n```\n\u003c/details\u003e\n\n`\"unordered\"` is useful if you want to ensure that specific tools are called at some point in the trajectory, but you don't necessarily need them to be in message order.\n\n**Note:** If you would like to configure the way this evaluator checks for tool call equality, see [this section](#tool-args-match-modes).\n\n#### Subset and superset match\n\nThe `\"subset\"` and `\"superset\"` modes match partial trajectories, ensuring that a trajectory contains a subset/superset of tool calls contained in a reference trajectory.\n\n\u003cdetails\u003e\n\u003csummary\u003ePython\u003c/summary\u003e\n\n```python\nimport json\nfrom openevals import create_trajectory_match_evaluator\n\noutputs = [\n {\"role\": \"user\", \"content\": \"What is the weather in SF and London?\"},\n {\n \"role\": \"assistant\",\n \"content\": \"\",\n \"tool_calls\": [\n {\"function\": {\"name\": \"get_weather\", \"arguments\": json.dumps({\"city\": \"SF and London\"})}},\n {\"function\": {\"name\": \"accuweather_forecast\", \"arguments\": json.dumps({\"city\": \"SF and London\"})}}\n ],\n },\n {\"role\": \"tool\", \"content\": \"It's 80 degrees and sunny in SF, and 90 degrees and rainy in London.\"},\n {\"role\": \"tool\", \"content\": \"Unknown.\"},\n {\"role\": \"assistant\", \"content\": \"The weather in SF is 80 degrees and sunny. In London, it's 90 degrees and rainy.\"},\n]\nreference_outputs = [\n {\"role\": \"user\", \"content\": \"What is the weather in SF and London?\"},\n {\n \"role\": \"assistant\",\n \"content\": \"\",\n \"tool_calls\": [\n {\"function\": {\"name\": \"get_weather\", \"arguments\": json.dumps({\"city\": \"SF and London\"})}}\n ],\n },\n {\"role\": \"tool\", \"content\": \"It's 80 degrees and sunny in San Francisco, and 90 degrees and rainy in London.\"},\n {\"role\": \"assistant\", \"content\": \"The weather in SF is 80หš and sunny. In London, it's 90หš and rainy.\"},\n]\n\nevaluator = create_trajectory_match_evaluator(trajectory_match_mode=\"superset\") # or \"subset\"\nresult = evaluator(outputs=outputs, reference_outputs=reference_outputs)\nprint(result)\n```\n\n```\n{'key': 'trajectory_superset_match', 'score': True, 'comment': None}\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eTypeScript\u003c/summary\u003e\n\n```ts\nimport {\n createTrajectoryMatchEvaluator,\n type FlexibleChatCompletionMessage,\n} from \"openevals\";\n\nconst outputs = [\n { role: \"user\", content: \"What is the weather in SF and London?\" },\n {\n role: \"assistant\",\n content: \"\",\n tool_calls: [\n { function: { name: \"get_weather\", arguments: JSON.stringify({ city: \"SF and London\" }) } },\n { function: { name: \"accuweather_forecast\", arguments: JSON.stringify({ city: \"SF and London\" }) } },\n ],\n },\n { role: \"tool\", content: \"It's 80 degrees and sunny in SF, and 90 degrees and rainy in London.\" },\n { role: \"tool\", content: \"Unknown.\" },\n { role: \"assistant\", content: \"The weather in SF is 80 degrees and sunny. In London, it's 90 degrees and rainy.\" },\n] satisfies FlexibleChatCompletionMessage[];\n\nconst referenceOutputs = [\n { role: \"user\", content: \"What is the weather in SF and London?\" },\n {\n role: \"assistant\",\n content: \"\",\n tool_calls: [\n { function: { name: \"get_weather\", arguments: JSON.stringify({ city: \"SF and London\" }) } },\n ],\n },\n { role: \"tool\", content: \"It's 80 degrees and sunny in San Francisco, and 90 degrees and rainy in London.\" },\n { role: \"assistant\", content: \"The weather in SF is 80หš and sunny. In London, it's 90หš and rainy.\" },\n] satisfies FlexibleChatCompletionMessage[];\n\nconst evaluator = createTrajectoryMatchEvaluator({ trajectoryMatchMode: \"superset\" }); // or \"subset\"\nconst result = await evaluator({ outputs, referenceOutputs });\nconsole.log(result);\n```\n\n```\n{ key: 'trajectory_superset_match', score: true }\n```\n\u003c/details\u003e\n\n`\"superset\"` is useful if you want to ensure that some key tools were called at some point in the trajectory, but an agent calling extra tools is still acceptable. `\"subset\"` is the inverse and is useful if you want to ensure that the agent did not call any tools beyond the expected ones.\n\n#### Tool args match modes\n\nWhen checking equality between tool calls, the above evaluators will require that all tool call arguments are the exact same by default. You can configure this behavior in the following ways:\n\n- Treating any two tool calls for the same tool as equivalent by setting `tool_args_match_mode=\"ignore\"` (Python) or `toolArgsMatchMode: \"ignore\"` (TypeScript)\n- Treating a tool call as equivalent if it contains a subset/superset of args compared to a reference tool call of the same name with `tool_args_match_mode=\"subset\"/\"superset\"` (Python) or `toolArgsMatchMode: \"subset\"/\"superset\"` (TypeScript)\n- Setting custom matchers for all calls of a given tool using the `tool_args_match_overrides` (Python) or `toolArgsMatchOverrides` (TypeScript) param\n\n`tool_args_match_overrides`/`toolArgsMatchOverrides` takes a dictionary whose keys are tool names and whose values are either `\"exact\"`, `\"ignore\"`, `\"subset\"`, `\"superset\"`, a list of field paths that must match exactly, or a comparator function:\n\nHere's an example that allows case insensitivity for the arguments to a tool named `get_weather`:\n\n\u003cdetails\u003e\n\u003csummary\u003ePython\u003c/summary\u003e\n\n```python\nimport json\nfrom openevals import create_trajectory_match_evaluator\n\noutputs = [\n {\"role\": \"user\", \"content\": \"What is the weather in SF?\"},\n {\n \"role\": \"assistant\",\n \"content\": \"\",\n \"tool_calls\": [\n {\"function\": {\"name\": \"get_weather\", \"arguments\": json.dumps({\"city\": \"san francisco\"})}}\n ],\n },\n {\"role\": \"tool\", \"content\": \"It's 80 degrees and sunny in SF.\"},\n {\"role\": \"assistant\", \"content\": \"The weather in SF is 80 degrees and sunny.\"},\n]\nreference_outputs = [\n {\"role\": \"user\", \"content\": \"What is the weather in San Francisco?\"},\n {\n \"role\": \"assistant\",\n \"content\": \"\",\n \"tool_calls\": [\n {\"function\": {\"name\": \"get_weather\", \"arguments\": json.dumps({\"city\": \"San Francisco\"})}}\n ],\n },\n {\"role\": \"tool\", \"content\": \"It's 80 degrees and sunny in San Francisco.\"},\n {\"role\": \"assistant\", \"content\": \"The weather in SF is 80หš and sunny.\"},\n]\n\nevaluator = create_trajectory_match_evaluator(\n trajectory_match_mode=\"strict\",\n tool_args_match_mode=\"exact\", \n tool_args_match_overrides={\n \"get_weather\": lambda x, y: x[\"city\"].lower() == y[\"city\"].lower()\n }\n)\n\nresult = evaluator(outputs=outputs, reference_outputs=reference_outputs)\nprint(result)\n```\n\n```\n{'key': 'trajectory_strict_match', 'score': True, 'comment': None}\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eTypeScript\u003c/summary\u003e\n\n```ts\nimport {\n createTrajectoryMatchEvaluator,\n type FlexibleChatCompletionMessage,\n} from \"openevals\";\n\nconst outputs = [\n { role: \"user\", content: \"What is the weather in SF?\" },\n {\n role: \"assistant\",\n content: \"\",\n tool_calls: [{ function: { name: \"get_weather\", arguments: JSON.stringify({ city: \"san francisco\" }) } }],\n },\n { role: \"tool\", content: \"It's 80 degrees and sunny in SF.\" },\n { role: \"assistant\", content: \"The weather in SF is 80 degrees and sunny.\" },\n] satisfies FlexibleChatCompletionMessage[];\n\nconst referenceOutputs = [\n { role: \"user\", content: \"What is the weather in San Francisco?\" },\n {\n role: \"assistant\",\n content: \"\",\n tool_calls: [{ function: { name: \"get_weather\", arguments: JSON.stringify({ city: \"San Francisco\" }) } }],\n },\n { role: \"tool\", content: \"It's 80 degrees and sunny in San Francisco.\" },\n { role: \"assistant\", content: \"The weather in SF is 80หš and sunny.\" },\n] satisfies FlexibleChatCompletionMessage[];\n\nconst evaluator = createTrajectoryMatchEvaluator({\n trajectoryMatchMode: \"strict\",\n toolArgsMatchOverrides: {\n get_weather: (x, y) =\u003e\n typeof x.city === \"string\" \u0026\u0026\n typeof y.city === \"string\" \u0026\u0026\n x.city.toLowerCase() === y.city.toLowerCase(),\n },\n});\n\nconst result = await evaluator({ outputs, referenceOutputs });\nconsole.log(result);\n```\n\n```\n{ key: 'trajectory_strict_match', score: true }\n```\n\u003c/details\u003e\n\nThis flexibility allows you to handle cases where you want looser equality for LLM generated arguments (`\"san francisco\"` to equal `\"San Francisco\"`) for only specific tool calls.\n\n### Trajectory LLM-as-judge\n\n`create_trajectory_llm_as_judge`/`createTrajectoryLLMAsJudge` uses an LLM to assess whether an agent's trajectory is accurate. Unlike the trajectory match evaluators, it doesn't require a reference trajectory. Use `TRAJECTORY_ACCURACY_PROMPT` for no-reference evaluation, or `TRAJECTORY_ACCURACY_PROMPT_WITH_REFERENCE` to compare against a reference:\n\n\u003cdetails\u003e\n\u003csummary\u003ePython\u003c/summary\u003e\n\n```python\nimport json\nfrom openevals import create_trajectory_llm_as_judge\nfrom openevals.prompts import TRAJECTORY_ACCURACY_PROMPT\n\nevaluator = create_trajectory_llm_as_judge(\n prompt=TRAJECTORY_ACCURACY_PROMPT,\n model=\"openai:gpt-5.4\",\n)\n\noutputs = [\n {\"role\": \"user\", \"content\": \"What is the weather in SF?\"},\n {\n \"role\": \"assistant\",\n \"content\": \"\",\n \"tool_calls\": [\n {\"function\": {\"name\": \"get_weather\", \"arguments\": json.dumps({\"city\": \"SF\"})}}\n ],\n },\n {\"role\": \"tool\", \"content\": \"It's 80 degrees and sunny in SF.\"},\n {\"role\": \"assistant\", \"content\": \"The weather in SF is 80 degrees and sunny.\"},\n]\n\nresult = evaluator(outputs=outputs)\nprint(result)\n```\n\n```\n{'key': 'trajectory_accuracy', 'score': True, 'comment': 'The trajectory is accurate...'}\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eTypeScript\u003c/summary\u003e\n\n```ts\nimport {\n createTrajectoryLLMAsJudge,\n TRAJECTORY_ACCURACY_PROMPT,\n type FlexibleChatCompletionMessage,\n} from \"openevals\";\n\nconst evaluator = createTrajectoryLLMAsJudge({\n prompt: TRAJECTORY_ACCURACY_PROMPT,\n model: \"openai:gpt-5.4\",\n});\n\nconst outputs = [\n { role: \"user\", content: \"What is the weather in SF?\" },\n {\n role: \"assistant\",\n content: \"\",\n tool_calls: [{ function: { name: \"get_weather\", arguments: JSON.stringify({ city: \"SF\" }) } }],\n },\n { role: \"tool\", content: \"It's 80 degrees and sunny in SF.\" },\n { role: \"assistant\", content: \"The weather in SF is 80 degrees and sunny.\" },\n] satisfies FlexibleChatCompletionMessage[];\n\nconst result = await evaluator({ outputs });\nconsole.log(result);\n```\n\n```\n{ key: 'trajectory_accuracy', score: true, comment: 'The trajectory is accurate...' }\n```\n\u003c/details\u003e\n\nIf you have a reference trajectory, use `TRAJECTORY_ACCURACY_PROMPT_WITH_REFERENCE` and pass `reference_outputs`/`referenceOutputs`:\n\n\u003cdetails\u003e\n\u003csummary\u003ePython\u003c/summary\u003e\n\n```python\nimport json\nfrom openevals import create_trajectory_llm_as_judge\nfrom openevals.prompts import TRAJECTORY_ACCURACY_PROMPT_WITH_REFERENCE\n\nevaluator = create_trajectory_llm_as_judge(\n prompt=TRAJECTORY_ACCURACY_PROMPT_WITH_REFERENCE,\n model=\"openai:gpt-5.4\",\n)\n\noutputs = [\n {\"role\": \"user\", \"content\": \"What is the weather in SF?\"},\n {\n \"role\": \"assistant\",\n \"content\": \"\",\n \"tool_calls\": [\n {\"function\": {\"name\": \"get_weather\", \"arguments\": json.dumps({\"city\": \"SF\"})}}\n ],\n },\n {\"role\": \"tool\", \"content\": \"It's 80 degrees and sunny in SF.\"},\n {\"role\": \"assistant\", \"content\": \"The weather in SF is 80 degrees and sunny.\"},\n]\nreference_outputs = [\n {\"role\": \"user\", \"content\": \"What is the weather in SF?\"},\n {\n \"role","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flangchain-ai%2Fopenevals","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flangchain-ai%2Fopenevals","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flangchain-ai%2Fopenevals/lists"}