{"id":14991773,"url":"https://github.com/langwatch/langevals","last_synced_at":"2025-09-25T14:30:33.467Z","repository":{"id":226510352,"uuid":"765149763","full_name":"langwatch/langevals","owner":"langwatch","description":"LangEvals aggregates various language model evaluators into a single platform, providing a standard interface for a multitude of scores and LLM guardrails, for you to protect and benchmark your LLM models and pipelines.","archived":false,"fork":false,"pushed_at":"2024-05-22T11:30:54.000Z","size":1698,"stargazers_count":14,"open_issues_count":2,"forks_count":3,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-05-22T12:35:28.296Z","etag":null,"topics":["evaluation","guardrails","llm","openai"],"latest_commit_sha":null,"homepage":"https://langwatch.ai/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/langwatch.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-02-29T11:25:20.000Z","updated_at":"2024-05-28T12:01:48.525Z","dependencies_parsed_at":"2024-05-22T12:45:23.775Z","dependency_job_id":null,"html_url":"https://github.com/langwatch/langevals","commit_stats":null,"previous_names":["langwatch/langevals"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/langwatch%2Flangevals","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/langwatch%2Flangevals/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/langwatch%2Flangevals/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/langwatch%2Flangevals/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/langwatch","download_url":"https://codeload.github.com/langwatch/langevals/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":234200153,"owners_count":18795139,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["evaluation","guardrails","llm","openai"],"created_at":"2024-09-24T14:59:47.975Z","updated_at":"2025-09-25T14:30:33.461Z","avatar_url":"https://github.com/langwatch.png","language":"Python","funding_links":[],"categories":["Others","Evaluation"],"sub_categories":[],"readme":"![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)\n[![Discord](https://img.shields.io/badge/LangWatch-Discord-%235865F2.svg)](https://discord.gg/kT4PhDS2gH)\n[![LangEvals Python version](https://img.shields.io/pypi/v/langevals?color=007EC6)](https://pypi.org/project/langevals/)\n\n# LangEvals\n\nLangEvals is the all-in-one library for LLM testing and evaluating in Python, it can be used in notebooks for exploration, in pytest for writting unit tests or as a server API for live-evaluations and guardrails. LangEvals is modular, including 20+ evaluators such as Ragas for RAG quality, OpenAI Moderation and Azure Jailbreak detection for safety and many others under the same interface.\n\nLangEvals is the backend that powers [LangWatch](https://github.com/langwatch/langwatch) evaluations.\n\n## Getting Started\n\nTo use LangEvals locally, install it as a dependency, together with the evaluators you are interested on:\n\n```bash\npip install \"langevals[all]\"\n# or select only the ones you are interested on, e.g.:\npip install \"langevals[azure,ragas,lingua]\"\n```\n\nThen right away you can start LangEvals as a server with:\n\n```\nlangevals-server\n```\n\nAnd navigate to `http://localhost:5562/` to try out the evaluators.\n\nAlternatively, you can use it as a library as the examples below show.\n\n### Running Batch Evaluations on Notebooks\n\nWhen exploring, it is usual to generate a number of outputs from your LLM, and then evaluate them all for performance score, for example on a Jupyter Notebook. You can use LangEvals `evaluate()` to score the results in batch using diverse evaluators:\n\n```python\nimport langevals\nfrom langevals_ragas.answer_relevancy import RagasAnswerRelevancyEvaluator\nfrom langevals_langevals.competitor_blocklist import (\n    CompetitorBlocklistEvaluator,\n    CompetitorBlocklistSettings,\n)\nimport pandas as pd\n\nentries = pd.DataFrame(\n    {\n        \"input\": [\"hello\", \"how are you?\", \"what is your name?\"],\n        \"output\": [\"hi\", \"I am a chatbot, no feelings\", \"My name is Bob\"],\n    }\n)\n\nresults = langevals.evaluate(\n    entries,\n    [\n        RagasAnswerRelevancyEvaluator(),\n        CompetitorBlocklistEvaluator(\n            settings=CompetitorBlocklistSettings(competitors=[\"Bob\"])\n        ),\n    ],\n)\n\nresults.to_pandas()\n```\n\nResults:\n\n| input              | output                      | answer_relevancy | competitor_blocklist | competitor_blocklist_details |\n| ------------------ | --------------------------- | ---------------- | -------------------- | ---------------------------- |\n| hello              | hi                          | 0.800714         | True                 | None                         |\n| how are you?       | I am a chatbot, no feelings | 0.813168         | True                 | None                         |\n| what is your name? | My name is Bob              | 0.971663         | False                | Competitors mentioned: Bob   |\n\n### Unit Test Evaluations with PyTest\n\nUsing various pytest plugins together with LangEvals makes a powerful combination to be able to write unit tests for LLMs and prevent regressions. Due to the probabilistic nature of LLMs, some extra care is needed as you will see below.\n\n#### Simple assertions - entity extraction test example\n\nThe first simple case is when LLMs are used where the expected output is fairly unambiguous, for example, extracting address entities from natural language text. In this example we use the [instructor library](https://github.com/jxnl/instructor), to use the LLM to easily extract values to a pydantic module, together with the [litellm](https://github.com/BerriAI/litellm) library, to call multiple LLM models:\n\n```python\n\nfrom itertools import product\nimport pytest\nimport pandas as pd\n\nimport instructor\n\nfrom litellm import completion\nfrom pydantic import BaseModel\n\n\nclass Address(BaseModel):\n    number: int\n    street_name: str\n    city: str\n    country: str\n\n\nentries = pd.DataFrame(\n    {\n        \"input\": [\n            \"Please send the package to 123 Main St, Springfield.\",\n            \"J'ai déménagé récemment à 56 Rue de l'Université, Paris.\",\n            \"A reunião será na Avenida Paulista, 900, São Paulo.\",\n        ],\n        \"expected_output\": [\n            Address(\n                number=123, street_name=\"Main St\", city=\"Springfield\", country=\"USA\"\n            ).model_dump_json(),\n            Address(\n                number=56,\n                street_name=\"Rue de l'Université\",\n                city=\"Paris\",\n                country=\"France\",\n            ).model_dump_json(),\n            Address(\n                number=900,\n                street_name=\"Avenida Paulista\",\n                city=\"São Paulo\",\n                country=\"Brazil\",\n            ).model_dump_json(),\n        ],\n    }\n)\n\nmodels = [\"gpt-3.5-turbo\", \"gpt-4-turbo\", \"groq/llama3-70b-8192\"]\n\nclient = instructor.from_litellm(completion)\n\n\n@pytest.mark.parametrize(\"entry, model\", product(entries.itertuples(), models))\n@pytest.mark.flaky(max_runs=3)\n@pytest.mark.pass_rate(0.6)\ndef test_extracts_the_right_address(entry, model):\n    address = client.chat.completions.create(\n        model=model,\n        response_model=Address,\n        messages=[\n            {\"role\": \"user\", \"content\": entry.input},\n        ],\n        temperature=0.0,\n    )\n\n    assert address.model_dump_json() == entry.expected_output\n```\n\nIn the example above, our test actually becomes 9 tests, checking for address extraction correctness in each of the 3 samples against 3 different models `gpt-3.5-turbo`, `gpt-4-turbo` and `groq/llama3`. This is done by the `@pytest.mark.parametrize` annotation and the `product` function to combine entries and models. The actual assertion is a simple `assert` with `==` comparison as you can see in the last line.\n\nAppart from `parametrize`, we also use the [flaky](https://github.com/box/flaky) library for retries with `@pytest.mark.flaky(max_runs=3)`, this allows us to effectively do a 3-shot prompting with our LLM. If you wish, you can also ensure the majority of the attempts are correct by using `@pytest.mark.flaky(max_runs=3, min_passes=2)`.\n\nLastly, we use the `@pytest.mark.pass_rate` annotation provided by LangEvals, this allow the test to pass even if some samples fail, as they do for example when the model guesses \"United States\" instead of \"USA\" for the country field. Since LLMs are probabilistic, this is necessary for bringing more stability to your test suite, while still ensuring a minimum threshold of accuracy, which in our case is defined as `0.6` (60%).\n\n#### Using LangEvals Evaluators - LLM-as-a-Judge\n\nAs things get more nuanced and less objective, exact string matches are no longer possible. We can then rely on LangEvals evaluators for validating many aspects of the LLM inputs and outputs. For complete flexibility, we can use for example a custom LLM-as-a-judge, with `CustomLLMBooleanEvaluator`. In the example below we validate that more than 80% of the recipes generated are vegetarian:\n\n```python\nfrom langevals import expect\n\nentries = pd.DataFrame(\n    {\n        \"input\": [\n            \"Generate me a recipe for a quick breakfast with bacon\",\n            \"Generate me a recipe for a lunch using lentils\",\n            \"Generate me a recipe for a vegetarian dessert\",\n        ],\n    }\n)\n\n@pytest.mark.parametrize(\"entry\", entries.itertuples())\n@pytest.mark.flaky(max_runs=3)\n@pytest.mark.pass_rate(0.8)\ndef test_extracts_the_right_address(entry):\n    response: ModelResponse = litellm.completion(\n        model=\"gpt-3.5-turbo\",\n        messages=[\n            {\n                \"role\": \"system\",\n                \"content\": \"You are a tweet-size recipe generator, just recipe name and ingredients, no yapping.\",\n            },\n            {\"role\": \"user\", \"content\": entry.input},\n        ],\n        temperature=0.0,\n    )  # type: ignore\n    recipe = response.choices[0].message.content  # type: ignore\n\n    vegetarian_checker = CustomLLMBooleanEvaluator(\n        settings=CustomLLMBooleanSettings(\n            prompt=\"Is the recipe vegetarian?\",\n        )\n    )\n\n    expect(input=entry.input, output=recipe).to_pass(vegetarian_checker)\n```\n\nThis test fails with a nice explanation from the LLM judge:\n\n```python\nFAILED tests/test_llm_as_judge.py::test_llm_as_judge[entry0] - AssertionError: Custom LLM Boolean Evaluator to_pass FAILED - The recipe for a quick breakfast with bacon includes bacon strips, making it a non-vegetarian recipe.\n```\n\nNotice we use the `expect` assertion util, this helps making it easier to run the evaluation and print a nice output with the detailed explanation in case of failures. The `expect` utility interface is modeled after Jest assertions, so you can expect a somewhat similar API if you are expericed with Jest.\n\n#### Using LangEvals Evaluators - Out of the box evaluators\n\nJust like `CustomLLMBooleanEvaluator`, you can use any other evaluator available from LangEvals to prevent regression on a variety of cases, for example, here we check that the LLM answers are always in english, regardless of the language used in the question, we also measure how relevant the answers are to the question:\n\n```python\nentries = pd.DataFrame(\n    {\n        \"input\": [\n            \"What's the connection between 'breaking the ice' and the Titanic's first voyage?\",\n            \"Comment la bataille de Verdun a-t-elle influencé la cuisine française?\",\n            \"¿Puede el musgo participar en la purificación del aire en espacios cerrados?\",\n        ],\n    }\n)\n\n\n@pytest.mark.parametrize(\"entry\", entries.itertuples())\n@pytest.mark.flaky(max_runs=3)\n@pytest.mark.pass_rate(0.8)\ndef test_language_and_relevancy(entry):\n    response: ModelResponse = litellm.completion(\n        model=\"gpt-3.5-turbo\",\n        messages=[\n            {\n                \"role\": \"system\",\n                \"content\": \"You reply questions only in english, no matter tha language the question was asked\",\n            },\n            {\"role\": \"user\", \"content\": entry.input},\n        ],\n        temperature=0.0,\n    )  # type: ignore\n    recipe = response.choices[0].message.content  # type: ignore\n\n    language_checker = LinguaLanguageDetectionEvaluator(\n        settings=LinguaLanguageDetectionSettings(\n            check_for=\"output_matches_language\",\n            expected_language=\"EN\",\n        )\n    )\n    answer_relevancy_checker = RagasAnswerRelevancyEvaluator()\n\n    expect(input=entry.input, output=recipe).to_pass(language_checker)\n    expect(input=entry.input, output=recipe).score(\n        answer_relevancy_checker\n    ).to_be_greater_than(0.8)\n```\n\nIn this example we are now not only validating a boolean assertion, but also making sure that 80% of our samples keep an answer relevancy score above 0.8 from the Ragas Answer Relevancy Evaluator.\n\n# Contributing\n\nLangEvals is a monorepo and has many subpackages with different dependencies for each evaluator library or provider. We use poetry to install all dependencies and create a virtual env for each sub-package to make sure they are fully isolated. Given this complexity, to make it easier to contribute to LangEvals we recommend using VS Code for the development. Before opening up on VS Code though, you need to make sure to install all dependencies, generating thus the .venv for each package:\n\n```\nmake install\n```\n\nThis will also generate the `langevals.code-workspace` file, creating a different workspace per evaluator and telling VS Code which venv to use for each. Then, open this file on vscode and click the \"Open Workspace\" button\n\n\n## Adding New Evaluators\n\nTo add a completely new evaluator for a library or API that is not already implemented, copy the `evaluators/example` folder, and follow the `example/word_count.py` boilerplate to implement your own evaluator, adding the dependencies on `pyproject.toml`, and testing it properly, following the `test_word_count.py` example.\n\nIf you want to add a new eval to an existing evaluator package (say, if OpenAI launches a new API for example), simply create a new Python file next to the existing ones.\n\nTo test it all together, run:\n\n```\nmake lock\nmake install\nmake test\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flangwatch%2Flangevals","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flangwatch%2Flangevals","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flangwatch%2Flangevals/lists"}