{"id":28533492,"url":"https://github.com/epam/ai-dial-rag-eval","last_synced_at":"2026-04-01T21:50:41.761Z","repository":{"id":295059121,"uuid":"952730840","full_name":"epam/ai-dial-rag-eval","owner":"epam","description":"A python library designed for RAG (Retrieval-Augmented Generation) evaluation, where retrieval and generation metrics are calculated.","archived":false,"fork":false,"pushed_at":"2026-03-27T15:05:32.000Z","size":693,"stargazers_count":5,"open_issues_count":12,"forks_count":0,"subscribers_count":13,"default_branch":"development","last_synced_at":"2026-03-27T23:42:00.858Z","etag":null,"topics":["ai-dial","llm"],"latest_commit_sha":null,"homepage":"https://dialx.ai","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/epam.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-03-21T19:25:22.000Z","updated_at":"2026-03-20T17:17:40.000Z","dependencies_parsed_at":"2025-05-23T13:21:05.645Z","dependency_job_id":"3b667aef-a586-4c85-a0df-1b0b36c41eb1","html_url":"https://github.com/epam/ai-dial-rag-eval","commit_stats":null,"previous_names":["epam/ai-dial-rag-eval"],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/epam/ai-dial-rag-eval","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/epam%2Fai-dial-rag-eval","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/epam%2Fai-dial-rag-eval/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/epam%2Fai-dial-rag-eval/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/epam%2Fai-dial-rag-eval/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/epam","download_url":"https://codeload.github.com/epam/ai-dial-rag-eval/tar.gz/refs/heads/development","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/epam%2Fai-dial-rag-eval/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31292573,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-01T21:15:39.731Z","status":"ssl_error","status_checked_at":"2026-04-01T21:15:34.046Z","response_time":53,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-dial","llm"],"created_at":"2025-06-09T17:07:15.878Z","updated_at":"2026-04-01T21:50:41.754Z","avatar_url":"https://github.com/epam.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AI DIAL RAG EVAL\n\n[![PyPI version](https://img.shields.io/pypi/v/aidial-rag-eval.svg)](https://pypi.org/project/aidial-rag-eval/)\n\n## Overview\n\nLibrary designed for RAG (Retrieval-Augmented Generation) evaluation, where retrieval and generation metrics are calculated.\n\n## Usage\n\nInstall the library using [pip](https://pip.pypa.org/en/stable/getting-started):\n\n```sh\npip install aidial-rag-eval\n```\n\n### Example\n\nThe example of how to get retrieval metrics along with answer inference based on the context.\n\n```python\nimport pandas as pd\nfrom langchain_openai import AzureChatOpenAI\nfrom aidial_rag_eval import create_rag_eval_metrics_report\nfrom aidial_rag_eval.metric_binds import CONTEXT_TO_ANSWER_INFERENCE,\\\n    ANSWER_TO_GROUND_TRUTH_INFERENCE, GROUND_TRUTH_TO_ANSWER_INFERENCE\n\nllm = AzureChatOpenAI(model=\"gemini-2.5-flash-lite\")\n\ndf_ground_truth = pd.DataFrame([\n    {\n        \"question\": \"What is the diameter of the Earth and the name of the biggest ocean?\",\n        \"documents\": [\"earth.pdf\"],\n        \"facts\": [\"The diameter of the Earth is approximately 12,742 kilometers.\", \"The biggest ocean on Earth is the Pacific Ocean.\"],\n        \"answer\": \"The Earth's diameter measures about 12,742 kilometers, and the Pacific Ocean is the largest ocean on our planet.\"\n    },])\ndf_answer = pd.DataFrame([\n    {\n        \"question\": \"What is the diameter of the Earth and the name of the biggest ocean?\",\n        \"documents\": [\"earth.pdf\"],\n        \"context\":  [\n            \"The Earth, our home planet, is the third planet from the sun. It's the only planet known to have an atmosphere containing free oxygen and oceans of liquid water on its surface. The diameter of the Earth is approximately 12,742 kilometers.\",\n            \"The Pacific Ocean is the largest and deepest of Earth's oceanic divisions, extending from the Arctic Ocean in the north to the Southern Ocean in the south.\"\n        ],\n        \"answer\": \"The Earth has a diameter of approximately 12,742 kilometers.\"\n    },\n])\n\ndf_metrics = create_rag_eval_metrics_report(\n    df_ground_truth,\n    df_answer,\n    llm=llm,\n    metric_binds=[\n        CONTEXT_TO_ANSWER_INFERENCE,\n        ANSWER_TO_GROUND_TRUTH_INFERENCE,\n        GROUND_TRUTH_TO_ANSWER_INFERENCE,\n    ],\n)\nprint(df_metrics[[\"facts_ranks\", \"recall\", 'precision', 'mrr', 'f1', 'ctx_ans_inference', 'ans_gt_inference', 'gt_ans_inference']])\n```\n\nIt is expected to see the following results:\n\n| recall | precision | mrr | f1  | ctx_ans_inference | ans_gt_inference | gt_ans_inference |\n| ------ | --------- | --- | --- | ----------------- | ---------------- | ---------------- |\n| 0.5    | 0.5       | 0.5 | 0.5 | 1.0               | 0.5              | 1.0              |\n\nIn this table:\n\n- \"recall\" of 0.5 indicates that only 1 out of 2 ground truth facts were found in the context.\n- \"precision\" of 0.5 reflects that just 1 context chunk out of 2 includes any ground truth facts.\n- The prefix of the inference metrics signifies the premise and hypothesis in the following format: *premise*_*hypothesis*_inference.\n  - \"ctx\" refers to 'context'\n  - \"ans\" refers to 'answer'\n  - \"gt\" refers to 'ground truth answer'\n- \"ctx_ans_inference\" and \"ans_gt_inference\" values of 1.0 mean our answer can be derived directly from the context and the ground truth answer, respectively.\n- \"gt_ans_inference\" of 0.5, denotes that the ground truth answer can only be partially inferred from our answer.\n\n## Recommended models\n\nThe algorithm is token-intensive. Considering the balance between quality and price, the following models are recommended:\n\n- **gemini-2.5-flash-lite**\n- **gpt-5-mini**\n- **gemini-2.0-flash-lite**\n- **gpt-5-nano**\n\n## Developer environment\n\nThis project uses [Python\u003e=3.11](https://www.python.org/downloads/) and [Poetry\u003e=2.2.1](https://python-poetry.org/) as a dependency manager.\n\nCheck out Poetry's [documentation on how to install it](https://python-poetry.org/docs/#installation) on your system before proceeding.\n\nTo install requirements:\n\n```sh\npoetry install\n```\n\nThis will install all requirements for running the package, linting, formatting and tests.\n\n## Lint\n\nRun the linting before committing:\n\n```sh\nmake lint\n```\n\nTo auto-fix formatting issues run:\n\n```sh\nmake format\n```\n\n## Test\n\nRun unit tests locally for available python versions:\n\n```sh\nmake test\n```\n\nRun unit tests for the specific python version:\n\n```sh\nmake test PYTHON=3.11\n```\n\nThe generation evaluation requires an access to the LLM. The generation evaluation tests (located in `tests/llm_tests` directory) use cached LLM responses by default. To run the tests with real LLM responses, you need add `--llm-mode=real` argument to the test command:\n\n```sh\nmake test PYTHON=3.11 ARGS=\"--llm-mode=real\"\n```\n\nThe test run with real LLM responses requires the following environment variables to be set:\n\n| Variable     | Description                      |\n| ------------ | -------------------------------- |\n| DIAL_URL     | The URL of the DIAL server.      |\n| DIAL_API_KEY | The API key for the DIAL server. |\n\nCopy `.env.example` to `.env` and customize it for your environment.\n\n## Clean\n\nTo remove the virtual environment and build artifacts run:\n\n```sh\nmake clean\n```\n\n## Build\n\nTo build the package run:\n\n```sh\nmake build\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fepam%2Fai-dial-rag-eval","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fepam%2Fai-dial-rag-eval","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fepam%2Fai-dial-rag-eval/lists"}