{"id":34060456,"url":"https://github.com/vero-labs-ai/vero-eval","last_synced_at":"2026-04-07T17:31:13.964Z","repository":{"id":314108946,"uuid":"1054028779","full_name":"vero-labs-ai/vero-eval","owner":"vero-labs-ai","description":"Open source framework for evaluating AI Agents","archived":false,"fork":false,"pushed_at":"2026-02-24T17:54:26.000Z","size":1096,"stargazers_count":28,"open_issues_count":1,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-02-24T21:55:11.575Z","etag":null,"topics":["dataset-generation","datasets","evals","evaluation","evaluation-framework","evaluation-metrics","langgraph","llm-evaluation","llm-evaluation-framework","python","rag-evaluation","rag-testing","synthetic-dataset-generation","testing","testing-framework","testing-library","user-persona"],"latest_commit_sha":null,"homepage":"https://vero.co.in/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vero-labs-ai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-10T09:09:25.000Z","updated_at":"2026-02-24T17:54:30.000Z","dependencies_parsed_at":"2025-09-11T17:16:45.454Z","dependency_job_id":"587495ca-30e0-4482-a831-105c55d6060b","html_url":"https://github.com/vero-labs-ai/vero-eval","commit_stats":null,"previous_names":["dark8203/vero-metrics","dark8203/vero-eval","vero-labs-ai/vero-eval"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/vero-labs-ai/vero-eval","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vero-labs-ai%2Fvero-eval","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vero-labs-ai%2Fvero-eval/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vero-labs-ai%2Fvero-eval/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vero-labs-ai%2Fvero-eval/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vero-labs-ai","download_url":"https://codeload.github.com/vero-labs-ai/vero-eval/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vero-labs-ai%2Fvero-eval/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31522204,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-07T16:28:08.000Z","status":"ssl_error","status_checked_at":"2026-04-07T16:28:06.951Z","response_time":105,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dataset-generation","datasets","evals","evaluation","evaluation-framework","evaluation-metrics","langgraph","llm-evaluation","llm-evaluation-framework","python","rag-evaluation","rag-testing","synthetic-dataset-generation","testing","testing-framework","testing-library","user-persona"],"created_at":"2025-12-14T04:06:06.806Z","updated_at":"2026-04-07T17:31:13.950Z","avatar_url":"https://github.com/vero-labs-ai.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Vero\n\n**Vero** is an open platform for evaluating and monitoring AI pipelines with real-world rigor.\n\u003cbr\u003e\nIt goes beyond standard benchmarking by understanding your business use-case to generate edge-case **user personas** and **stress-test** your Agent across challenging scenarios, help identify risks and build highly reliable AI systems.\n\n\u003e Most eval tools say \"You're broken\". Vero tells you where, and how to fix it\n\n\u003e ⭐ If you find this project helpful, consider giving it a star. It genuinely helps the project grow and encourages us to keep improving.\n\n# Index\n\n- [Key Features](#key-features)\n- [Flowchart](#flowchart)\n- [Project Structure](#project-structure)\n- [Getting Started](#getting-started)\n- [Metrics Overview](#metrics-overview)\n- [Evaluator](#evaluator)\n- [Test Dataset Generation](#test-dataset-generation)\n- [Report Generation](#report-generation)\n- [Containerised evaluation (Docker)](#containerised-evaluation-docker)\n\n\u003cbr\u003e\n\n# Key Features\n\n- **Trace \u0026 Log Execution**: Each query runs through the RAG pipeline is logged into an SQLite database, capturing the user query, retrieved context, reranked items, and the model’s output.\n- **Component-level Metrics**: Evaluate intermediate pipeline stages using metrics like Precision, Recall, Sufficiency, Citation, Overlap, and Ranking metrics (e.g. MRR, MAP, NDCG).\n- **Generation Metrics**: Measure semantic, factual, and alignment quality of generated outputs using metrics such as BERTScore, ROUGE, SEMScore, AlignScore, BLEURT, and G-Eval.\n- **Modular \u0026 Extensible**: Easily plug in new metric classes or custom scoring logic; the framework is designed to grow with your needs.\n- **End-to-End Evaluation**: Combine component metrics to understand the holistic performance of your RAG system — not just individual parts.\n\n# Flowchart\n\n![flowchart(v2).png](docs/flowchart%28v2%29.png)\n\n# Project Structure\n\n```\n.\n├── src/\n|   ├── vero/\n│   │   ├── evaluator         # Main package for evaluation\n│   │   ├── report generation workflow  # Report generation workflow\n│   │   ├── test dataset generator  # Test dataset generator\n|   │   ├── metrics/          # Main package for metrics\n|   └── └──  all the metrics  # All the metrics are in here\n└── tests/\n│   └── test_main.py/         # file for all the testing\n└── docker/\n│   ├── run_eval_pipeline.py  # script to run the evaluation pipeline\n│   └── Dockerfile/           # dockerfile for building the image    \n└── vero-deploy/\n│   ├── data/                 # folder for input and output data\n│   │   ├── inputs            # folder for input data\n│   │   │   ├── input.csv     # example input csv file\n│   │   │   └── ground_truth.csv # example ground truth csv file\n│   │   └── outputs           # folder for output data\n│   ├── config.yaml           # yaml file for configuration\n└── └── docker-compose.yaml   # docker compose file\n\n```\n\n# Getting Started\n\n### Setup\n\nInstall via pip (recommended inside a virtualenv):\n\n```py\npip install vero-eval\n```\n\n### Example Usage\n\n```py\nfrom vero.rag import SimpleRAGPipeline\nfrom vero.trace import TraceDB\nfrom vero.eval import Evaluator\n\ntrace_db = TraceDB(db_path=\"runs.db\")\npipeline = SimpleRAGPipeline(retriever=\"faiss\", generator=\"openai\", trace_db=trace_db)\n\n# Run your pipeline\nrun = pipeline.run(\"Who invented the transistor?\")\nprint(\"Answer:\", run.answer)\n\n# Later, compute metrics for all runs\nevaluator = Evaluator(trace_db=trace_db)\nresults = evaluator.evaluate()\nprint(results)\n```\n\n## Metrics Overview\n\nThe RAG Evaluation Framework supports three classes of metrics:\n\n- Generation Metrics — measure semantic/factual quality of answers.\n- Ranking Metrics — measure ranking quality of rerankers.\n- Retrieval Metrics — measure the quality and sufficiency of the retrieved context.\n\n# Evaluator\n\n## **Overview**\n\n- The Evaluator is a convenience wrapper to run multiple metrics over model outputs and retrieval results.\n- It orchestrates generation evaluation (text-generation metrics), retrieval evaluation (precision/recall/sufficiency), and reranker evaluation (NDCG, MAP, MRR). It produces CSV summaries by default.\n\n\u003e Quick notes\n\u003e\n\u003e - The Evaluator uses project metric classes (e.g., BartScore, BertScore, RougeScore, SemScore, PrecisionScore, RecallScore, MeanAP, MeanRR, RerankerNDCG, CumulativeNDCG, etc.). These metrics are in vero.metrics and are referenced internally.\n\u003e - Many methods expect particular CSV column names (see \"Expected CSV schemas\").\n\u003e - We highly recommend to install **gpu version** of torch library.\n\n### Steps to evaluate your pipeline\n\n**Step 1 - Generation evaluation**\n\n- Input: a CSV with \"Context Retrieved\" and \"Answer\" columns.\n- Result: Generation_Scores.csv with columns such as SemScore, BertScore, RougeLScore, BARTScore, BLUERTScore, G-Eval (Faithfulness).\n\nExample:\n\n```py\nfrom vero.evaluator import Evaluator\n\nevaluator = Evaluator()\n# data_path must point to a CSV with columns \"Context Retrieved\" and \"Answer\"\ndf_scores = evaluator.evaluate_generation(data_path='testing.csv')\nprint(df_scores.head())\n```\n\n**Step 2 - Preparing reranker inputs (parse ground truth + retriever output)**\n\n- Use parse_retriever_data to convert ground-truth chunk ids and retriever outputs into a ranked_chunks_data.csv suitable for reranker evaluation.\n\nExample:\n\n```py\nfrom vero.evaluator import Evaluator\n\nevaluator = Evaluator()\n# ground_truth_path: dataset with 'Chunk IDs' and 'Less Relevant Chunk IDs' columns\n# data_path: retriever output with 'Context Retrieved' containing \"id='...'\"\nevaluator.parse_retriever_data(\n    ground_truth_path='test_dataset_generator.csv',\n    data_path='testing.csv'\n)\n# This will produce 'ranked_chunks_data.csv'\n```\n\n**Step 3 - Retrieval evaluation (precision, recall, sufficiency)**\n\n- Inputs:\n  - retriever_data_path: a CSV that contains 'Retrieved Chunk IDs' and 'True Chunk IDs' columns (lists or strings).\n  - data_path: the generation CSV with 'Context Retrieved' and 'Question' (for sufficiency).\n- Result: Retrieval_Scores.csv\n\nExample:\n\n```py\nfrom vero.evaluator import Evaluator\n\nevaluator = Evaluator()\ndf_retrieval_scores = evaluator.evaluate_retrieval(\n    data_path='testing.csv',\n    retriever_data_path='ranked_chunks_data.csv'\n)\nprint(df_retrieval_scores.head())\n```\n\n**Step 4 - Reranker evaluation (MAP, MRR, NDCG)**\nExample:\n\n```py\nfrom vero.evaluator.evaluator import Evaluator\n\nevaluator = Evaluator()\ndf_reranker_scores = evaluator.evaluate_reranker(\n    ground_truth_path='test_dataset_generator.csv',\n    retriever_data_path='ranked_chunks_data.csv'\n)\nprint(df_reranker_scores)\n```\n\n#### Lower-level metric usage\n\nTo run a single metric directly you can instantiate the metric class. For example, to compute BARTScore or BertScore per pair:\n\n```py\nfrom vero.metrics import BartScore, BertScore\n\nwith BartScore() as bs:\n    bart_results = [bs.evaluate(context, answer) for context, answer in zip(contexts, answers)]\n\nwith BertScore() as bert:\n    bert_results = [bert.evaluate(context, answer) for context, answer in zip(contexts, answers)]\n```\n\n# Test Dataset Generation\n\n## **Overview**\n\n- The Test Dataset Generation module creates high-quality question-answer pairs derived from your document collection. It generates challenging queries designed to reveal retrieval and reasoning failures in RAG systems, and considering edge-case user personas.\n- Internally it chunks documents, clusters related chunks, and uses an LLM to produce QA items with ground-truth chunk IDs and metadata.\n\n### **Example**\n\n```py\nfrom vero.test_dataset_generator import generate_and_save\n\n# Generate 100 queries from PDFs stored in ./data/pdfs directory and save outputs in test_dataset directory\ngenerate_and_save(\n    data_path='./data/pdfs/',\n    usecase='Vitamin chatbot catering to general users for their daily queries',\n    save_path_dir='test_dataset',\n    n_queries=100\n)\n```\n\n# Report Generation\n\n## Overview\n\n- The Report Generation module consolidates evaluation outputs from generation, retrieval, and reranking into a final report.\n- It orchestrates a stateful workflow that processes CSV results from various evaluators and synthesizes comprehensive insights and recommendations.\n\n## Example Usage\n\n```py\nfrom vero.report_generation_workflow import ReportGenerator\n\n# Initialize the report generator\nreport_generator = ReportGenerator()\n\n# Generate the final report by providing:\n# - Pipeline configuration JSON file\n# - Generation, Retrieval, and Reranker evaluation CSV files\nreport_generator.generate_report(\n    'pipe_config_data.json',\n    'Generation_Scores.csv',\n    'Retrieval_Scores.csv',\n    'Reranked_Scores.csv'\n)\n```\n\n\n# Containerised evaluation (Docker)\n\n## Prequisites\n- Install [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/install/).\n- Disk spcae \u003e= 50 GB\n- CPU \u003e= 4 cores (GPU recommended for faster metric computation)\n- RAM \u003e= 8 GB\n\n**Step 1 - Clone the repository and switch to the deploy folder**\n```bash\ngit clone https://github.com/vero-eval/vero-eval.git\ncd vero-deploy\n```\n\n**Step 2 - Inspect example inputs**\n- Open the `data/inputs` folder inside `vero-deploy`.\n- Review the example CSV provided (for example `data/inputs/input.csv`) and structure your input CSV the same way. Common column names used across the project include `Context Retrieved`, `Answer`, `Question`, `Retrieved Chunk IDs`, `True Chunk IDs`, `Chunk IDs`, and `Less Relevant Chunk IDs`.\n\n**Step-3 Configure the input filename**\n- Open `config.yaml` in the `vero-deploy` directory and check the key that references the input CSV filename. Update it to point to your CSV if needed (ensure the filename you pick matches the file in `data/inputs`).\n\n**Step-4 Set your OpenAI key in Docker Compose**\n- Open `docker-compose.yaml` and set the `OPEN_AI_KEY` environment variable for the evaluation service. Example snippet:\n```yaml\nx-base-config: \u0026base-config\n  image: crimsonceres/demo-test\n  environment:\n    - HF_HOME=/root/.cache/huggingface\n    - TRANSFORMERS_CACHE=/root/.cache/huggingface/transformers\n    - OPENAI_API_KEY=\n```\nYou can also use environment substitution (for example `OPEN_AI_KEY=${OPEN_AI_KEY}`) and export the variable in your shell before running Docker.\n\n**Step-5 Run the evaluation container**\n- If you have a GPU available (recommended):\n```bash\ndocker compose run evaluation-runner-gpu\n```\n- If you are using CPU only:\n```bash\ndocker compose run evaluation-runner-cpu\n```\n\n\u003eThat is all required to run a containerised evaluation. Ensure `data/inputs` contains the CSV referenced in `config.yaml` and that `docker-compose.yaml` has `OPEN_AI_KEY` set before running the appropriate `docker compose run` command.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvero-labs-ai%2Fvero-eval","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvero-labs-ai%2Fvero-eval","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvero-labs-ai%2Fvero-eval/lists"}