{"id":31127134,"url":"https://github.com/patteg21/pigeon-evals","last_synced_at":"2026-04-28T12:05:15.731Z","repository":{"id":313103053,"uuid":"1041089395","full_name":"patteg21/pigeon-evals","owner":"patteg21","description":"A End-To-End RAG Pipeline that includes Evaluations, iterations, and swappable components. At its core it allows users to be able to try different embedding models and techniques.","archived":false,"fork":false,"pushed_at":"2025-10-03T02:13:34.000Z","size":34021,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-10-03T04:13:02.757Z","etag":null,"topics":["benchmarking","document-processing","embeddings","evaluation","llm","mcp","nlp","pipeline","processing","python","rag","retrieval-augmented-generation","vector-database","vector-search"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/patteg21.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-08-20T01:03:29.000Z","updated_at":"2025-09-16T04:30:59.000Z","dependencies_parsed_at":"2025-09-04T00:11:23.272Z","dependency_job_id":"bfa911c3-248d-4c99-b6e1-2f539478966d","html_url":"https://github.com/patteg21/pigeon-evals","commit_stats":null,"previous_names":["patteg21/pigeon-evals"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/patteg21/pigeon-evals","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/patteg21%2Fpigeon-evals","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/patteg21%2Fpigeon-evals/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/patteg21%2Fpigeon-evals/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/patteg21%2Fpigeon-evals/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/patteg21","download_url":"https://codeload.github.com/patteg21/pigeon-evals/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/patteg21%2Fpigeon-evals/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32379633,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-28T11:25:28.583Z","status":"ssl_error","status_checked_at":"2026-04-28T11:25:05.435Z","response_time":56,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmarking","document-processing","embeddings","evaluation","llm","mcp","nlp","pipeline","processing","python","rag","retrieval-augmented-generation","vector-database","vector-search"],"created_at":"2025-09-17T23:02:12.406Z","updated_at":"2026-04-28T12:05:15.692Z","avatar_url":"https://github.com/patteg21.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# Pigeon Evals\n\n**A comprehensive RAG (Retrieval-Augmented Generation) evaluation pipeline for document processing and retrieval system benchmarking**\n\n[![Python](https://img.shields.io/badge/Python-3.8%2B-blue.svg)](https://python.org)\n[![License](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)\n[![UV](https://img.shields.io/badge/Package%20Manager-uv-purple.svg)](https://docs.astral.sh/uv/)\n\n## Keywords\n`RAG evaluation`, `retrieval augmented generation`, `document processing pipeline`, `vector embeddings`, `embedding evaluation`, `semantic search`, `vector database`, `Pinecone`, `Qdrant`, `OpenAI embeddings`, `document chunking`, `retrieval benchmarking`, `MCP server`, `evaluation framework`, `Python pipeline`, `text processing`, `machine learning`, `NLP`, `retrieval system`, `AI evaluation`, `LLM evaluation`, `human evaluation`\n\n## What is Pigeon Evals?\n\nPigeon Evals is a **modular RAG evaluation framework** designed for comprehensive **document processing**, **retrieval system benchmarking**, and **LLM evaluation**. It provides end-to-end functionality from document ingestion to performance evaluation, with support for multiple storage backends, embedding providers, and evaluation methodologies.\n\n### Perfect for:\n- **RAG system evaluation** and performance benchmarking  \n- **Document processing pipeline** evaluation and optimization\n- **Semantic search** and retrieval quality assessment\n- **AI agent evaluation** with MCP (Model Context Protocol) integration\n- **Research and production** RAG system development\n- **Vector database** experimentation and comparison\n- **Multi-modal evaluation** (Human, LLM, and Agent-based)\n\n## Environment\n\nTo install uv: [https://docs.astral.sh/uv/getting-started/installation/](https://docs.astral.sh/uv/getting-started/installation/)\ncurl install: `curl -LsSf https://astral.sh/uv/install.sh | sh`\n\n### Pylance\nRecommended install [Ruff](https://docs.astral.sh/ruff/) and set PyLance to typeChecking `strict` or `basic`\n\n\nCreates a venv\n```bash\nuv venv\n```\n\nactivate venv\n```bash \nsource .venv/bin/activate\n```\n\nTo add dependencies + libraries\n```bash\nuv add \u003clibrary\u003e\n```\n\n\n# RAG Evaluation Pipeline (`/evals/`)\n\n## Overview\nA modular pipeline for processing documents with embeddings, storage, and comprehensive evaluation. The pipeline transforms raw documents into searchable vector representations using configurable processors and storage backends. The primary driver behind this setup is to create a proper RAG evaluation pipeline that can also operate in a production setting.\n\n## Usage\n```bash\npython src/main.py --config evals/configs/test.yml\n# or\nuv run src/main.py --config evals/configs/test.yml\n```\n\n### Dry Run Mode\n\nThe pipeline includes a comprehensive `--dry-run` mode for testing and development without making actual API calls or storage operations:\n\n```bash\npython src/main.py --config configs/test.yml --dry-run\n# or\nuv run src/main.py --config configs/test.yml --dry-run\n```\n\n#### What Dry Run Mode Does\n\n**Dry run mode provides mock implementations for all external operations:**\n\n- **Embedding Generation**: Generates deterministic mock embeddings instead of calling actual embedding providers (OpenAI/HuggingFace)\n- **Dimensionality Reduction**: Mocks PCA operations (fit, transform, save, load) without actual computation\n- **Storage Operations**: Simulates text and vector storage operations without persisting data\n- **Preserves Document Processing**: Still processes documents through the full pipeline, adding mock embeddings to chunks\n\n#### Key Benefits\n\n- **Cost-Free Testing**: No API costs from embedding providers\n- **Fast Development**: Skip expensive operations during development\n- **Pipeline Validation**: Verify configuration and data flow without side effects\n- **Deterministic Results**: Uses seeded random generation for consistent mock data\n\n#### Implementation Details\n\nThe dry run system uses the `@dry_response` decorator to automatically mock functions when dry run mode is enabled. Mock embeddings are:\n\n- **Deterministic**: Uses `random.seed(42)` for reproducible results\n- **Realistic**: Generated as random floats between -1.0 and 1.0\n- **Dimension-Aware**: Respects configured embedding dimensions\n- **Preserves Chunks**: Adds embeddings to original document chunks (not copies)\n\nYou can also enable dry run mode via environment variable:\n```bash\nexport DRY_RUN=true\npython src/main.py --config configs/test.yml\n```\n\n## Pipeline Architecture\nThe pipeline follows a sequential processing flow:\n\n1. **Document Loading** - Reads raw documents from specified directory\n2. **Text Processing** - Applies configurable processors (tables, page breaks, TOC parsing)\n3. **Embedding Generation** - Creates vector representations using OpenAI embeddings\n4. **Dimensionality Reduction** - Supports PCA for reducing embedding dimensions (configurable output dimensions). Ideally would add in alternatives for the future such as [UMAP](https://umap-learn.readthedocs.io/en/latest/) as well as several other methods.\n5. **Storage** - Saves processed data to both text (SQLite) and vector (Pinecone) stores\n6. **Report Generation** - Can run Human Eval, LLM Eval, and MCP Agent Eval pipelines dynamically based on runs or seperate from runs entirely. Currently works for my local mcp too!\n\n### Key Features\n- **Fully modular design** - Add/remove/swap any processing component\n- **Pydantic validation** - Type-safe configuration with automatic error detection\n- **Multi-threaded embedding** - Parallel processing for faster execution\n- **Flexible storage** - Support for multiple text and vector storage backends\n- **Flexible Framework** - Removing any piece of this does not break the system, it simply skips that part, meaning this can be a full end to end eval pipeline\n\n```bash\ntask: \"example_eval_task\"\ndataset: \n  path: \"data/processed_filings/META\"\n\nthreading:\n  max_workers: 8\n\npreprocess:\n  ocr: \"easyocr\"\n\nparser:\n  type: \"multistage\"\n  processes:\n    - name: \"table_extraction\"\n      steps:\n        - strategy: \"regex\"\n          regex_pattern: \"\\\\[TABLE_START\\\\][\\\\s\\\\S]*?\\\\[TABLE_END\\\\]\"\n          ignore_case: true\n\n    - name: \"paragraph_extraction\"\n      steps:\n        - strategy: \"regex\"\n          regex_pattern: \"\\\\[PAGE BREAK\\\\]\"\n          ignore_case: true\n        - strategy: \"paragraph\"\n          chunk_size: null\n          chunk_overlap: 256 \n\n\nembedding:\n  provider: \"huggingface\"\n  model: \"sentence-transformers/all-MiniLM-L6-v2\"\n  pooling_strategy: \"mean\"\n  dimension_reduction:\n    type: \"PCA\"\n    dims: 256\n  use_threading: true\n\n\nstorage:\n  text_store:\n    client: \"sqlite\"\n    path: \"./data/text_store.db\"\n    upload: true\n\n    # PostgreSQL options (commented out):\n    # host: \"localhost\"\n    # port: 5432\n    # database: \"pigeon_evals\"\n    # user: \"postgres\"\n    # password: \"\"\n\n    # S3 options (commented out):\n    # bucket_name: \"pigeon-evals-documents\"\n    # prefix: \"documents/\"\n    # access_key_id: null\n    # secret_access_key: null\n    # region: \"us-east-1\"\n    \n    # File store options (commented out):\n    # base_path: \"data/documents\"\n  vector:\n    provider: \"faiss\"  # Added explicit provider\n    path: \"./data/.faiss/index\"\n    upload: true\n    \n    clear: false\n    index: \"eval_index\"\n    dimension: 256  # Added to match embedding dimension_reduction\n    # Alternative vector options (commented out):\n    # index_name: \"alternative_index\"\n  outputs:\n    - \"chunks\"\n    - \"documents\"\n\neval:\n  top_k: 10\n  provider: \"openai\"\n  model: \"gpt-4o\"\n  evaluations: true\n  metrics:\n    - \"ndcg\"\n    - \"precision\"\n    - \"recall\"\n  \n  rerank:\n    provider: \"huggingface\"\n    model: \"cross-encoder/ms-marco-MiniLM-L-6-v2\"\n    top_k: 5\n  \n  test:\n    load:\n      path: \"data/tests/default.json\"\n      key: \"tests\"\n      \n    tests:\n      - type: \"llm\"\n        name: \"basic_llm_test\"\n        query: \"What is the main topic?\"\n        prompt: \"Analyze the retrieved documents and provide a summary.\"\n        eval_type:\n          - \"single\"\n      \n      - type: \"human\"\n        name: \"human_review\"\n        query: \"Quality assessment query\"\n      \n      - type: \"agent\"\n        name: \"agent_test\"\n        query: \"Test agent functionality\"\n        prompt: \"Execute the agent task\"\n        mcp:\n          command: \"python\"\n          args:\n            - \"agent_script.py\"\n            - \"--test\"\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpatteg21%2Fpigeon-evals","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpatteg21%2Fpigeon-evals","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpatteg21%2Fpigeon-evals/lists"}