{"id":48876900,"url":"https://github.com/hallengray/rag-forge","last_synced_at":"2026-04-18T03:00:50.256Z","repository":{"id":351034648,"uuid":"1207858383","full_name":"hallengray/rag-forge","owner":"hallengray","description":"Production-grade RAG pipelines with evaluation baked in","archived":false,"fork":false,"pushed_at":"2026-04-16T13:35:46.000Z","size":1485,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-17T02:03:40.707Z","etag":null,"topics":["cli","embeddings","llm","llm-evaluation","mcp","observability","python","rag","rag-evaluation","rag-pipeline","ragas","retrieval-augmented-generation","vector-database"],"latest_commit_sha":null,"homepage":"https://rag-forge-site.vercel.app/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hallengray.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":null},"created_at":"2026-04-11T13:56:02.000Z","updated_at":"2026-04-16T13:32:41.000Z","dependencies_parsed_at":null,"dependency_job_id":"38371016-7923-4423-91a8-81c8e38a9bb2","html_url":"https://github.com/hallengray/rag-forge","commit_stats":null,"previous_names":["hallengray/rag-forge"],"tags_count":8,"template":false,"template_full_name":null,"purl":"pkg:github/hallengray/rag-forge","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hallengray%2Frag-forge","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hallengray%2Frag-forge/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hallengray%2Frag-forge/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hallengray%2Frag-forge/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hallengray","download_url":"https://codeload.github.com/hallengray/rag-forge/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hallengray%2Frag-forge/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31954736,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-18T00:39:45.007Z","status":"online","status_checked_at":"2026-04-18T02:00:07.018Z","response_time":103,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","embeddings","llm","llm-evaluation","mcp","observability","python","rag","rag-evaluation","rag-pipeline","ragas","retrieval-augmented-generation","vector-database"],"created_at":"2026-04-16T01:00:24.123Z","updated_at":"2026-04-18T03:00:50.187Z","avatar_url":"https://github.com/hallengray.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n# RAG-Forge\n\n**Production-grade RAG pipelines with evaluation baked in — not bolted on after deployment.**\n\n[![npm version](https://img.shields.io/npm/v/@rag-forge/cli?label=%40rag-forge%2Fcli)](https://www.npmjs.com/package/@rag-forge/cli)\n[![PyPI version](https://img.shields.io/pypi/v/rag-forge-core?label=rag-forge-core)](https://pypi.org/project/rag-forge-core/)\n[![CI](https://img.shields.io/github/actions/workflow/status/hallengray/rag-forge/ci.yml?branch=main)](https://github.com/hallengray/rag-forge/actions)\n[![License: MIT](https://img.shields.io/github/license/hallengray/rag-forge)](./LICENSE)\n[![Discussions](https://img.shields.io/github/discussions/hallengray/rag-forge)](https://github.com/hallengray/rag-forge/discussions)\n\n[Docs](https://rag-forge-docs.vercel.app/) · [Website](https://rag-forge-site.vercel.app/) · [Discussions](https://github.com/hallengray/rag-forge/discussions) · [Changelog](./docs/release-notes)\n\n\u003c/div\u003e\n\n---\n\n## Why RAG-Forge?\n\nMost RAG projects ship without evaluation, and most evaluation libraries don't help you build the pipeline. Few tools score maturity end-to-end — so teams often don't know if they're at \"a demo that sometimes works\" or \"a system you can put in front of customers.\"\n\n- **Building a RAG pipeline is easy. Knowing whether it works is hard.** RAG-Forge closes that loop.\n- **Eval is a first-class citizen, not an afterthought.** Every template ships with a golden set and an audit gate.\n- **The RAG Maturity Model (RMM-0 → RMM-5)** gives you a concrete scorecard for any RAG system — yours or someone else's.\n\nRAG-Forge is one of the few toolkits that scaffolds production-ready RAG pipelines, runs continuous evaluation as a CI/CD gate, and scores any existing system against a published maturity model — all in one CLI.\n\n---\n\n## RAG Maturity Model\n\nThe RMM is the scoring framework at the heart of RAG-Forge. Run `rag-forge assess` on any audit report to see where your system sits.\n\n| Level | Name              | Exit Criteria                                              |\n|-------|-------------------|------------------------------------------------------------|\n| RMM-0 | Naive             | Basic vector search works                                  |\n| RMM-1 | Better Recall     | Hybrid search, Recall@5 \u003e 70%                              |\n| RMM-2 | Better Precision  | Reranker active, nDCG@10 +10%                              |\n| RMM-3 | Better Trust      | Guardrails, faithfulness \u003e 85%                             |\n| RMM-4 | Better Workflow   | Caching, P95 \u003c 4s, cost tracking                           |\n| RMM-5 | Enterprise        | Drift detection, CI/CD gates, adversarial tests            |\n\n---\n\n## Quick Start\n\n```bash\nnpm install -g @rag-forge/cli\n\n# Scaffold a project (use --directory to name the folder)\nrag-forge init basic --directory my-rag-project\ncd my-rag-project\n\n# Drop your documents into a folder of your choice (or use the example below)\nmkdir docs\necho \"RAG-Forge is a CLI for building and evaluating RAG pipelines.\" \u003e docs/example.md\n\nrag-forge index --source ./docs\nrag-forge audit --golden-set eval/golden_set.json\nrag-forge assess --audit-report reports/audit-report.json\n```\n\nFrom empty directory to a scored RAG system with a golden set and an audit report — in under a minute.\n\n---\n\n## Installation\n\n**CLI (Node.js 20+):**\n\n```bash\nnpm install -g @rag-forge/cli\n```\n\n**Python packages (Python 3.11+):**\n\n```bash\npip install rag-forge-core rag-forge-evaluator rag-forge-observability\n```\n\n---\n\n## Templates\n\n| Template     | Use Case                                             |\n|--------------|------------------------------------------------------|\n| `basic`      | First RAG project, simple Q\u0026A                        |\n| `hybrid`     | Production-ready document Q\u0026A with reranking         |\n| `agentic`    | Multi-hop reasoning with query decomposition         |\n| `enterprise` | Regulated industries with full security suite        |\n| `n8n`        | AI automation agency deployments                     |\n\nTemplates generate editable source code in your project — not framework dependencies. Fork the code, not the abstraction.\n\n---\n\n## Commands\n\n| Category         | Commands                                                             |\n|------------------|----------------------------------------------------------------------|\n| **Scaffolding**  | `init`, `add`                                                        |\n| **Ingestion**    | `parse`, `chunk`, `index`                                            |\n| **Query**        | `query`, `inspect`                                                   |\n| **Evaluation**   | `audit`, `assess`, `golden add`, `golden validate`                   |\n| **Operations**   | `report`, `cache stats`, `drift report`, `cost`                      |\n| **Security**     | `guardrails test`, `guardrails scan-pii`                             |\n| **Integration**  | `serve --mcp`, `n8n export`                                          |\n\nRun `rag-forge --help` for the full command reference.\n\n---\n\n## How RAG-Forge compares\n\nThere are great tools in this space. Here's an honest look at where each fits.\n\n| Capability                        | RAG-Forge | RAGAS  | LangChain Eval | Giskard |\n|-----------------------------------|:---------:|:------:|:--------------:|:-------:|\n| Scaffolds a RAG pipeline          |     ✓     |   —    |       —        |    —    |\n| Evaluation metrics                |     ✓     |   ✓    |       ✓        |    ✓    |\n| Maturity scoring (RMM-0 → 5)      |     ✓     |   —    |       —        |    —    |\n| CI gate workflow (audit action)   |     ✓     |   —    |    partial     | partial |\n| MCP server                        |     ✓     |   —    |       —        |    —    |\n| Guardrails / PII scanning         |     ✓     |   —    |    partial     |    ✓    |\n| Drift detection                   |     ✓     |   —    |       —        | partial |\n| Multi-language (TS + Python)      |     ✓     |   —    |       ✓        |    —    |\n| Framework-agnostic                |     ✓     |   ✓    |       —        |    ✓    |\n\n**Peer strengths worth knowing:**\n\n- **RAGAS** has deeper metric research and a large community. RAG-Forge's evaluator supports RAGAS as a backend — run `rag-forge audit --evaluator ragas` to use it directly.\n- **LangChain Eval** has the broadest ecosystem of integrations if you're already invested in LangChain.\n- **Giskard** has a strong general-purpose ML testing story beyond RAG.\n\nPick the tool that matches your stage. RAG-Forge's wedge is the full lifecycle — scaffold → evaluate → score → ship — in one CLI, with the RMM as the objective function.\n\n---\n\n## Architecture\n\nRAG-Forge is a polyglot monorepo. The CLI and MCP server are TypeScript; all RAG logic is Python. The CLI delegates to Python via a subprocess bridge so the two halves can be developed and versioned independently.\n\n```text\nrag-forge/\n├── packages/\n│   ├── cli/              TypeScript — Commander.js CLI (rag-forge command)\n│   ├── mcp/              TypeScript — MCP server (@modelcontextprotocol/sdk)\n│   ├── core/             Python    — RAG pipeline primitives\n│   ├── evaluator/        Python    — RAGAS + DeepEval + LLM-as-Judge\n│   └── observability/    Python    — OpenTelemetry + Langfuse\n├── templates/            Project templates (basic, hybrid, agentic, enterprise, n8n)\n└── apps/site/            Docs and marketing site (Next.js, deployed to Vercel)\n```\n\nSee [docs/architecture.md](./docs/architecture.md) for a deeper dive.\n\n---\n\n## Docs \u0026 Community\n\n- 📚 **Docs:** https://rag-forge-docs.vercel.app/\n- 🌐 **Website:** https://rag-forge-site.vercel.app/\n- 💬 **Discussions:** https://github.com/hallengray/rag-forge/discussions\n- 🔒 **Security:** see [SECURITY.md](./SECURITY.md)\n- 📝 **Changelog:** [docs/release-notes](./docs/release-notes)\n\n---\n\n## Contributing\n\nSee [CONTRIBUTING.md](./CONTRIBUTING.md) for development setup and contribution guidelines. All contributors are expected to follow our [Code of Conduct](./CODE_OF_CONDUCT.md).\n\n---\n\n## License\n\nMIT — see [LICENSE](./LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhallengray%2Frag-forge","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhallengray%2Frag-forge","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhallengray%2Frag-forge/lists"}