{"id":20208130,"url":"https://github.com/sciphi-ai/rag-performance","last_synced_at":"2026-01-28T01:35:11.564Z","repository":{"id":249298606,"uuid":"830106467","full_name":"SciPhi-AI/RAG-Performance","owner":"SciPhi-AI","description":"Measuring RAG solutions throughput and latency","archived":false,"fork":false,"pushed_at":"2024-07-23T13:56:56.000Z","size":51028,"stargazers_count":16,"open_issues_count":0,"forks_count":6,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-10T13:10:03.629Z","etag":null,"topics":["benchmarking","benchmarks","haystack","langchain","llamaindex","r2r","rag","ragflow","retrieval-augmented-generation"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SciPhi-AI.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-17T15:45:57.000Z","updated_at":"2025-03-29T16:08:02.000Z","dependencies_parsed_at":"2024-11-14T05:44:30.010Z","dependency_job_id":null,"html_url":"https://github.com/SciPhi-AI/RAG-Performance","commit_stats":null,"previous_names":["sciphi-ai/rag-performance"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/SciPhi-AI/RAG-Performance","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SciPhi-AI%2FRAG-Performance","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SciPhi-AI%2FRAG-Performance/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SciPhi-AI%2FRAG-Performance/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SciPhi-AI%2FRAG-Performance/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SciPhi-AI","download_url":"https://codeload.github.com/SciPhi-AI/RAG-Performance/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SciPhi-AI%2FRAG-Performance/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28831765,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-27T23:29:49.665Z","status":"ssl_error","status_checked_at":"2026-01-27T23:25:58.379Z","response_time":168,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmarking","benchmarks","haystack","langchain","llamaindex","r2r","rag","ragflow","retrieval-augmented-generation"],"created_at":"2024-11-14T05:34:15.218Z","updated_at":"2026-01-28T01:35:11.549Z","avatar_url":"https://github.com/SciPhi-AI.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Reliable, reproducable RAG solution benchmarks\n\nAn open-source benchmarking suite for benchmarking Retrieval-Augmented Generation (RAG) solutions.\n\n## About\nIn the rapidly evolving space of RAG, developers face a critical decision: which tool will best serve their project's needs? We've designed this benchmarking suite to provide a comparative analysis of popular RAG solutions.\n\n## Scalable Ingestion (Updated 7/15/2024)\nOur scalable ingestion test aimed to simulate a high-volume data environment, pushing each solution to its limits. This test provides\ncrucial insights into how these tools might perform in data-intensive, real-world scenarios.\n\nWe leveraged the [HuggingFace Legacy Datasets' Wikipedia corpus](https://huggingface.co/datasets/legacy-datasets/wikipedia) for this\nbenchmark. This dataset offers a diverse range of articles, making it an ideal candidate for testing RAG systems. Each framework was tasked\nwith ingesting articles as per their recommended practices, allowing us to measure their performance under optimal conditions.\n\n\n### Table 1: Ingestion Time over 10008026 Tokens\n| Solution | Time Elapsed (s) |\n|-----------|--------------|\n| [R2R](https://github.com/SciPhi-AI/R2R) | 62.97 |\n| LlamaIndex (Async) | 81.54 |\n| [LlamaIndex](https://github.com/run-llama/llama_index) | 171.93 |\n| [Haystack](https://github.com/deepset-ai/haystack) | 276.27 |\n| [LangChain](https://github.com/langchain-ai/langchain) | 510.04 |\n\n### Figure 1: Tokens Ingested vs. Time\n\nThe steeper the better, as the rate indicates faster ingestion\n\n\u003cimg src=\"./assets/ingestionchart.png\" alt=\"Benchmarks as of 7/15/2024\"\u003e\n\n\n## Individual File Ingestion (Updated 7/15/2024)\nTo complement our bulk ingestion test, we also examined how each solution handles individual file processing. This test simulates typical user interactions, such as uploading single documents or small batches of files.\n\nOur test suite included two text files of similar size but different content, and two PDF files of identical size but varying complexity.\nWe measured three key metrics for each file:\n- Ingestion Time\n- Tokens processed per second (for text files)\n- Megabytes processed per second\n\nAdditionally, we tested combined ingestion of identical file types to assess each solution's performance with multi-file uploads.\n\n### Table 2: Time taken to Ingest Files\n| Metric | Shakespeare | Churchill | Combined txt | University Physics | Introductory Statistics | Combined PDF |\n|---|---|---|---|---|---|---|\n| [**R2R**](https://github.com/SciPhi-AI/R2R) | 7.04 | 9.07 | 9.58 | 8.57 | 14.6 | 20.7 |\n| [**LlamaIndex**](https://github.com/run-llama/llama_index) | 67.0 | 57.7 | 189 | 7.92 | 18.0 | 25.3 |\n| [**Haystack**](https://github.com/deepset-ai/haystack) | 18.8 | 16.8 | 37.8 | 8.17 | 17.0 | 24.9 |\n| [**LangChain**](https://github.com/langchain-ai/langchain) | 65.2 | 64.7 | 134 | 9.31 | 24.7 | 35.2 |\n| [**RagFlow**](https://github.com/infiniflow/ragflow) | 1630 | 3800 | | | | |\n\n### Table 3: Megabytes per Second Throughput\n| Metric | Shakespeare | Churchill | Combined txt | University Physics | Introductory Statistics | Combined PDF |\n|---|---|---|---|---|---|---|\n| [**R2R**](https://github.com/SciPhi-AI/R2R) | 0.767 | 0.628 | 1.159 | 3.127 | 1.833 | 2.593 |\n| [**LlamaIndex**](https://github.com/run-llama/llama_index) | 0.081 | 0.099 | 0.056 | 3.384 | 1.490 | 2.121 |\n| [**Haystack**](https://github.com/deepset-ai/haystack) | 0.288 | 0.338 | 1.416 | 3.280 | 1.573 | 2.156 |\n| [**LangChain**](https://github.com/langchain-ai/langchain) | 0.083 | 0.088 | 0.082 | 2.879 | 1.086 | 1.523 |\n| [**RagFlow**](https://github.com/infiniflow/ragflow) | 0.003 | 0.001 | | | | |\n\n## Reproducing\nWe encourage you to reproduce the benchmarks that we report. Instructions to run benchmarking scripts are located in the source code for each benchmark.\n\n## Contributing\nWe welcome contributions of all sizes! Here's how you can help:\n- New benchmarks: Is there a benchmark that you think would be helpful? Feel free to add it in!\n- New Solutions: Add additional providers.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsciphi-ai%2Frag-performance","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsciphi-ai%2Frag-performance","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsciphi-ai%2Frag-performance/lists"}