{"id":49998406,"url":"https://github.com/shantanu-deshmukh/chunktuner","last_synced_at":"2026-07-03T15:36:27.564Z","repository":{"id":355187848,"uuid":"1227063460","full_name":"shantanu-deshmukh/chunktuner","owner":"shantanu-deshmukh","description":"Benchmark chunking strategies for your RAG corpus. Get a recommended config. CLI, Python library, and MCP server.","archived":false,"fork":false,"pushed_at":"2026-05-21T01:25:11.000Z","size":2682,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-21T07:55:37.238Z","etag":null,"topics":["chunking","embedding","evaluation","langchain","litellm","llamaindex","llm","mcp","optimization","rag","ragas","retrieval","text-splitting","vector-database"],"latest_commit_sha":null,"homepage":"https://shantanu-deshmukh.github.io/chunktuner/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/shantanu-deshmukh.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-02T06:35:01.000Z","updated_at":"2026-05-21T01:25:15.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/shantanu-deshmukh/chunktuner","commit_stats":null,"previous_names":["shantanu-deshmukh/chunk-tuner"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/shantanu-deshmukh/chunktuner","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shantanu-deshmukh%2Fchunktuner","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shantanu-deshmukh%2Fchunktuner/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shantanu-deshmukh%2Fchunktuner/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shantanu-deshmukh%2Fchunktuner/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/shantanu-deshmukh","download_url":"https://codeload.github.com/shantanu-deshmukh/chunktuner/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shantanu-deshmukh%2Fchunktuner/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35092210,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-03T02:00:05.635Z","response_time":110,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chunking","embedding","evaluation","langchain","litellm","llamaindex","llm","mcp","optimization","rag","ragas","retrieval","text-splitting","vector-database"],"created_at":"2026-05-19T09:00:24.684Z","updated_at":"2026-07-03T15:36:27.556Z","avatar_url":"https://github.com/shantanu-deshmukh.png","language":"Python","funding_links":[],"categories":["Agents","Uncategorized"],"sub_categories":["Research \u0026 Knowledge Retrieval","Uncategorized"],"readme":"# chunktuner\n\n[![PyPI version](https://img.shields.io/pypi/v/chunktuner.svg)](https://pypi.org/project/chunktuner/)\n[![Python versions](https://img.shields.io/pypi/pyversions/chunktuner.svg)](https://pypi.org/project/chunktuner/)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)\n[![CI](https://github.com/shantanu-deshmukh/chunktuner/actions/workflows/ci.yml/badge.svg)](https://github.com/shantanu-deshmukh/chunktuner/actions/workflows/ci.yml)\n[![Docs](https://img.shields.io/badge/docs-GitHub%20Pages-blue)](https://shantanu-deshmukh.github.io/chunktuner/)\n\nAuto chunking tuner and MCP server for RAG pipelines.\n\n**Give it your documents.** It tries multiple chunking strategies, measures which setup supports retrieval best, and recommends a configuration for your corpus and use case. Zero API cost to start — run `estimate` for a dry-run before any paid calls.\n\nFull documentation: [shantanu-deshmukh.github.io/chunktuner](https://shantanu-deshmukh.github.io/chunktuner/)\n\n```mermaid\nflowchart TD\n    Lib[\"Python library\"] --\u003e Ingest\n    CLI[\"CLI (chunk-tune)\"] --\u003e Ingest\n    MCP[\"MCP server\"] --\u003e Ingest\n\n    Ingest[\"Ingest your documents\u003cbr/\u003efiles, URLs, repos\"] --\u003e Tune\n\n    subgraph Tune [\"AutoTuner: for every strategy and param set\"]\n        direction LR\n        Chunk[\"Chunk document\"] --\u003e Embed[\"Embed chunks\u003cbr/\u003eand queries\"] --\u003e Score[\"Score retrieval\u003cbr/\u003erecall, MRR, NDCG\"]\n    end\n\n    Tune --\u003e Rank[\"Rank all configs\u003cbr/\u003eagainst baseline\"] --\u003e Best([\"Recommended config\u003cbr/\u003e.autochunk.yaml\"])\n```\n\n---\n\n## What it does\n\nWhen building a RAG pipeline, how you split documents into chunks directly impacts retrieval quality. `chunktuner` automates the process of finding the optimal chunking strategy for your specific corpus, embedding model, and use case.\n\nIt benchmarks strategies like fixed-token windows, recursive character splitting, semantic splitting, PDF structural chunking, and AST-based code chunking — then scores each one against real retrieval metrics (token recall, MRR, NDCG) and optional generation metrics (RAGAS faithfulness, answer relevancy).\n\n---\n\n## Interfaces\n\n- **Python library** — programmatic integration into your pipeline\n- **CLI** (`chunk-tune`) — human-driven tuning from the terminal\n- **MCP server** — use directly from Claude Desktop or any MCP host\n\n---\n\n## Quickstart\n\n```bash\n# Install (pick one)\nuv tool install chunktuner\npip install chunktuner\n\n# Initialize workspace (embedding_model defaults to null — no API calls)\nchunk-tune init\n\n# See cost estimate before running anything\nchunk-tune estimate ./my_docs --use-case rag_qa\n\n# Get a recommendation (dummy embeddings by default; add --embedding-model for real ones)\nchunk-tune recommend ./my_docs --use-case rag_qa\n```\n\n**Python API:**\n\n```python\nfrom pathlib import Path\nfrom chunktuner import FileIngestor, DummyEmbeddingFunction, LiteLLMEmbeddingFunction, AutoTuner\nfrom chunktuner import default_registry, Evaluator, ScoreCalculator\n\ndocs = FileIngestor().ingest_dir(Path(\"./my_docs\"))\n\n# Free/offline: use dummy embeddings for quick strategy comparison.\n# Swap in LiteLLMEmbeddingFunction for real embeddings with any provider:\n#   LiteLLMEmbeddingFunction(\"text-embedding-3-small\")          # OpenAI\n#   LiteLLMEmbeddingFunction(\"gemini/gemini-embedding-001\")     # Google\n#   LiteLLMEmbeddingFunction(\"openai/\u003cid\u003e\", api_base=\"http://localhost:1234/v1\")  # local\nembedding_fn = DummyEmbeddingFunction()\n\ntuner = AutoTuner(\n    strategies=default_registry,\n    evaluator=Evaluator(embedding_fn),\n    scorer=ScoreCalculator(use_case=\"rag_qa\"),\n)\nresult = tuner.recommend(docs, use_case=\"rag_qa\")\nprint(result.best.config)\n```\n\n---\n\n## Example output\n\nAfter running `recommend`, you get a ranked table with the winning config and how much it beats the baseline:\n\n```\n  Rank   Strategy              Params                  Score   Recall   MRR    IOU   AvgTok\n ────────────────────────────────────────────────────────────────────────────────────────\n   1 ★   recursive_character   1024 chr / 154 ov        0.821    0.950  0.880  0.062      212\n     2   fixed_tokens          512 tok / 51 ov           0.764    0.920  0.840  0.059      444\n   ...\n  Baseline  fixed_tokens  512 tok / 0 ov  →  score 0.682\n  Winner beats baseline by +0.139  (+20.4%)\n```\n\n---\n\n## Real-world example\n\nSee [examples/financial_analysis](examples/financial_analysis/README.md) for a full benchmark on S\u0026P 500 earnings call transcripts — a corpus where separator choice and chunk size make a measurable difference in retrieval quality.\n\nRun it offline with zero API cost:\n\n```bash\ncd examples/financial_analysis\nuv sync\nuv run python run_benchmark.py --fixture --num-transcripts 2\n```\n\n---\n\n## Supported strategies\n\n| Strategy              | Best for                              |\n| --------------------- | ------------------------------------- |\n| `fixed_tokens`        | Baseline; uniform token windows       |\n| `recursive_character` | General prose and documentation       |\n| `semantic`            | Theme-heavy articles                  |\n| `markdown_semantic`   | Structured Markdown docs              |\n| `pdf_structural`      | PDFs with layout regions and tables   |\n| `structural_semantic` | PDF/DOCX with mixed layout and text   |\n| `late_chunking`       | Long docs with dense cross-references |\n| `agentic`             | High-value narrative documents        |\n| `code_ast`            | Code repos (Python, JavaScript)       |\n| `code_window`         | Code baseline (sliding window)        |\n\n---\n\n## MCP server (Claude Desktop)\n\nPython **FastMCP** (`chunk-tune-mcp`, stdio). No Node.js build. See [docs/mcp_setup.md](docs/mcp_setup.md).\n\nAdd to your `.mcp.json`:\n\n```json\n{\n  \"mcpServers\": {\n    \"chunktuner\": {\n      \"command\": \"uvx\",\n      \"args\": [\"--from\", \"chunktuner[mcp]\", \"chunk-tune-mcp\"],\n      \"env\": {\n        \"CHUNK_TUNER_BASE_DIR\": \"/path/to/your/corpus\"\n      }\n    }\n  }\n}\n```\n\nTools available: `list_strategies`, `preview_chunks`, `evaluate_chunking`, `recommend_config`.\n\n---\n\n## CLI reference\n\n```\nchunk-tune init       Bootstrap workspace config\nchunk-tune analyze    Quick structural scan (no API cost)\nchunk-tune estimate   Dry-run cost/token estimate\nchunk-tune evaluate   Full evaluation across strategies\nchunk-tune recommend  Evaluation + best config recommendation\nchunk-tune compare    Side-by-side comparison of specific strategies\nchunk-tune preview    Inspect how a strategy splits a document\nchunk-tune cache      Manage embedding and chunk cache\n```\n\n---\n\n## Installation options\n\n```bash\npip install chunktuner                 # CLI + library\nuv add chunktuner                      # library\nuv tool install chunktuner             # global CLI\nuvx --from chunktuner chunk-tune …     # ephemeral CLI (no install)\n\n# With optional extras\npip install \"chunktuner[docling]\"      # PDF/DOCX support\nuv add \"chunktuner[docling]\"           # PDF/DOCX support\nuv add \"chunktuner[ragas]\"             # generation metrics\nuv add \"chunktuner[semantic]\"          # semantic chunking\nuv add \"chunktuner[code]\"              # AST code chunking\nuv add \"chunktuner[all]\"               # everything\n```\n\n---\n\n## Contributing\n\nSee [CONTRIBUTING.md](CONTRIBUTING.md).\n\n---\n\n## Author\n\n[Shantanu Deshmukh](https://shantanudeshmukh.com) — full stack developer building E2E AI applications.\n\n[LinkedIn](https://www.linkedin.com/in/shantanud/) / [Twitter](https://twitter.com/askshantanu)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshantanu-deshmukh%2Fchunktuner","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshantanu-deshmukh%2Fchunktuner","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshantanu-deshmukh%2Fchunktuner/lists"}