{"id":22701361,"url":"https://github.com/chonkie-ai/chonkie","last_synced_at":"2025-05-14T03:09:21.763Z","repository":{"id":262082605,"uuid":"881768979","full_name":"chonkie-ai/chonkie","owner":"chonkie-ai","description":"🦛 CHONK your texts with Chonkie ✨ - The no-nonsense RAG chunking library","archived":false,"fork":false,"pushed_at":"2025-03-18T00:27:35.000Z","size":1371,"stargazers_count":2803,"open_issues_count":21,"forks_count":121,"subscribers_count":21,"default_branch":"main","last_synced_at":"2025-03-18T01:27:15.510Z","etag":null,"topics":["ai","chunking","etl","nlp","python","rag","retrieval","semantic-segmentation","text-chunking","text-processing","text-splitting","vector-search"],"latest_commit_sha":null,"homepage":"https://docs.chonkie.ai","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/chonkie-ai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-11-01T07:26:05.000Z","updated_at":"2025-03-17T21:41:15.000Z","dependencies_parsed_at":"2024-11-10T12:37:37.657Z","dependency_job_id":"ca6f0572-4de3-402c-8396-403f59f61efd","html_url":"https://github.com/chonkie-ai/chonkie","commit_stats":{"total_commits":325,"total_committers":8,"mean_commits":40.625,"dds":0.09230769230769231,"last_synced_commit":"9389a4b288bf653e04825eeb2d7add33ccd8c476"},"previous_names":["bhavnicksm/chonkie","chonkie-ai/chonkie"],"tags_count":21,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chonkie-ai%2Fchonkie","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chonkie-ai%2Fchonkie/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chonkie-ai%2Fchonkie/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chonkie-ai%2Fchonkie/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/chonkie-ai","download_url":"https://codeload.github.com/chonkie-ai/chonkie/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245907524,"owners_count":20691953,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","chunking","etl","nlp","python","rag","retrieval","semantic-segmentation","text-chunking","text-processing","text-splitting","vector-search"],"created_at":"2024-12-10T07:01:43.302Z","updated_at":"2025-03-27T19:05:25.297Z","avatar_url":"https://github.com/chonkie-ai.png","language":"Python","funding_links":[],"categories":["Chunking \u0026 Document Processing","Python","LLM Application / RAG"],"sub_categories":["Frameworks"],"readme":"\u003cdiv align='center'\u003e\n\n![Chonkie Logo](/assets/chonkie_logo_br_transparent_bg.png)\n\n# 🦛 Chonkie ✨\n\n[![PyPI version](https://img.shields.io/pypi/v/chonkie.svg)](https://pypi.org/project/chonkie/)\n[![License](https://img.shields.io/github/license/bhavnicksm/chonkie.svg)](https://github.com/bhavnicksm/chonkie/blob/main/LICENSE)\n[![Documentation](https://img.shields.io/badge/docs-chonkie.ai-blue.svg)](https://docs.chonkie.ai)\n![Package size](https://img.shields.io/badge/size-15MB-blue)\n[![Downloads](https://static.pepy.tech/badge/chonkie)](https://pepy.tech/project/chonkie)\n[![Discord](https://dcbadge.limes.pink/api/server/https://discord.gg/rYYp6DC4cv?style=flat)](https://discord.gg/rYYp6DC4cv)\n[![GitHub stars](https://img.shields.io/github/stars/bhavnicksm/chonkie.svg)](https://github.com/bhavnicksm/chonkie/stargazers)\n\n_The no-nonsense RAG chunking library that's lightweight, lightning-fast, and ready to CHONK your texts_\n\n[Installation](#installation) •\n[Usage](#usage) •\n[Supported Methods](#supported-methods) •\n[Benchmarks](#benchmarks-️) •\n[Documentation](https://docs.chonkie.ai) •\n[Contributing](#contributing)\n\n\u003c/div\u003e\n\nEver found yourself building yet another RAG bot (your 2,342,148th one), only to hit that all-too-familiar wall? You know the one —— where you're stuck choosing between:\n\n- Library X: A behemoth that takes forever to install and probably includes three different kitchen sinks\n- Library Y: So bare-bones it might as well be a \"Hello World\" program\n- Writing it yourself? For the 2,342,149th time, _sigh_\n\nAnd you think to yourself:\n\n\u003e \"WHY CAN'T THIS JUST BE SIMPLE?!\" \u003c/br\u003e\n\u003e \"Why do I need to choose between bloated and bare-bones?\" \u003c/br\u003e\n\u003e \"Why can't I just install, import, and CHONK?!\" \u003c/br\u003e\n\nWell, look no further than Chonkie! (chonkie boi is a gud boi 🦛💕)\n\n**🚀 Feature-rich**: All the CHONKs you'd ever need \u003c/br\u003e\n**✨ Easy to use**: Install, Import, CHONK \u003c/br\u003e\n**⚡ Fast**: CHONK at the speed of light! zooooom \u003c/br\u003e\n**🌐 Wide support**: Supports all your favorite tokenizer CHONKS \u003c/br\u003e\n**🪶 Light-weight**: No bloat, just CHONK \u003c/br\u003e\n**🦛 Cute CHONK mascot**: psst it's a pygmy hippo btw \u003c/br\u003e\n**❤️ [Moto Moto](#acknowledgements)'s favorite python library** \u003c/br\u003e\n\n**Chonkie** is a chunking library that \"**just works™**\".\n\n# Installation\n\nTo install chonkie, simply run:\n\n```bash\npip install chonkie\n```\n\nChonkie follows the rule to have minimal default installs, read the [DOCS](https://docs.chonkie.ai) to know the installation for your required chunker, or simply install `all` if you don't want to think about it (not recommended).\n\n```bash\npip install chonkie[all]\n```\n\n# Usage\n\nHere's a basic example to get you started:\n\n```python\n# First import the chunker you want from Chonkie \nfrom chonkie import TokenChunker\n\n# Import your favorite tokenizer library\n# Also supports AutoTokenizers, TikToken and AutoTikTokenizer\nfrom tokenizers import Tokenizer \ntokenizer = Tokenizer.from_pretrained(\"gpt2\")\n\n# Initialize the chunker\nchunker = TokenChunker(tokenizer)\n\n# Chunk some text\nchunks = chunker(\"Woah! Chonkie, the chunking library is so cool! I love the tiny hippo hehe.\")\n\n# Access chunks\nfor chunk in chunks:\n    print(f\"Chunk: {chunk.text}\")\n    print(f\"Tokens: {chunk.token_count}\")\n```\n\nMore example usages given inside the [DOCS](https://docs.chonkie.ai)\n\n# Supported Methods\n\nChonkie provides several chunkers to help you split your text efficiently for RAG applications. Here's a quick overview of the available chunkers:\n\n- **TokenChunker**: Splits text into fixed-size token chunks.\n- **WordChunker**: Splits text into chunks based on words.\n- **SentenceChunker**: Splits text into chunks based on sentences.\n- **RecursiveChunker**: Splits text hierarchically using customizable rules to create semantically meaningful chunks.\n- **SemanticChunker**: Splits text into chunks based on semantic similarity.\n- **SDPMChunker**: Splits text using a Semantic Double-Pass Merge approach.\n- **LateChunker (experimental)**: Embeds text and then splits it to have better chunk embeddings.\n\nMore on these methods and the approaches taken inside the [DOCS](https://docs.chonkie.ai)\n\n# Benchmarks 🏃‍♂️\n\n\u003e \"I may be smol hippo, but I pack a punch!\" 🦛\n\nHere's a quick peek at how Chonkie performs:\n\n**Size**📦\n\n- **Default Install:** 15MB (vs 80-171MB for alternatives)\n- **With Semantic:** Still lighter than the competition!\n\n**Speed**⚡\n\n- **Token Chunking:** 33x faster than the slowest alternative\n- **Sentence Chunking:** Almost 2x faster than competitors\n- **Semantic Chunking:** Up to 2.5x faster than others\n\nCheck out our detailed [benchmarks](https://docs.chonkie.ai/benchmarks) to see how Chonkie races past the competition! 🏃‍♂️💨\n\n# Contributing\n\nWant to help make Chonkie even better? Check out our [CONTRIBUTING.md](CONTRIBUTING.md) guide! Whether you're fixing bugs, adding features, or improving docs, every contribution helps make Chonkie a better CHONK for everyone.\n\nRemember: No contribution is too small for this tiny hippo! 🦛\n\n# Acknowledgements\n\nChonkie would like to CHONK its way through a special thanks to all the users and contributors who have helped make this library what it is today! Your feedback, issue reports, and improvements have helped make Chonkie the CHONKIEST it can be.\n\nAnd of course, special thanks to [Moto Moto](https://www.youtube.com/watch?v=I0zZC4wtqDQ\u0026t=5s) for endorsing Chonkie with his famous quote:\n\u003e \"I like them big, I like them chonkie.\"\n\u003e                                         ~ Moto Moto\n\n\n# Citation\n\nIf you use Chonkie in your research, please cite it as follows:\n\n```bibtex\n@misc{chonkie2024,\n  author = {Minhas, Bhavnick AND Nigam, Shreyash},\n  title = {Chonkie: A Fast Feature-full Chunking Library for RAG Bots},\n  year = {2024},\n  publisher = {GitHub},\n  howpublished = {\\url{https://github.com/bhavnick/chonkie}},\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchonkie-ai%2Fchonkie","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchonkie-ai%2Fchonkie","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchonkie-ai%2Fchonkie/lists"}