{"id":27777764,"url":"https://github.com/revelrylabs/text_chunker_ex","last_synced_at":"2026-02-18T23:32:21.783Z","repository":{"id":224772317,"uuid":"760059381","full_name":"revelrylabs/text_chunker_ex","owner":"revelrylabs","description":"A library for semantically coherent text chunking","archived":false,"fork":false,"pushed_at":"2026-02-03T16:45:06.000Z","size":225,"stargazers_count":108,"open_issues_count":5,"forks_count":9,"subscribers_count":14,"default_branch":"main","last_synced_at":"2026-02-04T06:23:50.533Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/revelrylabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":"NOTICE","maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-02-19T17:52:46.000Z","updated_at":"2026-02-03T16:54:22.000Z","dependencies_parsed_at":"2024-04-18T17:01:51.951Z","dependency_job_id":"fb36cc3b-5ccd-4576-a567-8f1e817a3f18","html_url":"https://github.com/revelrylabs/text_chunker_ex","commit_stats":null,"previous_names":["revelrylabs/text_chunker_ex"],"tags_count":12,"template":false,"template_full_name":null,"purl":"pkg:github/revelrylabs/text_chunker_ex","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/revelrylabs%2Ftext_chunker_ex","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/revelrylabs%2Ftext_chunker_ex/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/revelrylabs%2Ftext_chunker_ex/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/revelrylabs%2Ftext_chunker_ex/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/revelrylabs","download_url":"https://codeload.github.com/revelrylabs/text_chunker_ex/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/revelrylabs%2Ftext_chunker_ex/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29598239,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-18T22:25:43.180Z","status":"ssl_error","status_checked_at":"2026-02-18T22:25:42.766Z","response_time":162,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-04-30T07:02:30.039Z","updated_at":"2026-02-18T23:32:21.760Z","avatar_url":"https://github.com/revelrylabs.png","language":"Elixir","funding_links":[],"categories":["RAG \u0026 Embeddings","Vector Databases and RAG","Generative AI"],"sub_categories":["How to Join","LLM Tools"],"readme":"# TextChunker: Flexible Text Chunking for Elixir\n\n## About\nTextChunker is an Elixir library for segmenting large text documents, optimizing them for efficient embedding and storage within vector databases for use in retrieval augmented generation (RAG) applications.\n\nIt prioritizes context preservation and adaptability, and is therefore ideal for analytical, NLP, and other applications where understanding the relationship between text segments is crucial.\n\n## Motivation\n\nFill the gap in the Elixir ecosystem for a good semantic text chunker, and give access to langchain-style chunking.\n\n## Build Status\n\n![tests](https://github.com/revelrylabs/text_chunker_ex/actions/workflows/test.yml/badge.svg)\n\n## Key Features\n\n- Semantic Chunking: Prioritizes chunking text into meaningful blocks based on separators relevant to the specified format (e.g., headings, paragraphs in Markdown).\n- Configurable Chunking: Fine-tune the chunking process with options for, text chunk size, overlap and format.\n- Metadata Tracking: Automatically generates Chunk structs containing byte range information for accurately reassembling the original text if needed.\n- Extensibility: Designed to accommodate additional chunking strategies in the future.\n\n## Installation\n\nAdd TextChunker to your mix.exs:\n\n```elixir\ndef deps do\n  [\n    {:text_chunker, \"~\u003e 0.3.2\"}\n  ]\nend\n```\n\nFetch dependencies:\n\n```\nmix deps.get\n```\n\n## Usage\n\nChunk your text using the `split` function:\n\n```elixir\ntext = \"Your text to be split...\"\n\nchunks = TextChunker.split(text)\n```\n\nThis will chunk up your text using the default parameters - a chunk size of `1000`, chunk overlap of `200`, format of `:plaintext` and using the `RecursiveChunk` strategy.\n\nThe split method returns `Chunks` of your text. These chunks include the start and end bytes of each chunk.\n\n```elixir\n%TextChunker.Chunk{\n    start_byte: 0,\n    end_byte: 44,\n    text: \"This is a sample text. It will be split into\",\n  }\n```\n\n### Options\n\nIf you wish to adjust these parameters, configuration can optionally be passed via a keyword list. \n\n  - `chunk_size` -  The approximate target chunk size, as measured per code points. This means that both `a` and `👻` count as one. Chunks will not exceed this maximum, but may sometimes be smaller. **Important note** This means that graphemes *may* be split. For example, `👩‍🚒` may be split into `👩,🚒` or not depending on the split boundary.\n  - `chunk_overlap` - The contextual overlap between chunks, as measured per code point. Overlap is *not* guaranteed; again this should be treated as a maximum. The size of an individual overlap will depend on the semantics of the text being split.\n  - `format` - What informs separator selection. Because we are trying to preserve meaning between the chunks, the format of the text we are splitting is important. It's important to split newlines in plain text; it's important to split `###` headings in markdown.\n\n```elixir\ntext = \"\"\"\n## Your text to be split\n\nLet's split your text up properly!\n\"\"\"\nopts = [chunk_size: 10, chunk_overlap: 5, format: :markdown]\nchunks = TextChunker.split(text, opts)\n```\n\n### Chunking Strategies\n\nCurrently, we only implement one strategy choice: Recursive Chunk. This was reverse-engineered from LangChain, with plans to add more methods in the future. \n\n#### Recursive Chunk (current default)\n\nYou can use Recursive Chunk to split text up into any chunk size you wish, with or without overlap. It is important to note that this overlap is not guaranteed - rather, if the overlap makes sense, this is the max length for that overlap. Recursive Chunk prioritizes keeping the semantics intact (as defined by the separators derived from the input format). The overlap does not occur when such an overlap would break those semantics. See below for examples.\n\n## Examples\n\n```elixir\ntext = \"This is a sample text. It will be split into properly-sized chunks using the TextChunker library.\"\n\niex(10)\u003e TextChunker.split(text)\n\n[\n  %TextChunker.Chunk{\n    start_byte: 0,\n    end_byte: 97,\n    text: \"This is a sample text. It will be split into properly-sized chunks using the TextChunker library.\"\n  }\n]\n\ntext = \"This is a sample text. It will be split into properly-sized chunks using the TextChunker library.\"\nopts = [chunk_size: 50, chunk_overlap: 5, format: :plaintext, strategy: TextChunker.Strategies.RecursiveChunk]\n\niex(10)\u003e TextChunker.split(text, opts)\n\n[\n  %TextChunker.Chunk{\n    start_byte: 0,\n    end_byte: 44,\n    text: \"This is a sample text. It will be split into\"\n  },\n  %TextChunker.Chunk{\n    start_byte: 39,\n    end_byte: 88,\n    text: \" into properly-sized chunks using the TextChunker\"\n  },\n  %TextChunker.Chunk{\n    start_byte: 88,\n    end_byte: 97,\n    text: \" library.\"\n  }\n]\n```\n\n## Contributing and Development\n\nBug reports and pull requests are welcome on GitHub at https://github.com/revelrylabs/text_chunker_ex. Check out the [contributing guidelines](CONTRIBUTING.md) for more info.\n\nEveryone is welcome to participate in the project. We expect contributors to adhere to the [Contributor Covenant Code of Conduct](CODE_OF_CONDUCT.md).\n\n## Releases\n\nSee [RELEASES.md](RELEASES.md) for details about the release process.\n\n\n## Acknowledgments\n\nSpecial thanks to the creators of Langchain for their initial approach to recursive text splitting, which inspired this library. See the [NOTICE](NOTICE) file for details.\n\n\n## License\n\nTextChunker is released under the MIT License. See the [LICENSE](LICENSE) file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frevelrylabs%2Ftext_chunker_ex","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frevelrylabs%2Ftext_chunker_ex","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frevelrylabs%2Ftext_chunker_ex/lists"}