{"id":17538627,"url":"https://github.com/oramasearch/chunker","last_synced_at":"2025-04-22T17:31:28.331Z","repository":{"id":222030191,"uuid":"755897271","full_name":"oramasearch/chunker","owner":"oramasearch","description":"Split large texts into chunks with a maximum number of tokens. Split by fixed size or by sentence.","archived":false,"fork":false,"pushed_at":"2024-03-01T05:55:49.000Z","size":11592,"stargazers_count":30,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-16T01:50:42.106Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oramasearch.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-02-11T12:20:38.000Z","updated_at":"2025-04-07T12:49:08.000Z","dependencies_parsed_at":"2024-11-10T01:25:08.583Z","dependency_job_id":"3162a00f-73c1-4a77-9960-4e91197ecaa5","html_url":"https://github.com/oramasearch/chunker","commit_stats":null,"previous_names":["oramasearch/chunker","askorama/chunker"],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oramasearch%2Fchunker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oramasearch%2Fchunker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oramasearch%2Fchunker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oramasearch%2Fchunker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oramasearch","download_url":"https://codeload.github.com/oramasearch/chunker/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250287532,"owners_count":21405633,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-20T21:01:42.895Z","updated_at":"2025-04-22T17:31:27.417Z","avatar_url":"https://github.com/oramasearch.png","language":"TypeScript","funding_links":[],"categories":["others"],"sub_categories":[],"readme":"# Orama Chunker\n\n[![Node.js Tests](https://github.com/askorama/chunker/actions/workflows/nodejs.yml/badge.svg)](https://github.com/askorama/chunker/actions/workflows/nodejs.yml)\n[![Deno Tests](https://github.com/oramasearch/chunker/actions/workflows/deno.yml/badge.svg)](https://github.com/oramasearch/chunker/actions/workflows/deno.yml)\n\nWhen engaging with ChatGPT or other Large Language Models (LLMs), breaking down your input into smaller chunks is a strategy that significantly enhances the interaction experience. This approach is not just about managing the technical constraints of these models, such as input length limitations, but also about improving the quality of the dialogue. By dividing a complex query or detailed discussion into more digestible parts, users can guide the model through the conversation in a step-by-step manner. This method allows for a more nuanced understanding of the context and the specifics of each query, leading to responses that are not only accurate but also highly relevant to the user's needs.\n\nOrama Chunker lets you achieve this by providing a simple yet powerful interface to two different chunking strategies: **fixed chunking**, and **NLP chunking**.\n\n## Installation\n\nYou can install Orama Chunker via `npm`:\n\n```sh\nnpm i @orama/chunker\n```\n\nIf you're using Deno, you can import it via JSR:\n\n```js\nimport { FixedChunker, NLPChunker } from 'jsr:@orama/chunker' \n```\n\n## Usage\n\nOrama Chunker is really easy to use. First of all, you have to decide which chunking strategy you want to adopt.\n\n![[Chunking Strategies](/misc/chunking-strategies.png)](https://raw.githubusercontent.com/askorama/chunker/main/misc/chunking-strategies.png)\n\nThe **Fixed Chunker** will divide your input text into several pieces of a specified size. It does not consider the semantics of your input text, as its sole purpose is to divide the text and ensure that each piece contains a maximum number of tokens. It is slightly faster and lighter as it requires fewer computations to determine the chunking strategy.\n\nOn the other hand, the **NLP Chunker** divides your input into multiple sentences while preserving the overall context of each individual chunk. This method requires a bit more computing power, but it's usually the preferred option.\n\nOnce you have decided which chunking strategy you want to adopt, you can import the `FixedChunker` or the `NLPChunker` into your project:\n\n```js\nimport { FixedChunker } from '@orama/chunker'\n\nconst input = `My large text input...`\nconst maxTokens = 512\n\nconst chunker = new FixedChunker()\nconst result = await chunker.chunk(input, maxTokens)\n```\n\nIn this example, `result` will return an array of strings, where each string contains at most `512` tokens.\n\nThe **NLP Chunker** works in the same way:\n\n```js\nimport { NLPChunker } from '@orama/chunker'\n\nconst input = `My large text input...`\nconst maxTokens = 512\n\nconst chunker = new NLPChunker()\nconst result = await chunker.chunk(input, maxTokens)\n```\n\nThe only difference is that it will return all the sentences found in the text input, where each sentence will have at most `512` tokens.\n\n## License\n\n[Apache 2.0](/LICENSE.md)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foramasearch%2Fchunker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foramasearch%2Fchunker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foramasearch%2Fchunker/lists"}