{"id":22773746,"url":"https://github.com/moekiorg/baran","last_synced_at":"2025-04-06T21:15:15.211Z","repository":{"id":170187328,"uuid":"646313748","full_name":"moekiorg/baran","owner":"moekiorg","description":"Text Splitter for Large Language Model (LLM) datasets.","archived":false,"fork":false,"pushed_at":"2024-12-09T05:07:37.000Z","size":52,"stargazers_count":81,"open_issues_count":3,"forks_count":9,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-30T19:09:35.820Z","etag":null,"topics":["ai","gem","llm","markdown","ruby"],"latest_commit_sha":null,"homepage":"https://rubygems.org/gems/baran","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/moekiorg.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-05-28T01:33:12.000Z","updated_at":"2025-03-25T08:44:29.000Z","dependencies_parsed_at":null,"dependency_job_id":"41c499cb-3487-4387-a30e-8ac5ae146456","html_url":"https://github.com/moekiorg/baran","commit_stats":null,"previous_names":["moekidev/baran","kawakamimoeki/baran","hackluckcat/baran"],"tags_count":15,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/moekiorg%2Fbaran","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/moekiorg%2Fbaran/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/moekiorg%2Fbaran/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/moekiorg%2Fbaran/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/moekiorg","download_url":"https://codeload.github.com/moekiorg/baran/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247550690,"owners_count":20956987,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","gem","llm","markdown","ruby"],"created_at":"2024-12-11T17:59:25.158Z","updated_at":"2025-04-06T21:15:15.179Z","avatar_url":"https://github.com/moekiorg.png","language":"Ruby","funding_links":[],"categories":["Ruby"],"sub_categories":[],"readme":"# Baran\n\n![v](https://badgen.net/rubygems/v/baran)\n![dt](https://badgen.net/rubygems/dt/baran)\n![license](https://badgen.net/github/license/kawakamimoeki/baran)\n\nText Splitter for Large Language Model datasets.\n\nTo avoid token constraints and improve the accuracy of vector search in the Large Language Model, it is necessary to divide the document. This gem supports splitting the text in the specified manner.\n\n## Installation\n\nInstall the gem and add to the application's Gemfile by executing:\n\n    $ bundle add baran\n\nIf bundler is not being used to manage dependencies, install the gem by executing:\n\n    $ gem install baran\n\n## Usage\n\n### Default Paramters\n\n- `chunk_size`: 1024 (characters)\n- `chunk_overlap`: 64 (characters)\n\n### Character Text Splitter\n\nSplitting by the specified character.\n\n```ruby\nsplitter = Baran::CharacterTextSplitter.new(\n    chunk_size: 1024,\n    chunk_overlap: 64,\n    separator: \"\\n\\n\"\n)\nsplitter.chunks(text, metadata: { ... })\n# =\u003e [{ cursor: 0, text: \"...\", metadata: { ... } }, ...]\n```\n\n### Recursive Character Text Splitter\n\nSplitting by the specified characters recursively.\n\n```ruby\nsplitter = Baran::RecursiveCharacterTextSplitter.new(\n    separators: [\"\\n\\n\", \"\\n\", \" \", \"\"]\n)\nsplitter.chunks(text, metadata: { ... })\n# =\u003e [{ cursor: 0, text: \"...\", metadata: { ... } }, ...]\n```\n\n### Markdown Text Splitter\n\nSplitting by the Markdown descriptions.\n\n```ruby\nsplitter = Baran::MarkdownSplitter.new\nsplitter.chunks(markdown, metadata: { ... })\n# =\u003e [{ cursor: 0, text: \"...\", metadata: { ... } }, ...]\n```\n\nSplit with the following priority.\n\n```ruby\n[\n    \"\\n# \", # h1\n    \"\\n## \", # h2\n    \"\\n### \", # h3\n    \"\\n#### \", # h4\n    \"\\n##### \", # h5\n    \"\\n###### \", # h6\n    \"```\\n\\n\", # code block\n    \"\\n\\n***\\n\\n\", # horizontal rule\n    \"\\n\\n---\\n\\n\", # horizontal rule\n    \"\\n\\n___\\n\\n\", # horizontal rule\n    \"\\n\\n\", # new line\n    \"\\n\", # new line\n    \" \", # space\n    \"\" # empty\n]\n```\n\n## Development\n\nAfter checking out the repo, run `bin/setup` to install dependencies. Then, run `bundle exec rake` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.\n\n## Contributing\n\nBug reports and pull requests are welcome on GitHub at https://github.com/kawakamimoeki/baran. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [code of conduct](https://github.com/kawakamimoeki/baran/blob/main/CODE_OF_CONDUCT.md).\n\n## License\n\nThe gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).\n\n## Code of Conduct\n\nEveryone interacting in the Baran project's codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/kawakamimoeki/baran/blob/main/CODE_OF_CONDUCT.md).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmoekiorg%2Fbaran","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmoekiorg%2Fbaran","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmoekiorg%2Fbaran/lists"}