{"id":16872728,"url":"https://github.com/schneiderfelipe/chat-splitter","last_synced_at":"2025-06-20T00:04:17.856Z","repository":{"id":181252667,"uuid":"665227740","full_name":"schneiderfelipe/chat-splitter","owner":"schneiderfelipe","description":"Split chat messages by maximum chat completion token count","archived":false,"fork":false,"pushed_at":"2024-04-20T23:48:52.000Z","size":22,"stargazers_count":3,"open_issues_count":3,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-31T04:12:05.121Z","etag":null,"topics":["ai","artificial-intelligence","chat","chatgpt","gpt-4","nlp","openai","split","text","tiktoken","tokenizer"],"latest_commit_sha":null,"homepage":"https://schneiderfelipe.github.io/posts/chat-splitter-first-release/","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/schneiderfelipe.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-07-11T18:17:45.000Z","updated_at":"2024-09-10T14:33:47.000Z","dependencies_parsed_at":null,"dependency_job_id":"f7371ffb-4623-4559-81a8-fcb8d57dd210","html_url":"https://github.com/schneiderfelipe/chat-splitter","commit_stats":{"total_commits":17,"total_committers":3,"mean_commits":5.666666666666667,"dds":0.2941176470588235,"last_synced_commit":"36ec9bd25a3607e05731bae7baaf68a1696adb80"},"previous_names":["schneiderfelipe/chat-splitter"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/schneiderfelipe/chat-splitter","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/schneiderfelipe%2Fchat-splitter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/schneiderfelipe%2Fchat-splitter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/schneiderfelipe%2Fchat-splitter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/schneiderfelipe%2Fchat-splitter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/schneiderfelipe","download_url":"https://codeload.github.com/schneiderfelipe/chat-splitter/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/schneiderfelipe%2Fchat-splitter/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260852095,"owners_count":23072587,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","artificial-intelligence","chat","chatgpt","gpt-4","nlp","openai","split","text","tiktoken","tokenizer"],"created_at":"2024-10-13T15:15:54.876Z","updated_at":"2025-06-20T00:04:12.837Z","avatar_url":"https://github.com/schneiderfelipe.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# chat-splitter\n\n[![Build Status]][actions]\n[![Latest Version]][crates.io]\n[![Documentation]][docs.rs]\n\n[Build Status]: https://github.com/schneiderfelipe/chat-splitter/actions/workflows/rust.yml/badge.svg\n[actions]: https://github.com/schneiderfelipe/chat-splitter/actions/workflows/rust.yml\n[Latest Version]: https://img.shields.io/crates/v/chat_splitter.svg\n[crates.io]: https://crates.io/crates/chat_splitter\n[Documentation]: https://img.shields.io/docsrs/chat-splitter\n[docs.rs]: https://docs.rs/chat-splitter\n\n\u003e For more information,\n\u003e please refer to the [blog announcement](https://schneiderfelipe.github.io/posts/chat-splitter-first-release/).\n\nWhen utilizing the [`async_openai`](https://github.com/64bit/async-openai) [Rust](https://www.rust-lang.org/) crate,\nit is crucial to ensure that you do not exceed\nthe [maximum number of tokens](https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them) specified by [OpenAI](https://openai.com/)'s [chat models](https://platform.openai.com/docs/api-reference/chat).\n\n[`chat-splitter`](https://crates.io/crates/chat_splitter) categorizes chat messages into 'outdated' and 'recent' messages,\nallowing you to split them based on both the maximum\nmessage count and the maximum chat completion token count.\nThe token counting functionality is provided by\n[`tiktoken_rs`](https://github.com/zurawiki/tiktoken-rs).\n\n## Usage\n\nHere's a basic example:\n\n```rust\n// Get all your previously stored chat messages...\nlet mut stored_messages = /* get_stored_messages()? */;\n\n// ...and split into 'outdated' and 'recent',\n// where 'recent' always fits the context size.\nlet (outdated_messages, recent_messages) =\n    ChatSplitter::default().split(\u0026stored_messages);\n```\n\nFor a more detailed example,\nsee [`examples/chat.rs`](https://github.com/schneiderfelipe/chat-splitter/blob/main/examples/chat.rs).\n\n## Contributing\n\nContributions to `chat-splitter` are welcome!\nIf you find a bug or have a feature request,\nplease [submit an issue](https://github.com/schneiderfelipe/chat-splitter/issues).\nIf you'd like to contribute code,\nplease feel free to [submit a pull request](https://github.com/schneiderfelipe/chat-splitter/pulls).\n\nLicense: MIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fschneiderfelipe%2Fchat-splitter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fschneiderfelipe%2Fchat-splitter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fschneiderfelipe%2Fchat-splitter/lists"}