{"id":49499540,"url":"https://github.com/sergeivaskov/logs-tokenizer","last_synced_at":"2026-05-01T12:01:00.405Z","repository":{"id":352645987,"uuid":"1216016107","full_name":"sergeivaskov/logs-tokenizer","owner":"sergeivaskov","description":"Logs Tokenizer is a utility for automatically compressing logs by up to 80% upon insertion to save tokens when working with LLM models.","archived":false,"fork":false,"pushed_at":"2026-05-01T11:27:35.000Z","size":219,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-01T11:32:59.070Z","etag":null,"topics":["cli","compression","compressor","llm","productivity","prompt","prompt-engineering","prompt-toolkit","rust"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sergeivaskov.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-20T13:34:20.000Z","updated_at":"2026-05-01T11:26:37.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/sergeivaskov/logs-tokenizer","commit_stats":null,"previous_names":["sergeivaskov/logs-tokenizer"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/sergeivaskov/logs-tokenizer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sergeivaskov%2Flogs-tokenizer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sergeivaskov%2Flogs-tokenizer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sergeivaskov%2Flogs-tokenizer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sergeivaskov%2Flogs-tokenizer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sergeivaskov","download_url":"https://codeload.github.com/sergeivaskov/logs-tokenizer/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sergeivaskov%2Flogs-tokenizer/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32495949,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-30T13:12:12.517Z","status":"online","status_checked_at":"2026-05-01T02:00:05.856Z","response_time":64,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","compression","compressor","llm","productivity","prompt","prompt-engineering","prompt-toolkit","rust"],"created_at":"2026-05-01T12:00:32.387Z","updated_at":"2026-05-01T12:01:00.397Z","avatar_url":"https://github.com/sergeivaskov.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Logs Tokenizer 🗜️\n\n[📥 Download for Windows](https://github.com/sergeivaskov/logs-tokenizer/releases/latest/download/Logs.Tokenizer.exe)\n\n[📥 Download for Mac (Apple Silicon)](https://github.com/sergeivaskov/logs-tokenizer/releases/latest/download/Logs.Tokenizer.macos-arm64.dmg)[(Intel)](https://github.com/sergeivaskov/logs-tokenizer/releases/latest/download/Logs.Tokenizer.macos-x64.dmg)\n\n**Logs Tokenizer** is a smart background utility for developers, coders, and DevOps engineers that extremely compresses logs directly in the clipboard, keeping them in a readable text format.\n\n## ⚠️ Problem\n\nYou need to save tokens when interacting with AI, especially with Claude Code, but you don't want to sacrifice volume. Logs contain a large amount of duplicated information that can be factored out.\n\n## 💡 Solution\n\nYou simply copy logs, press `Ctrl+Alt+V`, and the utility instantly compresses the text up to 80% in the clipboard, replacing repeating patterns with short tokens and automatically pastes the result, which remains completely understandable for neural networks. Additionally, this structure allows you to highlight the problem for the agent without unnecessary noise and context window overflow.\n\n---\n\n## 🔍 Example\n\n### Log lines before compression (Original)\n\n```text\n2026-04-20T19:58:52.970 [StageDiag] \u003e handleTokenTracking vk=70 keyDown=0\n2026-04-20T19:58:52.970 [TTTrack] htt: enter vk=70 keyDown=0\n2026-04-20T19:58:52.970 [TTTrack] htt: after modifier check\n2026-04-20T19:58:52.970 [TTTrack] htt: KeyUp branch, checking cachedSettings\n2026-04-20T19:58:52.970 [TTTrack] htt: KeyUp autoReplEnabled=1 isProcessing=0\n2026-04-20T19:58:52.970 [TTTrack] htt: KeyUp postRepl=0\n2026-04-20T19:58:52.971 [TTTrack] htt: KeyUp hasPlannedTargetLayout=0\n2026-04-20T19:58:52.971 [TTTrack] htt: KeyUp branch returning false\n2026-04-20T19:58:53.017 [HookDiag] LowLevelKeyboardProc: vk=32 keyDown=1 injected=0 extraInfo=0 ctrlLogical=0 ctrlPhysical=0 heldMask=0x0 shouldTrack=1 skipInjected=0\n2026-04-20T19:58:53.018 [StageDiag] \u003e entry vk=32 keyDown=1\n2026-04-20T19:58:53.018 [StageDiag] \u003e drainPendingReset vk=32 keyDown=1\n2026-04-20T19:58:53.018 [StageDiag] \u003e shouldBlockAllFeatures vk=32 keyDown=1\n2026-04-20T19:58:53.018 [StageDiag] \u003e executePendingReplacementIfAny vk=32 keyDown=1\n2026-04-20T19:58:53.019 [StageDiag] \u003e hotkeyManager.processKeyEvent vk=32 keyDown=1\n```\n\n### Log lines after compression (Logs Tokenizer)\n\n```text\n!16!!t!83!2!\n\u00261f:!A!\n\u00261f:!F!\n\u00261f:!D!\n\u00261f:!B!\n\u00261f:!w!\n\u00261f:!y!\n\u00261u:!z!\n\u00261u:!E!\n\u00261u:!C!\n\u00261V:!G!\n\u00261V:!P!\n\u00261V:!Q!\n\u00261V:!J!\n!4!209 !v!\n```\n\n*At the beginning of the compressed version, a compact Legend (dictionary) is added, and the log text itself is radically shortened.*\n\n---\n\n## ✨ Key Features\n\n- **Semantic log compression:** The algorithm understands log structure. It automatically finds timestamps, `[App]` components, `id=` keys, and trims leading zeros in hex addresses.\n- **Macro templating:** A unique feature that finds machine-generated patterns (e.g., `User {ID} logged in`) and collapses them into short macros with parameters.\n- **Inline deduplication:** Identical spam lines are collapsed into a compact `... xN` format.\n- **Seamless UX:** The program sits in the system tray. You simply press a global hotkey, and it reads the buffer, compresses the text, shows a system notification with the compression percentage, and simulates pressing `Ctrl+V` (Paste).\n- **Lightning speed:** Written in Rust. The core works at the token array level (without string allocations in hot loops), compressing megabytes of text in fractions of a second.\n\n## 🚀 How to Use\n\n1. Launch `Logs Tokenizer.exe` (an icon will appear in the tray).\n2. Select and copy (`Ctrl+C`) any large chunk of logs.\n3. Go to a messenger or ChatGPT.\n4. Press `Ctrl + Alt + V` (or `Cmd + Alt + V` on macOS)\n5. The compressed text will automatically be pasted into the input field.\n\n## 🛠️ Building from Source\n\nMake sure you have Rust (cargo) installed.\n\n### Windows \u0026 Linux\n\n```bash\ngit clone https://github.com/sergeivaskov/logs-tokenizer.git\ncd logstokenizer\ncargo build --release\n```\n\nThe executable file will be located at `target/release/logstokenizer.exe` (on Windows) or `target/release/logstokenizer` (on Linux).\n\n### macOS\n\nTo build the macOS App Bundle (which runs in the background and has an icon):\n\n```bash\ngit clone https://github.com/sergeivaskov/logs-tokenizer.git\ncd logstokenizer\nbash scripts/build_macos.sh\n```\n\nThe compiled application will be located at `target/release/LogsTokenizer.app`. You can move it to your `/Applications` folder.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsergeivaskov%2Flogs-tokenizer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsergeivaskov%2Flogs-tokenizer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsergeivaskov%2Flogs-tokenizer/lists"}