{"id":19994834,"url":"https://github.com/kyr0/brotli-unicode","last_synced_at":"2025-05-04T13:31:40.349Z","repository":{"id":57700792,"uuid":"501357249","full_name":"kyr0/brotli-unicode","owner":"kyr0","description":"A novel algorithm for encoding/decoding and character level compression/decompression with character-level compression rates of \u003e500%. The compressed data is copy-pasteable without data loss.","archived":false,"fork":false,"pushed_at":"2022-06-08T19:12:11.000Z","size":1617,"stargazers_count":17,"open_issues_count":5,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-11-03T09:34:09.557Z","etag":null,"topics":["brotli","browser","compression","decompression","js","nodejs","typescript","unicode","webassembly"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kyr0.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-06-08T17:56:46.000Z","updated_at":"2024-10-09T01:12:06.000Z","dependencies_parsed_at":"2022-08-28T03:20:23.915Z","dependency_job_id":null,"html_url":"https://github.com/kyr0/brotli-unicode","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kyr0%2Fbrotli-unicode","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kyr0%2Fbrotli-unicode/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kyr0%2Fbrotli-unicode/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kyr0%2Fbrotli-unicode/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kyr0","download_url":"https://codeload.github.com/kyr0/brotli-unicode/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224393950,"owners_count":17303725,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["brotli","browser","compression","decompression","js","nodejs","typescript","unicode","webassembly"],"created_at":"2024-11-13T04:57:47.890Z","updated_at":"2024-11-13T04:58:06.298Z","avatar_url":"https://github.com/kyr0.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# brotli-unicode\n\nThis library compresses using the Brotli algorithm, based on WebAssembly.\nAfter the compression has been done, another character encoding/compression algorithm is applied: `base-unicode`\n`base-unicode` transcodes the `Uint8Array` into a Unicode string that is shorter than the original text (character wise).\nData in this form can also be copy-pasted when modern system fonts are used. Also, modern browsers allow Unicode\nin URIs. Therefore, data compressed and encoded with this library can be transmitted via URLs.\n\nFor decoding, you can either use the JS variant, which is much smaller in code size, or you can also use the\nWebAssembly implementation.\n\nThis algorithm allows for stellar compression ratios on text and binary data.\nIn our test scenario we're proud to present a compression rate of 558%.\n\n\u003cimg src=\"jest_results.png\" /\u003e\n\n## Setup\n\nAs a package for development (Node.js, Browsers):\n\n```bash\n  yarn add brotli-unicode\n\n  # or\n\n  npm i brotli-unicode\n```\n\n## Usage of the WASM variant\n\nThe usage in a Node.js or Browser environment is trivial:\n\n```ts\n// import size (uncompressed, but minified) / WASM version / max performance: 1.8M\nimport { compress, decompress } from 'brotli-unicode'\n\n// Node.js or using the buffer package\nlet input = Buffer.from('Hello🤖!')\n\n// alternatively, in-browser (without any third-party libraries)\ninput = TextEncoder.encode('Hello🤖!')\n\n// it takes a Uint8Array and returns a base-unicode encoded string (copy and pasteable)\nconst compressed = await compress(input)\n\n// it takes the base-unicode encoded string and returns a Uint8Array\nconst decompressed = await decompress(compressed)\n\n// Node.js or using the buffer package\nlet output = Buffer.from(decompressed)\n\n// alternatively, in-browser (without any third-party libraries)\noutput = TextDecoder.decode(decompressed)\n```\n\nPlease note that the WASM version comes with a whopping size of (minified)\n1.8MiB. This is, because the binary is base64 encoded and inlined.\n\nIf you prefer maximum performance and memory efficiency over small bundle size,\nchoose the WASM variant. Also, if you need compression, use the WASM version.\n\n## Usage of the pure JS variant\n\nIf you need a small bundle size, can effort the slowdown and\nonly need decompression, use the hard-written JavaScript decompressor:\n\n```ts\n// import size (uncompressed, but minified) / JS version / only decompress / slower: 152K\nimport { decompress } from 'brotli-unicode/js'\n\n// please also note that the pure JS variant is synchronous\n// for large inputs, you could optimize the execution by moving\n// this call into a Worker\n\n// it takes a base-unicode encoded string and returns a Uint8Array\nconst decompressed = decompress(compressed)\n\n// Node.js or using the buffer package\nlet output = Buffer.from(decompressed)\n\n// alternatively, in-browser (without any third-party libraries)\noutput = TextDecoder.decode(decompressed)\n```\n\n## Options\n\nThe `compress` method comes with a second `options` parameter.\n\n### Quality level\n\nThe most common setting is `quality` with a scale from 0 to 11.\nBy default, the quality is set to best quality (11).\n\n```ts\nconst compressed = await compress(Buffer.from('foobar'), { quality: 9 })\n```\n\nA lower quality value makes the output bigger but improves compression time.\nIn 99.9% of the cases, you don't want to change this value.\n\n### Custom dictionary\n\nThe relevant options here is `customDictionary`. You can set this to an Uint8Array string\nof tokens which should be part of the `a priori` known dictionary. This can be useful\nif you have power over both, the sender and the receiver part and if you know exactly\nwhich tokens will be used alot in the input. For example, if you know that you'll\nbe compressing text, encoded as UTF16/UCS-2 and you know that the content is TypeScript code,\nyou could include the keywords of the TypeScript language in the custom dictionary.\n\nPlease mind, that you need to set the same value for decoding as well.\n\n```ts\n// with this configuration, \"let\" must not be encoded in the dictionary and carried as part of the\n// payload. The binary tree (huffman coding tree)\nconst customDictionary = Buffer.from('let')\nconst compressed = await compress('let foo = 123; let bar = \"foo\";', { customDictionary })\nconst decompressed = await decompress(compressed, { customDictionary })\n```\n\n## Limitations\n\nThere is no streaming compression/decompression yet. It can be simply done by exposing the API from the WASM implementation.\nIf you need that, pls. ping via Issue.\n\n## Build\n\n    yarn build\n\n## Test\n\n    yarn test\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkyr0%2Fbrotli-unicode","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkyr0%2Fbrotli-unicode","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkyr0%2Fbrotli-unicode/lists"}