{"id":13929625,"url":"https://github.com/daniel5151/compressed-emoji-shortcodes","last_synced_at":"2025-04-09T15:32:49.867Z","repository":{"id":79236051,"uuid":"328465679","full_name":"daniel5151/compressed-emoji-shortcodes","owner":"daniel5151","description":"A Quest to Find a Highly Compressed Emoji :shortcode: Lookup Function","archived":false,"fork":false,"pushed_at":"2021-01-27T23:27:47.000Z","size":239,"stargazers_count":17,"open_issues_count":1,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-03-24T08:54:41.811Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://prilik.com/blog/post/emoji-shortcodes/","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/daniel5151.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2021-01-10T19:47:22.000Z","updated_at":"2025-02-21T08:53:24.000Z","dependencies_parsed_at":"2023-03-12T07:25:11.465Z","dependency_job_id":null,"html_url":"https://github.com/daniel5151/compressed-emoji-shortcodes","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/daniel5151%2Fcompressed-emoji-shortcodes","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/daniel5151%2Fcompressed-emoji-shortcodes/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/daniel5151%2Fcompressed-emoji-shortcodes/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/daniel5151%2Fcompressed-emoji-shortcodes/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/daniel5151","download_url":"https://codeload.github.com/daniel5151/compressed-emoji-shortcodes/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248058201,"owners_count":21040717,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-07T18:02:27.123Z","updated_at":"2025-04-09T15:32:49.009Z","avatar_url":"https://github.com/daniel5151.png","language":"Rust","funding_links":[],"categories":["others"],"sub_categories":[],"readme":"# Highly Compressed Emoji Shortcode Mapping\n\nAn experiment to try and find a highly compressed representation of the\nentire\n[unicode shortcodes-to-emoji](https://raw.githubusercontent.com/github/gemoji/master/db/emoji.json)\nmapping that can be indexed _without requiring any dynamic allocation_. In other\nwords: what's the smallest amount of static storage(code and data) required to\nwrite a function with the following signature:\n\n```rust\nfn shortcode_to_emoji(input: \u0026str) -\u003e Option\u003c\u0026str\u003e\n```\n\nCheck out the project's [writeup](https://prilik.com/blog/post/emoji-shortcodes/)\nfor more details on the rationale and idea behind this project.\n\n\u003e **DISCLAIMER:** This is a project that was hacked together over the course of\n\u003e a couple weekends, where iteration speed took priority over code\n\u003e quality/robustness. The implementation is a mess, and the build system is\n\u003e incredibly brittle. If you're even _remotely_ considering building off this\n\u003e repo - please don't. You'd be much better off building off these _ideas_ and\n\u003e writing a new implementation from scratch.\n\nMake sure to try out the\n[online demo](https://prilik.com/compressed-emoji-shortcodes)!\n\n## Building and Running\n\nAt the moment, building this library requires running a \\*nix OS with `curl`\ninstalled, which is required to download the emoji database used as part of the\nbuild process. Aside from that, the core `maximally-compressed-emoji-shortcodes`\nlibrary is a bog-standard Rust crate which can be built with `cargo build`.\n\nGiven that this is more of a quick-and-dirty experiment rather than a proper\nready-to-use library, there isn't any easy-to-use config file to play around\nwith. If you're seriously interested in playing around with this mess of code,\nget ready to do some spelunking. Some potentially useful starting off points:\n\n-   Changing the size of the hash - `./rust-phf/phf/src/map.rs:18` (the\n    `keys: Slice\u003cu16\u003e` field) and `./rust-phf/phf_shared/src/lib.rs:43` (the\n    `checksum` function).\n-   Fixup Table search times - `build.rs`\n-   Swapping out the key/value data set - `build.rs`\n\nThis repo includes several test/example projects that use the\n`maximally-compressed-emoji-shortcodes` library:\n\n-   `kowalski-analysis`: a _very_ messy playground for testing various\n    properties of the library (e.g: false-positive rates, accuracy, compression\n    ratio, etc...).\n-   `example_no_std`: a `no_std` Rust binary that serves as a rough benchmark\n    for how much space the library will occupy when deployed in an embedded\n    context.\n-   `shortcode-web`: using the magic of `wasm-pack`, you can play with this\n    project via a neat little online demo! Try it out at\n    [here](https://prilik.com/compressed-emoji-shortcodes)!\n    -   _Note:_ the `.wasm` size of the `shortcode-web` demo is not\n        representative of the binary size on an proper embedded platform, since\n        `wasm-bindgen` introduces almost 15kb of overhead for some reason (i.e:\n        when the single exported function is replaced with a noop). I could\n        probably slim this down by bypassing `wasm-bindgen` entirely, and\n        figuring out how to accept Javascript `String`s over the FFI, but that's\n        _high effort_. So yeah, just subtract 15kb from the (uncompressed)\n        `.wasm` size to get a better idea of the compression factor.\n\n## Future Work?\n\nI'm pretty much done with this project for now, but there are still a few ideas\nthat might be worth exploring to compress things down even more:\n\n-   Compressing the emoji UTF-8 strings using some sort of domain specific\n    representation\n    -   e.g: it seems like most emoji fall under a\n        [small subrange](https://stackoverflow.com/questions/30470079/emoji-value-range)\n        of unicode codepoints, it might be possible to shave a couple bits of\n        overhead from each emoji mapping by adding/removing an offset from the\n        stored value.\n-   Using non-standard key hash sizes (i.e: 9-bit, 10-bit, etc...).\n    -   Follow up: automatically trying out various key hash sizes to minimize\n        space overhead while maintaining a favorable hash-collision rate.\n-   Actually cleaning up this abomination of a codebase and releasing a proper\n    library that employs these various techniques\n\nOh, and of course, I should probably port it to\n[one of my keyboards](https://github.com/daniel5151/qmk_firmware). After all,\nthat was the whole inspiration for this endeavor!\n\n## Acknowledgements\n\nThis project wouldn't have been possible without the incredible\n[rust-phf](https://github.com/sfackler/rust-phf) library. The in-tree version of\n`rust-phf` is a stripped down version and heavily modified version the library,\noptimizing the map's internal representation for this particular use-case.\n\nThe initial POC of this project was based off of\nhttps://github.com/kornelski/gh-emoji.\n\nThe emoji shortcode database is downloaded directly from Github's\n[gemoji](https://github.com/github/gemoji/tree/master) library.\n\nSpecial thanks to [Matt D'Souza](https://github.com/DSouzaM) and\n[Ethan Hardy](https://github.com/ethan-hardy), who I nerd-sniped into helping me\nwith this funky little project.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdaniel5151%2Fcompressed-emoji-shortcodes","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdaniel5151%2Fcompressed-emoji-shortcodes","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdaniel5151%2Fcompressed-emoji-shortcodes/lists"}