{"id":51493246,"url":"https://github.com/unrays/sharded-string-interner","last_synced_at":"2026-07-07T12:30:36.157Z","repository":{"id":360063715,"uuid":"1248498767","full_name":"unrays/sharded-string-interner","owner":"unrays","description":"High-performance sharded string interner using atomic memory resources, custom allocators, and low-contention concurrency design.","archived":false,"fork":false,"pushed_at":"2026-05-24T20:16:47.000Z","size":38,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-24T21:25:45.544Z","etag":null,"topics":["cache-friendly","chunk-allocator","compiler-infrastructure","concurrency","custom-allocator","exotic","high-performance","lock-free","low-contention","low-latency","memory-management","monotonic-allocator","multithreading","performance-critical","quebec","sharding","string-interner","string-interning","systems-programming"],"latest_commit_sha":null,"homepage":"https://unrays.github.io/sharded-string-interner/","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsl-1.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/unrays.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-24T18:09:47.000Z","updated_at":"2026-05-24T20:27:00.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/unrays/sharded-string-interner","commit_stats":null,"previous_names":["unrays/sharded-string-interner"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/unrays/sharded-string-interner","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/unrays%2Fsharded-string-interner","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/unrays%2Fsharded-string-interner/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/unrays%2Fsharded-string-interner/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/unrays%2Fsharded-string-interner/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/unrays","download_url":"https://codeload.github.com/unrays/sharded-string-interner/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/unrays%2Fsharded-string-interner/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35228621,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-07T02:00:07.222Z","response_time":90,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cache-friendly","chunk-allocator","compiler-infrastructure","concurrency","custom-allocator","exotic","high-performance","lock-free","low-contention","low-latency","memory-management","monotonic-allocator","multithreading","performance-critical","quebec","sharding","string-interner","string-interning","systems-programming"],"created_at":"2026-07-07T12:30:35.604Z","updated_at":"2026-07-07T12:30:36.146Z","avatar_url":"https://github.com/unrays.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Sharded String Interner\n\nA high-performance sharded string interning system with low-contention concurrency design, custom memory resources, and chunk-based allocation strategies. This project is part of the **EXOTIC::memory** / **EXOTIC::intern** ecosystem.\n\n---\n\n## Overview\n\nThis project was originally intended to be integrated into a compiler, but it remains a fully generic system that can be used in other performance-critical environments.\n\nIt took approximately three weeks to complete during an end-of-term period, from initial design to full implementation. This work has been extremely beneficial in strengthening my understanding of low-level systems, memory management, and multithreaded programming.\n\nBefore this project, my knowledge of multithreading was very limited. It served as a foundational step toward understanding professional systems programming concepts such as contention control, memory ownership, concurrent access patterns, and low-level allocation strategies.\n\nFurthermore, it's in this project that I implemented my personal allocation system inspired by `std::pmr`. In fact, I have a repository that deals with it!\n\n---\n\n### Architecture\n\nAt its core, this project implements a sharded string interning system designed to reduce contention as much as possible while keeping lookups fast and lightweight.\n\nStrings are distributed across shards using a hash-based sharding function. Each shard is completely independent and contains:\n\n- An `unordered_set\u003cstd::string_view\u003e` used as the intern registry\n- Its own `unsynchronized_chunk_allocator`\n- A dedicated `shared_mutex`\n\nThe idea behind the design is fairly simple: most operations are reads, so the synchronization model is optimized around that.\n\n- `Shared` locking is used during lookups\n- `Exclusive` locking only happens when a new string must be inserted\n\nThis allows already-interned strings to be retrieved extremely quickly with very little contention.\n\n---\n\n### Lookup Flow\n\nWhen a string enters a shard, the system first checks if it already exists inside the shard registry under a shared lock.\n\nIf the string is already interned:\n\n- The existing `string_view` is retrieved\n- Its `.data()` pointer is returned directly\n- No allocation is performed\n- No exclusive locking is required\n\nIn practice, repeated lookups become extremely cheap since they mostly stay inside the shared-lock path.\n\n---\n\n### Allocation Flow\n\nIf the string is not found during lookup, the system switches to an exclusive lock.\n\nAt this stage:\n\n1. Memory is requested from the shard's `unsynchronized_chunk_allocator`\n2. The allocator itself acquires memory from a shared upstream atomic memory resource\n3. The string is copied into the allocated memory region\n4. A new `std::string_view` pointing to this stable memory is inserted into the registry\n5. The resulting pointer is returned\n\nOnce interned, future lookups will directly reuse the same pointer without allocating again.\n\n---\n\n### Memory Model\n\nEach shard owns its own chunk allocator, but all allocators ultimately share the same upstream atomic memory resource.\n\nThe goal here was to keep shard logic local while still centralizing the actual memory acquisition in a concurrency-safe way.\n\nThis design provides:\n\n- Local allocation behavior per shard\n- Thread-safe upstream memory acquisition\n- Stable string lifetime guarantees\n- Reduced allocation overhead through chunk-based allocation\n\nThe atomic memory resource essentially acts as a global pool from which all shards acquire their memory ranges.\n\n---\n\n## Example Usage\n\n```cpp\nexotic::memory::monotonic_atomic_buffer upstream(1 \u003c\u003c 20);\nexotic::memory::unsynchronized_chunk_allocator\u003cchar\u003e alloc(\u0026upstream);\n\nexotic::intern::ShardedStringInterner\u003c8\u003e interner(\u0026alloc); // 8 shards\n\nconst char* a = interner.intern(\"hello\");\nconst char* b = interner.intern(\"hello\"); // same pointer\n```\n\n---\n\n## Notes\n\nThis project is intentionally designed as a learning-focused system rather than a drop-in standard library replacement. That’s how I learn, and it seems to work very well.\n\n---\n\n## License\n\nThis project is licensed under the Boost Software License. See the [LICENSE](LICENSE) file for details.\n\n\u003cp align=\"center\"\u003e\u003csub\u003e© Félix-Olivier Dumas 2026\u003c/sub\u003e\u003c/p\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Funrays%2Fsharded-string-interner","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Funrays%2Fsharded-string-interner","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Funrays%2Fsharded-string-interner/lists"}