{"id":31762390,"url":"https://github.com/houseme/starshard","last_synced_at":"2025-10-09T22:18:41.020Z","repository":{"id":316609067,"uuid":"1064112776","full_name":"houseme/starshard","owner":"houseme","description":"A blazing-fast sharded concurrent HashMap using hashbrown and RwLock, with lazy shards, atomic length cache, and async support.","archived":false,"fork":false,"pushed_at":"2025-09-25T15:43:43.000Z","size":21,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-25T16:41:01.319Z","etag":null,"topics":["async","atomic","concurrency","hashbrown","hashmap","rust","rust-lang","rwlock"],"latest_commit_sha":null,"homepage":"https://houseme.github.io/starshard/","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/houseme.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE-APACHE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":{"github":null,"patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":"houseme","issuehunt":null,"otechie":null,"lfx_crowdfunding":null,"custom":["https://paypal.me/houseme"]}},"created_at":"2025-09-25T14:58:09.000Z","updated_at":"2025-09-25T15:48:31.000Z","dependencies_parsed_at":"2025-09-25T16:41:06.445Z","dependency_job_id":null,"html_url":"https://github.com/houseme/starshard","commit_stats":null,"previous_names":["houseme/starshard"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/houseme/starshard","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/houseme%2Fstarshard","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/houseme%2Fstarshard/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/houseme%2Fstarshard/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/houseme%2Fstarshard/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/houseme","download_url":"https://codeload.github.com/houseme/starshard/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/houseme%2Fstarshard/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279002114,"owners_count":26083307,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-09T02:00:07.460Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["async","atomic","concurrency","hashbrown","hashmap","rust","rust-lang","rwlock"],"created_at":"2025-10-09T22:18:27.704Z","updated_at":"2025-10-09T22:18:41.011Z","avatar_url":"https://github.com/houseme.png","language":"Rust","funding_links":["https://liberapay.com/houseme","https://paypal.me/houseme"],"categories":[],"sub_categories":[],"readme":"# Starshard\n\n[![Build](https://github.com/houseme/starshard/workflows/Build/badge.svg)](https://github.com/houseme/starshard/actions?query=workflow%3ABuild)\n[![crates.io](https://img.shields.io/crates/v/starshard.svg)](https://crates.io/crates/starshard)\n[![docs.rs](https://docs.rs/starshard/badge.svg)](https://docs.rs/starshard/)\n[![License](https://img.shields.io/crates/l/starshard)](./LICENSE-APACHE)\n[![Downloads](https://img.shields.io/crates/d/starshard)](https://crates.io/crates/starshard)\n\nEnglish | [简体中文](README_CN.md)\n\n\u003cb\u003eStarshard\u003c/b\u003e: a high-performance, lazily sharded concurrent HashMap for Rust.\n\n\u003ccode\u003eSync + Async + Optional Rayon + Optional Serde\u003c/code\u003e\n\n---\n\n## Status\n\nEarly stage. API may still evolve (semantic stability prioritized; minor naming changes possible).\n\n## Motivation\n\nYou often need a fast concurrent map:\n\n- Standard single `RwLock\u003cHashMap\u003c..\u003e\u003e` becomes contended under mixed read/write load.\n- Fully lock-free or CHM designs can add memory + complexity cost.\n- Sharding with lazy initialization offers a pragmatic middle ground.\n\nStarshard focuses on:\n\n1. Minimal uncontended overhead.\n2. Lazy shard allocation (memory proportional to actually touched shards).\n3. Atomic cached length.\n4. Snapshot iteration (parallel if `rayon`).\n5. Symmetric sync / async APIs.\n6. Extensible design (future: rebalancing, eviction, metrics).\n\n---\n\n## Features\n\n| Feature | Description                                          | Notes                          |\n|---------|------------------------------------------------------|--------------------------------|\n| `async` | Adds `AsyncShardedHashMap` (Tokio `RwLock`)          | Independent of `rayon`         |\n| `rayon` | Parallel snapshot flatten for large iteration        | Used internally; API unchanged |\n| `serde` | Serialize/Deserialize (sync) + async snapshot helper | Hasher not persisted           |\n| (none)  | Pure sync core                                       | Lowest dependency surface      |\n\nEnable all in docs.rs via:\n\n```toml\n[package.metadata.docs.rs]\nall-features = true\n```\n\n---\n\n## Installation\n\n```toml\n[dependencies]\nstarshard = { version = \"0.5\", features = [\"async\", \"rayon\", \"serde\"] }\n# or minimal:\n# starshard = \"0.5\"\n```\n\n`serde_json` (tests / examples):\n\n```toml\n[dev-dependencies]\nserde_json = \"1\"\n```\n\n---\n\n## Quick Start (Sync)\n\n```rust\nuse starshard::ShardedHashMap;\nuse rustc_hash::FxBuildHasher;\n\nlet map: ShardedHashMap\u003cString, i32, FxBuildHasher\u003e = ShardedHashMap::new(64);\nmap.insert(\"a\".into(), 1);\nassert_eq!(map.get(\u0026\"a\".into()), Some(1));\nassert_eq!(map.len(), 1);\n```\n\n### Custom Hasher (defense against adversarial keys)\n\n```rust\nuse starshard::ShardedHashMap;\nuse std::collections::hash_map::RandomState;\n\nlet secure = ShardedHashMap::\u003cString, u64, RandomState\u003e\n::with_shards_and_hasher(128, RandomState::default ());\nsecure.insert(\"k\".into(), 7);\n```\n\n---\n\n## Async Usage\n\n```rust\n#[cfg(feature = \"async\")]\n#[tokio::main]\nasync fn main() {\n    use starshard::AsyncShardedHashMap;\n    let m: AsyncShardedHashMap\u003cString, u32\u003e = AsyncShardedHashMap::new(64);\n    m.insert(\"x\".into(), 42).await;\n    assert_eq!(m.get(\u0026\"x\".into()).await, Some(42));\n}\n```\n\n---\n\n## Parallel Iteration (`rayon`)\n\n```rust\n#[cfg(feature = \"rayon\")]\n{\nuse starshard::ShardedHashMap;\nlet m: ShardedHashMap\u003cString, u32\u003e = ShardedHashMap::new(32);\nfor i in 0..50_000 {\nm.insert(format ! (\"k{i}\"), i);\n}\nlet count = m.iter().count(); // internal parallel flatten\nassert_eq!(count, 50_000);\n}\n```\n\n---\n\n## Serde Semantics\n\nSync:\n\n- Serialized shape: `{ \"shard_count\": usize, \"entries\": [[K,V], ...] }`.\n- Hasher internal state not preserved; recreated with `S::default()`.\n- Requirements: `K: Eq + Hash + Clone + Serialize + Deserialize`, `V: Clone + Serialize + Deserialize`,\n  `S: BuildHasher + Default + Clone`.\n\nAsync:\n\n- No direct `Serialize`; call:\n\n```rust\n#[cfg(all(feature = \"async\", feature = \"serde\"))]\n{\nlet snap = async_map.async_snapshot_serializable().await;\nlet json = serde_json::to_string( \u0026 snap).unwrap();\n}\n```\n\n- To reconstruct: create a new async map and bulk insert.\n\n---\n\n## Consistency Model\n\n- Per-shard ops are linearizable w.r.t that shard.\n- Global iteration builds a per-shard snapshot as each shard lock is taken (not a fully atomic global view).\n- `len()` is maintained atomically (structural insert/remove only).\n- Iteration after concurrent writes may omit late inserts performed after a shard snapshot was captured.\n\n---\n\n## Performance Notes (Indicative)\n\n| Scenario                                              | Observation (relative)       |\n|-------------------------------------------------------|------------------------------|\n| Read-heavy mixed workload vs global `RwLock\u003cHashMap\u003e` | Reduced contention           |\n| Large snapshot iteration with `rayon` (100k+)         | 3-4x speedup flattening      |\n| Sparse shard usage                                    | Only touched shards allocate |\n\nDo benchmark with your own key/value distribution and CPU topology.\n\n---\n\n## Safety / Concurrency\n\n- No nested multi-shard lock ordering -\u003e avoids deadlocks.\n- Each shard single `RwLock`; iteration snapshots avoid long-lived global blocking.\n- Cloning values required (trade memory for contention isolation).\n- Not lock-free: intense write focus on one shard can still serialize.\n\n---\n\n## Limitations\n\n- No dynamic shard rebalancing.\n- No eviction / TTL.\n- Snapshot iteration allocates intermediate vectors.\n- Hasher state not serialized.\n- No lock-free progress guarantees.\n\n---\n\n## Roadmap (Potential)\n\n- Optional adaptive shard expansion / rebalancing.\n- Per-shard eviction strategies (LRU / segmented).\n- Metrics hooks (pre/post op instrumentation).\n- Batched multi-insert API.\n- Zero-copy or COW snapshot mode.\n\n---\n\n## Design Sketch\n\n```\nArc -\u003e RwLock\u003cVec\u003cOption\u003cArc\u003cRwLock\u003cHashMap\u003cK,V,S\u003e\u003e\u003e\u003e\u003e\u003e + AtomicUsize(len)\n```\n\nLazy fill of inner `Option` slot when first key hashes into shard.\n\n---\n\n## Examples Summary\n\n| Goal                  | Snippet                         |\n|-----------------------|---------------------------------|\n| Basic sync            | see Quick Start                 |\n| Async insert/get      | see Async Usage                 |\n| Parallel iterate      | enable `rayon`                  |\n| Serde snapshot (sync) | `serde_json::to_string(\u0026map)`   |\n| Async serde snapshot  | `async_snapshot_serializable()` |\n| Custom hasher         | `with_shards_and_hasher(..)`    |\n\n---\n\n## License\n\nDual license: [MIT](LICENSE-MIT) OR [Apache-2.0](LICENSE-APACHE) (choose either).\n\n---\n\n## Contribution\n\nPRs welcome: focus on correctness (tests), simplicity, and documentation clarity.  \nRun:\n\n```bash\ncargo clippy --all-features -- -D warnings\ncargo test --all-features\n```\n\n---\n\n## Minimal Example (All Features)\n\n```rust\nuse starshard::{ShardedHashMap, AsyncShardedHashMap};\n#[cfg(feature = \"async\")]\n#[tokio::main]\nasync fn main() {\n    let sync_map: ShardedHashMap\u003cu64, u64\u003e = ShardedHashMap::new(32);\n    sync_map.insert(1, 10);\n\n    #[cfg(feature = \"serde\")]\n    {\n        let json = serde_json::to_string(\u0026sync_map).unwrap();\n        let _de: ShardedHashMap\u003cu64, u64\u003e = serde_json::from_str(\u0026json).unwrap();\n    }\n\n    let async_map: AsyncShardedHashMap\u003cu64, u64\u003e = AsyncShardedHashMap::new(32);\n    async_map.insert(2, 20).await;\n}\n```\n\n---\n\n## Disclaimer\n\nBenchmarks and behavior notes are indicative only; validate under production load patterns.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhouseme%2Fstarshard","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhouseme%2Fstarshard","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhouseme%2Fstarshard/lists"}