{"id":17724309,"url":"https://github.com/zutto/shardedmap","last_synced_at":"2025-03-14T05:32:04.778Z","repository":{"id":57546117,"uuid":"117074266","full_name":"zutto/shardedmap","owner":"zutto","description":"Sharding of golang's built in map for better concurrency and performance","archived":false,"fork":false,"pushed_at":"2018-02-01T16:43:52.000Z","size":37,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-11T08:04:54.476Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zutto.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-01-11T08:49:58.000Z","updated_at":"2024-03-07T17:22:24.000Z","dependencies_parsed_at":"2022-09-05T10:50:35.481Z","dependency_job_id":null,"html_url":"https://github.com/zutto/shardedmap","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zutto%2Fshardedmap","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zutto%2Fshardedmap/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zutto%2Fshardedmap/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zutto%2Fshardedmap/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zutto","download_url":"https://codeload.github.com/zutto/shardedmap/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243532517,"owners_count":20306151,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-25T15:46:29.568Z","updated_at":"2025-03-14T05:32:04.374Z","avatar_url":"https://github.com/zutto.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Golangs built in map sharded!\n#### For better concurrency.\n\nThe whole point of this project was to assist me in a scenario where map access I/O became a bottleneck.\n\n( godoc: https://godoc.org/github.com/zutto/shardedmap )\n\n### 1. Installation\n`go get github.com/zutto/shardedmap`\n\n\n### 2. Usage\n#### Intialization is easy as:\n`shardmap := shardedmap.NewShardMap(24)`\n\n Wondering what that 24 is? It's the amount of 'shards' shardedmap should initialize.\nGeneral rule of thumb: Lower your I/O is, the less shards you need. Each shard holds their own mutex lock for rwd access.\n\n##### Adding item to shardmap\n\n```\ntype demo struct {\n   str1 string\n   str2 string\n}\n\nvar data interface{} = demo{\"Hello\", \"World\"}\n\nshardmap.Set(\"I am a string key\", \u0026data)\n```\n\nInput data type is interface{}, key datatype is always a string.\n\n##### Reading items from shardmap\n```\nretrievedData := shardmap.Get(\"I am a string key\")\n```\n\n#### Deleting items from shardmap\n```\nshardmap.Delete(\"I am a string key\")\n```\n\n###### for more documentation, please look at godoc or the source code!\n\n### 3. Performance\nThere are benchmarks comparing normal map vs shardedmap included in the shardmap_test.go file.\nGenerally, shardedmap only becomes faster the more I/O you require, and amount of shards you are willing to create. Also, don't forget that slices exist! use slices if you can!!\n\nHeres the benchmarks, i ran them on t460p laptop w/i7-6700HQ CPU @ 2.60GHz. Your mileage may vary, and the performance difference is best seen\n```\n$ go test -bench=.\ngoos: linux\ngoarch: amd64\npkg: github.com/zutto/shardedmap\nBenchmarkShardedMapWrite-8               1000000              2047 ns/op\nBenchmarkShardedMapRead-8                1000000              1218 ns/op\nBenchmarkShardedMapWR-8                   500000              3197 ns/op\nBenchmarkShardedMapWRForceGet-8           300000              4242 ns/op\nBenchmarkShardedMapDel-8                 1000000              1420 ns/op\nBenchmarkPlainMapWrite-8                  500000              5481 ns/op\nBenchmarkPlainMapRead-8                  1000000              2363 ns/op\nBenchmarkPlainMapDel-8                    500000              4518 ns/op\n```\n\nShards are stored in a slice. The slice where your data is added to is computed by djbhash - djbhash is dubbed as:\n\u003e\"It is one of the most efficient hash functions ever published.\"\n\nDjbhash is very simple and insecure hash, its sole purpose is to split the I/O access across the shards, thus why fast hashing function was deemed necessary. \n\n\n### 4. Limitations\nLimitations mainly come from golangs std rwmutex \u0026 slices \u0026 maps. If you find more limitations in this project, do please let me know!\n\n\n\n\n\n### ----------------\n\n\nTODO:\n- [x] license\n- [x] documentation \u0026 readme\n- [ ] clean the code\n- [ ] more tests?\n- [ ] better error handling graceful error handling to be done.\n- [ ] look for more entrypoints for fuzzing\n- [ ] documentation \u0026 readme of the sub-libraries (shardreduce \u0026 shardquest)\n\n\n\n\n\n\n\nThis project is licensed under MIT license.\n\nAlso if you use shardedmap somewhere, I would be delighted to hear about it!\n\nSupport available via issues, enterprise support available upon request.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzutto%2Fshardedmap","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzutto%2Fshardedmap","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzutto%2Fshardedmap/lists"}