https://github.com/zutto/shardedmap
Sharding of golang's built in map for better concurrency and performance
https://github.com/zutto/shardedmap
Last synced: over 1 year ago
JSON representation
Sharding of golang's built in map for better concurrency and performance
- Host: GitHub
- URL: https://github.com/zutto/shardedmap
- Owner: zutto
- License: mit
- Created: 2018-01-11T08:49:58.000Z (over 8 years ago)
- Default Branch: master
- Last Pushed: 2018-02-01T16:43:52.000Z (over 8 years ago)
- Last Synced: 2025-03-11T08:04:54.476Z (over 1 year ago)
- Language: Go
- Size: 36.1 KB
- Stars: 5
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Golangs built in map sharded!
#### For better concurrency.
The whole point of this project was to assist me in a scenario where map access I/O became a bottleneck.
( godoc: https://godoc.org/github.com/zutto/shardedmap )
### 1. Installation
`go get github.com/zutto/shardedmap`
### 2. Usage
#### Intialization is easy as:
`shardmap := shardedmap.NewShardMap(24)`
Wondering what that 24 is? It's the amount of 'shards' shardedmap should initialize.
General rule of thumb: Lower your I/O is, the less shards you need. Each shard holds their own mutex lock for rwd access.
##### Adding item to shardmap
```
type demo struct {
str1 string
str2 string
}
var data interface{} = demo{"Hello", "World"}
shardmap.Set("I am a string key", &data)
```
Input data type is interface{}, key datatype is always a string.
##### Reading items from shardmap
```
retrievedData := shardmap.Get("I am a string key")
```
#### Deleting items from shardmap
```
shardmap.Delete("I am a string key")
```
###### for more documentation, please look at godoc or the source code!
### 3. Performance
There are benchmarks comparing normal map vs shardedmap included in the shardmap_test.go file.
Generally, shardedmap only becomes faster the more I/O you require, and amount of shards you are willing to create. Also, don't forget that slices exist! use slices if you can!!
Heres the benchmarks, i ran them on t460p laptop w/i7-6700HQ CPU @ 2.60GHz. Your mileage may vary, and the performance difference is best seen
```
$ go test -bench=.
goos: linux
goarch: amd64
pkg: github.com/zutto/shardedmap
BenchmarkShardedMapWrite-8 1000000 2047 ns/op
BenchmarkShardedMapRead-8 1000000 1218 ns/op
BenchmarkShardedMapWR-8 500000 3197 ns/op
BenchmarkShardedMapWRForceGet-8 300000 4242 ns/op
BenchmarkShardedMapDel-8 1000000 1420 ns/op
BenchmarkPlainMapWrite-8 500000 5481 ns/op
BenchmarkPlainMapRead-8 1000000 2363 ns/op
BenchmarkPlainMapDel-8 500000 4518 ns/op
```
Shards are stored in a slice. The slice where your data is added to is computed by djbhash - djbhash is dubbed as:
>"It is one of the most efficient hash functions ever published."
Djbhash is very simple and insecure hash, its sole purpose is to split the I/O access across the shards, thus why fast hashing function was deemed necessary.
### 4. Limitations
Limitations mainly come from golangs std rwmutex & slices & maps. If you find more limitations in this project, do please let me know!
### ----------------
TODO:
- [x] license
- [x] documentation & readme
- [ ] clean the code
- [ ] more tests?
- [ ] better error handling graceful error handling to be done.
- [ ] look for more entrypoints for fuzzing
- [ ] documentation & readme of the sub-libraries (shardreduce & shardquest)
This project is licensed under MIT license.
Also if you use shardedmap somewhere, I would be delighted to hear about it!
Support available via issues, enterprise support available upon request.