{"id":26384583,"url":"https://github.com/douglasmakey/cache-example","last_synced_at":"2025-03-17T07:29:42.723Z","repository":{"id":90269349,"uuid":"218352501","full_name":"douglasmakey/cache-example","owner":"douglasmakey","description":"A few days ago, I read an article about BigCache and I was interested to know how they avoided these 2 problems: concurrent access and expensive GC cycles","archived":false,"fork":false,"pushed_at":"2019-10-29T18:11:02.000Z","size":5,"stargazers_count":13,"open_issues_count":0,"forks_count":3,"subscribers_count":2,"default_branch":"master","last_synced_at":"2023-02-28T00:46:09.429Z","etag":null,"topics":["cache","go","golang"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/douglasmakey.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-10-29T18:09:53.000Z","updated_at":"2024-06-19T08:59:33.825Z","dependencies_parsed_at":null,"dependency_job_id":"75bf0fb2-2e0f-4849-809f-352414bbd49e","html_url":"https://github.com/douglasmakey/cache-example","commit_stats":null,"previous_names":[],"tags_count":0,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/douglasmakey%2Fcache-example","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/douglasmakey%2Fcache-example/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/douglasmakey%2Fcache-example/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/douglasmakey%2Fcache-example/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/douglasmakey","download_url":"https://codeload.github.com/douglasmakey/cache-example/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243991775,"owners_count":20380051,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cache","go","golang"],"created_at":"2025-03-17T07:29:42.149Z","updated_at":"2025-03-17T07:29:42.714Z","avatar_url":"https://github.com/douglasmakey.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# How BigCache avoids expensive GC cycles and speeds up concurrent access\n\nA few days ago, I read an article about BigCache and I was interested to know how they avoided these 2 problems:\n\n* concurrent access\n* expensive GC cycles\n\nI went to their repository and read the code to understand how they achieved it. I think it's amazing so I would like to share it with you.\n\n*'Fast, concurrent, evicting in-memory cache written to keep big number of entries without impact on performance. BigCache keeps entries on heap but omits GC for them. To achieve that operations on bytes arrays take place, therefore entries (de)serialization in front of the cache will be needed in most use cases.'*\n\n[BigCache](https://github.com/allegro/bigcache)\n\n## Concurrent access\n\nSurely you will need concurrent access, either your program uses goroutines, or you have an HTTP server that allocates goroutines for each request. The most common approach to achieve it would be to use sync.RWMutex in front of the cache access function to ensure that only one goroutine could modify it at a time, but if you use this approach and other goroutine try to make modifications in the cache, the second goroutine would be blocked until the first goroutine unlock the lock, causing undesirable contention periods. \n\n\nTo solve this problem, they used shards, but what is a shard? A shard is a struct that contains its instance of the cache with a lock.\n\nThen they use an array of N shards to distribute the data into them, so when you are going to put or get data from the cache, a shard for that data is chosen by a function that we will talk later, in this way the locks contention can be minimized, because each shard has its lock.\n\n\n```go\ntype cacheShard struct {\n\titems        map[uint64]uint32\n\tlock         sync.RWMutex\n\tarray        []byte\n\ttail         int\n}\n```\n\n## Expensive GC cycles\n\n```go\nvar map[string]Item\n```\n\nThe most common pattern in a simple implementation of cache in Go is using a map to save the items, but if you are using a map the garbage collector (GC) will touch every single item of that map during the mark phase, this can be very expensive on the application performance when the map is very large.\n\n*After go version 1.5, if you use a map without pointers in keys and values, the GC will omit its content.*\n\n```go\nvar map[int]int\n```\n\nTo avoid this, they used a map without pointers in keys and values, with this the GC will omit the entries in the map and use an array of bytes, where they can put the entry serialized in bytes, then they can store in the map the hashedkey like key and the index of the entry into the array like the value.\n\n\nUsing an array of bytes is a smart solution because it only adds one additional object to the mark phase. Since a byte array doesn’t have any pointers (other than the object itself), the GC can mark the entire object in O(1) time.\n\n\n# Let's start coding\n\nIt will be a fairly simple implementation of cache, I avoided eviction, capacity and other things, the code will be simple just to demonstrate how they solved the problems I talked above.\n\nFirst, the hasher this is a `copy \u0026 paste` from their repository, you can find the code [Here](https://github.com/allegro/bigcache/blob/master/fnv.go), it is a **Hasher which makes no memory allocations.**\n\nhasher.go\n\n```go\npackage main\n\n// newDefaultHasher returns a new 64-bit FNV-1a Hasher which makes no memory allocations.\n// Its Sum64 method will lay the value out in big-endian byte order.\n// See https://en.wikipedia.org/wiki/Fowler–Noll–Vo_hash_function\nfunc newDefaultHasher() fnv64a {\n\treturn fnv64a{}\n}\n\ntype fnv64a struct{}\n\nconst (\n\t// offset64 FNVa offset basis. See https://en.wikipedia.org/wiki/Fowler–Noll–Vo_hash_function#FNV-1a_hash\n\toffset64 = 14695981039346656037\n\t// prime64 FNVa prime value. See https://en.wikipedia.org/wiki/Fowler–Noll–Vo_hash_function#FNV-1a_hash\n\tprime64 = 1099511628211\n)\n\n// Sum64 gets the string and returns its uint64 hash value.\nfunc (f fnv64a) Sum64(key string) uint64 {\n\tvar hash uint64 = offset64\n\tfor i := 0; i \u003c len(key); i++ {\n\t\thash ^= uint64(key[i])\n\t\thash *= prime64\n\t}\n\n\treturn hash\n}\n```\n\n\n\nSecond, the cache struct contains the logic to get the shards and functions get\u0026set.\n\n\nI talked above in the **Concurrent access** section about a function to choose a shard for the data, to achived this they use the hasher above to hash the key and with the hashedkey get a shard for the the key, to achived that they do a bitwise operation with `AND` operator, using a mask based on the size of shards to turn off certain bits to get a value into the range of shards.\n\n\n```\nhashedkey\u0026mask\n\n    0111\nAND 1101  (mask)\n  = 0101\n```\n\n\ncache.go\n\n```go\npackage main\n\nvar minShards = 1024\n\ntype cache struct {\n\tshards []*cacheShard\n\thash   fnv64a\n}\n\nfunc newCache() *cache {\n\tcache := \u0026cache{\n\t\thash:   newDefaultHasher(),\n\t\tshards: make([]*cacheShard, minShards),\n\t}\n\tfor i := 0; i \u003c minShards; i++ {\n\t\tcache.shards[i] = initNewShard()\n\t}\n\n\treturn cache\n}\n\nfunc (c *cache) getShard(hashedKey uint64) (shard *cacheShard) {\n\treturn c.shards[hashedKey\u0026uint64(minShards-1)]\n}\n\nfunc (c *cache) set(key string, value []byte) {\n\thashedKey := c.hash.Sum64(key)\n\tshard := c.getShard(hashedKey)\n\tshard.set(hashedKey, value)\n}\n\nfunc (c *cache) get(key string) ([]byte, error) {\n\thashedKey := c.hash.Sum64(key)\n\tshard := c.getShard(hashedKey)\n\treturn shard.get(key, hashedKey)\n}\n```\n\nFinally, where the magic occurs, in each shard have an array of bytes `[]byte` and a map `map[uint64]uint32`. In the map, they put the index for each entry like value and in the array save the entry in bytes.\n\nThey use the tail to keep the index in the array of bytes.\n\nshard.go\n\n```go\npackage main\n\nimport (\n\t\"encoding/binary\"\n\t\"errors\"\n\t\"sync\"\n)\n\nconst (\n\theaderEntrySize = 4\n\tdefaultValue    = 1024 // For this example we use 1024 like default value.\n)\n\ntype cacheShard struct {\n\titems        map[uint64]uint32\n\tlock         sync.RWMutex\n\tarray        []byte\n\ttail         int\n\theaderBuffer []byte\n}\n\nfunc initNewShard() *cacheShard {\n\treturn \u0026cacheShard{\n\t\titems:        make(map[uint64]uint32, defaultValue),\n\t\tarray:        make([]byte, defaultValue),\n\t\ttail:         1,\n\t\theaderBuffer: make([]byte, headerEntrySize),\n\t}\n}\n\nfunc (s *cacheShard) set(hashedKey uint64, entry []byte) {\n\tw := wrapEntry(entry)\n\ts.lock.Lock()\n\tindex := s.push(w)\n\ts.items[hashedKey] = uint32(index)\n\ts.lock.Unlock()\n}\n\nfunc (s *cacheShard) push(data []byte) int {\n\tdataLen := len(data)\n\tindex := s.tail\n\ts.save(data, dataLen)\n\treturn index\n}\n\nfunc (s *cacheShard) save(data []byte, len int) {\n\t// Put in the first 4 bytes the size of the value\n\tbinary.LittleEndian.PutUint32(s.headerBuffer, uint32(len))\n\ts.copy(s.headerBuffer, headerEntrySize)\n\ts.copy(data, len)\n}\n\nfunc (s *cacheShard) copy(data []byte, len int) {\n\t// Using the tail to keep the order to write in the array\n\ts.tail += copy(s.array[s.tail:], data[:len])\n}\n\nfunc (s *cacheShard) get(key string, hashedKey uint64) ([]byte, error) {\n\ts.lock.RLock()\n\titemIndex := int(s.items[hashedKey])\n\tif itemIndex == 0 {\n\t\ts.lock.RUnlock()\n\t\treturn nil, errors.New(\"key not found\")\n\t}\n\n\t// Read the first 4 bytes after the index, remember these 4 bytes have the size of the value, so\n\t// you can use this to get the size and get the value in the array using index+blockSize to know until what point\n\t// you need to read\n\tblockSize := int(binary.LittleEndian.Uint32(s.array[itemIndex : itemIndex+headerEntrySize]))\n\tentry := s.array[itemIndex+headerEntrySize : itemIndex+headerEntrySize+blockSize]\n\ts.lock.RUnlock()\n\treturn readEntry(entry), nil\n}\n\nfunc readEntry(data []byte) []byte {\n\tdst := make([]byte, len(data))\n\tcopy(dst, data)\n\n\treturn dst\n}\n\nfunc wrapEntry(entry []byte) []byte {\n\t// You can put more information like a timestamp if you want.\n\tblobLength := len(entry)\n\tblob := make([]byte, blobLength)\n\tcopy(blob, entry)\n\treturn blob\n}\n\n```\n\nmain.go\n\n```go\npackage main\n\nimport \"fmt\"\n\nfunc main() {\n\tcache := newCache()\n\tcache.set(\"key\", []byte(\"the value\"))\n\n\tvalue, err := cache.get(\"key\")\n\tif err != nil {\n\t\tfmt.Println(err)\n\t}\n\n\tfmt.Println(string(value))\n\t// OUTPUT:\n\t// the value\n}\n\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdouglasmakey%2Fcache-example","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdouglasmakey%2Fcache-example","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdouglasmakey%2Fcache-example/lists"}