{"id":23369770,"url":"https://github.com/viccon/sturdyc","last_synced_at":"2025-05-14T23:06:35.141Z","repository":{"id":231906402,"uuid":"783002688","full_name":"viccon/sturdyc","owner":"viccon","description":"A caching library with advanced concurrency features designed to make I/O heavy applications robust and highly performant","archived":false,"fork":false,"pushed_at":"2025-04-04T21:02:59.000Z","size":433,"stargazers_count":1198,"open_issues_count":5,"forks_count":30,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-05-10T14:16:37.202Z","etag":null,"topics":["cache","concurrency","go","golang","performance"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/viccon.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-04-06T17:00:40.000Z","updated_at":"2025-05-10T04:29:31.000Z","dependencies_parsed_at":"2024-06-07T07:42:32.664Z","dependency_job_id":"8761abc2-dd3e-41ea-b331-8dc7c4f94e0c","html_url":"https://github.com/viccon/sturdyc","commit_stats":null,"previous_names":["creativecreature/sturdyc","viccon/sturdyc"],"tags_count":34,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/viccon%2Fsturdyc","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/viccon%2Fsturdyc/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/viccon%2Fsturdyc/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/viccon%2Fsturdyc/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/viccon","download_url":"https://codeload.github.com/viccon/sturdyc/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254243360,"owners_count":22038046,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cache","concurrency","go","golang","performance"],"created_at":"2024-12-21T15:05:57.745Z","updated_at":"2025-05-14T23:06:30.134Z","avatar_url":"https://github.com/viccon.png","language":"Go","readme":"![sturdyC-fn-2](https://github.com/user-attachments/assets/2def120a-ad2b-4590-bef0-83c461af1b07)\n\u003e A sturdy gopher shielding data sources from rapidly incoming requests.\n\n# `sturdyc`: a caching library for building sturdy systems\n\n[![Mentioned in Awesome Go](https://awesome.re/mentioned-badge.svg)](https://github.com/avelino/awesome-go)\n[![Go Reference](https://pkg.go.dev/badge/github.com/viccon/sturdyc.svg)](https://pkg.go.dev/github.com/viccon/sturdyc)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/viccon/sturdyc/blob/master/LICENSE)\n[![Go Report Card](https://goreportcard.com/badge/github.com/viccon/sturdyc)](https://goreportcard.com/report/github.com/viccon/sturdyc)\n[![Test](https://github.com/viccon/sturdyc/actions/workflows/main.yml/badge.svg)](https://github.com/viccon/sturdyc/actions/workflows/main.yml)\n[![codecov](https://codecov.io/gh/viccon/sturdyc/graph/badge.svg?token=CYSKW3Z7E6)](https://codecov.io/gh/viccon/sturdyc)\n\n\n`sturdyc` eliminates cache stampedes and can minimize data source load in\nhigh-throughput systems through features such as request coalescing and\nasynchronous refreshes. It combines the speed of in-memory caching with\ngranular control over data freshness. At its core, `sturdyc` provides\n**non-blocking reads** and **sharded writes** for minimal lock contention. The\n[xxhash](https://github.com/cespare/xxhash) algorithm is used for efficient key\ndistribution.\n\nIt has all the functionality you would expect from a caching library, but what\n**sets it apart** are the flexible configurations that have been designed to\nmake I/O heavy applications both _robust_ and _highly performant_.\n\nWe have been using this package in production to enhance both the performance\nand reliability of our services that retrieve data from distributed caches,\ndatabases, and external APIs. While the API surface of sturdyc is tiny, it\noffers extensive configuration options. I encourage you to read through this\nREADME and experiment with the examples in order to understand its full\ncapabilities.\n\nThis screenshot shows the P95 latency improvements we observed after adding\nthis package in front of a distributed key-value store:\n\n\u0026nbsp;\n\u003cimg width=\"1554\" alt=\"Screenshot 2024-05-10 at 10 15 18\" src=\"https://github.com/viccon/sturdyc/assets/12787673/adad1d4c-e966-4db1-969a-eda4fd75653a\"\u003e\n\u0026nbsp;\n\nAnd through a combination of inflight-tracking, asynchronous refreshes, and\nrefresh coalescing, we reduced load on underlying data sources by more than\n90%. This reduction in outgoing requests has enabled us to operate with fewer\ncontainers and significantly cheaper database clusters.\n\n# Table of contents\n\nBelow is the table of contents for what this README is going to cover. However,\nif this is your first time using this package, I encourage you to **read these\nexamples in the order they appear**. Most of them build on each other, and many\nshare configurations.\n\n- [**installing**](https://github.com/viccon/sturdyc?tab=readme-ov-file#installing)\n- [**creating a cache client**](https://github.com/viccon/sturdyc?tab=readme-ov-file#creating-a-cache-client)\n- [**evictions**](https://github.com/viccon/sturdyc?tab=readme-ov-file#evictions)\n- [**get or fetch**](https://github.com/viccon/sturdyc?tab=readme-ov-file#get-or-fetch)\n- [**stampede protection**](https://github.com/viccon/sturdyc?tab=readme-ov-file#stampede-protection)\n- [**early refreshes**](https://github.com/viccon/sturdyc?tab=readme-ov-file#early-refreshes)\n- [**deletions**](https://github.com/viccon/sturdyc?tab=readme-ov-file#deletions)\n- [**caching non-existent records**](https://github.com/viccon/sturdyc?tab=readme-ov-file#non-existent-records)\n- [**caching batch endpoints per record**](https://github.com/viccon/sturdyc?tab=readme-ov-file#batch-endpoints)\n- [**cache key permutations**](https://github.com/viccon/sturdyc?tab=readme-ov-file#cache-key-permutations)\n- [**refresh coalescing**](https://github.com/viccon/sturdyc?tab=readme-ov-file#refresh-coalescing)\n- [**request passthrough**](https://github.com/viccon/sturdyc?tab=readme-ov-file#passthrough)\n- [**distributed storage**](https://github.com/viccon/sturdyc?tab=readme-ov-file#distributed-storage)\n- [**custom metrics**](https://github.com/viccon/sturdyc?tab=readme-ov-file#custom-metrics)\n- [**generics**](https://github.com/viccon/sturdyc?tab=readme-ov-file#generics)\n\n# Installing\n\n```sh\ngo get github.com/viccon/sturdyc\n```\n\n# Creating a cache client\n\nThe first thing you will have to do is to create a cache client to hold your\nconfiguration:\n\n```go\n// Maximum number of entries in the cache. Exceeding this number will trigger\n// an eviction (as long as the \"evictionPercentage\" is greater than 0).\ncapacity := 10000\n// Number of shards to use. Increasing this number will reduce write lock collisions.\nnumShards := 10\n// Time-to-live for cache entries.\nttl := 2 * time.Hour\n// Percentage of entries to evict when the cache reaches its capacity. Setting this\n// to 0 will make writes a no-op until an item has either expired or been deleted.\nevictionPercentage := 10\n\n// Create a cache client with the specified configuration.\ncacheClient := sturdyc.New[int](capacity, numShards, ttl, evictionPercentage)\n\ncacheClient.Set(\"key1\", 99)\nlog.Println(cacheClient.Size())\nlog.Println(cacheClient.Get(\"key1\"))\n\ncacheClient.Delete(\"key1\")\nlog.Println(cacheClient.Size())\nlog.Println(cacheClient.Get(\"key1\"))\n```\n\n\nThe `New` function is variadic, and as the final argument we're also able to\nprovide a wide range of configuration options, which we will explore in detail\nin the sections to follow.\n\n# Evictions\n\nThe cache has two eviction strategies. One is a background job which\ncontinuously evicts expired records from each shard. However, there are options\nto both tweak the interval at which the job runs:\n\n```go\ncacheClient := sturdyc.New[int](capacity, numShards, ttl, evictionPercentage,\n    sturdyc.WithEvictionInterval(time.Second),\n)\n```\n\nas well as disabling the functionality altogether:\n\n```go\ncacheClient := sturdyc.New[int](capacity, numShards, ttl, evictionPercentage,\n    sturdyc.WithNoContinuousEvictions()\n)\n```\n\nThe latter can give you a slight performance boost in situations where you're\nunlikely to ever exceed the capacity you've assigned to your cache.\n\nHowever, if the capacity is reached, the second eviction strategy is triggered.\nThis process performs evictions on a per-shard basis, selecting records for\nremoval based on recency. The eviction algorithm uses\n[quickselect](https://en.wikipedia.org/wiki/Quickselect), which has an O(N)\ntime complexity without the overhead of requiring write locks on reads to\nupdate a recency list, as many LRU caches do.\n\nNext, we'll start to look at some of the more _advanced features_.\n\n# Get or fetch\n\nI have tried to design the API in a way that should make the process of\nintegrating `sturdyc` with any data source as straightforward as possible.\nWhile it provides the basic get/set methods you would expect from a cache, the\nadvanced functionality is accessed through just two core functions:\n`GetOrFetch` and `GetOrFetchBatch`\n\nAs an example, let's say that we had the following code for fetching orders\nfrom an API:\n\n```go\nfunc (c *Client) Order(ctx context.Context, id string) (Order, error) {\n\ttimeoutCtx, cancel := context.WithTimeout(ctx, c.timeout)\n\tdefer cancel()\n\n\tvar response Order\n\terr := requests.URL(c.orderURL).\n\t\tPathf(\"/order/%s\", id).\n\t\tToJSON(\u0026response).\n\t\tFetch(timeoutCtx)\n\n\treturn response, err\n}\n```\n\nAll we would have to do is wrap the lines of code that retrieves the data in a\nfunction, and then hand that over to our cache client:\n\n```go\nfunc (c *Client) Order(ctx context.Context, id string) (Order, error) {\n\tfetchFunc := func(ctx context.Context) (Order, error) {\n\t\ttimeoutCtx, cancel := context.WithTimeout(ctx, c.timeout)\n\t\tdefer cancel()\n\n\t\tvar response Order\n\t\terr := requests.URL(c.orderURL).\n\t\t\tPathf(\"/order/%s\", id).\n\t\t\tToJSON(\u0026response).\n\t\t\tFetch(timeoutCtx)\n\n\t\treturn response, err\n\t}\n\n\treturn c.cache.GetOrFetch(ctx, id, fetchFunc)\n}\n```\n\nThe cache is going to return the value from memory if it's available, and\notherwise will call the `fetchFn` to retrieve the data from the underlying data\nsource.\n\nMost of our examples are going to be retrieving data from HTTP APIs, but it's\njust as easy to wrap a database query, a remote procedure call, a disk read, or\nany other I/O operation.\n\nThe `fetchFn` that we pass to `GetOrFetch` has the following function\nsignature:\n\n```go\ntype FetchFn[T any] func(ctx context.Context) (T, error)\n```\n\nFor data sources capable of handling requests for multiple records at once,\nwe'll use `GetOrFetchBatch`:\n\n```go\ntype KeyFn func(id string) string\n\ntype BatchFetchFn[T any] func(ctx context.Context, ids []string) (map[string]T, error)\n\nfunc (c *Client[T]) GetOrFetchBatch(ctx context.Context, ids []string, keyFn KeyFn, fetchFn BatchFetchFn[T]) (map[string]T, error) {\n\t// ...\n}\n```\n\nThere are a few things to unpack here, so let's start with the `KeyFn`. When\nadding an in-memory cache to an API client capable of calling multiple\nendpoints, it's highly unlikely that an ID alone is going to be enough to\nuniquely identify a record.\n\nTo illustrate, let's say that we're building a Github client and want to use\nthis package to get around their rate limit. The username itself wouldn't make\nfor a good cache key because we could use it to fetch gists, commits,\nrepositories, etc. Therefore, `GetOrFetchBatch` takes a `KeyFn` that prefixes\neach ID with something to identify the data source so that we don't end up with\ncache key collisions:\n\n```go\ngistPrefixFn := cacheClient.BatchKeyFn(\"gists\")\ncommitPrefixFn := cacheClient.BatchKeyFn(\"commits\")\ngists, err := cacheClient.GetOrFetchBatch(ctx, userIDs, gistPrefixFn, fetchGists)\ncommits, err := cacheClient.GetOrFetchBatch(ctx, userIDs, commitPrefixFn, fetchCommits)\n```\n\nWe're now able to use the _same_ cache for _multiple_ data sources, and\ninternally we'd get cache keys of this format:\n\n```\ngists-ID-viccon\ngists-ID-some-other-user\ncommits-ID-viccon\ncommits-ID-some-other-user\n```\n\nNow, let's use a bit of our imagination because Github doesn't actually allow\nus to fetch gists from multiple users at once. However, if they did, our client\nwould probably look something like this:\n\n```go\nfunc (client *GithubClient) Gists(ctx context.Context, usernames []string) (map[string]Gist, error) {\n\tcacheKeyFn := client.cache.BatchKeyFn(\"gists\")\n\tfetchFunc := func(ctx context.Context, cacheMisses []string) (map[string]Gist, error) {\n\t\ttimeoutCtx, cancel := context.WithTimeout(ctx, client.timeout)\n\t\tdefer cancel()\n\n\t\tvar response map[string]Gist\n\t\terr := requests.URL(c.baseURL).\n\t\t\tPath(\"/gists\").\n\t\t\tParam(\"usernames\", strings.Join(cacheMisses, \",\")).\n\t\t\tToJSON(\u0026response).\n\t\t\tFetch(timeoutCtx)\n\t\treturn response, err\n\t}\n\treturn sturdyc.GetOrFetchBatch(ctx, client.cache, usernames, cacheKeyFn, fetchFunc)\n}\n```\n\nIn the example above, the `fetchFunc` would get called for users where we don't\nhave their gists in our cache, and the cacheMisses slice would contain their\nactual usernames (without the prefix from the keyFn).\n\nThe map that we return from our `fetchFunc` should have the IDs (in this case the\nusernames) as keys, and the actual data that we want to cache (the gist) as the\nvalue.\n\nLater, we'll see how we can use closures to pass query parameters and options\nto our fetch functions, as well as how to use the `PermutatedBatchKeyFn` to\ncreate unique cache keys for each permutation of them.\n\n# Stampede protection\n\n`sturdyc` provides automatic protection against cache stampedes (also known as\nthundering herd) - a situation that occurs when many requests for a particular\npiece of data, which has just expired or been evicted from the cache, come in\nat once.\n\nPreventing this has been one of the key objectives. We do not want to cause a\nsignificant load on an underlying data source every time one of our keys\nexpire. To address this, `sturdyc` performs _in-flight_ tracking for every key.\n\nWe can demonstrate this using the `GetOrFetch` function which, as I mentioned\nearlier, takes a key, and a function for retrieving the data if it's not in the\ncache. The cache is going to ensure that we never have more than a single\nin-flight request per key:\n\n```go\n\tvar count atomic.Int32\n\tfetchFn := func(_ context.Context) (int, error) {\n\t\t// Increment the count so that we can assert how many times this function was called.\n\t\tcount.Add(1)\n\t\ttime.Sleep(time.Second)\n\t\treturn 1337, nil\n\t}\n\n\t// Fetch the same key from 5 goroutines.\n\tvar wg sync.WaitGroup\n\tfor i := 0; i \u003c 5; i++ {\n\t\twg.Add(1)\n\t\tgo func() {\n\t\t\t// We'll ignore the error here for brevity.\n\t\t\tval, _ := cacheClient.GetOrFetch(context.Background(), \"key2\", fetchFn)\n\t\t\tlog.Printf(\"got value: %d\\n\", val)\n\t\t\twg.Done()\n\t\t}()\n\t}\n\twg.Wait()\n\n\tlog.Printf(\"fetchFn was called %d time\\n\", count.Load())\n\tlog.Println(cacheClient.Get(\"key2\"))\n\n```\n\nRunning this program we can see that we were able to retrieve the value for all\n5 goroutines, and that the fetchFn only got called once:\n\n```sh\n❯ go run .\n2024/05/21 08:06:29 got value: 1337\n2024/05/21 08:06:29 got value: 1337\n2024/05/21 08:06:29 got value: 1337\n2024/05/21 08:06:29 got value: 1337\n2024/05/21 08:06:29 got value: 1337\n2024/05/21 08:06:29 fetchFn was called 1 time\n2024/05/21 08:06:29 1337 true\n```\n\nThe in-flight tracking works for batch operations too where the cache is able\nto deduplicate a batch of cache misses, and then assemble the response by\npicking records from **multiple** in-flight requests.\n\nTo demonstrate this, we'll use the `GetOrFetchBatch` function, which as mentioned\nearlier, can be used to retrieve data from a data source capable of handling\nrequests for multiple records at once.\n\nWe'll start by creating a mock function that sleeps for `5` seconds, and then\nreturns a map with a numerical value for every ID:\n\n```go\nvar count atomic.Int32\nfetchFn := func(_ context.Context, ids []string) (map[string]int, error) {\n\t// Increment the counter so that we can assert how many times this function was called.\n\tcount.Add(1)\n\ttime.Sleep(time.Second * 5)\n\n\tresponse := make(map[string]int, len(ids))\n\tfor _, id := range ids {\n\t\tnum, _ := strconv.Atoi(id)\n\t\tresponse[id] = num\n\t}\n\n\treturn response, nil\n}\n```\n\nNext, we'll need some batches to test with, so here I've created three batches\nwith five IDs each:\n\n```go\nbatches := [][]string{\n\t{\"1\", \"2\", \"3\", \"4\", \"5\"},\n\t{\"6\", \"7\", \"8\", \"9\", \"10\"},\n\t{\"11\", \"12\", \"13\", \"14\", \"15\"},\n}\n```\n\nand we can now request each batch in a separate goroutine:\n\n```go\nfor _, batch := range batches {\n\tgo func() {\n\t\tres, _ := cacheClient.GetOrFetchBatch(context.Background(), batch, keyPrefixFn, fetchFn)\n\t\tlog.Printf(\"got batch: %v\\n\", res)\n\t}()\n}\n\n// Just to ensure that these batches are in fact in-flight, we'll sleep to give the goroutines a chance to run.\ntime.Sleep(time.Second * 2)\n```\n\nAt this point, the cache should have 3 in-flight requests for IDs 1-15:\n\n```sh\n[1,2,3,4,5]      =\u003e REQUEST 1 (IN-FLIGHT)\n[6,7,8,9,10]     =\u003e REQUEST 2 (IN-FLIGHT)\n[11,12,13,14,15] =\u003e REQUEST 3 (IN-FLIGHT)\n```\n\nKnowing this, let's test the stampede protection by launching another five\ngoroutines. Each of these goroutines will request two random IDs from our\nprevious batches. For example, they could request one ID from the first\nrequest, and another from the second or third.\n\n```go\n// Launch another 5 goroutines that are going to pick two random IDs from any of our in-flight batches.\n// e.g:\n// [1,8]\n// [4,11]\n// [14,2]\n// [6,15]\n\nfunc pickRandomValue(batches [][]string) string {\n\tbatch := batches[rand.IntN(len(batches))]\n\treturn batch[rand.IntN(len(batch))]\n}\n\nvar wg sync.WaitGroup\nfor i := 0; i \u003c 5; i++ {\n\twg.Add(1)\n\tgo func() {\n\t\tids := []string{pickRandomValue(batches), pickRandomValue(batches)}\n\t\tres, _ := cacheClient.GetOrFetchBatch(context.Background(), ids, keyPrefixFn, fetchFn)\n\t\tlog.Printf(\"got batch: %v\\n\", res)\n\t\twg.Done()\n\t}()\n}\n\nwg.Wait()\nlog.Printf(\"fetchFn was called %d times\\n\", count.Load())\n```\n\nRunning this program, and looking at the logs, we'll see that the cache is able\nto resolve all of the ids from these new goroutines without generating any\nadditional requests even though we're picking IDs from different in-flight\nrequests:\n\n```sh\n❯ go run .\n2024/05/21 09:14:23 got batch: map[8:8 9:9]\n2024/05/21 09:14:23 got batch: map[4:4 9:9] \u003c---- NOTE: ID 4 and 9 are part of different batches\n2024/05/21 09:14:23 got batch: map[11:11 12:12 13:13 14:14 15:15]\n2024/05/21 09:14:23 got batch: map[1:1 7:7] \u003c---- NOTE: ID 1 and 7 are part of different batches\n2024/05/21 09:14:23 got batch: map[10:10 6:6 7:7 8:8 9:9]\n2024/05/21 09:14:23 got batch: map[3:3 9:9] \u003c---- NOTE: ID 3 and 9 are part of different batches\n2024/05/21 09:14:23 got batch: map[1:1 2:2 3:3 4:4 5:5]\n2024/05/21 09:14:23 got batch: map[4:4 9:9] \u003c---- NOTE: ID 4 and 9 are part of different batches\n2024/05/21 09:14:23 fetchFn was called 3 times \u003c---- NOTE: We only generated 3 outgoing requests.\n```\n\nThe entire example is available [here.](https://github.com/viccon/sturdyc/tree/main/examples/basic)\n\n# Early refreshes\n\nServing data from memory is typically at least one to two orders of magnitude\nfaster than reading from disk, and if you have to retrieve the data across a\nnetwork the difference can grow even larger. Consequently, we're often able to\nsignificantly improve our applications performance by adding an in-memory\ncache.\n\nHowever, one has to be aware of the usual trade-offs. Suppose we use a TTL of\n10 seconds. That means the cached data can be up to 10 seconds old. In many\napplications this may be acceptable, but in others it can introduce stale\nreads. Additionally, once the cached value expires, the first request after\nexpiration must refresh the cache, resulting in a longer response time for that\nuser. This can make the average latency look very different from the P90–P99\ntail latencies, since those percentiles capture the delays of having to go to\nthe actual data source in order to refresh the cache. This in turn can make it\ndifficult to configure appropriate alarms for your applications response times.\n\n`sturdyc` aims to give you a lot of control over these choices when you enable\nthe **early refreshes** functionality. It will prevent your most frequently\nused records from ever expiring by continuously refreshing them in the\nbackground. This can have a significant impact on your applications latency.\nWe've seen the P99 of some of our applications go from 50ms down to 1.\n\nOne thing to note about these background refreshes is that they are scheduled\nif a key is **requested again** after a configurable amount of time has passed.\nThis is an important distinction because it means that the cache doesn't just\nnaively refresh every key it's ever seen. Instead, it only refreshes the\nrecords that are actually in active rotation, while allowing unused keys to be\ndeleted once their TTL expires. This also means that the request that gets\nchosen to refresh the value won’t retrieve the updated data right away as the\nrefresh happens asynchronously.\n\nHowever, asynchronous refreshes present challenges with infrequently requested\nkeys. While background refreshes keep latency low by serving cached values\nduring updates, this can lead to perpetually stale data. If a key isn't\nrequested again before its next scheduled refresh, we remain permanently one\nupdate behind, as each read triggers a refresh that won't be seen until the\nnext request. This is similar to a burger restaurant that prepares a new burger\nafter each customer's order - if the next customer arrives too late, they'll\nreceive a cold burger, despite the restaurant's proactive cooking strategy.\n\nTo solve this, you also get to provide a synchronous refresh time. This\nessentially tells the cache: \"If the data is older than x, I want the refresh\nto be blocking and have the user wait for the response.\" Or using the burger\nanalogy: if a burger has been sitting for more than X minutes, the restaurant\nstarts making a fresh one while the customer waits. Unlike a real restaurant\nthough, the cache keeps the old value as a fallback - if the refresh fails,\nwe'll still serve the \"cold burger\" rather than letting the customer go hungry.\n\nBelow is an example configuration that you can use to enable this\nfunctionality:\n\n```go\nfunc main() {\n\t// Set a minimum and maximum refresh delay for the records. This is used to\n\t// spread out the refreshes of our records evenly over time. If we're running\n\t// our application across 100 containers, we don't want to send a spike of\n\t// refreshes from every container every 30 ms. Instead, we'll use some\n\t// randomization to spread them out evenly between 10 and 30 ms.\n\tminRefreshDelay := time.Millisecond * 10\n\tmaxRefreshDelay := time.Millisecond * 30\n\t// Set a synchronous refresh delay for when we want a refresh to happen synchronously.\n\tsynchronousRefreshDelay := time.Second * 30\n\t// The base used for exponential backoff when retrying a background refresh.\n\t// Most of the time, we perform refreshes well in advance of the records\n\t// expiry time. Hence, we can use this to make it easier for a system that\n\t// is having trouble to get back on it's feet by making fewer refreshes when\n\t// we're seeing a lot of errors. Once we receive a successful response, the\n\t// refreshes return to their original frequency. You can set this to 0\n\t// if you don't want this behavior.\n\tretryBaseDelay := time.Millisecond * 10\n\n\t// Create a cache client with the specified configuration.\n\tcacheClient := sturdyc.New[string](capacity, numShards, ttl, evictionPercentage,\n\t\tsturdyc.WithEarlyRefreshes(minRefreshDelay, maxRefreshDelay, retryBaseDelay),\n\t)\n}\n```\n\nLet's build a simple API client that embeds the cache using our configuration:\n\n```go\ntype API struct {\n\t*sturdyc.Client[string]\n}\n\nfunc NewAPI(c *sturdyc.Client[string]) *API {\n\treturn \u0026API{c}\n}\n\nfunc (a *API) Get(ctx context.Context, key string) (string, error) {\n\t// This could be an API call, a database query, etc.\n\tfetchFn := func(_ context.Context) (string, error) {\n\t\tlog.Printf(\"Fetching value for key: %s\\n\", key)\n\t\treturn \"value\", nil\n\t}\n\treturn a.GetOrFetch(ctx, key, fetchFn)\n}\n```\n\nNow we can return to our `main` function to create an instance of it, and then\ncall the `Get` method in a loop:\n\n```go\nfunc main() {\n\t// ...\n\n\tcacheClient := sturdyc.New[string](...)\n\n\t// Create a new API instance with the cache client.\n\tapi := NewAPI(cacheClient)\n\n\t// We are going to retrieve the values every 10 milliseconds, however the\n\t// logs will reveal that actual refreshes fluctuate randomly within a 10-30\n\t// millisecond range.\n\tfor i := 0; i \u003c 100; i++ {\n\t\tval, err := api.Get(context.Background(), \"key\")\n\t\tif err != nil {\n\t\t\tlog.Println(\"Failed to  retrieve the record from the cache.\")\n\t\t\tcontinue\n\t\t}\n\t\tlog.Printf(\"Value: %s\\n\", val)\n\t\ttime.Sleep(minRefreshDelay)\n\t}\n}\n```\n\nRunning this program, we're going to see that the value gets refreshed\nasynchronously once every 2-3 retrievals:\n\n```sh\ncd examples/refreshes\ngo run .\n2024/04/07 09:05:29 Fetching value for key: key\n2024/04/07 09:05:29 Value: value\n2024/04/07 09:05:29 Value: value\n2024/04/07 09:05:29 Fetching value for key: key\n2024/04/07 09:05:29 Value: value\n2024/04/07 09:05:29 Value: value\n2024/04/07 09:05:29 Value: value\n2024/04/07 09:05:29 Fetching value for key: key\n...\n```\n\nIf this was a real application it would have reduced our response times\nsignificantly because none of our users would have to wait for the I/O\noperation that refreshes the data. It's always performed in the background as\nlong as the key is being continuously requested.\n\nWe don't have to be afraid that the data for infrequently used keys gets stale\neither, given that we set the synchronous refresh delay like this:\n\n```go\n\tsynchronousRefreshDelay := time.Second * 30\n```\n\nWhich means that if a key isn't requested again within 30 seconds, the cache\nwill make the refresh synchronous. Even if a minute has passed and 1,000\nrequests suddenly come in for this key, the stampede protection will kick in\nand make the refresh synchronous for all of them, while also ensuring that only\na single request is made to the underlying data source.\n\nSometimes I like to use this feature to provide a degraded experience when an\nupstream system encounters issues. For this, I choose a high TTL and a low\nrefresh time, so that when everything is working as expected, the records are\nrefreshed continuously. However, if the upstream system stops responding, I can\nrely on cached records for the entire duration of the TTL.\n\nThis also brings us to the final argument of the `WithEarlyRefreshes` function\nwhich is the retry base delay. This delay is used to create an exponential\nbackoff for our background requests if a data source starts to return errors.\nPlease note that this **only** applies to background refreshes. If we reach a\npoint where all of the records are older than the synchronous refresh time,\nwe're going to send a steady stream of outgoing requests. That is because I\nthink of the synchronous refresh time as \"I really don’t want the data to be\nolder than this, but I want the possibility of using an even higher TTL in\norder to serve stale.\" Therefore, if a synchronous refresh fails, I want the\nvery next request for that key to attempt another refresh.\n\nAlso, if you don't want any of this serve stale functionality you could just\nuse short TTLs. The cache will never return a record where the TTL has expired.\nI'm just trying to showcase some different ways to leverage this functionality!\n\nThe entire example is available [here.](https://github.com/viccon/sturdyc/tree/main/examples/refreshes)\n\n# Deletions\n\nWhat if a record gets deleted at the underlying data source? Our cache might\nuse a 2-hour-long TTL, and we definitely don't want it to take that long for\nthe deletion to propagate.\n\nHowever, if we were to modify our client from the previous example so that it\nreturns an error after the first request:\n\n```go\ntype API struct {\n\tcount int\n\t*sturdyc.Client[string]\n}\n\nfunc NewAPI(c *sturdyc.Client[string]) *API {\n\treturn \u0026API{0, c}\n}\n\nfunc (a *API) Get(ctx context.Context, key string) (string, error) {\n\tfetchFn := func(_ context.Context) (string, error) {\n\t\ta.count++\n\t\tlog.Printf(\"Fetching value for key: %s\\n\", key)\n\t\tif a.count == 1 {\n\t\t\treturn \"value\", nil\n\t\t}\n\t\treturn \"\", errors.New(\"error this key does not exist\")\n\t}\n\treturn a.GetOrFetch(ctx, key, fetchFn)\n}\n```\n\nand then run the program again:\n\n```sh\ncd examples/refreshes\ngo run .\n```\n\nWe'll see that the exponential backoff kicks in, delaying our background\nrefreshes which results in more iterations for every refresh, but the value is\nstill being printed:\n\n```sh\n2024/05/09 13:22:03 Fetching value for key: key\n2024/05/09 13:22:03 Value: value\n2024/05/09 13:22:03 Value: value\n2024/05/09 13:22:03 Value: value\n2024/05/09 13:22:03 Value: value\n2024/05/09 13:22:03 Value: value\n2024/05/09 13:22:03 Value: value\n2024/05/09 13:22:03 Value: value\n2024/05/09 13:22:04 Value: value\n2024/05/09 13:22:04 Value: value\n2024/05/09 13:22:04 Value: value\n2024/05/09 13:22:04 Value: value\n2024/05/09 13:22:04 Value: value\n2024/05/09 13:22:04 Value: value\n2024/05/09 13:22:04 Value: value\n2024/05/09 13:22:04 Value: value\n2024/05/09 13:22:04 Value: value\n2024/05/09 13:22:04 Value: value\n2024/05/09 13:22:04 Value: value\n2024/05/09 13:22:04 Value: value\n2024/05/09 13:22:04 Value: value\n2024/05/09 13:22:04 Value: value\n2024/05/09 13:22:04 Value: value\n2024/05/09 13:22:04 Value: value\n2024/05/09 13:22:04 Value: value\n2024/05/09 13:22:04 Value: value\n2024/05/09 13:22:04 Value: value\n2024/05/09 13:22:04 Value: value\n2024/05/09 13:22:04 Value: value\n2024/05/09 13:22:04 Value: value\n2024/05/09 13:22:04 Value: value\n2024/05/09 13:22:04 Fetching value for key: key\n```\n\nThis is a bit tricky because how you determine if a record has been deleted\ncould vary based on your data source. It could be a status code, zero value,\nempty list, specific error message, etc. There is no easy way for the cache to\nfigure this out implicitly.\n\nIt couldn't simply delete a record every time it receives an error. If an\nupstream system goes down, we want to be able to serve the data for the\nduration of the TTL, while reducing the frequency of our refreshes to make it\neasier for them to recover.\n\nTherefore, if a record is deleted, we'll have to explicitly inform the cache\nabout it by returning a custom error:\n\n```go\nfetchFn := func(_ context.Context) (string, error) {\n\t\ta.count++\n\t\tlog.Printf(\"Fetching value for key: %s\\n\", key)\n\t\tif a.count == 1 {\n\t\t\treturn \"value\", nil\n\t\t}\n\t\treturn \"\", sturdyc.ErrNotFound\n\t}\n```\n\nThis tells the cache that the record is no longer available at the underlying data source.\n\nIf we run this application again we'll see that it works, and that we're no\nlonger getting any cache hits. This leads to outgoing requests for every\niteration:\n\n```go\n2024/05/09 13:40:47 Fetching value for key: key\n2024/05/09 13:40:47 Value: value\n2024/05/09 13:40:47 Value: value\n2024/05/09 13:40:47 Value: value\n2024/05/09 13:40:47 Fetching value for key: key\n2024/05/09 13:40:47 Failed to  retrieve the record from the cache.\n2024/05/09 13:40:47 Fetching value for key: key\n2024/05/09 13:40:47 Failed to  retrieve the record from the cache.\n2024/05/09 13:40:47 Fetching value for key: key\n2024/05/09 13:40:47 Failed to  retrieve the record from the cache.\n2024/05/09 13:40:47 Fetching value for key: key\n2024/05/09 13:40:47 Failed to  retrieve the record from the cache.\n```\n\n**Please note** that we only have to return the `sturdyc.ErrNotFound` when\nwe're using `GetOrFetch`. For `GetOrFetchBatch`, we'll simply omit the key from the\nmap we're returning. I think this inconsistency is a little unfortunate, but it\nwas the best API I could come up with. Having to return an error like this if\njust a single ID wasn't found:\n\n```go\n\tbatchFetchFn := func(_ context.Context, cacheMisses []string) (map[string]string, error) {\n\t\tresponse, err := myDataSource(cacheMisses)\n\t\tfor _, id := range cacheMisses {\n\t\t\t// NOTE: Don't do this, it's just an example.\n\t\t\tif response[id]; !id {\n\t\t\t\treturn response, sturdyc.ErrNotFound\n\t\t\t}\n\t\t}\n\t\treturn response, nil\n\t}\n```\n\nand then have the cache either swallow that error and return nil, or return the\nmap with the error, felt much less intuitive.\n\nThis code is based on the example available [here.](https://github.com/viccon/sturdyc/tree/main/examples/refreshes)\n\n# Non-existent records\n\nIn the example above, we could see that once we delete the key, the following\niterations lead to a continuous stream of outgoing requests. This would also\nhappen for every ID that doesn't exist at the underlying data source. If we\ncan't retrieve it, we can't cache it. If we can't cache it, we can't serve it\nfrom memory. If this happens frequently, we'll experience a lot of I/O\noperations, which will significantly increase our system's latency.\n\nThe reasons why someone might request IDs that don't exist can vary. It could\nbe due to a faulty CMS configuration, or perhaps it's caused by a slow\ningestion process where it takes time for a new entity to propagate through a\ndistributed system. Regardless, this will negatively impact our systems\nperformance.\n\nTo address this issue, we can instruct the cache to mark these IDs as missing\nrecords. If you're using this functionality in combination with the\n`WithEarlyRefreshes` option, they are going to get refreshed at the same\nfrequency as regular records. Hence, if an ID is continuously requested, and\nthe upstream eventually returns a valid response, we'll see it propagate to our\ncache.\n\nTo illustrate, I'll make some small modifications to the code from the previous\nexample. I'm going to to make the API client return a `ErrNotFound` for the\nfirst three requests:\n\n```go\ntype API struct {\n\t*sturdyc.Client[string]\n\tcount int\n}\n\nfunc NewAPI(c *sturdyc.Client[string]) *API {\n\treturn \u0026API{c, 0}\n}\n\nfunc (a *API) Get(ctx context.Context, key string) (string, error) {\n\tfetchFn := func(_ context.Context) (string, error) {\n\t\ta.count++\n\t\tlog.Printf(\"Fetching value for key: %s\\n\", key)\n\t\tif a.count \u003e 3 {\n\t\t\treturn \"value\", nil\n\t\t}\n\t\t// This error tells the cache that the data does not exist at the source.\n\t\treturn \"\", sturdyc.ErrNotFound\n\t}\n\treturn a.GetOrFetch(ctx, key, fetchFn)\n}\n```\n\nNext, we'll just have to enable missing record storage which tells the cache\nthat anytime it gets a `ErrNotFound` error it should mark the key as missing:\n\n```go\nfunc main() {\n\t// ...\n\n\tcacheClient := sturdyc.New[string](capacity, numShards, ttl, evictionPercentage,\n\t\tsturdyc.WithEarlyRefreshes(minRefreshDelay, maxRefreshDelay, retryBaseDelay),\n\t\tsturdyc.WithMissingRecordStorage(),\n\t)\n\n\tapi := NewAPI(cacheClient)\n\n\t// ...\n\tfor i := 0; i \u003c 100; i++ {\n\t\tval, err := api.Get(context.Background(), \"key\")\n\t\t// The cache returns ErrMissingRecord for any key that has been marked as missing.\n\t\t// You can use this to exit-early, or return some type of default state.\n\t\tif errors.Is(err, sturdyc.ErrMissingRecord) {\n\t\t\tlog.Println(\"Record does not exist.\")\n\t\t}\n\t\tif err == nil {\n\t\t\tlog.Printf(\"Value: %s\\n\", val)\n\t\t}\n\t\ttime.Sleep(minRefreshDelay)\n\t}\n}\n```\n\nRunning this program, we'll see that the record is missing during the first 3\nrefreshes, and then transitions into having a value:\n\n```sh\n❯ go run .\n2024/05/09 21:25:28 Fetching value for key: key\n2024/05/09 21:25:28 Record does not exist.\n2024/05/09 21:25:28 Record does not exist.\n2024/05/09 21:25:28 Record does not exist.\n2024/05/09 21:25:28 Fetching value for key: key\n2024/05/09 21:25:28 Record does not exist.\n2024/05/09 21:25:28 Record does not exist.\n2024/05/09 21:25:28 Fetching value for key: key\n2024/05/09 21:25:28 Record does not exist.\n2024/05/09 21:25:28 Record does not exist.\n2024/05/09 21:25:28 Fetching value for key: key\n2024/05/09 21:25:28 Value: value // Look, the value exists now!\n2024/05/09 21:25:28 Value: value\n2024/05/09 21:25:28 Value: value\n2024/05/09 21:25:28 Fetching value for key: key\n...\n```\n\n**Please note** that this functionality is _implicit_ for `GetOrFetchBatch`.\nYou simply just have to omit the key from the map:\n\n```go\n\tbatchFetchFn := func(_ context.Context, cacheMisses []string) (map[string]string, error) {\n\t\t// The cache will check if every ID in cacheMisses is present in the response.\n\t\t// If it finds any IDs that are missing it will proceed to mark them as missing\n\t\t// if missing record storage is enabled.\n\t\tresponse, err := myDataSource(cacheMisses)\n\t\treturn response, nil\n\t}\n```\n\nThe entire example is available [here.](https://github.com/viccon/sturdyc/tree/main/examples/missing)\n\n# Batch endpoints\n\nOne challenge with caching batchable endpoints is that you have to find a way\nto reduce the number of cache keys. Consider an endpoint that allows fetching\n10,000 records in batches of 20. The IDs for the batch are supplied as query\nparameters, for example, `https://example.com?ids=1,2,3,4,5,...20`. If we were\nto use this as the cache key, the way many CDNs would, we could quickly\ncalculate the number of keys we would generate like this:\n\n$$ C(n, k) = \\binom{n}{k} = \\frac{n!}{k!(n-k)!} $$\n\nFor $n = 10,000$ and $k = 20$, this becomes:\n\n$$ C(10,000, 20) = \\binom{10,000}{20} = \\frac{10,000!}{20!(10,000-20)!} $$\n\nThis results in an approximate value of:\n\n$$ \\approx 4.032 \\times 10^{61} $$\n\nand this is if we're sending perfect batches of 20. If we were to do 1 to 20\nIDs (not just exactly 20 each time) the total number of combinations would be\nthe sum of combinations for each k from 1 to 20.\n\nAt this point, the hit rate for each key would be so low that we'd have better\nodds of winning the lottery.\n\nTo prevent this, `sturdyc` pulls the response apart and caches each record\nindividually. This effectively prevents super-polynomial growth in the number\nof cache keys because the batch itself is never going to be included in the\nkey.\n\nTo get a better feeling for how this works, we can look at the function signature\nfor the `GetOrFetchBatch` function:\n\n```go\nfunc (c *Client[T]) GetOrFetchBatch(ctx context.Context, ids []string, keyFn KeyFn, fetchFn BatchFetchFn[T]) (map[string]T, error) {}\n```\n\nWhat the cache does is that it takes the IDs, applies the `keyFn` to them, and\nthen checks each key individually if it's present in the cache. The keys that\naren't present will be passed to the `fetchFn`.\n\nThe `fetchFn` has this signature where it returns a map where the ID is the\nkey:\n\n```go\ntype BatchFetchFn[T any] func(ctx context.Context, ids []string) (map[string]T, error)\n```\n\nThe cache can use this to iterate through the response map, again apply the\n`keyFn` to each ID, and then store each record individually.\n\nSometimes, the function signature for the `BatchFetchFn` can feel too limited.\nYou may need additional options and not just the IDs to retrieve the data. But\ndon't worry, we'll look at how to solve this in the next section!\n\nFor now, to get some code to play around with, let's once again build a small\nexample application. This time, we'll start with the API client:\n\n```go\ntype API struct {\n\t*sturdyc.Client[string]\n}\n\nfunc NewAPI(c *sturdyc.Client[string]) *API {\n\treturn \u0026API{c}\n}\n\nfunc (a *API) GetBatch(ctx context.Context, ids []string) (map[string]string, error) {\n\t// We are going to pass a cache a key function that prefixes each id with\n\t// the string \"some-prefix\", and adds an -ID- separator before the actual\n\t// id. This makes it possible to save the same id for different data\n\t// sources as the keys would look something like this: some-prefix-ID-1234\n\tcacheKeyFn := a.BatchKeyFn(\"some-prefix\")\n\n\t// The fetchFn is only going to retrieve the IDs that are not in the cache. Please\n\t// note that the cacheMisses is going to contain the actual IDs, not the cache keys.\n\tfetchFn := func(_ context.Context, cacheMisses []string) (map[string]string, error) {\n\t\tlog.Printf(\"Cache miss. Fetching ids: %s\\n\", strings.Join(cacheMisses, \", \"))\n\t\t// Batch functions should return a map where the key is the id of the record.\n\t\tresponse := make(map[string]string, len(cacheMisses))\n\t\tfor _, id := range cacheMisses {\n\t\t\tresponse[id] = \"value\"\n\t\t}\n\t\treturn response, nil\n\t}\n\n\treturn a.GetOrFetchBatch(ctx, ids, cacheKeyFn, fetchFn)\n}\n```\n\nand we're going to use the same cache configuration as the previous example, so\nI've omitted it for brevity:\n\n```go\nfunc main() {\n\t// ...\n\n\t// Create a new API instance with the cache client.\n\tapi := NewAPI(cacheClient)\n\n\t// Make an initial call to make sure that IDs 1-10 are retrieved and cached.\n\tlog.Println(\"Seeding ids 1-10\")\n\tids := []string{\"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\", \"10\"}\n\tapi.GetBatch(context.Background(), ids)\n\tlog.Println(\"Seed completed\")\n\n\t// To demonstrate that the records have been cached individually, we can continue\n\t// fetching a random subset of records from the original batch, plus a new\n\t// ID. By examining the logs, we should be able to see that the cache only\n\t// fetches the ID that wasn't present in the original batch, indicating that\n\t// the batch itself isn't part of the key.\n\tfor i := 1; i \u003c= 100; i++ {\n\t\t// Get N ids from the original batch.\n\t\trecordsToFetch := rand.IntN(10) + 1\n\t\tbatch := make([]string, recordsToFetch)\n\t\tcopy(batch, ids[:recordsToFetch])\n\t\t// Add a random ID between 1 and 100 to the batch.\n\t\tbatch = append(batch, strconv.Itoa(rand.IntN(1000)+10))\n\t\tvalues, _ := api.GetBatch(context.Background(), batch)\n\t\t// Print the records we retrieved from the cache.\n\t\tlog.Println(values)\n\t}\n}\n```\n\nRunning this code, we can see that we only end up fetching the randomized ID,\nwhile continuously getting cache hits for IDs 1-10, regardless of what the\nbatch looks like:\n\n```sh\n2024/04/07 11:09:58 Seed completed\n2024/04/07 11:09:58 Cache miss. Fetching ids: 173\n2024/04/07 11:09:58 map[1:value 173:value 2:value 3:value 4:value]\n2024/04/07 11:09:58 Cache miss. Fetching ids: 12\n2024/04/07 11:09:58 map[1:value 12:value 2:value 3:value 4:value]\n2024/04/07 11:09:58 Cache miss. Fetching ids: 730\n2024/04/07 11:09:58 map[1:value 2:value 3:value 4:value 730:value]\n2024/04/07 11:09:58 Cache miss. Fetching ids: 520\n2024/04/07 11:09:58 map[1:value 2:value 3:value 4:value 5:value 520:value 6:value 7:value 8:value]\n...\n```\n\nThe entire example is available [here.](https://github.com/viccon/sturdyc/tree/main/examples/batch)\n\n# Cache key permutations\n\nAs I mentioned in the previous section, the function signature for the\n`BatchFetchFn`, which the `GetOrFetchBatch` function uses, can feel too limited:\n\n```go\ntype BatchFetchFn[T any] func(ctx context.Context, ids []string) (map[string]T, error)\n```\n\nWhat if you're fetching data from some endpoint that accepts a variety of query\nparameters? Or perhaps you're doing a database query and want to apply some\nordering and filtering to the data?\n\nClosures provide an elegant solution to this limitation. Let's illustrate this\nby looking at an actual API client I've written:\n\n```go\nconst moviesByIDsCacheKeyPrefix = \"movies-by-ids\"\n\ntype MoviesByIDsOpts struct {\n\tIncludeUpcoming bool\n\tIncludeUpsell   bool\n}\n\nfunc (c *Client) MoviesByIDs(ctx context.Context, ids []string, opts MoviesByIDsOpts) (map[string]Movie, error) {\n\tcacheKeyFunc := c.cache.PermutatedBatchKeyFn(moviesByIDsCacheKeyPrefix, opts)\n\tfetchFunc := func(ctx context.Context, cacheMisses []string) (map[string]Movie, error) {\n\t\ttimeoutCtx, cancel := context.WithTimeout(ctx, c.timeout)\n\t\tdefer cancel()\n\n\t\tvar response map[string]Movie\n\t\terr := requests.URL(c.baseURL).\n\t\t\tPath(\"/movies\").\n\t\t\tParam(\"ids\", strings.Join(cacheMisses, \",\")).\n\t\t\tParam(\"include_upcoming\", strconv.FormatBool(opts.IncludeUpcoming)).\n\t\t\tParam(\"include_upsell\", strconv.FormatBool(opts.IncludeUpsell)).\n\t\t\tToJSON(\u0026response).\n\t\t\tFetch(timeoutCtx)\n\t\treturn response, err\n\t}\n\treturn sturdyc.GetOrFetchBatch(ctx, c.cache, ids, cacheKeyFunc, fetchFunc)\n}\n```\n\nThe API clients `MoviesByIDs` method calls an external API to fetch movies by\nIDs, and the `BatchFetchFn` that we're passing to `sturdyc` has a closure over\nthe query parameters we need.\n\nHowever, one **important** thing to note here is that the ID is **no longer**\nenough to _uniquely_ identify a record in our cache even with the basic prefix\nfunction we've used before. It will no longer work to just have cache keys that\nlooks like this:\n\n```\nmovies-ID-1\nmovies-ID-2\nmovies-ID-3\n```\n\nNow why is that? If you think about it, the query parameters will most likely\nbe used by the system we're calling to transform the data in various ways.\nHence, we need to store a movie not only once per ID, but also once per\ntransformation. In other terms, we should cache each movie once for each\npermutation of our options:\n\n```\nID 1 IncludeUpcoming: true  IncludeUpsell: true\nID 1 IncludeUpcoming: false IncludeUpsell: false\nID 1 IncludeUpcoming: true  IncludeUpsell: false\nID 1 IncludeUpcoming: false IncludeUpsell: true\n```\n\nThis is what the `PermutatedBatchKeyFn` is used for. It takes a prefix and a\nstruct which internally it uses reflection on in order to concatenate the\n**exported** fields to form a unique cache key that would look like this:\n\n```\n// movies-by-ids is our prefix that we passed as the\n// first argument to the PermutatedBatchKeyFn function.\nmovies-by-ids-true-true-ID-1\nmovies-by-ids-false-false-ID-1\nmovies-by-ids-true-false-ID-1\nmovies-by-ids-false-true-ID-1\n```\n\nPlease note that the struct should be flat without nesting. The fields can be\n`time.Time` values, as well as any basic types, pointers to these types, and\nslices containing them.\n\nOnce again, I'll provide a small example application that you can play around\nwith to get a deeper understanding of this functionality. We're essentially\ngoing to use the same API client as before, but this time we're going to use\nthe `PermutatedBatchKeyFn` rather than the `BatchKeyFn`:\n\n```go\ntype OrderOptions struct {\n\tCarrierName        string\n\tLatestDeliveryTime string\n}\n\ntype OrderAPI struct {\n\t*sturdyc.Client[string]\n}\n\nfunc NewOrderAPI(c *sturdyc.Client[string]) *OrderAPI {\n\treturn \u0026OrderAPI{c}\n}\n\nfunc (a *OrderAPI) OrderStatus(ctx context.Context, ids []string, opts OrderOptions) (map[string]string, error) {\n\t// We use the PermutedBatchKeyFn when an ID isn't enough to uniquely identify a\n\t// record. The cache is going to store each id once per set of options.\n\tcacheKeyFn := a.PermutatedBatchKeyFn(\"key\", opts)\n\n\t// We'll create a fetchFn with a closure that captures the options. For this\n\t// simple example, it logs and returns the status for each order, but you could\n\t// just as easily have called an external API.\n\tfetchFn := func(_ context.Context, cacheMisses []string) (map[string]string, error) {\n\t\tlog.Printf(\"Fetching: %v, carrier: %s, delivery time: %s\\n\", cacheMisses, opts.CarrierName, opts.LatestDeliveryTime)\n\t\tresponse := map[string]string{}\n\t\tfor _, id := range cacheMisses {\n\t\t\tresponse[id] = fmt.Sprintf(\"Available for %s\", opts.CarrierName)\n\t\t}\n\t\treturn response, nil\n\t}\n\treturn a.GetOrFetchBatch(ctx, ids, cacheKeyFn, fetchFn)\n}\n```\n\nNow, let's try to use this client:\n\n```go\nfunc main() {\n\t// ...\n\n\t// Create a new cache client with the specified configuration.\n\tcacheClient := sturdyc.New[string](capacity, numShards, ttl, evictionPercentage,\n\t\tsturdyc.WithEarlyRefreshes(minRefreshDelay, maxRefreshDelay, retryBaseDelay),\n\t)\n\n\t// We will fetch these IDs using three different option sets.\n\tids := []string{\"id1\", \"id2\", \"id3\"}\n\toptionSetOne := OrderOptions{CarrierName: \"FEDEX\", LatestDeliveryTime: \"2024-04-06\"}\n\toptionSetTwo := OrderOptions{CarrierName: \"DHL\", LatestDeliveryTime: \"2024-04-07\"}\n\toptionSetThree := OrderOptions{CarrierName: \"UPS\", LatestDeliveryTime: \"2024-04-08\"}\n\n\torderClient := NewOrderAPI(cacheClient)\n\tctx := context.Background()\n\n\t// Next, we'll call the orderClient to make sure that we've retrieved and cached\n\t// these IDs for all of our option sets.\n\tlog.Println(\"Filling the cache with all IDs for all option sets\")\n\torderClient.OrderStatus(ctx, ids, optionSetOne)\n\torderClient.OrderStatus(ctx, ids, optionSetTwo)\n\torderClient.OrderStatus(ctx, ids, optionSetThree)\n\tlog.Println(\"Cache filled\")\n}\n```\n\nAt this point, the cache has stored each record individually for each option\nset. The keys would look something like this:\n\n```\nFEDEX-2024-04-06-ID-1\nDHL-2024-04-07-ID-1\nUPS-2024-04-08-ID-1\netc..\n```\n\nNext, we'll add a sleep to make sure that all of the records are due for a\nrefresh, and then request the ids individually for each set of options:\n\n```go\nfunc main() {\n\t// ...\n\n\t// Sleep to make sure that all records are due for a refresh.\n\ttime.Sleep(maxRefreshDelay + 1)\n\n\t// Fetch each id for each option set.\n\tfor i := 0; i \u003c len(ids); i++ {\n\t\t// NOTE: We're using the same ID for these requests.\n\t\torderClient.OrderStatus(ctx, []string{ids[i]}, optionSetOne)\n\t\torderClient.OrderStatus(ctx, []string{ids[i]}, optionSetTwo)\n\t\torderClient.OrderStatus(ctx, []string{ids[i]}, optionSetThree)\n\t}\n\n\t// Sleep for a second to allow the refresh logs to print.\n\ttime.Sleep(time.Second)\n}\n```\n\nRunning this program, we can see that the records are refreshed once per unique\nid+option combination:\n\n```sh\ngo run .\n2024/04/07 13:33:56 Filling the cache with all IDs for all option sets\n2024/04/07 13:33:56 Fetching: [id1 id2 id3], carrier: FEDEX, delivery time: 2024-04-06\n2024/04/07 13:33:56 Fetching: [id1 id2 id3], carrier: DHL, delivery time: 2024-04-07\n2024/04/07 13:33:56 Fetching: [id1 id2 id3], carrier: UPS, delivery time: 2024-04-08\n2024/04/07 13:33:56 Cache filled\n2024/04/07 13:33:58 Fetching: [id1], carrier: FEDEX, delivery time: 2024-04-06\n2024/04/07 13:33:58 Fetching: [id1], carrier: UPS, delivery time: 2024-04-08\n2024/04/07 13:33:58 Fetching: [id1], carrier: DHL, delivery time: 2024-04-07\n2024/04/07 13:33:58 Fetching: [id2], carrier: UPS, delivery time: 2024-04-08\n2024/04/07 13:33:58 Fetching: [id2], carrier: FEDEX, delivery time: 2024-04-06\n2024/04/07 13:33:58 Fetching: [id2], carrier: DHL, delivery time: 2024-04-07\n2024/04/07 13:33:58 Fetching: [id3], carrier: FEDEX, delivery time: 2024-04-06\n2024/04/07 13:33:58 Fetching: [id3], carrier: UPS, delivery time: 2024-04-08\n2024/04/07 13:33:58 Fetching: [id3], carrier: DHL, delivery time: 2024-04-07\n```\n\nThe entire example is available [here.](https://github.com/viccon/sturdyc/tree/main/examples/permutations)\n\n# Refresh coalescing\n\nAs you may recall, our client is using the `WithEarlyRefreshes` option to\nrefresh the records in the background whenever their keys are requested again\nafter a certain amount of time has passed. And as seen in the example above,\nwe're successfully storing and refreshing the records once for every\npermutation of the options we used to retrieve it. However, we're not taking\nadvantage of the endpoint's batch capabilities.\n\nTo make this more efficient, we can enable the **refresh coalescing**\nfunctionality, but before we'll update our example to use it let's just take a\nmoment to understand how it works.\n\nTo start, we need to understand what determines whether two IDs can be\ncoalesced for a refresh: *the options*. E.g, do we want to perform the same\ndata transformations for both IDs? If so, they can be sent in the same batch.\nThis applies when we use the cache in front of a database too. Do we want to\nuse the same filters, sorting, etc?\n\nIf we look at the movie example from before, you can see that I've extracted\nthese options into a struct:\n\n```go\nconst moviesByIDsCacheKeyPrefix = \"movies-by-ids\"\n\ntype MoviesByIDsOpts struct {\n\tIncludeUpcoming bool\n\tIncludeUpsell   bool\n}\n\nfunc (c *Client) MoviesByIDs(ctx context.Context, ids []string, opts MoviesByIDsOpts) (map[string]Movie, error) {\n\tcacheKeyFunc := c.cache.PermutatedBatchKeyFn(moviesByIDsCacheKeyPrefix, opts)\n\tfetchFunc := func(ctx context.Context, cacheMisses []string) (map[string]Movie, error) {\n\t\t// ...\n\t\tdefer cancel()\n\t}\n\treturn sturdyc.GetOrFetchBatch(ctx, c.cache, ids, cacheKeyFunc, fetchFunc)\n}\n```\n\nAnd as I mentioned before, the `PermutatedBatchKeyFn` is going to perform\nreflection on this struct to create cache keys that look something like this:\n\n```\nmovies-by-ids-true-true-ID-1\nmovies-by-ids-false-false-ID-1\nmovies-by-ids-true-false-ID-1\nmovies-by-ids-false-true-ID-1\n```\n\nWhat the refresh coalescing functionality then does is that it removes the ID\nbut keeps the permutation string and uses it to create and uniquely\nidentifiable buffer where it can gather IDs that should be refreshed with the\nsame options:\n\n```\nmovies-by-ids-true-true\nmovies-by-ids-false-false\nmovies-by-ids-true-false\nmovies-by-ids-false-true\n```\n\nThe only change we have to make to the previous example is to enable this\nfeature:\n\n```go\nfunc main() {\n\t// ...\n\n\t// With refresh coalescing enabled, the cache will buffer refreshes\n\t// until the batch size is reached or the buffer timeout is hit.\n\tbatchSize := 3\n\tbatchBufferTimeout := time.Second * 30\n\n\t// Create a new cache client with the specified configuration.\n\tcacheClient := sturdyc.New[string](capacity, numShards, ttl, evictionPercentage,\n\t\tsturdyc.WithEarlyRefreshes(minRefreshDelay, maxRefreshDelay, retryBaseDelay),\n\t\tsturdyc.WithRefreshCoalescing(batchSize, batchBufferTimeout),\n\t)\n\n\t// ...\n}\n```\n\nSo now we're saying that we want to coalesce the refreshes for each\npermutation, and try to process them in batches of 3. However, if it's not able\nto reach that size within 30 seconds we want the refresh to happen anyway.\n\nThe previous output revealed that the refreshes happened one by one:\n\n```sh\ngo run .\n2024/04/07 13:33:56 Filling the cache with all IDs for all option sets\n2024/04/07 13:33:56 Fetching: [id1 id2 id3], carrier: FEDEX, delivery time: 2024-04-06\n2024/04/07 13:33:56 Fetching: [id1 id2 id3], carrier: DHL, delivery time: 2024-04-07\n2024/04/07 13:33:56 Fetching: [id1 id2 id3], carrier: UPS, delivery time: 2024-04-08\n2024/04/07 13:33:56 Cache filled\n2024/04/07 13:33:58 Fetching: [id1], carrier: FEDEX, delivery time: 2024-04-06\n2024/04/07 13:33:58 Fetching: [id1], carrier: UPS, delivery time: 2024-04-08\n2024/04/07 13:33:58 Fetching: [id1], carrier: DHL, delivery time: 2024-04-07\n2024/04/07 13:33:58 Fetching: [id2], carrier: UPS, delivery time: 2024-04-08\n2024/04/07 13:33:58 Fetching: [id2], carrier: FEDEX, delivery time: 2024-04-06\n2024/04/07 13:33:58 Fetching: [id2], carrier: DHL, delivery time: 2024-04-07\n2024/04/07 13:33:58 Fetching: [id3], carrier: FEDEX, delivery time: 2024-04-06\n2024/04/07 13:33:58 Fetching: [id3], carrier: UPS, delivery time: 2024-04-08\n2024/04/07 13:33:58 Fetching: [id3], carrier: DHL, delivery time: 2024-04-07\n```\n\nWe'll now try to run this code again, but with the `WithRefreshCoalescing`\noption enabled:\n\n```sh\ngo run .\n2024/04/07 13:45:42 Filling the cache with all IDs for all option sets\n2024/04/07 13:45:42 Fetching: [id1 id2 id3], carrier: FEDEX, delivery time: 2024-04-06\n2024/04/07 13:45:42 Fetching: [id1 id2 id3], carrier: DHL, delivery time: 2024-04-07\n2024/04/07 13:45:42 Fetching: [id1 id2 id3], carrier: UPS, delivery time: 2024-04-08\n2024/04/07 13:45:42 Cache filled\n2024/04/07 13:45:44 Fetching: [id1 id2 id3], carrier: FEDEX, delivery time: 2024-04-06\n2024/04/07 13:45:44 Fetching: [id1 id3 id2], carrier: DHL, delivery time: 2024-04-07\n2024/04/07 13:45:44 Fetching: [id1 id2 id3], carrier: UPS, delivery time: 2024-04-08\n```\n\nThe number of refreshes went from **9** to **3**. Imagine what a batch size of\n50 could do for your applications performance!\n\nThere is more information about this in the section about metrics, but for our\nproduction applications we're also using the `WithMetrics` option so that we\ncan monitor how well our refreshes are performing:\n\n\u003cimg width=\"941\" alt=\"Screenshot 2024-05-04 at 12 38 04\" src=\"https://github.com/viccon/sturdyc/assets/12787673/b1359867-f1ef-4a09-8c75-d7d2360726f1\"\u003e\nThis chart shows the batch sizes for our coalesced refreshes.\n\n\u003cimg width=\"940\" alt=\"Screenshot 2024-05-04 at 12 38 20\" src=\"https://github.com/viccon/sturdyc/assets/12787673/de7f00ee-b14d-443b-b69e-91e19665c252\"\u003e\nThis chart shows the average batch size of our refreshes for two different data sources\n\nThe entire example is available [here.](https://github.com/viccon/sturdyc/tree/main/examples/buffering)\n\nAnother point to note is how effectively the options we've seen so far can be\ncombined to create high-performing, flexible, and robust caching solutions:\n\n```go\ncapacity := 10000\nnumShards := 10\nttl := 2 * time.Hour\nevictionPercentage := 10\nminRefreshDelay := time.Second\nmaxRefreshDelay := time.Second * 2\nsynchronousRefreshDelay := time.Second * 120 // 2 minutes.\nretryBaseDelay := time.Millisecond * 10\nbatchSize := 10\nbatchBufferTimeout := time.Second * 15\n\ncacheClient := sturdyc.New[string](capacity, numShards, ttl, evictionPercentage,\n\tsturdyc.WithEarlyRefreshes(minRefreshDelay, maxRefreshDelay, synchronousRefreshDelay, retryBaseDelay),\n\tsturdyc.WithRefreshCoalescing(batchSize, batchBufferTimeout),\n)\n```\n\nWith the configuration above, the keys in active rotation are going to be\nscheduled for a refresh every 1-2 seconds. For batchable data sources, where we\nare making use of the `GetOrFetchBatch` function, we'll ask the cache (using\nthe `WithRefreshCoalescing` option) to delay them for up to 15 seconds or until\na batch size of 10 is reached.\n\nWhat if we get a request for a key that hasn't been refreshed in the last 120\nseconds? Given the `synchronousRefreshDelay` passed to the `WithEarlyRefreshes`\noption, the cache will skip any background refresh and instead perform a\nsynchronous refresh to ensure that the data is fresh. Did 1000 requests\nsuddenly arrive for this key? No problem, the in-flight tracking makes sure\nthat we only make **one** request to the underlying data source. This works for\nrefreshes too by the way. If 1000 requests arrived for a key that was 3 seconds\nold (greater than our `maxRefreshDelay`) we would only schedule a single\nrefresh for it.\n\nIs the underlying data source experiencing downtime? With our TTL of two-hours\nwe'll be able to provide a degraded experience to our users by serving the data\nwe have in our cache.\n\n# Passthrough\n\nThere are times when you want to always retrieve the latest data from the\nsource and only use the in-memory cache as a _fallback_. In such scenarios, you\ncan use the `Passthrough` and `PassthroughBatch` functions. The cache will\nstill perform in-flight tracking and deduplicate your requests.\n\n# Distributed storage\n\nIt's important to read the previous sections before jumping here in order to\nunderstand how `sturdyc` works when it comes to creating cache keys, tracking\nin-flight requests, refreshing records in the background, and\nbuffering/coalescing requests to minimize the number of round trips we have to\nmake to an underlying data source. As you'll soon see, we'll leverage all of\nthese features for the distributed storage too.\n\nHowever, let's first understand when this functionality can be useful. This\nfeature is particularly valuable when building applications that can achieve a\nhigh cache hit rate while also being subject to large bursts of requests.\n\nAs an example, I've used this in production for a large streaming application.\nThe content was fairly static - new movies, series, and episodes were only\ningested a couple of times an hour. This meant that we could achieve a very\nhigh hit rate for our data sources. However, during the evenings, when a\npopular football match or TV show was about to start, our traffic could spike\nby a factor of 20 within less than a minute.\n\nTo illustrate the problem further, let’s say the hit rate for our in-memory\ncache was 99.8%. Then, when we received that large burst of traffic, our\nauto-scaling would begin provisioning new containers. These containers would\nobviously be brand new, with an initial hit rate of 0%. This would cause a\nsignificant load on our underlying data sources as soon as they came online,\nbecause every request they received led to a cache miss so that we had to make\nan outgoing request to the data source. And these data sources had gotten used\nto being shielded from most of the traffic by the older containers high\nhit-rate and refresh coalescing usage. Hence, what was a 20x spike for us could\neasily become a 200x spike for them until our new containers had warmed their\ncaches.\n\nTherefore, I decided to add the ability to have the containers sync their\nin-memory cache with a distributed key-value store that would have an easier\ntime absorbing these bursts.\n\nAdding distributed storage to the cache is, from the package's point of view,\nessentially just another data source with a higher priority. Hence, we're still\nable to take great advantage of all the features we've seen so far, and these\nefficiency gains will hopefully allow us to use a much cheaper cluster.\n\nA bit simplified, we can think of the cache's interaction with the\ndistributed storage like this:\n\n```go\n// NOTE: This is an example. The cache has similar functionality internally.\nfunc (o *OrderAPI) OrderStatus(ctx context.Context, id string) (string, error) {\n\tcacheKey := \"order-status-\" + id\n\tfetchFn := func(ctx context.Context) (string, error) {\n\t\t// Check Redis cache first.\n\t\tif orderStatus, ok := o.redisClient.Get(cacheKey); ok {\n\t\t\treturn orderStatus, nil\n\t\t}\n\n\t\t// Fetch the order status from the underlying data source.\n\t\tvar response OrderStatusResponse\n\t\terr := requests.URL(o.baseURL).\n\t\t\tParam(\"id\", id).\n\t\t\tToJSON(\u0026response).\n\t\t\tFetch(ctx)\n\t\tif err != nil {\n\t\t\treturn \"\", err\n\t\t}\n\n\t\t// Add the order status to the Redis cache so that it becomes available for the other containers.\n\t\tgo func() { o.RedisClient.Set(cacheKey, response.OrderStatus, time.Hour) }()\n\n\t\treturn response.OrderStatus, nil\n\t}\n\n\treturn o.GetOrFetch(ctx, id, fetchFn)\n}\n```\n\nThe real implementation interacts with the distributed storage through an\nabstraction so that you're able to use any key-value store you want. All you\nwould have to do is implement this interface:\n\n```go\ntype DistributedStorage interface {\n\tGet(ctx context.Context, key string) ([]byte, bool)\n\tSet(ctx context.Context, key string, value []byte)\n\tGetBatch(ctx context.Context, keys []string) map[string][]byte\n\tSetBatch(ctx context.Context, records map[string][]byte)\n}\n```\n\nand then pass it to the `WithDistributedStorage` option when you create your\ncache client:\n\n```go\ncacheClient := sturdyc.New[string](capacity, numShards, ttl, evictionPercentage,\n\t// Other options...\n\tsturdyc.WithDistributedStorage(storage),\n)\n```\n\n**Please note** that you are responsible for configuring the TTL and eviction\npolicies of this storage. `sturdyc` will only make sure that it queries this\ndata source first, and then writes the keys and values to this storage as soon\nas it has gone out to an underlying data source and refreshed them. Therefore,\nI'd advice you to use the configuration above with short TTLs for the\ndistributed storage, or things might get too stale. I mostly think it's useful\nif you're consuming data sources that are rate limited or don't handle brief\nbursts from new containers very well.\n\nI've included an example to showcase this functionality\n[here.](https://github.com/viccon/sturdyc/tree/main/examples/distribution)\n\nWhen running that application, you should see output that looks something like\nthis:\n\n```go\n❯ go run .\n2024/06/07 10:32:56 Getting key shipping-options-1234-asc from the distributed storage\n2024/06/07 10:32:56 Fetching shipping options from the underlying data source\n2024/06/07 10:32:56 The shipping options were retrieved successfully!\n2024/06/07 10:32:56 Writing key shipping-options-1234-asc to the distributed storage\n2024/06/07 10:32:56 The shipping options were retrieved successfully!\n2024/06/07 10:32:57 The shipping options were retrieved successfully!\n2024/06/07 10:32:57 Getting key shipping-options-1234-asc from the distributed storage\n2024/06/07 10:32:57 The shipping options were retrieved successfully!\n2024/06/07 10:32:57 Getting key shipping-options-1234-asc from the distributed storage\n2024/06/07 10:32:57 The shipping options were retrieved successfully!\n2024/06/07 10:32:57 The shipping options were retrieved successfully!\n2024/06/07 10:32:57 Getting key shipping-options-1234-asc from the distributed storage\n2024/06/07 10:32:58 The shipping options were retrieved successfully!\n2024/06/07 10:32:58 The shipping options were retrieved successfully!\n2024/06/07 10:32:58 Getting key shipping-options-1234-asc from the distributed storage\n2024/06/07 10:32:58 The shipping options were retrieved successfully!\n2024/06/07 10:32:58 Getting key shipping-options-1234-asc from the distributed storage\n2024/06/07 10:32:58 The shipping options were retrieved successfully!\n```\n\nAbove we can see that the underlying data source was only visited **once**, and\nthat the remaining background refreshes that the in-memory cache performed only\nwent to the distributed storage.\n\n# Distributed storage early refreshes\n\nAs I mentioned before, the configuration from the section above works well as\nlong as you're using short TTLs for the distributed key-value store. However,\nI've also built systems where I wanted to leverage the distributed storage as\nan additional robustness feature with long TTLs. That way, if an upstream\nsystem goes down, newly provisioned containers could still retrieve the latest\ndata that the old containers had cached from something like a Redis.\n\nIf you have a similar use case, you could use the following\nconfiguration instead:\n\n```go\ncacheClient := sturdyc.New[string](capacity, numShards, ttl, evictionPercentage,\n\tsturdyc.WithDistributedStorageEarlyRefreshes(storage, time.Minute),\n)\n```\n\nWith a configuration like this, I would usually set the TTL for the distributed\nstorage to something like an hour. However, if `sturdyc` queries the\ndistributed storage and finds that a record is older than 1 minute (the second\nargument to the function), it will refresh the record from the underlying data\nsource, and then write the updated value back to it. So the interaction with\nthe distributed storage would look something like this:\n\n- Start by trying to retrieve the key from the distributeted storage. If the\n  data is fresh, it's returned immediately and written to the in-memory cache.\n- If the key was found in the distributed storage, but wasn't fresh enough,\n  we'll visit the underlying data source, and then write the response to both\n  the distributed cache and the one we have in-memory.\n- If the call to refresh the data failed, the cache will use the value from the\n  distributed storage as a fallback.\n\nHowever, there is one more scenario we must cover now that requires two\nadditional methods to be implemented:\n\n```go\ntype DistributedStorageEarlyRefreshes interface {\n\tDistributedStorage\n\tDelete(ctx context.Context, key string)\n\tDeleteBatch(ctx context.Context, keys []string)\n}\n```\n\nThese delete methods will be called when a refresh occurs, and the cache\nnotices that it can no longer retrieve the key at the underlying data source.\nThis indicates that the key has been deleted, and we will want this change to\npropagate to the distributed key-value store as soon as possible, and not have\nto wait for the TTL to expire.\n\n**Please note** that you are still responsible for setting the TTL and eviction\npolicies for the distributed store. The cache will only invoke the delete\nmethods when a record has gone missing from the underlying data source. If\nyou're using **missing record storage**, it will write the key as a missing\nrecord instead.\n\nI've included an example to showcase this functionality\n[here.](https://github.com/viccon/sturdyc/tree/main/examples/distributed-early-refreshes)\n\n# Custom metrics\n\nThe cache can be configured to report custom metrics for:\n\n- Size of the cache\n- Cache hits\n- Cache misses\n- Background refreshes\n- Synchronous refreshes\n- Missing records\n- Evictions\n- Forced evictions\n- The number of entries evicted\n- Shard distribution\n- The batch size of a coalesced refresh\n\nThere are also distributed metrics if you're using the cache with a\n_distributed storage_, which adds the following metrics in addition to what\nwe've seen above:\n\n- Distributed cache hits\n- Distributed cache misses\n- Distributed refreshes\n- Distributed missing records\n- Distributed stale fallback\n\nAll you have to do is implement one of these interfaces:\n\n```go\ntype MetricsRecorder interface {\n\tCacheHit()\n\tCacheMiss()\n\tAsynchronousRefresh()\n\tSynchronousRefresh()\n\tMissingRecord()\n\tForcedEviction()\n\tEntriesEvicted(int)\n\tShardIndex(int)\n\tCacheBatchRefreshSize(size int)\n\tObserveCacheSize(callback func() int)\n}\n\ntype DistributedMetricsRecorder interface {\n\tMetricsRecorder\n\tDistributedCacheHit()\n\tDistributedCacheMiss()\n\tDistributedRefresh()\n\tDistributedMissingRecord()\n\tDistributedFallback()\n}\n```\n\nand pass it as an option when you create the client:\n\n```go\ncacheBasicMetrics := sturdyc.New[any](\n\tcacheSize,\n\tshardSize,\n\tcacheTTL,\n\tevictWhenFullPercentage,\n\tsturdyc.WithMetrics(metricsRecorder),\n)\n\ncacheDistributedMetrics := sturdyc.New[any](\n\tcacheSize,\n\tshardSize,\n\tcacheTTL,\n\tevictWhenFullPercentage,\n\tsturdyc.WithDistributedStorage(metricsRecorder),\n\tsturdyc.WithDistributedMetrics(metricsRecorder),\n)\n```\n\nBelow are a few images where some of these metrics have been visualized in Grafana:\n\n\u003cimg width=\"939\" alt=\"Screenshot 2024-05-04 at 12 36 43\" src=\"https://github.com/viccon/sturdyc/assets/12787673/1f630aed-2322-4d3a-9510-d582e0294488\"\u003e\nHere we can how often we're able to serve from memory.\n\n\u003cimg width=\"942\" alt=\"Screenshot 2024-05-04 at 12 37 39\" src=\"https://github.com/viccon/sturdyc/assets/12787673/25187529-28fb-4c4e-8fe9-9fb48772e0c0\"\u003e\nThis image displays the number of items we have cached.\n\n\u003cimg width=\"941\" alt=\"Screenshot 2024-05-04 at 12 38 04\" src=\"https://github.com/viccon/sturdyc/assets/12787673/b1359867-f1ef-4a09-8c75-d7d2360726f1\"\u003e\nThis chart shows the batch sizes for the buffered refreshes.\n\n\u003cimg width=\"940\" alt=\"Screenshot 2024-05-04 at 12 38 20\" src=\"https://github.com/viccon/sturdyc/assets/12787673/de7f00ee-b14d-443b-b69e-91e19665c252\"\u003e\nAnd lastly, we can see the average batch size of our refreshes for two different data sources.\n\n# Generics\n\nPersonally, I tend to create caches based on how frequently the data needs to\nbe refreshed rather than what type of data it stores. I'll often have one\ntransient cache which refreshes the data every 2-5 milliseconds, and another\ncache where I'm fine if the data is up to a minute old.\n\nHence, I don't want to tie the cache to any specific type so I'll often just\nuse `any`:\n\n```go\n\tcacheClient := sturdyc.New[any](capacity, numShards, ttl, evictionPercentage,\n\t\tsturdyc.WithEarlyRefreshes(minRefreshDelay, maxRefreshDelay, retryBaseDelay),\n\t\tsturdyc.WithRefreshCoalescing(10, time.Second*15),\n\t)\n```\n\nHowever, having all client methods return `any` can quickly add a lot of\nboilerplate if you're storing more than a handful of types, and need to make\ntype assertions.\n\nIf you want to avoid this, you can use any of the package level exports:\n\n- [`GetOrFetch`](https://pkg.go.dev/github.com/viccon/sturdyc#GetOrFetch)\n- [`GetOrFetchBatch`](https://pkg.go.dev/github.com/viccon/sturdyc#GetOrFetchBatch)\n- [`Passthrough`](https://pkg.go.dev/github.com/viccon/sturdyc#Passthrough)\n- [`PassthroughBatch`](https://pkg.go.dev/github.com/viccon/sturdyc#PassthroughBatch)\n\nThey will take the cache, call the function for you, and perform the type\nconversions internally. If the type conversions were to fail, you'll get a\n[`ErrInvalidType`](https://pkg.go.dev/github.com/viccon/sturdyc#pkg-variables) error.\n\nBelow is an example of what an API client that uses these functions could look\nlike:\n\n```go\ntype OrderAPI struct {\n\tcacheClient *sturdyc.Client[any]\n}\n\nfunc NewOrderAPI(c *sturdyc.Client[any]) *OrderAPI {\n\treturn \u0026OrderAPI{cacheClient: c}\n}\n\nfunc (a *OrderAPI) OrderStatus(ctx context.Context, ids []string) (map[string]string, error) {\n\tcacheKeyFn := a.cacheClient.BatchKeyFn(\"order-status\")\n\tfetchFn := func(_ context.Context, cacheMisses []string) (map[string]string, error) {\n\t\tresponse := make(map[string]string, len(ids))\n\t\tfor _, id := range cacheMisses {\n\t\t\tresponse[id] = \"Order status: pending\"\n\t\t}\n\t\treturn response, nil\n\t}\n\treturn sturdyc.GetOrFetchBatch(ctx, a.cacheClient, ids, cacheKeyFn, fetchFn)\n}\n\nfunc (a *OrderAPI) DeliveryTime(ctx context.Context, ids []string) (map[string]time.Time, error) {\n\tcacheKeyFn := a.cacheClient.BatchKeyFn(\"delivery-time\")\n\tfetchFn := func(_ context.Context, cacheMisses []string) (map[string]time.Time, error) {\n\t\tresponse := make(map[string]time.Time, len(ids))\n\t\tfor _, id := range cacheMisses {\n\t\t\tresponse[id] = time.Now()\n\t\t}\n\t\treturn response, nil\n\t}\n\treturn sturdyc.GetOrFetchBatch(ctx, a.cacheClient, ids, cacheKeyFn, fetchFn)\n}\n```\n\nThe entire example is available [here.](https://github.com/viccon/sturdyc/tree/main/examples/generics)\n","funding_links":[],"categories":["Database","Data Integration Frameworks","Go","数据库"],"sub_categories":["Caches","缓存"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fviccon%2Fsturdyc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fviccon%2Fsturdyc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fviccon%2Fsturdyc/lists"}