{"id":17760801,"url":"https://github.com/modfin/cove","last_synced_at":"2025-06-23T10:35:16.586Z","repository":{"id":259524472,"uuid":"877395865","full_name":"modfin/cove","owner":"modfin","description":"Embedded cache lib using sqlite for storage","archived":false,"fork":false,"pushed_at":"2024-10-30T07:18:45.000Z","size":146,"stargazers_count":19,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-15T14:13:42.319Z","etag":null,"topics":["cache","generics","go","golang","library","sqlite","sqlite3"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/modfin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-10-23T15:19:19.000Z","updated_at":"2025-02-01T19:02:57.000Z","dependencies_parsed_at":"2024-10-26T07:31:10.101Z","dependency_job_id":"113ba8f5-377c-4dc2-9746-4c74f0708505","html_url":"https://github.com/modfin/cove","commit_stats":null,"previous_names":["modfin/cove"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/modfin/cove","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/modfin%2Fcove","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/modfin%2Fcove/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/modfin%2Fcove/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/modfin%2Fcove/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/modfin","download_url":"https://codeload.github.com/modfin/cove/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/modfin%2Fcove/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261462624,"owners_count":23162000,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cache","generics","go","golang","library","sqlite","sqlite3"],"created_at":"2024-10-26T19:13:22.446Z","updated_at":"2025-06-23T10:35:11.546Z","avatar_url":"https://github.com/modfin.png","language":"Go","readme":"# Cove\n\n[![goreportcard.com](https://goreportcard.com/badge/github.com/modfin/cove)](https://goreportcard.com/report/github.com/modfin/cove)\n[![PkgGoDev](https://pkg.go.dev/badge/github.com/modfin/cove)](https://pkg.go.dev/github.com/modfin/cove)\n\n\n`cove` is a caching library for Go that utilizes SQLite as the storage backend. It provides a TTL cache for key-value pairs with support for namespaces, batch operations, range scans and eviction callbacks.\n\n\n## TL;DR\n\nA TTL caching for Go backed by SQLite. (See examples for usage)\n\n```bash \ngo get github.com/modfin/cove\n```\n\n```go\npackage main\n\nimport (\n\t\"fmt\"\n\t\"github.com/modfin/cove\"\n)\n\nfunc main() {\n\tcache, err := cove.New(\n\t\tcove.URITemp(),\n\t)\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\tdefer cache.Close()\n\n\tns, err := cache.NS(\"strings\")\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\t// When using generic, use a separate namespace for each type\n\tstringCache := cove.Of[string](ns)\n\n\tstringCache.Set(\"key\", \"the string\")\n\n\tstr, err := stringCache.Get(\"key\")\n\thit, err := cove.Hit(err)\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\tif hit {\n\t\tfmt.Println(str) // Output: the string\n\t}\n\n\tstr, err = stringCache.GetOr(\"async-key\", func(key string) (string, error) {\n\t\treturn \"refreshed string\", nil\n\t})\n\thit, err = cove.Hit(err)\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\tif hit {\n\t\tfmt.Println(str) // Output: refreshed string\n\t}\n\n}\n\n```\n\n\n## Use case\n`cove` is meant to be embedded into your application, and not as a standalone service. \nIt is a simple key-value store that is meant to be used for caching data that is expensive to compute or retrieve. \n`cove` can also be used as a key-value store\n\nSo why SQLite?\n\nThere are plenty of in memory and other caches build in go, \\\neg https://github.com/avelino/awesome-go#cache, performance, concurrency, fast, LFU, LRU, ARC and so on. \\\nThere are also a few key-value stores build in go that can be embedded (just like cove), \\\neg https://github.com/dgraph-io/badger or https://github.com/boltdb/bolt and probably quite a few more, \\\nhttps://github.com/avelino/awesome-go#databases-implemented-in-go.\n\nWell if these alternatives suits your use case, use them. \nThe main benefit of using a cache/kv, from my perspective and the reason for building cove, is that a cache backed by sqlite should be decent \nand fast enough in most case. \nIts generically just a good solution while probably being outperformed by others in niche cases.\n- you can have very large K/V pairs\n- you can tune it for your use case\n- it should perform decently\n- you can cache hundreds of GB. SSD are fast these days.\n- page caching and tuning will help you out.\n\nWhile sqlite has come a long way since its inception and particular with it running in WAL mode, there are some limitations.\nEg only one writer is allowed at a time. So if you have a write heavy cache, you might want to consider another solution. \nWith that said it should be fine for most with some tuning to increase write performance, eg `synchronous = off`.\n\n## Installation\n\nTo install `cove`, use `go get`:\n\n```sh\ngo get github.com/modfin/cove\n```\n\n### Considerations\nSince `cove` uses SQLite as the storage backend, it is important realize that you project now will depend on cgo and that the SQLite library will be compiled into your project. This might not be a problem at all, but it could cause problems in some modern \"magic\" build tools used in CD/CI pipelines for go.\n\n## Usage\n\n### Tuning \n\ncove uses sqlite in WAL mode and writes the data to disk. While probably :memory: works for the most part, it does not have all the cool performance stuff that comes with sqlite in WAL mode on disk and probably will result in som SQL_BUSY errors.\n\nIn general the default tuning is the following\n```sqlite\nPRAGMA journal_mode = WAL;\nPRAGMA synchronous = normal;\nPRAGMA temp_store = memory;\nPRAGMA auto_vacuum = incremental;\nPRAGMA incremental_vacuum;\n```\n\nHave a look at https://www.sqlite.org/pragma.html for tuning your cache to your needs.\n\nIf you are write heavy, you might want to consider `synchronous = off` and dabble with some other settings, eg `wal_autocheckpoint`, to increase write performance. The tradeoff is that you might lose some read performance instead.\n\n```go\n    cache, err := cove.New(\n\t\tcove.URITemp(), \n\t\tcove.DBSyncOff(), \n\t\t// Yes, yes, this can be used to inject sql, but I trust you\n\t\t// to not let your users arbitrarily configure pragma on your \n\t\t// sqlite instance.\n\t\tcove.DBPragma(\"wal_autocheckpoint = 1000\"),\n    )\n```\n\n\n### Creating a Cache\n\nTo create a cache, use the `New` function. You can specify various options such as TTL, vacuum interval, and eviction callbacks.\n\n\n#### Config\n\n`cove.URITemp()` Creates a temporary directory in `/tmp` or similar for the database. If combined with a `cove.DBRemoveOnClose()` and a gracefull shutdown, the database will be removed on close\n\nif you want the cache to persist over restarts or such, you can use `cove.URIFromPath(\"/path/to/db/cache.db\")` instead.\n\nThere are a few options that can be set when creating a cache, see `cove.With*` and `cove.DB*` functions for more information.\n\n#### Example\n\n```go\npackage main\n\nimport (\n    \"github.com/modfin/cove\"\n    \"time\"\n)\n\nfunc main() {\n    cache, err := cove.New(\n        cove.URITemp(),\n        cove.DBRemoveOnClose(),\n        cove.WithTTL(time.Minute*10),\n    )\n    if err != nil {\n        panic(err)\n    }\n    defer cache.Close()\n}\n```\n\n### Setting and Getting Values\n\nYou can set and get values from the cache using the `Set` and `Get` methods.\n\n```go\npackage main\n\nimport (\n    \"fmt\"\n    \"github.com/modfin/cove\"\n    \"time\"\n)\n\nfunc main() {\n    cache, err := cove.New(\n        cove.URITemp(),\n        cove.DBRemoveOnClose(),\n        cove.WithTTL(time.Minute*10),\n    )\n    if err != nil {\n        panic(err)\n    }\n    defer cache.Close()\n\n    // Set a value\n    err = cache.Set(\"key\", []byte(\"value0\"))\n    if err != nil {\n        panic(err)\n    }\n\n    // Get the value\n    value, err := cache.Get(\"key\")\n\thit, err := cove.Hit(err)\n    if err != nil {\n        panic(err)\n    }\n\n    fmt.Println(\"[Hit]:\", hit, \"[Value]:\", string(value)) \n\t// Output: \"[Hit]: true [Value]: value0\n}\n```\n\n\n\n### Handling `NotFound` errors\n\nIf a key is not found in the cache, the `Get` method will return an `NotFound` error.\n\nYou can handle not found errors using the `Hit` and `Miss` helper functions.\n\n```go\npackage main\n\nimport (\n    \"errors\"\n    \"fmt\"\n    \"github.com/modfin/cove\"\n    \"time\"\n)\n\nfunc main() {\n    cache, err := cove.New(\n        cove.URITemp(),\n        cove.DBRemoveOnClose(),\n        cove.WithTTL(time.Minute*10),\n    )\n    if err != nil {\n        panic(err)\n    }\n    defer cache.Close()\n\n    _, err = cache.Get(\"key\")\n    fmt.Println(\"err == cove.NotFound:\", err == cove.NotFound)\n    fmt.Println(\"errors.Is(err, cove.NotFound):\", errors.Is(err, cove.NotFound))\n\n    _, err = cache.Get(\"key\")\n    hit, err := cove.Hit(err)\n    if err != nil { // A \"real\" error has occurred\n        panic(err)\n    }\n    if !hit {\n        fmt.Println(\"key miss\")\n    }\n\n    _, err = cache.Get(\"key\")\n    miss, err := cove.Miss(err)\n    if err != nil { // A \"real\" error has occurred\n        panic(err)\n    }\n    if miss {\n        fmt.Println(\"key miss\")\n    }\n}\n```\n\n\n\n\n\n### Using Namespaces\n\nNamespaces allow you to isolate different sets of keys within the same cache.\n\n```go\npackage main\n\nimport (\n    \"github.com/modfin/cove\"\n    \"time\"\n\t\"fmt\"\n)\n\nfunc main() {\n    cache, err := cove.New(\n        cove.URITemp(),\n        cove.DBRemoveOnClose(),\n        cove.WithTTL(time.Minute*10),\n    )\n    if err != nil {\n        panic(err)\n    }\n    defer cache.Close()\n\n\n\terr = cache.Set(\"key\", []byte(\"value0\"))\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\t\n    ns, err := cache.NS(\"namespace1\")\n    if err != nil {\n        panic(err)\n    }\n\n    err = ns.Set(\"key\", []byte(\"value1\"))\n    if err != nil {\n        panic(err)\n    }\n\n\n\n\n\tvalue, err := cache.Get(\"key\")\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\tfmt.Println(string(value)) // Output: value0\n\t\n    value, err = ns.Get(\"key\")\n    if err != nil {\n        panic(err)\n    }\n    fmt.Println(string(value)) // Output: value1\n\t\n\t\n}\n```\n\n### Eviction Callbacks\n\nYou can set a callback function to be called when a key is evicted from the cache.\n\n```go\npackage main\n\nimport (\n    \"fmt\"\n    \"github.com/modfin/cove\"\n)\n\nfunc main() {\n    cache, err := cove.New(\n        cove.URITemp(),\n        cove.DBRemoveOnClose(),\n        cove.WithEvictCallback(func(key string, val []byte) {\n            fmt.Printf(\"evicted %s: %s\\n\", key, string(val))\n        }),\n    )\n    if err != nil {\n        panic(err)\n    }\n    defer cache.Close()\n\n    err = cache.Set(\"key\", []byte(\"evict me\"))\n    if err != nil {\n        panic(err)\n    }\n\n    _, err = cache.Evict(\"key\")\n    if err != nil {\n        panic(err)\n    }\n    // Output: evicted key: evict me\n}\n```\n\n### Using Iterators\n\nIterators allow you to scan through keys and values without loading all rows into memory.\n\n\u003e **WARNING** \\\n\u003e Since iterators don't really have any way of communication errors \\\n\u003e the Con is that errors are dropped when using iterators. \\\n\u003e the Pro is that it is very easy to use, and scan row by row (ie. no need to load all rows into memory)\n\n```go\npackage main\n\nimport (\n    \"fmt\"\n    \"github.com/modfin/cove\"\n    \"time\"\n)\n\nfunc main() {\n    cache, err := cove.New(\n        cove.URITemp(),\n        cove.DBRemoveOnClose(),\n        cove.WithTTL(time.Minute*10),\n    )\n    if err != nil {\n        panic(err)\n    }\n    defer cache.Close()\n\n    for i := 0; i \u003c 100; i++ {\n        err = cache.Set(fmt.Sprintf(\"key%d\", i), []byte(fmt.Sprintf(\"value%d\", i)))\n        if err != nil {\n            panic(err)\n        }\n    }\n\n    // KV iterator\n    for k, v := range cache.ItrRange(\"key97\", cove.RANGE_MAX) {\n        fmt.Println(k, string(v))\n    }\n\n    // Key iterator\n    for key := range cache.ItrKeys(cove.RANGE_MIN, \"key1\") {\n        fmt.Println(key)\n    }\n\n    // Value iterator\n    for value := range cache.ItrValues(cove.RANGE_MIN, \"key1\") {\n        fmt.Println(string(value))\n    }\n}\n```\n\n\n### Batch Operations\n\n`cove` provides batch operations to efficiently handle multiple keys in a single operation. This includes `BatchSet`, `BatchGet`, and `BatchEvict`.\n\n#### BatchSet\n\nThe `BatchSet` method allows you to set multiple key-value pairs in the cache in a single operation. This method ensures atomicity by splitting the batch into sub-batches if necessary.\n\n```go\npackage main\n\nimport (\n    \"fmt\"\n    \"github.com/modfin/cove\"\n)\n\nfunc assertNoErr(err error) {\n    if err != nil {\n        panic(err)\n    }\n}\n\nfunc main() {\n    cache, err := cove.New(\n        cove.URITemp(),\n        cove.DBRemoveOnClose(),\n    )\n    assertNoErr(err)\n    defer cache.Close()\n\n    err = cache.BatchSet([]cove.KV[[]byte]{\n        {K: \"key1\", V: []byte(\"val1\")},\n        {K: \"key2\", V: []byte(\"val2\")},\n    })\n    assertNoErr(err)\n}\n```\n\n#### BatchGet\n\nThe `BatchGet` method allows you to retrieve multiple keys from the cache in a single operation. This method ensures atomicity by splitting the batch into sub-batches if necessary.\n\n```go\npackage main\n\nimport (\n    \"fmt\"\n    \"github.com/modfin/cove\"\n)\n\nfunc assertNoErr(err error) {\n    if err != nil {\n        panic(err)\n    }\n}\n\nfunc main() {\n    cache, err := cove.New(\n        cove.URITemp(),\n        cove.DBRemoveOnClose(),\n    )\n    assertNoErr(err)\n    defer cache.Close()\n\n    err = cache.BatchSet([]cove.KV[[]byte]{\n        {K: \"key1\", V: []byte(\"val1\")},\n        {K: \"key2\", V: []byte(\"val2\")},\n    })\n    assertNoErr(err)\n\n    kvs, err := cache.BatchGet([]string{\"key1\", \"key2\", \"key3\"})\n    assertNoErr(err)\n\n    for _, kv := range kvs {\n        fmt.Println(kv.K, \"-\", string(kv.V))\n        // Output:\n        // key1 - val1\n        // key2 - val2\n    }\n}\n```\n\n#### BatchEvict\n\nThe `BatchEvict` method allows you to evict multiple keys from the cache in a single operation. This method ensures atomicity by splitting the batch into sub-batches if necessary.\n\n```go\npackage main\n\nimport (\n    \"fmt\"\n    \"github.com/modfin/cove\"\n)\n\nfunc assertNoErr(err error) {\n    if err != nil {\n        panic(err)\n    }\n}\n\nfunc main() {\n    cache, err := cove.New(\n        cove.URITemp(),\n        cove.DBRemoveOnClose(),\n    )\n    assertNoErr(err)\n    defer cache.Close()\n\n    err = cache.BatchSet([]cove.KV[[]byte]{\n        {K: \"key1\", V: []byte(\"val1\")},\n        {K: \"key2\", V: []byte(\"val2\")},\n    })\n    assertNoErr(err)\n\n    evicted, err := cache.BatchEvict([]string{\"key1\", \"key2\", \"key3\"})\n    assertNoErr(err)\n\n    for _, kv := range evicted {\n        fmt.Println(\"Evicted,\", kv.K, \"-\", string(kv.V))\n        // Output:\n        // Evicted, key1 - val1\n        // Evicted, key2 - val2\n    }\n}\n```\n\nThese batch operations help in efficiently managing multiple keys in the cache, ensuring atomicity and reducing the number of individual operations.\n\n\n\n### Typed Cache\n\nThe `TypedCache` in `cove` provides a way to work with strongly-typed values in the cache, using Go generics. This allows you to avoid manual serialization and deserialization of values, making the code cleaner and less error-prone.\n\n#### Creating a Typed Cache\n\nA Typed Cache is simply to use golang generics to wrap the cache and provide type safety and ease of use.\nThe Typed Cache comes with the same fetchers and api as the untyped cache but adds a marshalling and unmarshalling layer on top of it. \nencoding/gob is used for serialization and deserialization of values.\n\nTo create a typed cache, use the `Of` function, passing the existing cache/namespace instance:\n\n```go\npackage main\n\nimport (\n\t\"fmt\"\n\t\"github.com/modfin/cove\"\n\t\"time\"\n)\n\nfunc assertNoErr(err error) {\n\tif err != nil {\n\t\tpanic(err)\n\t}\n}\n\ntype Person struct {\n\tName string\n\tAge  int\n}\n\nfunc main() {\n\t// Create a base cache\n\tcache, err := cove.New(\n\t\tcove.URITemp(),\n\t\tcove.DBRemoveOnClose(),\n\t)\n\tassertNoErr(err)\n\tdefer cache.Close()\n\n\tpersonNamespace, err := cache.NS(\"person\")\n\tassertNoErr(err)\n\t\n\t// Create a typed cache for Person struct\n\ttypedCache := cove.Of[Person](personNamespace)\n\n\t// Set a value in the typed cache\n\terr = typedCache.Set(\"alice\", Person{Name: \"Alice\", Age: 30})\n\tassertNoErr(err)\n\n\t// Get a value from the typed cache\n\talice, err := typedCache.Get(\"alice\")\n\tassertNoErr(err)\n\tfmt.Printf(\"%+v\\n\", alice) // Output: {Name:Alice Age:30}\n}\n```\n\n#### Methods\n\n- **Set**: Sets a value in the cache with the default TTL.\n- **Get**: Retrieves a value from the cache.\n- **SetTTL**: Sets a value in the cache with a custom TTL.\n- **GetOr**: Retrieves a value from the cache or calls a fetch function if the key does not exist.\n- **BatchSet**: Sets a batch of key/value pairs in the cache.\n- **BatchGet**: Retrieves a batch of keys from the cache.\n- **BatchEvict**: Evicts a batch of keys from the cache.\n- **Evict**: Evicts a key from the cache.\n- **EvictAll**: Evicts all keys in the cache.\n- **Range**: Returns all key-value pairs in a specified range.\n- **Keys**: Returns all keys in a specified range.\n- **Values**: Returns all values in a specified range.\n- **ItrRange**: Returns an iterator for key-value pairs in a specified range.\n- **ItrKeys**: Returns an iterator for keys in a specified range.\n- **ItrValues**: Returns an iterator for values in a specified range.\n- **Raw**: Returns the underlying untyped cache.\n\nThe `TypedCache` uses `encoding/gob` for serialization and deserialization of values, ensuring type safety and ease of use.\n\n\n\n\n## Benchmarks\n\nAll models are wrong but some are useful. Not sure what category this falls under,\nbut here are some benchmarks `inserts/sec`, `reads/sec`, `write mb/sec` and `read mb/sec`.\n\nIn general Linux, 4 cores, a ssd and 32 gb of ram it seems to do some \n- 20-30k inserts/sec\n- 200k reads/sec.\n- writes 100-200 mb/sec\n- reads 1000-2000 mb/sec. \n\nIt seems fast enough...\n\n```txt \n \nBenchmarkSetParallel/default-4                         28_256 insert/sec\nBenchmarkSetParallel/sync-off-4                        36_523 insert/sec\nBenchmarkSetParallel/sync-off+autocheckpoint-4         25_480 insert/sec\n\nBenchmarkGetParallel/default-4                        192_668 reads/sec\nBenchmarkGetParallel/sync-off-4                       238_714 reads/sec\nBenchmarkGetParallel/sync-off+autocheckpoint-4        193_778 reads/sec\n\nBenchmarkSetMemParallel/default+0.1mb-4                   273 write-mb/sec\nBenchmarkSetMemParallel/default+1mb-4                     261 write-mb/sec\nBenchmarkSetMemParallel/sync-off+0.1mb-4                  238 write-mb/sec\nBenchmarkSetMemParallel/sync-off+1mb-4                    212 write-mb/sec\n\nBenchmarkSetMem/default+0.1mb-4                           104 write-mb/sec\nBenchmarkSetMem/default+1mb-4                             122 write-mb/sec\nBenchmarkSetMem/sync-off+0.1mb-4                          219 write-mb/sec\nBenchmarkSetMem/sync-off+1mb-4                            249 write-mb/sec\n\nBenchmarkGetMemParallel/default+0.1mb-4                2_189 read-mb/sec\nBenchmarkGetMemParallel/default+1mb-4                  1_566 read-mb/sec\nBenchmarkGetMemParallel/sync-off+0.1mb-4               2_194 read-mb/sec\nBenchmarkGetMemParallel/sync-off+1mb-4                 1_501 read-mb/sec\n\nBenchmarkGetMem/default+0.1mb-4                          764 read-mb/sec\nBenchmarkGetMem/default+1mb-4                            520 read-mb/sec\nBenchmarkGetMem/sync-off+0.1mb-4                         719 read-mb/sec\nBenchmarkGetMem/sync-off+1mb-4                           530 read-mb/sec\n\n```\n\n\n## TODO\n\n- [ ] Add hooks or middleware for logging, metrics, eviction strategy etc.\n- [ ] More testing\n\n## License\n\nThis project is licensed under the MIT License - see the `LICENSE` file for details.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmodfin%2Fcove","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmodfin%2Fcove","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmodfin%2Fcove/lists"}