{"id":16920516,"url":"https://github.com/ostafen/zipcache","last_synced_at":"2025-04-11T16:39:52.653Z","repository":{"id":65732929,"uuid":"597434866","full_name":"ostafen/zipcache","owner":"ostafen","description":"An in-memory compressed cache for gigabytes of data written in Go.","archived":false,"fork":false,"pushed_at":"2023-02-06T10:46:41.000Z","size":617,"stargazers_count":18,"open_issues_count":1,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-25T12:51:04.650Z","etag":null,"topics":["cache","compression","golang","hashtable"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ostafen.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2023-02-04T14:44:57.000Z","updated_at":"2024-10-15T13:29:54.000Z","dependencies_parsed_at":"2023-02-19T06:30:36.957Z","dependency_job_id":null,"html_url":"https://github.com/ostafen/zipcache","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ostafen%2Fzipcache","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ostafen%2Fzipcache/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ostafen%2Fzipcache/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ostafen%2Fzipcache/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ostafen","download_url":"https://codeload.github.com/ostafen/zipcache/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248441932,"owners_count":21104104,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cache","compression","golang","hashtable"],"created_at":"2024-10-13T19:48:33.009Z","updated_at":"2025-04-11T16:39:52.635Z","avatar_url":"https://github.com/ostafen.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# zipcache\nAn in-memory compressed cache\n\n# Why?\n\nCaching many items in memory can considerably increase the space requirements of an application. A simple way to mitigate this problem is to compress each cache entry using some compression algorithm (deflate, gzip, etc...). Howewer this approach is not effective when the average size of each entry is relatively small.\n\n## ZipCache Layout\nThe goal of ZipCache is to achieve a better compression rate by arranging multiple values (represented as **byte slices**) within a single byte chunk, whose maximum size is fixed (**4096** by default). Values are appended in the current chunk in the same order are they are inserted. When a chunk is full, a new one is appended to the **chunk list** and the old one gets compressed in the background. Values whose size exceeds the chunk size are split across multiple blocks. A separate map is kept (which is a standard golang **map**) to easily locate values within blocks, which associates a key to a triplet **\u003cchunkNumber, byteOffset, length\u003e**.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"./diagram.png\" /\u003e\n\u003c/p\u003e\n\n\n# Sample use case\n\nSuppose you want to cache a set of json encoded entries of relatively small size.\n\nLet's try to firstly follow the naive approach, where each entry is compressed separatedly and stored in a standard golang map. \n\n```golang\npackage main\n\n...\n\nfunc main() {\n\tf, err := os.Open(\"../airlines.json\")\n\tfatalIfErr(err)\n\n\tdata, err := io.ReadAll(f)\n\tfatalIfErr(err)\n\n\tm := make(map[string]string)\n\n\tx := make([]map[string]any, 0)\n\terr = json.Unmarshal(data, \u0026x)\n\tfatalIfErr(err)\n\n\ttotalTime := time.Now()\n\tfor i, item := range x {\n\t\tdata, err := json.Marshal(item)\n\t\tfatalIfErr(err)\n\n\t\tstart := time.Now()\n\t\tm[strconv.Itoa(i)] = string(compress(data))\n\t\ttotalTime += time.Since(start)\n\t\tfatalIfErr(err)\n\t}\n\n\tsize := 0\n\tfor _, v := range m {\n\t\tsize += len(v)\n\t}\n\tfmt.Printf(\"time (seconds):\\t %f\\n\", totalTime.Seconds())\n\tfmt.Printf(\"ratio (%%):\\t %.2f\\n\", float64(size)/float64(len(data)))\n}\n\n```\nThe output is the following:\n\n```bash\ntime (seconds):\t 1.503385\nratio (%):\t 0.34\n```\n\nNow, let's try to use ZipCache to accomplish the same task.\n\n```golang\n\nimport (\n\t...\n\t\"github.com/ostafen/zipcache\"\n)\n\n...\n\nfunc main() {\n\tf, err := os.Open(\"./airlines.json\")\n\tfatalIfErr(err)\n\n\tdata, err := io.ReadAll(f)\n\tfatalIfErr(err)\n\n\tcfg := zipcache.DefaultConfig().\n\t\t\tWithChunkSize(4096*4). // defines how many entries will be compressed together. Set this according to average entry size.\n\t\t\tWithReaderWriter(gzipReader, gzipWriter) // use gzip compression algorithm. Default is deflate.\n\n\tcache := zipcache.New(cfg)\n\n\tx := make([]map[string]any, 0)\n\terr = json.Unmarshal(data, \u0026x)\n\tfatalIfErr(err)\n\n\ttotal := time.Duration(0)\n\tfor i, item := range x {\n\t\tdata, err := json.Marshal(item)\n\t\tfatalIfErr(err)\n\n\t\tstart := time.Now()\n\t\terr = cache.Put([]byte(strconv.Itoa(i)), data)\n\t\ttotal += time.Since(start)\n\t\tfatalIfErr(err)\n\t}\n\n\tfmt.Printf(\"time (seconds):\\t %f\\n\", total.Seconds())\n\tfmt.Printf(\"ratio (%%):\\t %.2f\\n\", float64(cache.Size())/float64(len(data)))\n}\n```\n\nThis is what we get:\n\n```bash\ntime (seconds):\t 0.006377\nratio (%):\t 0.11\n```\n\nNot only we save about 23% additional space, but code is ~200x faster, since compression is invoked less times.\n\n# Limitations\n\nZipCache is a grow-only cache. Cache items are not evicted and mappings cannot be updated nor deleted. However, support for update/deletion is planned.\n\n# When to use ZipCache\n\nUse ZipCache when:\n\n- you are more concerned about RAM consumption rather then absolute lookup performance;\n- you never change your key-value mappings once you enter your items in the cache;\n- you don't need to expire your items.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fostafen%2Fzipcache","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fostafen%2Fzipcache","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fostafen%2Fzipcache/lists"}