{"id":50970095,"url":"https://github.com/bytadaniel/clickcache","last_synced_at":"2026-06-19T01:02:25.123Z","repository":{"id":61319557,"uuid":"550521685","full_name":"bytadaniel/clickcache","owner":"bytadaniel","description":"Clickhouse data collector for delayed batch insert","archived":false,"fork":false,"pushed_at":"2024-10-18T11:44:28.000Z","size":193,"stargazers_count":7,"open_issues_count":1,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2026-02-18T11:52:15.640Z","etag":null,"topics":["batch","cache","clickhouse","clickhouse-ts","disk","in-memory","insert","ram","s3"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bytadaniel.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2022-10-12T22:59:46.000Z","updated_at":"2025-02-14T13:09:56.000Z","dependencies_parsed_at":"2023-10-16T03:42:18.114Z","dependency_job_id":null,"html_url":"https://github.com/bytadaniel/clickcache","commit_stats":{"total_commits":25,"total_committers":3,"mean_commits":8.333333333333334,"dds":"0.16000000000000003","last_synced_commit":"0bbcd76b996ea8d4dd3b8cfef4fe28fe73918934"},"previous_names":[],"tags_count":16,"template":false,"template_full_name":null,"purl":"pkg:github/bytadaniel/clickcache","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bytadaniel%2Fclickcache","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bytadaniel%2Fclickcache/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bytadaniel%2Fclickcache/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bytadaniel%2Fclickcache/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bytadaniel","download_url":"https://codeload.github.com/bytadaniel/clickcache/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bytadaniel%2Fclickcache/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34513029,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-18T02:00:06.871Z","response_time":128,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["batch","cache","clickhouse","clickhouse-ts","disk","in-memory","insert","ram","s3"],"created_at":"2026-06-19T01:02:24.339Z","updated_at":"2026-06-19T01:02:25.057Z","avatar_url":"https://github.com/bytadaniel.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# clickcache by @bytadaniel\n\n\u003ca href=\"https://www.npmjs.com/package/clickcache\" alt=\"NPM latest version\"\u003e\u003cimg src=\"https://img.shields.io/npm/v/clickcache.svg\"\u003e\u003c/a\u003e\n\u003ca href=\"https://www.npmjs.com/package/clickcache\" alt=\"NPM total downloads\"\u003e\u003cimg src=\"https://img.shields.io/npm/dt/clickcache.svg\"\u003e\u003c/a\u003e\n\u003ca href=\"https://github.com/bytadaniel/clickcache\" alt=\"Github stars\"\u003e\u003cimg src=\"https://img.shields.io/github/stars/bytadaniel/clickcache.svg?style=social\u0026label=Star\"\u003e\u003c/a\u003e\n\u003ca href=\"https://github.com/bytadaniel/clickcache\" alt=\"Github forks\"\u003e\u003cimg src=\"https://img.shields.io/github/forks/bytadaniel/clickcache.svg?style=social\u0026label=Fork\"\u003e\u003c/a\u003e\n\u003ca href=\"https://github.com/bytadaniel/clickcache\" alt=\"Github contributors\"\u003e\u003cimg src=\"https://img.shields.io/github/contributors/bytadaniel/clickcache.svg\"\u003e\u003c/a\u003e\n\n![GitHub followers](https://img.shields.io/github/followers/bytadaniel?style=social)\n\n## Run the clickcache dockerized server working from scratch\n- https://github.com/bytadaniel/clickcache-server.git\n\n## Contact me\nPlease be free to open issues and create pull requests.\nJoin the clickcache dev channel on Telegram and ask be about the project directly\n- https://t.me/clickcache\n\n## Why\n\nIn my data work, I encountered the need for a straightforward way to asynchronously gather small pieces of data into larger batches and efficiently transmit them to Clickhouse.\nTo address this requirement, I developed the `clickcache` package.\n\n`сlickcache` excels at working not only with the [official clickhouse client](https://github.com/ClickHouse/clickhouse-js) but also with third-party clients.\nIt does so by delegating the read/write work to them while focusing on data aggregation in one central location and preparing it for insertion.\n\n## Roadmap\n\nThis cache collector will support of is actually supporting caching data\n\n- ✅ store data in the runtime process memory\n- ✅ store data in the system memory storage (on your disk)\n- 🏗 store data in the cloud (s3)\n\n## Usage\n\n```bash\nnpm install clickcache\n```\n\n```js\nconst config: ResolverOptions = {\n  chunkLifeMs: 60000,           // Set the time to live limit for chunks\n  chunkSize: 1000,              // Set the max size limit for chunks\n  checkIntervalMs: 10000,       // Set the check interval. It is normal to check batches state 5-10 times per TTL\n  dataWatcher: 'disk',          // Choose the way to store data\n  disk: {\n    outputDirectory: './chunks' // Both absolute and relative path work\n  }\n}\n\n// define the singleton resolver instance\nconst resolver = new ChunkResolver(config)\n\n// set as much handlers as you need\n\n// sync handler to log chunk output\nresolver.onResolved(chunk =\u003e {\n  console.log(chunk.id)\n  console.log(chunk.size)\n})\n\n// async handler to pass data in clickhouse storage\nresolver.onResolved(async chunk =\u003e {\n  const myRows = await chunk.loadRows()\n  await clickhouseClient\n    .insertFunction(chunk.table, myRows)\n    .then(() =\u003e console.log('Hurrah! My data is saved!'))\n    .catch(e =\u003e resolver.cache(chunk.table, myRows))\n})\n\n// use this method to cache a few rows or a single row\n// it will be stored and collected to a huuuge batch of data\nconst chunk = await resolver.cache(myTable, rows)\n```\n\n# How it works\n\nThis package contains some enities\n\n- `ChunkResolver`\n- `ChunkRegistry`\n- `DataWatcher`\n- `Chunk`\n\nIt collects many single rows by uning `ChunkResolver`, then arranges these rows to chunks. When the chunk is ready, `ChunkResolver` passes it to your your handlers, where you are able to process database insertion\n\n`Chunk` has a relation to `ChunkRegistry` and `DataWatcher`\n\n`ChunkRegistry` is a in-memory storage shared within all parts of the core functionality. It contains chunk metadata such as chunk state (is blocked or not, is consistent or not, is expired or not etc) and chunk refs itself\n\n`Chunk` has a relation with the stored data though `DataWatcher` and can `load` it according your need\n\n`DataWatcher` is an abstract entity which interacts with the data. Data can be stored in `process memory`, `disk storage` and `cloud`. Data watcher can store and restore your data.\n\nFor example, you are using  the `disk storage` watcher. You are caching your data and someshing goes wrong with the main process. It restarts, restores the last state of data and concistently resolves it\n\nIt is not possible to restore the data by using `process memory` data watcher\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbytadaniel%2Fclickcache","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbytadaniel%2Fclickcache","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbytadaniel%2Fclickcache/lists"}