{"id":23773395,"url":"https://github.com/beenotung/pre-compute","last_synced_at":"2026-04-03T02:30:22.635Z","repository":{"id":57152368,"uuid":"263841952","full_name":"beenotung/pre-compute","owner":"beenotung","description":"pre-compute or pre-fetch resources into buffer/cache to maximize concurrent processing capacity. With automatic back-pressure management.","archived":false,"fork":false,"pushed_at":"2023-06-06T01:35:37.000Z","size":28,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-01T07:18:08.504Z","etag":null,"topics":["buffer","concurrent","pre-computer","prefetch","producer-consumer"],"latest_commit_sha":null,"homepage":"https://www.npmjs.com/package/pre-compute","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/beenotung.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-05-14T07:10:05.000Z","updated_at":"2023-06-06T00:25:41.000Z","dependencies_parsed_at":"2022-08-27T15:24:06.870Z","dependency_job_id":null,"html_url":"https://github.com/beenotung/pre-compute","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/beenotung%2Fpre-compute","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/beenotung%2Fpre-compute/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/beenotung%2Fpre-compute/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/beenotung%2Fpre-compute/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/beenotung","download_url":"https://codeload.github.com/beenotung/pre-compute/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239960667,"owners_count":19725358,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["buffer","concurrent","pre-computer","prefetch","producer-consumer"],"created_at":"2025-01-01T05:39:37.593Z","updated_at":"2026-04-03T02:30:22.575Z","avatar_url":"https://github.com/beenotung.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pre-compute\n\npre-compute or pre-fetch resources into buffer/cache to maximize concurrent processing capacity.\nWith automatic back-pressure management.\n\n[![npm Package Version](https://img.shields.io/npm/v/pre-compute)](https://www.npmjs.com/package/pre-compute)\n[![Minified Package Size](https://img.shields.io/bundlephobia/min/pre-compute)](https://bundlephobia.com/package/pre-compute)\n[![Minified and Gzipped Package Size](https://img.shields.io/bundlephobia/minzip/pre-compute)](https://bundlephobia.com/package/pre-compute)\n\n## Use Case\n\nproducer: pull-based resources, e.g. network/disk IO\n\nconsumer: CPU intensive processing logic / async processor, that cannot catch up the data production rate\n\n## Features\n\n- optimized for IO-intensive task with latency, e.g. network/disk IO\n- isomorphic (support browser, node.js in CommonJS and ESM)\n- memory-efficient (using ring-buffer)\n- lightweight (\u003c1KB)\n- built-in typescript support\n\n## Installation\n\n### Install with package manager\n\n```bash\nnpm i pre-compute\n```\n\nYou can also install it with [pnpm](https://www.npmjs.com/package/pnpm) or [yarn](https://www.npmjs.com/package/yarn)\n\nimport from typescript or javascript (ESM)\n\n```typescript\nimport { PreCompute } from 'pre-compute'\n```\n\nimport from node.js (CommonJS)\n\n```javascript\nconst { PreCompute } = require('pre-compute')\n```\n\n### To use in browser directly\n\n```html\n\u003cscript src=\"https://cdn.jsdelivr.net/npm/pre-compute@1/dist/browser.min.js\"\u003e\u003c/script\u003e\n\u003cscript\u003e\n  const compute =  new PreCompute({...})\n\u003c/script\u003e\n```\n\n## Usage Example\n\n```typescript\nimport { PreCompute } from 'pre-compute'\n\nlet n = 100\n// a list of resources we want to fetch and process\nlet urls = new Array(n).fill('').map((_, i) =\u003e 'http://example.com/post/' + i)\n\n// fetch a resource\nfunction producer(i: number) {\n  return fetch(urls[i]).then(res =\u003e res.text())\n}\n\n// the state of the processor\nlet wordCounts = new Map\u003cstring, number\u003e()\n\n// process a fetched resource\nfunction consume(text: string) {\n  text.split(' ').forEach(word =\u003e {\n    let count = wordCounts.get(word) || 0\n    wordCounts.set(word, count + 1)\n  })\n}\n\nconst compute = new PreCompute({\n  // pre-fetch this amount of resources for the downstream consumer\n  bufferSize: 20,\n  // the resource producer, takes idx as parameter\n  producer,\n  // eagerly pre-fetch until this index inclusively\n  // optional, default is unlimited\n  max: n,\n})\n\nasync function main() {\n  // the main loop\n  for (let i = 0; i \u003c n; i++) {\n    // get i^th resources, and pre-fetch next 20 resources (sliding window)\n    let text = await compute.get(i)\n    consume(text) // can put await if needed, this can cause back-pressure on the pre-fetching\n  }\n  // print the final result\n  let total = Array.from(wordCounts.values()).reduce((acc, c) =\u003e acc + c)\n  console.log('number of words:', total)\n}\n\nmain()\n```\n\nComplete example in: [test/pre-compute-test.ts](./test/pre-compute-test.ts)\n\n## Tips to speed up consumer\n\nThis library applies concurrent computing for consumer that need to mutate the same memory.\n\nThe IO calls in producer are executed in parallel underneath while the consumer is executed sequentially.\n\nFor CPU-intensive work that can be parallelized without mutex lock, you can use [workerpool](https://www.npmjs.com/package/workerpool) for the consumer, then both the producer and consumer will be multiple-threaded.\n\n## Todo\n\n- [ ] auto adjust the buffer size by time/memory constraint\n\n## License\n\nThis project is licensed with [BSD-2-Clause](./LICENSE)\n\nThis is free, libre, and open-source software. It comes down to four essential freedoms [[ref]](https://seirdy.one/2021/01/27/whatsapp-and-the-domestication-of-users.html#fnref:2):\n\n- The freedom to run the program as you wish, for any purpose\n- The freedom to study how the program works, and change it so it does your computing as you wish\n- The freedom to redistribute copies so you can help others\n- The freedom to distribute copies of your modified versions to others\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbeenotung%2Fpre-compute","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbeenotung%2Fpre-compute","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbeenotung%2Fpre-compute/lists"}