{"id":24472924,"url":"https://github.com/shutterstock/p-map-iterable","last_synced_at":"2025-05-07T04:06:34.459Z","repository":{"id":172027664,"uuid":"648745770","full_name":"shutterstock/p-map-iterable","owner":"shutterstock","description":"Performs a concurrent mapping with back pressure (won't iterate all source items if the consumer is not reading).","archived":false,"fork":false,"pushed_at":"2025-02-27T03:05:53.000Z","size":618,"stargazers_count":2,"open_issues_count":3,"forks_count":0,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-05-07T04:06:27.963Z","etag":null,"topics":["async-iterator","back-pressure","blocking-queue","concurrency","nodejs","p-map","parallel","queue"],"latest_commit_sha":null,"homepage":"https://tech.shutterstock.com/p-map-iterable/","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/shutterstock.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-06-02T17:49:16.000Z","updated_at":"2025-02-18T16:21:59.000Z","dependencies_parsed_at":"2025-02-18T17:20:30.207Z","dependency_job_id":"1edadd4d-22aa-4183-8132-bc337027cfee","html_url":"https://github.com/shutterstock/p-map-iterable","commit_stats":{"total_commits":11,"total_committers":2,"mean_commits":5.5,"dds":"0.36363636363636365","last_synced_commit":"deb10d8c70edf28dbbc4815292931c23bbcacdf8"},"previous_names":["shutterstock/p-map-iterable"],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shutterstock%2Fp-map-iterable","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shutterstock%2Fp-map-iterable/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shutterstock%2Fp-map-iterable/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shutterstock%2Fp-map-iterable/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/shutterstock","download_url":"https://codeload.github.com/shutterstock/p-map-iterable/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252810273,"owners_count":21807759,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["async-iterator","back-pressure","blocking-queue","concurrency","nodejs","p-map","parallel","queue"],"created_at":"2025-01-21T08:14:08.924Z","updated_at":"2025-05-07T04:06:34.437Z","avatar_url":"https://github.com/shutterstock.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![npm (scoped)](https://img.shields.io/npm/v/%40shutterstock/p-map-iterable)](https://www.npmjs.com/package/@shutterstock/p-map-iterable) [![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT) [![API Docs](https://img.shields.io/badge/API%20Docs-View%20Here-blue)](https://tech.shutterstock.com/p-map-iterable/) [![Build - CI](https://github.com/shutterstock/p-map-iterable/actions/workflows/ci.yml/badge.svg)](https://github.com/shutterstock/p-map-iterable/actions/workflows/ci.yml) [![Package and Publish](https://github.com/shutterstock/p-map-iterable/actions/workflows/publish.yml/badge.svg)](https://github.com/shutterstock/p-map-iterable/actions/workflows/publish.yml) [![Publish Docs](https://github.com/shutterstock/p-map-iterable/actions/workflows/docs.yml/badge.svg)](https://github.com/shutterstock/p-map-iterable/actions/workflows/docs.yml)\n\n# Overview\n\n`@shutterstock/p-map-iterable` provides several classes that allow processing results of `p-map`-style mapper functions by iterating the results as they are completed, with backpressure to limit the number of items that are processed ahead of the consumer.\n\nA common use case for `@shutterstock/p-map-iterable` is as a \"prefetcher\" that will fetch, for example, AWS S3 files in an AWS Lambda function. By prefetching large files the consumer is able to use 100% of the paid-for Lambda CPU time for the JS thread, rather than waiting idle while the next file is fetched. The backpressure (set by `maxUnread`) prevents the prefetcher from consuming unlimited memory or disk space by racing ahead of the consumer.\n\nThese classes will typically be helpful in batch or queue consumers, not as much in request/response services.\n\n# Example Usage Scenarios\n\n## Typical Processing Loop without `IterableMapper`\n\n```typescript\nconst source = new SomeSource();\nconst sourceIds = [1, 2,... 1000];\nconst sink = new SomeSink();\nfor (const sourceId of sourceIds) {\n  const item = await source.read(sourceId);     // takes 300 ms of I/O wait, no CPU\n  const outputItem = doSomeOperation(item);     // takes 20 ms of CPU\n  await sink.write(outputItem);                 // takes 500 ms of I/O wait, no CPU\n}\n```\n\nEach iteration takes 820ms total, but we waste time waiting for I/O. We could prefetch the next read (300ms) while processing (20ms) and writing (500ms), without changing the order of reads or writes.\n\n## Using `IterableMapper` as Prefetcher with Blocking Sequential Writes\n\n`concurrency: 1` on the prefetcher preserves the order of the reads and and writes are sequential and blocking (unchanged).\n\n```typescript\nconst source = new SomeSource();\nconst sourceIds = [1, 2,... 1000];\n// Pre-reads up to 8 items serially and releases in sequential order\nconst sourcePrefetcher = new IterableMapper(sourceIds,\n  async (sourceId) =\u003e source.read(sourceId),\n  { concurrency: 1, maxUnread: 10 }\n);\nconst sink = new SomeSink();\nfor await (const item of sourcePrefetcher) {    // may not block for fast sources\n  const outputItem = doSomeOperation(item);     // takes 20 ms of CPU\n  await sink.write(outputItem);                 // takes 500 ms of I/O wait, no CPU\n}\n```\n\nThis reduces iteration time to 520ms by overlapping reads with processing/writing.\n\n## Using `IterableMapper` as Prefetcher with Background Sequential Writes with `IterableQueueMapperSimple`\n\n`concurrency: 1` on the prefetcher preserves the order of the reads.\n`concurrency: 1` on the flusher preserves the order of the writes, but allows the loop to iterate while last write is completing.\n\n```typescript\nconst source = new SomeSource();\nconst sourceIds = [1, 2,... 1000];\nconst sourcePrefetcher = new IterableMapper(sourceIds,\n  async (sourceId) =\u003e source.read(sourceId),\n  { concurrency: 1, maxUnread: 10 }\n);\nconst sink = new SomeSink();\nconst flusher = new IterableQueueMapperSimple(\n  async (outputItem) =\u003e sink.write(outputItem),\n  { concurrency: 1 }\n);\nfor await (const item of sourcePrefetcher) {    // may not block for fast sources\n  const outputItem = doSomeOperation(item);     // takes 20 ms of CPU\n  await flusher.enqueue(outputItem);            // will periodically block for portion of write time\n}\n// Wait for all writes to complete\nawait flusher.onIdle();\n// Check for errors\nif (flusher.errors.length \u003e 0) {\n// ...\n}\n```\n\nThis reduces iteration time to about `max((max(readTime, writeTime) - cpuOpTime, cpuOpTime))`\nby overlapping reads and writes with the CPU processing step.\nIn this contrived example, the loop time is reduced to 500ms - 20ms = 480ms.\nIn cases where the CPU usage time is higher, the impact can be greater.\n\n## Using `IterableMapper` as Prefetcher with Out of Order Reads and Background Out of Order Writes with `IterableQueueMapperSimple`\n\nFor maximum throughput, allow out of order reads and writes with\n`IterableQueueMapper` (to iterate results with backpressure when too many unread items) or\n`IterableQueueMapperSimple` (to handle errors at end without custom iteration and applying backpressure to block further enqueues when `concurrency` items are in process):\n\n```typescript\nconst source = new SomeSource();\nconst sourceIds = [1, 2,... 1000];\nconst sourcePrefetcher = new IterableMapper(sourceIds,\n  async (sourceId) =\u003e source.read(sourceId),\n  { concurrency: 10, maxUnread: 20 }\n);\nconst sink = new SomeSink();\nconst flusher = new IterableQueueMapperSimple(\n  async (outputItem) =\u003e sink.write(outputItem),\n  { concurrency: 10 }\n);\nfor await (const item of sourcePrefetcher) {    // typically will not block\n  const outputItem = doSomeOperation(item);     // takes 20 ms of CPU\n  await flusher.enqueue(outputItem);            // typically will not block\n}\n// Wait for all writes to complete\nawait flusher.onIdle();\n// Check for errors\nif (flusher.errors.length \u003e 0) {\n // ...\n}\n```\n\nThis reduces iteration time to about 20ms by overlapping reads and writes with the CPU processing step. In this contrived (but common) example we would get a 41x improvement in throughput, removing 97.5% of the time to process each item and fully utilizing the CPU time available in the JS event loop.\n\n# Getting Started\n\n## Installation\n\nThe package is available on npm as [@shutterstock/p-map-iterable](https://www.npmjs.com/package/@shutterstock/p-map-iterable)\n\n`npm i @shutterstock/p-map-iterable`\n\n## Importing\n\n```typescript\nimport {\n  IterableMapper,\n  IterableQueueMapper,\n  IterableQueueMapperSimple } from '@shutterstock/p-map-iterable';\n```\n\n## API Documentation\n\nAfter installing the package, you might want to look at our [API Documentation](https://tech.shutterstock.com/p-map-iterable/) to learn about all the features available.\n\n# `p-map-iterable` vs `p-map` vs `p-queue`\n\nThese diagrams illustrate the differences in operation betweeen `p-map`, `p-queue`, and `p-map-iterable`.\n\n## `p-map-iterable`\n\n![p-map-iterable operations overview](https://github.com/shutterstock/p-map-iterable/assets/5617868/abdc7079-8c12-4518-8135-867fc5085e60)\n\n## `p-map`\n\n![p-map operations overview](https://github.com/shutterstock/p-map-iterable/assets/5617868/2fd88213-3135-4de8-8ec2-224555c08d65)\n\n## `p-queue`\n\n![p-queue operations overview](https://github.com/shutterstock/p-map-iterable/assets/5617868/88300edb-7bfe-41f0-ae5b-1cd5723bc255)\n\n# Features\n\n- [IterableMapper](https://tech.shutterstock.com/p-map-iterable/classes/IterableMapper.html)\n  - Interface and concept based on: [p-map](https://github.com/sindresorhus/p-map)\n  - Allows a sync or async iterable input\n  - User supplied sync or async mapper function\n  - Exposes an async iterable interface for consuming mapped items\n  - Allows a maximum queue depth of mapped items - if the consumer stops consuming, the queue will fill up, at which point the mapper will stop being invoked until an item is consumed from the queue\n  - This allows mapping with backpressure so that the mapper does not consume unlimited resources (e.g. memory, disk, network, event loop time) by racing ahead of the consumer\n- [IterableQueueMapper](https://tech.shutterstock.com/p-map-iterable/classes/IterableQueueMapper.html)\n  - Wraps `IterableMapper`\n  - Adds items to the queue via the `enqueue` method\n- [IterableQueueMapperSimple](https://tech.shutterstock.com/p-map-iterable/classes/IterableQueueMapperSimple.html)\n  - Wraps `IterableQueueMapper`\n  - Discards results as they become available\n  - Exposes any accumulated errors through the `errors` property instead of throwing an `AggregateError`\n  - Not actually `Iterable` - May rename this before 1.0.0\n\n## Lower Level Utilities\n- [IterableQueue](https://tech.shutterstock.com/p-map-iterable/classes/IterableQueue.html)\n  - Lower level utility class\n  - Wraps `BlockingQueue`\n  - Exposes an async iterable interface for consuming items in the queue\n- [BlockingQueue](https://tech.shutterstock.com/p-map-iterable/classes/BlockingQueue.html)\n  - Lower level utility class\n  - `dequeue` blocks until an item is available or until all items have been removed, then returns `undefined`\n  - `enqueue` blocks if the queue is full\n  - `done` signals that no more items will be added to the queue\n\n# `IterableMapper`\n\nSee [p-map](https://github.com/sindresorhus/p-map) docs for a good start in understanding what this does.\n\nThe key difference between `IterableMapper` and `pMap` are that `IterableMapper` does not return when the entire mapping is done, rather it exposes an iterable that the caller loops through. This enables results to be processed while the mapping is still happening, while optionally allowing for backpressure to slow or stop the mapping if the caller is not consuming items fast enough. Common use cases include `prefetching` items from a remote service - the next set of requests are dispatched asyncronously while the current responses are processed and the prefetch requests will pause when the unread queue fills up.\n\nSee [examples/iterable-mapper.ts](./examples/iterable-mapper.ts) for an example.\n\nRun the example with `npm run example:iterable-mapper`\n\n# `IterableQueueMapper`\n\n`IterableQueueMapper` is similar to `IterableMapper` but instead of taking an iterable input it instead adds data via the `enqueue` method which will block if `maxUnread` will be reached by the current number of `mapper`'s running in parallel.\n\nSee [examples/iterable-queue-mapper.ts](./examples/iterable-queue-mapper.ts) for an example.\n\nRun the example with `npm run example:iterable-queue-mapper`\n\n# `IterableQueueMapperSimple`\n\n`IterableQueueMapperSimple` is similar to `IterableQueueMapper` but instead exposing the results as an iterable it discards the results as soon as they are ready and exposes any errors through the `errors` property.\n\nSee [examples/iterable-queue-mapper-simple.ts](./examples/iterable-queue-mapper-simple.ts) for an example.\n\nRun the example with `npm run example:iterable-queue-mapper-simple`\n\n# Contributing - Setting up Build Environment\n\n- `nvm use`\n- `npm i`\n- `npm run build`\n- `npm run lint`\n- `npm run test`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshutterstock%2Fp-map-iterable","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshutterstock%2Fp-map-iterable","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshutterstock%2Fp-map-iterable/lists"}