https://github.com/daanv2/queue-processor

A promise like processor that tries to keep the event-loop as clean as possible
https://github.com/daanv2/queue-processor

event eventloop js npm processor queue ts

Last synced: 26 days ago
JSON representation

A promise like processor that tries to keep the event-loop as clean as possible

• Host: GitHub
• URL: https://github.com/daanv2/queue-processor
• Owner: DaanV2
• License: mit
• Created: 2021-10-15T13:49:06.000Z (over 4 years ago)
• Default Branch: main
• Last Pushed: 2025-02-18T15:41:55.000Z (about 1 year ago)
• Last Synced: 2025-08-31T19:40:56.625Z (7 months ago)
• Topics: event, eventloop, js, npm, processor, queue, ts
• Language: TypeScript
• Homepage: https://www.npmjs.com/package/@daanv2/queue-processor
• Size: 187 KB
• Stars: 2
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE
  • Security: SECURITY.md

Awesome Lists containing this project

README

          
# Queue-Processor

A processor of collections that chunks the items into subsections. Each chunk is scheduled after when the previous chunk or item has been processed. This allows other workflows or scheduled work to execute before continuing on a large dataset. These processors try to keep the event loop as clean as possible. 

[![npm-publish](https://github.com/DaanV2/queue-processor/actions/workflows/npm-publish.yml/badge.svg)](https://github.com/DaanV2/queue-processor/actions/workflows/npm-publish.yml)

[![npm-test](https://github.com/DaanV2/queue-processor/actions/workflows/npm-test.yml/badge.svg)](https://github.com/DaanV2/queue-processor/actions/workflows/npm-test.yml)

[![tagged-release](https://github.com/DaanV2/queue-processor/actions/workflows/tagged-release.yml/badge.svg)](https://github.com/DaanV2/queue-processor/actions/workflows/tagged-release.yml)

## Why would you need this?

I have an extension that takes a ~30 to process a couple of hundred files, summarize them and diagnose problems; this all takes CPU time. While at the same time, I wanted to give priority to smaller and faster takes, in which I needed a library that could ensure that things got chunked into smaller pieces of work. So other smaller tasks could be done in between the large parts.

Pros:

- Faster response times on other tasks/processes as they also get their time to shine.

Cons:

- Slightly slower. Chunking a process and allowing other tasks to go first determines how long it takes to finish something.

## Example

```ts

  //The array of items to process

  const items : T[] = getItems();

  //Processes for each

  QueueProcessor.forEach(items, (item)=>console.log(item));

  //Map each item

  const mapped : U[] = QueueProcessor.map(items, (item)=>convert(item));

  //Filter each item

  const filtered : T[] = QueueProcessor.filter(items, (item)=>item === ...);

  //Processes for each, then use the promise to wait for the result

  QueueProcessor.forEach(items, (item)=>console.log(item)).then(items=>{...});

  //Async await code

  const items = await QueueProcessor.forEach(items, (item)=>console.log(item));

```