https://github.com/rixrix/task-nibbler
Sequentially or concurrently process an array of items, easily. your call.
https://github.com/rixrix/task-nibbler
concurrency javascript nodejs queue sequence task
Last synced: 11 months ago
JSON representation
Sequentially or concurrently process an array of items, easily. your call.
- Host: GitHub
- URL: https://github.com/rixrix/task-nibbler
- Owner: rixrix
- License: mit
- Created: 2017-06-26T04:13:17.000Z (over 8 years ago)
- Default Branch: master
- Last Pushed: 2017-07-15T09:52:22.000Z (over 8 years ago)
- Last Synced: 2025-03-05T15:39:07.659Z (12 months ago)
- Topics: concurrency, javascript, nodejs, queue, sequence, task
- Language: JavaScript
- Homepage:
- Size: 10.7 KB
- Stars: 0
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
[](https://travis-ci.org/rixrix/task-nibbler)
[](https://badge.fury.io/js/task-nibbler)
# Task Nibbler
Sequentially or concurrently process an array of items, easily. your call.
## Use Case
I have a few tasks that pulls out data from 3rd party APIs. These tasks are a mixed of:
* short-lived
* long-running tasks
* Some 3rd party API providers with rate limiting applied is tricky to work with. This library helps out in pulling data in burst mode - 5 concurrent downloads - or just one after the other.
## Install
### NPM
```bash
$> npm install task-nibbler
```
### Yarn
```bash
$> yarn add task-nibbler
```
## Usage
```javascript
var taskNibbler = require('task-nibbler').nibbler;
var arrayOfThings = [
1, 2, 3,...., 10000
];
// This is your callback method for processing each item
// Note that it should return a Promise so that `task-nibbler` knows
// when to call or process the next task from the queue
function myItemThingProcessor(arrayOfThingsItem) {
return new Promise(function(resolve, reject) {
// work on each item like:
// - query data from database
// - download file from AWS S3 bucket
// - query data from your analytics API, etc
resolve();
});
}
taskNibbler(arrayOfThings, myItemThingProcessor)
.then(function() {
// done, do your post processing
})
.catch(function(error) {
// when an error occurs
});
```
## API
### nibbler(items, [opts], callback(string))
Returns a Promise.
`items`
String[], array of items to process
opts - Object, optional flags for various flags
`opts.concurrencyLevel` is a number for setting up the concurrency level. Default level is 1, of which, roughly equates to sequential processing of each item.
Default 1
`callback`
Object, required callback method wrapped in Promise. Task Nibbler calls this function once the item is in queue and passes the item value as the first.
## Todo
* Webpack
* Test
* Hook with TravisCI
* Auto-publish to NPM, NuGet, Bower
* ES6-ify
* TypeScript definition file
## References
* [Async#queue](https://caolan.github.io/async/docs.html#queue)
## License
MIT