{"id":23438548,"url":"https://github.com/simple-dev-tools/super-loop","last_synced_at":"2025-10-19T23:50:55.601Z","repository":{"id":57677334,"uuid":"487540765","full_name":"simple-dev-tools/super-loop","owner":"simple-dev-tools","description":"Easy programming interface for producer-consumer problem, leveraging nodejs stream.","archived":false,"fork":false,"pushed_at":"2022-12-17T23:40:07.000Z","size":889,"stargazers_count":0,"open_issues_count":2,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-16T02:06:48.127Z","etag":null,"topics":["backpressure","concurrency","nodejs","producer-consumer","stream","tps"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/simple-dev-tools.png","metadata":{"files":{"readme":"README.md","changelog":"Changelog.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-05-01T13:09:39.000Z","updated_at":"2022-05-01T13:35:46.000Z","dependencies_parsed_at":"2023-01-29T18:16:03.015Z","dependency_job_id":null,"html_url":"https://github.com/simple-dev-tools/super-loop","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simple-dev-tools%2Fsuper-loop","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simple-dev-tools%2Fsuper-loop/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simple-dev-tools%2Fsuper-loop/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simple-dev-tools%2Fsuper-loop/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/simple-dev-tools","download_url":"https://codeload.github.com/simple-dev-tools/super-loop/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248097979,"owners_count":21047346,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backpressure","concurrency","nodejs","producer-consumer","stream","tps"],"created_at":"2024-12-23T14:50:01.338Z","updated_at":"2025-10-19T23:50:50.555Z","avatar_url":"https://github.com/simple-dev-tools.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![NPM](https://nodei.co/npm/super-loop.png)](https://www.npmjs.com/package/super-loop)\n\n# Super Loop\n\nEasy programming interface for producer-consumer problem and enhanced loop, leveraging nodejs stream. \n\nIt provides the following features: \n* repeats cycle for `n` times;\n* repeats cycle until future moment;\n* repeats cycle until \"ender\" function returns true;\n* supports tps controlling;\n* supports backpressure;\n* supports max concurrency configuration;\n\n## Why\n\nAsynchronous event-based processing is quite common in modern microservice architecture as well as data ETL. Here are some typical use cases: \n\n* Consume SQS messages at given max tps and max concurrency to fully utilize system capacity (e.g. CPU);\n* Read upstream events and persist them to a NoSQL db at fast and controlled pace;\n* Purge NoSQL db records selectively at fast and controlled pace;\n* Process CSV files and call REST API at fast and controlled pace;\n* Performance-test HTTP endpoints with customized data, i.e. a simplified version of K6;\n\n\nThe goal of this library is to provide easy programming interfaces for those common and popular use cases, so as to increase developer efficiency. \n\n\n## Installation\n\n```shell\nnpm i --save super-loop\n```\n\n## Usage Examples\n\n### Example - producer consumer problem\n\n```js\nconst SuperLoop =  require('super-loop');\nconst stats = require('measured-core').createCollection();\n\nasync function main() {\n    const intv = setInterval(function () {\n        console.log(JSON.stringify(stats, null, 2));\n    }, 1000);\n\n    try {\n        const producer = () =\u003e {\n            stats.meter('producerTps').mark();\n            return [{name: 'John'}, {name: 'Alex'}]\n        }\n\n        const consumer = (data) =\u003e {\n            stats.meter('consumerTps').mark();\n            // processing ...\n            //console.log(data);\n        }\n\n        const loop = new SuperLoop();\n        const lastFor = 120_000;\n\n        loop.on('warn', (err) =\u003e {\n            console.error(err);\n        });\n\n        await loop.producedBy(producer)\n            .consumedBy(consumer)\n            .concurrency(200)\n            .rate(1000)\n            //.repeat(100)\n            .until(Date.now() + lastFor)\n            .exec();\n\n        console.log('loop ends')\n\n    } catch (e) {\n        console.error('something went wrong', e)\n    }\n    clearInterval(intv);\n}\n\nmain().catch(console.error)\n```\n\n### More examples\n\n* [CSV Processing](https://github.com/simple-dev-tools/super-loop/blob/main/examples/csv_processor.js)\n* [Enhanced for/while Loop](https://github.com/simple-dev-tools/super-loop/blob/main/examples/simple_loop.js)\n* [Performance-test your HTTP endpoints](https://github.com/simple-dev-tools/super-loop/blob/main/examples/performance_test1.js)\n\n\n## API\n\nSuper Loop API design follows [Fluent Interface](https://en.wikipedia.org/wiki/Fluent_interface).  All methods return `this`, except `exec` which kicks off the execution of the loop.\n\n### consumedBy(func)\n\nConfigure consumer function. \n\nArguments: \n* `func` is a function that take nodejs stream chunk as param, and returns nothing. e.g. `(data) =\u003e { console.log(data) }` \n\n### producedBy(func)\n\nConfigure producer function.\n\nArguments: \n* `func` is a function that takes no param, but returns an Array of data chunks. e.g. `() =\u003e [1, 2, 'a', 'b']`\n\n### invoke(func)\n\nAlias to `consumedBy`. \n\n### endedBy(func)\n\nConfigure ender function. When the ender function returns `true`, the loop ends. e.g. `() =\u003e false`\n\nArguments: \n* `func` is a function that takes no param, but returns a boolean.  \n\n### concurrency(maxC)\n\nConfigure max concurrency for consumer function.\n\nArguments: \n* `maxC` is the max concurrecy the consumer function would run at. \n\n### rate(tps)\n\nConfigure max tps for consumer function.\n\nArguments: \n* `tps` is the max tps the consumer function would run at. \n\n### until(endTime)\n\nConfigure timestamp (in `ms`) when the loop should stop. \n\nArguments: \n* `endTime` timestamp in `ms`\n\n### repeat(times)\n\nConfigure max repeats the producer function should be called. After max repeats are reached, loop ends. \n\nArguments: \n* `times` max repeats\n\n### pipeFrom(upstream) \n\nConfigure custom upstream rather than using super-loop internal `Readable` as upstream. \n\nA good example is processing file stream.\n\nArguments:\n* `upstream` Transform or Readable stream \n\n### pipeTo(downstream) \n\nConfigure custom downstream to continue streaming.\n\nArguments:\n* `downstream` Transform or Writable stream \n\n### exec() \n\nStart the loop for execution, until one of the ending conditions is met. The ending conditions are specified by `endedBy`, `until`, `repeat`. \n\nReturns:\n\n* __Error__ if critical error happens in the internal nodejs streams\n\n### Events\n\n* `warn`. The loop emits non-critical errors when processing individual data chunks. Here is an example to catch them and log them properly: \n\n```js\nloop.on('warn', (err) =\u003e {\n    myLogger.error(err);\n});\n```\n\n\n## License\nsuper-loop is licensed under the MIT license.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsimple-dev-tools%2Fsuper-loop","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsimple-dev-tools%2Fsuper-loop","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsimple-dev-tools%2Fsuper-loop/lists"}