{"id":41284016,"url":"https://github.com/funny-bytes/continuous-streams","last_synced_at":"2026-01-23T02:58:35.837Z","repository":{"id":41806812,"uuid":"323420634","full_name":"funny-bytes/continuous-streams","owner":"funny-bytes","description":"Special purpose Node streams","archived":false,"fork":false,"pushed_at":"2025-09-24T13:03:05.000Z","size":425,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-11-03T00:17:22.739Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/funny-bytes.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2020-12-21T18:45:33.000Z","updated_at":"2025-09-24T13:03:03.000Z","dependencies_parsed_at":"2025-08-18T22:33:02.124Z","dependency_job_id":"65d9ce83-e58e-4242-a11d-010b149874bf","html_url":"https://github.com/funny-bytes/continuous-streams","commit_stats":{"total_commits":38,"total_committers":2,"mean_commits":19.0,"dds":0.3157894736842105,"last_synced_commit":"e3d3fbd303a55d2e2947e8939e66f339c29ca982"},"previous_names":["frankthelen/continuous-streams"],"tags_count":19,"template":false,"template_full_name":null,"purl":"pkg:github/funny-bytes/continuous-streams","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/funny-bytes%2Fcontinuous-streams","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/funny-bytes%2Fcontinuous-streams/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/funny-bytes%2Fcontinuous-streams/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/funny-bytes%2Fcontinuous-streams/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/funny-bytes","download_url":"https://codeload.github.com/funny-bytes/continuous-streams/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/funny-bytes%2Fcontinuous-streams/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28679139,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-23T01:00:35.747Z","status":"online","status_checked_at":"2026-01-23T02:00:08.296Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-23T02:58:34.638Z","updated_at":"2026-01-23T02:58:35.823Z","avatar_url":"https://github.com/funny-bytes.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Continuous Streams\n\nStream classes with specific behavior:\n\n* continuous reading, i.e., it doesn't stop if there is temporarily no data available\n* configurable chunk size to read\n* specific error handling (error vs. skip/retry)\n* configurable parallel processing\n* timeouts\n* graceful shutdown after `SIGTERM` or `SIGINT`\n\n![main workflow](https://github.com/funny-bytes/continuous-streams/actions/workflows/main.yml/badge.svg)\n[![Coverage Status](https://coveralls.io/repos/github/funny-bytes/continuous-streams/badge.svg?branch=main)](https://coveralls.io/github/funny-bytes/continuous-streams?branch=main)\n[![Maintainability](https://api.codeclimate.com/v1/badges/d0f823493c0977615c21/maintainability)](https://codeclimate.com/github/funny-bytes/continuous-streams/maintainability)\n[![node](https://img.shields.io/node/v/continuous-streams.svg)](https://nodejs.org)\n[![code style](https://img.shields.io/badge/code_style-airbnb-brightgreen.svg)](https://github.com/airbnb/javascript)\n[![Types](https://img.shields.io/npm/types/continuous-streams.svg)](https://www.npmjs.com/package/continuous-streams)\n[![License Status](http://img.shields.io/npm/l/continuous-streams.svg)]()\n\n## Install\n\n```bash\nnpm install continuous-streams\n```\n\n## Usage\n\nThis is a basic example (ES6/ES2015).\n\n```javascript\nconst { pipeline } = require('stream');\nconst { ContinuousReader, ContinuousWriter } = require('continuous-streams');\n\nconst reader = new ContinuousReader({\n  chunkSize: 100, // default `50`\n});\nreader.readData = async (count) =\u003e {\n  // read `count` items from resource\n  // if rejects, a `skip` event is emitted (unless `skipOnError` is `false`)\n  return items; // resolve with array of data items (can be empty)\n};\n\nconst writer = new ContinuousWriter({\n  parallelOps: 10, // max number of `writeData()` to fire off in parallel\n});\nwriter.writeData = async (item) =\u003e {\n  // process a single data item\n  // if rejects, a `skip` event is emitted (unless `skipOnError` is `false`)\n};\n\npipeline( // go!\n  reader,\n  writer,\n  (error) =\u003e { ... }, // pipeline stopped\n);\n\n['SIGTERM', 'SIGINT'].forEach((eventType) =\u003e {\n  process.on(eventType, () =\u003e reader.stop());\n});\n```\n\nOr if you prefer **TypeScript** (types included):\n\n```typescript\nimport { pipeline } from \"stream\";\nimport { ContinuousReader, ContinuousWriter } from \"continuous-streams\";\n\nconst reader = new ContinuousReader\u003cFoo\u003e();\nreader.readData = async (count: number): Promise\u003cFoo[]\u003e =\u003e {\n  // ...\n  return items;\n}\n\nconst writer = new ContinuousWriter\u003cFoo\u003e();\nwriter.writeData = async (item: Foo): Promise\u003cvoid\u003e =\u003e {\n  // ...\n}\n\n// ...\n```\n\nFor this module to work, your **TypeScript compiler options** must include\n`\"target\": \"ES2015\"` (or later), `\"moduleResolution\": \"node\"`, and\n`\"esModuleInterop\": true`.\n\n## Stream Classes\n\n### Class `ContinuousReader`\n\nExtends `stream.Readable`.\nIt reads from an underlying data source (`objectMode` only) -- please implement or assign `readData()` for that purpose.\nIt does not `end` when the underlying data source is (temporarily) empty but waits for new data to arrive.\nIt is robust/reluctant regarding (temporary) reading errors.\nIt supports gracefully stopping the pipeline.\n\n#### Options\n\n* `chunkSize` - Whenever the number of objects in the internal buffer is dropping below `chunkSize`, a new chunk of data is read from the underlying resource. Higher means fewer polling but less real-time. Default is `50`.\n* `skipOnError` - If `true` (default), a `skip` event is emitted when `readData()` rejects. If `false`, an `error` event is emitted when `readData()` rejects and the pipeline will stop. Default is `true`.\n* `waitAfterEmpty` - Delay in milliseconds if there is (temporarily) no data available. Default is `5000`.\n* `waitAfterLow` - Delay in milliseconds if there is (temporarily) less data available than chunk size. Default is `1000`.\n* `waitAfterError` - Delay in milliseconds if there is a (temporary) reading problem. Default is `10000`.\n* `autoStop` - The intention of this package was to provide easy streaming without stopping the stream when the source is (temporarily) empty. But sometimes, however, this is exactly what you want. Default is `false`. Set to `true` to automatically and gracefully stop streaming when `readData()` returns no data or less than requested.\n\n#### Methods\n\n* `async readData(count)` -- Reads `count` data items from the underlying resource. To be implemented or assigned. `count` is usually equals `chunkSize`. It resolves with an array of data items -- which may be empty if there is temporarily no data available. If it rejects, an `error` or `skip` event is emitted (depending on `skipOnError`).\n* `stop()` - To be called after `SIGINT` or `SIGTERM` for gracefully stopping the pipeline. The `end` event is emitted either immediately (if the read buffer is empty) or at the next reading attempt. Graceful shutdown means that all data that has been read so far will be fully processed throughout the entire pipeline. Example: `process.on('SIGTERM', () =\u003e reader.stop())`.\n\n#### Events\n\n* `skip` - When reading from the underlying resource failed (if `skipOnError` is `true`). The stream continues to read after a delay of `waitAfterError`. Example handler: `reader.on('skip', ({ error }) =\u003e { ... })`.\n* `error` - When reading from the underlying resource failed (if `skipOnError` is `false`). This will stop the pipeline. Example handler: `reader.on('error', (error) =\u003e { ... })`.\n* `end` - When `stop()` was called for gracefully stopping the pipeline.\n* `close` - When the stream is closed (as usual).\n* `debug` - After each successful reading attempt providing some debug information. Example handler: `reader.on('debug', ({ items, requested, total, elapsed }) =\u003e { ... })`. `items` is the number of read data items. `requested` is the number of requested data items (normally equals `count`). `total` is an overall counter. `elapsed` is the number of milliseconds of `readData()` to resolve.\n\n### Class `ContinuousWriter`\n\nExtends `stream.Writable`.\nIt processes data items (`objectMode` only) for some sort of write operation at the end of a pipeline  -- please implement or assign `writeData()` for that purpose.\nIt is robust/reluctant regarding errors.\nIf `skipOnError` is `true` (default), an error during a write operation emits a `skip` event and stream processing will continue.\nIf `skipOnError` is `false`, an error during a write operation emits an `error` event and stream processing will stop.\nSupports gracefully stopping the entire pipeline, i.e., it waits until all asynchronous operations in-flight are returned before emitting the `finish` event.\n\n#### Options\n\n* `parallelOps` - The max number of asynchronous `writeData()` operations to fire off in parallel. Default is `10`.\n* `skipOnError` - If `true` (default), a `skip` event is emitted when `writeData()` rejects. If `false`, an `error` event is emitted when `writeData()` rejects and the pipeline will stop. Default is `true`.\n* `timeoutMillis` - Timeout in milliseconds for `writeData()`. Default is `60000`.\n\n#### Methods\n\n* `async writeData(item)` -- Processes a single data item. To be implemented or assigned. If it rejects, an `error` or `skip` event is emitted (depending on `skipOnError`).\n\n#### Events\n\n* `skip` - If `skipOnError` is `true` (default). Example handler: `writer.on('skip', ({ data, error }) =\u003e { ... })`.\n* `error` - If `skipOnError` is `false`. This will stop the pipeline.\n* `finish` - After graceful shutdown and all asynchronous write operations are returned.\n* `close` - After `error` or `finish` (as usual).\n* `debug` - After each successful write operation providing some debug information. Example handler: `writer.on('debug', ({ inflight, total, elapsed }) =\u003e { ... })`. `inflight` is the number of asynchronous `writeData()` operations currently inflight. `total` is an overall counter. `elapsed` is the number of milliseconds of `writeData()` to resolve.\n\n### Class `ContinuousTransformer`\n\nExtends `stream.Transform`.\nIt processes data items (`objectMode` only) for some sort of transform operation in the midst of a pipeline  -- please implement or assign `transformData()` for that purpose.\nIt is robust/reluctant regarding errors.\nIf `skipOnError` is `true` (default), an error during a transform operation emits a `skip` event and stream processing will continue.\nIf `skipOnError` is `false`, an error during a transform operation emits an `error` event and stream processing will stop.\n\n#### Options\n\n* `parallelOps` - The max number of asynchronous `transformData()` operations to fire off in parallel. Default is `10`.\n* `skipOnError` - If `true` (default), a `skip` event is emitted when `transformData()` rejects. If `false`, an `error` event is emitted when `transformData()` rejects and the pipeline will stop. Default is `true`.\n* `timeoutMillis` - Timeout in milliseconds for `transformData()`. Default is `60000`.\n\n#### Methods\n\n* `async transformData(item)` -- Processes a single data item. Resolves with the transformed data item (or an array of items for splitting the item into multiple items). To be implemented or assigned. If it rejects, an `error` or `skip` event is emitted (depending on `skipOnError`).\n\n#### Events\n\n* `skip` - If `skipOnError` is `true` (default). Example handler: `transformer.on('skip', ({ data, error }) =\u003e { ... })`.\n* `error` - If `skipOnError` is `false`. This will stop the pipeline.\n* `close` - When the stream is closed (as usual).\n* `debug` - After each successful transform operation providing some debug information. Example handler: `transformer.on('debug', ({ inflight, total, elapsed }) =\u003e { ... })`. `inflight` is the number of asynchronous `transformData()` operations currently inflight. `total` is an overall counter. `elapsed` is the number of milliseconds of `transformData()` to resolve.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffunny-bytes%2Fcontinuous-streams","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffunny-bytes%2Fcontinuous-streams","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffunny-bytes%2Fcontinuous-streams/lists"}