{"id":15710148,"url":"https://github.com/willfarrell/datastream","last_synced_at":"2026-03-12T06:03:07.297Z","repository":{"id":49349418,"uuid":"505292588","full_name":"willfarrell/datastream","owner":"willfarrell","description":"Commonly used stream patterns for Web Streams API and NodeJS Streams","archived":false,"fork":false,"pushed_at":"2024-09-25T18:45:02.000Z","size":4102,"stargazers_count":7,"open_issues_count":0,"forks_count":2,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-06T13:01:36.552Z","etag":null,"topics":["nodejs-stream","stream","streams","web-stream"],"latest_commit_sha":null,"homepage":"https://datastream.js.org","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/willfarrell.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":".github/CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":".github/CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["willfarrell"],"patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"custom":null}},"created_at":"2022-06-20T04:22:26.000Z","updated_at":"2024-12-30T01:02:16.000Z","dependencies_parsed_at":"2024-06-10T03:42:00.445Z","dependency_job_id":"8ba22e8a-3726-444e-adb5-8da62fb80caf","html_url":"https://github.com/willfarrell/datastream","commit_stats":{"total_commits":79,"total_committers":4,"mean_commits":19.75,"dds":"0.30379746835443033","last_synced_commit":"5cd6e3f16b820bfd8055c1b9403e5e5f69dc7f75"},"previous_names":[],"tags_count":39,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/willfarrell%2Fdatastream","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/willfarrell%2Fdatastream/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/willfarrell%2Fdatastream/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/willfarrell%2Fdatastream/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/willfarrell","download_url":"https://codeload.github.com/willfarrell/datastream/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249207219,"owners_count":21230028,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["nodejs-stream","stream","streams","web-stream"],"created_at":"2024-10-03T21:03:43.705Z","updated_at":"2026-03-12T06:03:07.290Z","avatar_url":"https://github.com/willfarrell.png","language":"JavaScript","readme":"\u003cdiv align=\"center\"\u003e\n\u003c!--\u003cbr/\u003e\u003cbr/\u003e\u003cbr/\u003e\u003cbr/\u003e\u003cbr/\u003e\u003cbr/\u003e\u003cbr/\u003e\n\u003cbr/\u003e\u003cbr/\u003e\u003cbr/\u003e\u003cbr/\u003e\u003cbr/\u003e\u003cbr/\u003e\u003cbr/\u003e--\u003e\n\u003ch1\u003e\u0026lt;datastream\u0026gt;\u003c/h1\u003e\n\u003cp\u003eCommonly used stream patterns for Web Streams API and NodeJS Stream.\u003c/p\u003e\n\u003cp\u003eIf you're iterating over an array more than once, it's time to use streams.\u003c/p\u003e\n\u003cbr /\u003e\n\u003cp\u003e\n  \u003ca href=\"https://github.com/willfarrell/datastream/actions/workflows/test-unit.yml\"\u003e\u003cimg src=\"https://github.com/willfarrell/datastream/actions/workflows/test-unit.yml/badge.svg\" alt=\"GitHub Actions unit test status\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/willfarrell/datastream/actions/workflows/test-dast.yml\"\u003e\u003cimg src=\"https://github.com/willfarrell/datastream/actions/workflows/test-dast.yml/badge.svg\" alt=\"GitHub Actions dast test status\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/willfarrell/datastream/actions/workflows/test-perf.yml\"\u003e\u003cimg src=\"https://github.com/willfarrell/datastream/actions/workflows/test-perf.yml/badge.svg\" alt=\"GitHub Actions perf test status\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/willfarrell/datastream/actions/workflows/test-sast.yml\"\u003e\u003cimg src=\"https://github.com/willfarrell/datastream/actions/workflows/test-sast.yml/badge.svg\" alt=\"GitHub Actions SAST test status\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/willfarrell/datastream/actions/workflows/test-lint.yml\"\u003e\u003cimg src=\"https://github.com/willfarrell/datastream/actions/workflows/test-lint.yml/badge.svg\" alt=\"GitHub Actions lint test status\"\u003e\u003c/a\u003e\n  \u003cbr/\u003e\n  \u003ca href=\"https://www.npmjs.com/package/@datastream/core\"\u003e\u003cimg alt=\"npm version\" src=\"https://img.shields.io/npm/v/@datastream/core.svg\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://packagephobia.com/result?p=@datastream/core\"\u003e\u003cimg src=\"https://packagephobia.com/badge?p=@datastream/core\" alt=\"npm install size\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://www.npmjs.com/package/@datastream/core\"\u003e\n  \u003cimg alt=\"npm weekly downloads\" src=\"https://img.shields.io/npm/dw/@datastream/core.svg\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://www.npmjs.com/package/@datastream/core#provenance\"\u003e\n  \u003cimg alt=\"npm provenance\" src=\"https://img.shields.io/badge/provenance-Yes-brightgreen\"\u003e\u003c/a\u003e\n  \u003cbr/\u003e\n  \u003ca href=\"https://scorecard.dev/viewer/?uri=github.com/willfarrell/datastream\"\u003e\u003cimg src=\"https://api.scorecard.dev/projects/github.com/willfarrell/datastream/badge\" alt=\"Open Source Security Foundation (OpenSSF) Scorecard\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://slsa.dev\"\u003e\u003cimg src=\"https://slsa.dev/images/gh-badge-level3.svg\" alt=\"SLSA 3\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/willfarrell/datastream/blob/main/docs/CODE_OF_CONDUCT.md\"\u003e\u003cimg src=\"https://img.shields.io/badge/Contributor%20Covenant-2.1-4baaaa.svg\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://biomejs.dev\"\u003e\u003cimg alt=\"Checked with Biome\" src=\"https://img.shields.io/badge/Checked_with-Biome-60a5fa?style=flat\u0026logo=biome\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://conventionalcommits.org\"\u003e\u003cimg alt=\"Conventional Commits\" src=\"https://img.shields.io/badge/Conventional%20Commits-1.0.0-%23FE5196?logo=conventionalcommits\u0026logoColor=white\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/willfarrell/datastream/blob/main/package.json#L32\"\u003e\n  \u003cimg alt=\"code coverage\" src=\"https://img.shields.io/badge/code%20coverage-95%25-brightgreen\"\u003e\u003c/a\u003e\n  \u003cbr/\u003e\n  \u003ca href=\"https://gitter.im/willfarrell/Lobby\"\u003e\u003cimg src=\"https://badges.gitter.im/gitterHQ/gitter.svg\" alt=\"Chat on Gitter\" style=\"max-width:100%;\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://stackoverflow.com/questions/tagged/datastream?sort=Newest\u0026uqlId=35052\"\u003e\u003cimg src=\"https://img.shields.io/badge/StackOverflow-[datastream]-yellow\" alt=\"Ask questions on StackOverflow\" style=\"max-width:100%;\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\u003c/div\u003e\n\n- [`@datastream/core`](#core)\n  - pipeline\n  - pipejoin\n  - streamToArray\n  - streamToString\n  - isReadable\n  - isWritable\n  - makeOptions\n  - createReadableStream\n  - createTransformStream\n  - createWritableStream\n\n## Streams\n\n- Readable: The start of a pipeline of streams that injects data into a stream.\n- PassThrough: Does not modify the data, but listens to the data and prepares a result that can be retrieved.\n- Transform: Modifies data as it passes through.\n- Writable: The end of a pipeline of streams that stores data from the stream.\n\n### Basics\n\n- [`@datastream/string`](#string)\n  - stringReadableStream [Readable]\n  - stringLengthStream [PassThrough]\n  - stringOutputStream [PassThrough]\n- [`@datastream/object`](#object)\n  - objectReadableStream [Readable]\n  - objectCountStream [PassThrough]\n  - objectBatchStream [Transform]\n  - objectOutputStream [PassThrough]\n\n### Common\n\n- [`@datastream/fetch`](#fetch)\n  - fetchResponseStream [Readable]\n- [`@datastream/charset[/{detect,decode,encode}]`](#charset)\n  - charsetDetectStream [PassThrough]\n  - charsetDecodeStream [Transform]\n  - charsetEncodeStream [Transform]\n- [`@datastream/compression[/{gzip,deflate}]`](#compression)\n  - gzipCompressionStream [Transform]\n  - gzipDecompressionStream [Transform]\n  - deflateCompressionStream [Transform]\n  - deflateDecompressionStream [Transform]\n- [`@datastream/digest`](#digest)\n  - digestStream [PassThrough]\n\n### Advanced\n\n- [`@datastream/csv[/{parse,format}]`](#csv)\n  - csvParseStream [Transform]\n  - csvFormatStream [Transform]\n- [`@datastream/validate`](#validate)\n  - validateStream [Transform]\n\n## Setup\n\n```bash\nnpm install @datastream/core @datastream/{module}\n```\n\n## Flows\n\n```mermaid\nstateDiagram-v2\n\n    [*] --\u003e fileRead*: path\n    [*] --\u003e fetchResponse: URL\n    [*] --\u003e sqlCopyTo*: SQL\n    [*] --\u003e stringReadable: string\n    [*] --\u003e stringReadable: string[]\n    [*] --\u003e objectReadable: object[]\n    [*] --\u003e createReadable: blob\n\n    readable --\u003e charsetDetect: binary\n    charsetDetect --\u003e [*]\n\n    readable --\u003e decryption\n    decryption --\u003e passThroughBuffer: buffer\n\n    readable --\u003e decompression\n    decompression --\u003e passThroughBuffer: buffer\n    passThroughBuffer --\u003e charsetDecode: buffer\n    charsetDecode --\u003e passThroughString: string\n    passThroughString --\u003e parse: string\n    parse --\u003e validate: object\n    validate --\u003e passThroughObject: object\n    passThroughObject --\u003e transform: object\n\n    transform --\u003e format: object\n    format --\u003e charsetEncode: string\n    charsetEncode --\u003e compression: buffer\n    compression --\u003e writable: buffer\n\n    charsetEncode --\u003e encryption: buffer\n    encryption --\u003e writable: buffer\n\n    state readable {\n        fileRead*\n        fetchResponse\n        sqlCopyTo*\n        createReadable\n        stringReadable\n        objectReadable\n        awsS3Get\n        awsDynamoDBQuery\n        awsDynamoDBScan\n        awsDynamoDBGet\n    }\n\n    state decompression {\n        brotliDeompression\n        gzipDeompression\n        deflateDeompression\n        zstdDecompression*\n        protobufDecompression*\n    }\n\n    state decryption {\n      decryption*\n    }\n\n    state parse {\n      csvParse\n      jsonParse*\n      xmlParse*\n    }\n\n    state passThroughBuffer {\n      digest\n    }\n\n    state passThroughString {\n      stringLength\n      stringOutput\n    }\n\n    state passThroughObject {\n      objectCount\n      objectOutput\n    }\n\n    state transform {\n      objectBatch\n      objectPivotLongToWide\n      objectPivotWideToLong\n      objectKeyValue\n      objectKeyValue\n    }\n\n    state format {\n      csvFormat\n      jsonFormat*\n      xmlFormat*\n    }\n\n    state compression {\n        brotliCompression\n        gzipCompression\n        deflateCompression\n        zstdCompression*\n        protobufCompression*\n    }\n\n    state encryption {\n      encryption*\n    }\n\n    state writable {\n        fileWrite*\n        fetchRequest*\n        sqlCopyFrom*\n        awsS3Put\n        awsDynamoDBPut\n        awsDynamoDBDelete\n    }\n    writable --\u003e [*]\n```\n\n\\* possible future package\n\n## Write your own\n\n### Readable\n\n#### NodeJS Streams\n\n- [NodeJS](https://nodejs.org/api/stream.html#class-streamreadable)\n\n#### Web Streams API\n\n- [MDN](https://developer.mozilla.org/en-US/docs/Web/API/ReadableStream)\n- [NodeJS](https://nodejs.org/api/webstreams.html#class-readablestream)\n\n### Transform\n\n#### NodeJS Streams\n\n- [NodeJS](https://nodejs.org/api/stream.html#class-streamtransform)\n\n#### Web Streams API\n\n- [MDN](https://developer.mozilla.org/en-US/docs/Web/API/TransformStream)\n- [NodeJS](https://nodejs.org/api/webstreams.html#class-transformstream)\n\n### Writeable\n\n#### NodeJS Streams\n\n- [NodeJS](https://nodejs.org/api/stream.html#class-streamwritable)\n\n#### Web Streams API\n\n- [MDN](https://developer.mozilla.org/en-US/docs/Web/API/WritableStream)\n- [NodeJS](https://nodejs.org/api/webstreams.html#class-writablestream)\n\n## End-to-End Examples\n\n### NodeJS: Import CSV into SQL database\n\nRead a CSV file, validate the structure, pivot data, then save compressed.\n\n- fs.creatReadStream\n- gzip\n- cryptoDigest\n- charsetDecode\n- csvParse\n- countChunks\n- validate\n- changeCase (pascal to snake)\n- parquet?\n- csvFormat\n- postgesCopyFrom\n\n### WebWorker: Validate and collect metadata about file prior to upload\n\n- \u003cinput type=\"file\"\u003e\n- cryptoDigest\n- charsetDetect\n- jsonParse?\n- validate\n\n### WebWorker: Upload file compressed\n\nUpload file with brotli compression?\n\n### WebWorker: Decompress protobuf compressed JSON requests\n\nFetch protobuf file, decompress, parse JSON\n\n### streams\n\n- filter\n\n- file (docs only?)\n\n### examples\n\n- fetch\n- node:fs\n- input type=file\n- readable string/array/etc\n\n## License\n\nLicensed under [MIT License](LICENSE). Copyright (c) 2026 [will Farrell](https://github.com/willfarrell) and [contributors](https://github.com/middyjs/middy/graphs/contributors).\n","funding_links":["https://github.com/sponsors/willfarrell"],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwillfarrell%2Fdatastream","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwillfarrell%2Fdatastream","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwillfarrell%2Fdatastream/lists"}