{"id":15673598,"url":"https://github.com/lmammino/stream-accumulator","last_synced_at":"2025-05-07T18:24:51.092Z","repository":{"id":57371938,"uuid":"125659460","full_name":"lmammino/stream-accumulator","owner":"lmammino","description":"Accumulate all the data flowing through a stream and emit it as a single chunk or as a promise","archived":false,"fork":false,"pushed_at":"2018-04-07T16:37:44.000Z","size":113,"stargazers_count":9,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-15T17:54:11.351Z","etag":null,"topics":["accumulator","buffer","data-pipeline","data-processing","library","module","node","nodejs","stream","streams"],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lmammino.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-03-17T18:51:11.000Z","updated_at":"2019-05-01T06:41:59.000Z","dependencies_parsed_at":"2022-08-30T14:51:59.864Z","dependency_job_id":null,"html_url":"https://github.com/lmammino/stream-accumulator","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lmammino%2Fstream-accumulator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lmammino%2Fstream-accumulator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lmammino%2Fstream-accumulator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lmammino%2Fstream-accumulator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lmammino","download_url":"https://codeload.github.com/lmammino/stream-accumulator/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252932620,"owners_count":21827341,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["accumulator","buffer","data-pipeline","data-processing","library","module","node","nodejs","stream","streams"],"created_at":"2024-10-03T15:41:24.501Z","updated_at":"2025-05-07T18:24:51.069Z","avatar_url":"https://github.com/lmammino.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🥒 stream-accumulator\n\n[![npm version](https://badge.fury.io/js/stream-accumulator.svg)](http://badge.fury.io/js/stream-accumulator)\n[![CircleCI](https://circleci.com/gh/lmammino/stream-accumulator.svg?style=shield)](https://circleci.com/gh/lmammino/stream-accumulator)\n[![codecov.io](https://codecov.io/gh/lmammino/stream-accumulator/coverage.svg?branch=master)](https://codecov.io/gh/lmammino/stream-accumulator)\n[![JavaScript Style Guide](https://img.shields.io/badge/code_style-standard-brightgreen.svg)](https://standardjs.com)\n\n\nAccumulate all the data flowing through a stream and emit it as a single chunk or as a promise.\n\n⚠️ **Warning**: This module will buffer all the data from the source stream in memory, so watch out for infinite (or very large) streams. Using this library might not be ideal in such cases!\n\n\n## 💽 Install \u0026 quick usage\n\n**Requirements**: this library is written for Node.js \u003e= 6.0\n\nAs usual, this happens through NPM:\n\n```bash\nnpm install --save stream-accumulator\n```\n\nThen, in your code:\n\n```javascript\nconst StreamAccumulator = require('stream-accumulator')\n\n// sourceStream is some readable stream\n\n// ... inside an async function\nconst streamContent = await StreamAccumulator.promise(sourceStream)\n\n// do stuff with streamContent (which would be a buffer)\n```\n\n\n## 🤔 Rationale\n\nOftentimes you are receiving data through a stream, but you have to accumulate all the data (waiting the stream to end) before you can use it. For instance, when you have to apply a transformation that requires the full data to be loaded in memory for the transformation to be possible.\n\nThis library allows you to accumulate the data of a stream into a single Buffer and allows you to consume the resulting Buffer through a stream or a promise based interface.\n\n\n## 😲 An example\n\nLet's say you are receiving some JavaScript source code through a streaming interface and you want to minify the code using [UglifyJS](https://www.npmjs.com/package/uglify-js). This is how you generally solve the problem:\n\n```javascript\nconst { createReadStream } = require('fs')\nconst { minify } = require('uglify-js')\n\nlet data = ''\ncreateReadStream('source.js')\n  .on('data', (d) =\u003e data += d.toString())\n  .on('error', (e) =\u003e {\n    console.error(e)\n    process.exit(1)\n  })\n  .on('end', () =\u003e {\n    const { code: minifiedCode } = minify(data)\n    console.log('Minified code', minifiedCode)\n  })\n```\n\nWhile this implementation is fine and it's not particularly complicated, it might not be the most composable one it's a bit verbose to read.\n\nUsing `StreamAccumulator` you can solve the same problem as follows:\n\n```javascript\nconst { createReadStream } = require('fs')\nconst StreamAccumulator = require('stream-accumulator')\n\nconst source = createReadStream('source.js')\nsource\n  .pipe(new StreamAccumulator())\n  .on('end', (data) =\u003e {\n    const { code: minifiedCode } = minify(data)\n    console.log('Minified code', minifiedCode)\n  })\n```\n\nAnd, if you use the promise based interface you can even use `async/await` and have a more streamlined and easy to read implementation:\n\n```javascript\nconst { createReadStream } = require('fs')\nconst StreamAccumulator = require('stream-accumulator')\n\n// ... inside an async function\nconst source = createReadStream('source.js')\nconst code = await StreamAccumulator.promise(source)\nconst { code: minifiedCode } = minify(data)\nconsole.log('Minified code', minifiedCode)\n```\n\n\n## 🕹 Usage\n\nThis library offers 2 APIs, a stream based one (transform stream) and a promise based one.\n\n\n### 🌊 Stream based API\n\nThe stream based API allows you to pipe a `Readable` stream into `StreamAccumulator`. `StreamAccumulator` will buffer the entire readable stream and will emit a single chunk when the source stream is ended.\n\n\n#### Example:\n\n```javascript\nconst StreamAccumulator = require('stream-accumulator')\n\n// initialize someReadableStream\n\nsomeReadableStream\n  .pipe(new StreamAccumulator())\n  .on('end', (data) =\u003e {\n    // data is a buffer\n  })\n```\n\n\n### 🤞Promise based API\n\nThe promise based API allows you to wait for the source stream to finish (or emit an error.\n\n\n#### Example:\n\n```javascript\nconst StreamAccumulator = require('stream-accumulator')\n\n// initialize someSourceStream\n\n// ... inside an async function\nconst data = await StreamAccumulator.promise(someSourceStream)\n// data is a buffer\n```\n\n\n## 👯‍ Contributing\n\nEveryone is very welcome to contribute to this project.\nYou can contribute just by submitting bugs or suggesting improvements by\n[opening an issue on GitHub](https://github.com/lmammino/stream-accumulator/issues).\n\n\n## 🤦‍ License\n\nLicensed under [MIT License](LICENSE). © Luciano Mammino.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flmammino%2Fstream-accumulator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flmammino%2Fstream-accumulator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flmammino%2Fstream-accumulator/lists"}