{"id":13493467,"url":"https://github.com/MattMorgis/async-stream-generator","last_synced_at":"2025-03-28T11:32:46.652Z","repository":{"id":57185830,"uuid":"131325832","full_name":"MattMorgis/async-stream-generator","owner":"MattMorgis","description":"Pipe ES6 Async Generators through Node.js Streams","archived":false,"fork":false,"pushed_at":"2019-06-27T15:08:32.000Z","size":168,"stargazers_count":54,"open_issues_count":0,"forks_count":3,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-02T05:19:29.917Z","etag":null,"topics":["async-generator","async-iterators","nodejs","streams"],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MattMorgis.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-04-27T17:24:35.000Z","updated_at":"2024-05-27T14:27:15.000Z","dependencies_parsed_at":"2022-09-06T04:11:35.960Z","dependency_job_id":null,"html_url":"https://github.com/MattMorgis/async-stream-generator","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MattMorgis%2Fasync-stream-generator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MattMorgis%2Fasync-stream-generator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MattMorgis%2Fasync-stream-generator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MattMorgis%2Fasync-stream-generator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MattMorgis","download_url":"https://codeload.github.com/MattMorgis/async-stream-generator/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246021433,"owners_count":20710938,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["async-generator","async-iterators","nodejs","streams"],"created_at":"2024-07-31T19:01:15.486Z","updated_at":"2025-03-28T11:32:46.332Z","avatar_url":"https://github.com/MattMorgis.png","language":"JavaScript","funding_links":[],"categories":["JavaScript"],"sub_categories":[],"readme":"# async-stream-generator\n\nPipe ES6 Async Generators through Node.js [Streams](https://nodejs.org/api/stream.html).\n\n## 10 Second Tutorial\n\n`streamify` is a function that takes an async generator function and when invoked, returns a Readable Stream.\n\n```javascript\nconst fs = require(\"fs\");\nconst streamify = require(\"async-stream-generator\");\n\nasync function* generator(stream) {\n  for await (const chunk of stream) {\n    yield chunk;\n  }\n}\n\nconst main = () =\u003e {\n  const readStream = fs.createReadStream(\"path-to-data.json\");\n  streamify(generator(readStream)).pipe(process.stdout);\n};\n\nmain();\n```\n\n## What are Streams and Why Should I Care?\n\nI/O in node is asynchronous. The early days of Node.js required interacting with the disk and network by passing callbacks to functions.\n\nFor example, here is code that serves up a file from disk:\n\n```javascript\nconst http = require(\"http\");\nconst fs = require(\"fs\");\n\nconst server = http.createServer((request, response) =\u003e {\n  fs.readFile(__dirname + \"/mock-data.json\", (error, data) =\u003e {\n    response.end(data);\n  });\n});\nserver.listen(8000);\n```\n\nThis code works but it buffers up the entire file into memory for every request before writing the result back to clients. If the file is very large, your program could start eating a lot of memory as it serves lots of users concurrently, particularly for users on slow connections.\n\nThe user experience is poor too because users will need to wait for the whole file to be buffered into memory on your server before they can start receiving any content.\n\nHowever, both `request` and `response` are **streams**.\n\n```javascript\nconst http = require(\"http\");\nconst fs = require(\"fs\");\n\nconst server = http.createServer((req, res) =\u003e {\n  const stream = fs.createReadStream(__dirname + \"/mock-data.json\");\n  stream.pipe(res);\n});\nserver.listen(8000);\n```\n\nThis is where Node.js shines. `.pipe()` will write to clients one chunk at at a time immediately as they are received from disk.\n\nUsing `.pipe()` has other benefits too, like handling backpressure automatically so that node won't buffer chunks into memory needlessly when the remote client is on a really slow or high-latency connection.\n\nThis is very much like what you might do on the command-line to pipe programs together except in node instead of the shell!\n\n```\na | b | c | d\n```\n\nOnce you learn the stream api, you can just snap together streaming modules like lego bricks instead of having to remember how to push data through non-streaming, custom APIs.\n\nStreams make programming in node simple, elegant, and composable.\n\n## What are Async Iterators and Generators?\n\nPreviously to read the contents of a stream asynchronously, you used callbacks:\n\n```javascript\nconst fs = require(\"fs\");\n\nconst main = inputFilePath =\u003e {\n  const readStream = fs.createReadStream(inputFilePath, {\n    encoding: \"utf8\",\n    highWaterMark: 256\n  });\n\n  readStream.on(\"data\", chunk =\u003e {\n    console.log(\"\u003e\u003e\u003e \" + chunk);\n    console.log(\"\\n\");\n  });\n\n  readStream.on(\"end\", () =\u003e {\n    console.log(\"### DONE ###\");\n  });\n};\n\nmain(\"./mock-data.json\");\n```\n\nAs of Node.js v10, you can use [`asynchronous iteration`](https://github.com/tc39/proposal-async-iteration) to read the stream of a file, which enables the `for-await-of` syntax:\n\n```javascript\nconst fs = require(\"fs\");\n\nconst main = async inputFilePath =\u003e {\n  const readStream = fs.createReadStream(inputFilePath, {\n    encoding: \"utf8\",\n    highWaterMark: 256\n  });\n\n  for await (const chunk of readStream) {\n    console.log(\"\u003e\u003e\u003e \" + chunk);\n    console.log(\"\\n\");\n  }\n\n  console.log(\"### DONE ###\");\n};\n\nmain(\"./mock-data.json\");\n```\n\nOutput for both:\n\n```\n...\n\n\u003e\u003e\u003e ld\":\"Indonesia\",\"customer_title\":\"Honorable\"}\n{\"guid\":\"bf62800e-b3b1-46f2-a3f2-dc17c66c90a1\",\"car_make\":\"Ford\",\"car_model\":\"Bronco II\",\"car_model_year\":1986,\"car_color\":\"Pink\",\"car_country_cold\":\"Philippines\",\"customer_title\":\"Rev\"}\n{\"guid\":\"32a2f79b-5a0b-\n\n\n\u003e\u003e\u003e 4072-9ebb-0e3600d0f714\",\"car_make\":\"Toyota\",\"car_model\":\"RAV4\",\"car_model_year\":2001,\"car_color\":\"Purple\",\"car_country_cold\":\"China\",\"customer_title\":\"Mr\"}\n{\"guid\":\"6d52f031-c7e7-4167-81bc-e2879d6630e2\",\"car_make\":\"Lexus\",\"car_model\":\"SC\",\"car_model_year\":\n\n\n\u003e\u003e\u003e 1998,\"car_color\":\"Teal\",\"car_country_cold\":\"Russia\",\"customer_title\":\"Rev\"}\n\n\n\n### DONE ###\n```\n\nYou can use async generators to process input similiar to Unix piping. Generator functions use the `async` and `function*` keywords, consume an async iterator and use `yield` instead of `return`.\n\nExample of Generator #1, which will process our chunks of data into lines:\n\n```javascript\nasync function* chunksToLines(chunks) {\n  let previous = \"\";\n\n  for await (const chunk of chunks) {\n    previous += chunk;\n    let eolIndex;\n\n    while ((eolIndex = previous.indexOf(\"\\n\")) \u003e= 0) {\n      // this line includes the EOL\n      const line = previous.slice(0, eolIndex + 1);\n      yield line;\n      previous = previous.slice(eolIndex + 1);\n    }\n  }\n\n  if (previous.length \u003e 0) {\n    yield previous;\n  }\n}\n```\n\nExample of Generator #2, which will number each line\n\n```javascript\nasync function* numberOfLines(lines) {\n  let counter = 1;\n  for await (const line of lines) {\n    yield counter + \": \" + line;\n    counter++;\n  }\n}\n```\n\nNow you can snap these generators together using function composition to stream the file to the console line by line.\n\nThe whole program will read in the file 256 bytes at a time (defined by `highWaterMark`). Break each chunk into lines, number them, print them, and repeat.\n\n```javascript\nconst printAsyncIterable = async numberedLines =\u003e {\n  for await (const line of numberedLines) {\n    console.log(line);\n  }\n};\n\nconst main = () =\u003e {\n  const readStream = fs.createReadStream(\"./mock-data.json\", {\n    encoding: \"utf8\",\n    highWaterMark: 256\n  });\n  printAsyncIterable(numberOfLines(chunksToLines(readStream)));\n};\n\nmain();\n```\n\nOutput\n\n```\n...\n3999: {\"guid\":\"32a2f79b-5a0b-4072-9ebb-0e3600d0f714\",\"car_make\":\"Toyota\",\"car_model\":\"RAV4\",\"car_model_year\":2001,\"car_color\":\"Purple\",\"car_country_cold\":\"China\",\"customer_title\":\"Mr\"}\n\n4000: {\"guid\":\"6d52f031-c7e7-4167-81bc-e2879d6630e2\",\"car_make\":\"Lexus\",\"car_model\":\"SC\",\"car_model_year\":1998,\"car_color\":\"Teal\",\"car_country_cold\":\"Russia\",\"customer_title\":\"Rev\"}\n```\n\n## Where Async Generators Fall Short\n\nThese new tools are great for _reading_ streams, however, it's still not clear how to `write()` to another stream or create a processing pipeline with `pipe()`.\n\nThis was discussed [here](https://github.com/tc39/proposal-async-iteration/issues/74).\n\nEnter this module.\n\nUsing the same generators from above, we can `pipe()` the results to a writeable stream.\n\n```javascript\nconst http = require(\"http\");\nconst fs = require(\"fs\");\nconst streamify = require(\"async-stream-generator\");\n\nconst server = http.createServer(async (req, res) =\u003e {\n  const readStream = fs.createReadStream(\"./mock-data.json\", {\n    encoding: \"utf8\",\n    highWaterMark: 256\n  });\n\n  streamify(numberOfLines(chunksToLines(readStream))).pipe(res);\n});\n\nserver.listen(8000);\n```\n\n## References and Thank Yous\n\n* All code can be found in the `examples` directory.\n\n* This was forked from [@mimetnet](https://github.com/mimetnet)'s module [stream-generators](https://github.com/mimetnet/node-stream-generators), which offers the same functionality to synchronous generators.\n\n* Early stream examples and a deeper dive into streams can be found at [@substack](https://github.com/substack)'s [Stream Handbook](https://github.com/substack/stream-handbook).\n\n* Async Generator and Iterator examples from [2ality.com](http://2ality.com/2018/04/async-iter-nodejs.html)\n\n* [TC39 Proposal](https://github.com/tc39/proposal-async-iteration).\n\n* Node.js Support for [Symbol.asyncIterator](https://github.com/nodejs/readable-stream/issues/254).\n\n* [Node.js Stream Meeting Notes - Async Iterators](https://github.com/tc39/proposal-async-iteration/issues/74).\n\n## License\n\n[MIT](https://github.com/MattMorgis/async-stream-generator/blob/master/LICENSEd)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMattMorgis%2Fasync-stream-generator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FMattMorgis%2Fasync-stream-generator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMattMorgis%2Fasync-stream-generator/lists"}