{"id":19907542,"url":"https://github.com/mattiasbuelens/remote-web-streams","last_synced_at":"2025-05-08T00:26:29.873Z","repository":{"id":42236396,"uuid":"135857539","full_name":"MattiasBuelens/remote-web-streams","owner":"MattiasBuelens","description":"Web streams that work across web workers and iframes.","archived":false,"fork":false,"pushed_at":"2023-10-19T11:20:29.000Z","size":645,"stargazers_count":80,"open_issues_count":0,"forks_count":9,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-12-26T16:06:33.246Z","etag":null,"topics":["remote","stream","streams","transferable-objects","transferable-streams","web-worker","whatwg-streams","worker"],"latest_commit_sha":null,"homepage":"https://mattiasbuelens.github.io/remote-web-streams/examples/","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MattiasBuelens.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-06-02T22:48:53.000Z","updated_at":"2024-11-19T05:48:08.000Z","dependencies_parsed_at":"2024-06-19T04:07:53.448Z","dependency_job_id":"67d8b1b0-2a13-4489-98bc-02386a4a6867","html_url":"https://github.com/MattiasBuelens/remote-web-streams","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MattiasBuelens%2Fremote-web-streams","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MattiasBuelens%2Fremote-web-streams/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MattiasBuelens%2Fremote-web-streams/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MattiasBuelens%2Fremote-web-streams/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MattiasBuelens","download_url":"https://codeload.github.com/MattiasBuelens/remote-web-streams/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":231468021,"owners_count":18381174,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["remote","stream","streams","transferable-objects","transferable-streams","web-worker","whatwg-streams","worker"],"created_at":"2024-11-12T20:41:57.633Z","updated_at":"2024-12-27T10:07:54.378Z","avatar_url":"https://github.com/MattiasBuelens.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# remote-web-streams\n[Web streams][streams-spec] that work across web workers and `\u003ciframe\u003e`s.\n\n## Problem\nSuppose you want to process some data that you've downloaded somewhere. The processing is quite CPU-intensive,\nso you want to do it inside a worker. No problem, the web has you covered with `postMessage`!\n\n```js\n// main.js\n(async () =\u003e {\n  const response = await fetch('./some-data.txt');\n  const data = await response.text();\n  const worker = new Worker('./worker.js');\n  worker.onmessage = (event) =\u003e {\n    const output = event.data;\n    const results = document.getElementById('results');\n    results.appendChild(document.createTextNode(output)); // tadaa!\n  };\n  worker.postMessage(data);\n})();\n\n// worker.js\nself.onmessage = (event) =\u003e {\n  const input = event.data;\n  const output = process(input); // do the actual work\n  self.postMessage(output);\n}\n```\n\nAll is good: your processing does not block the main thread, so your web page remains responsive. However, it takes\nquite a long time before the results show up: first *all* of the data needs to be downloaded, then *all* that data\nneeds to be processed, and *finally* everything is shown on the page. Wouldn't it be nice if we could already show\nsomething as soon as *some* of the data has been downloaded and processed?\n\nNormally, you'd tackle this with by reading the input as a stream, piping it through one or more transform streams\nand finally displaying the results as they come in.\n\n```js\n// main.js\n(async () =\u003e {\n  const response = await fetch('./some-data.txt');\n  await response.body\n    .pipeThrough(new TransformStream({\n      transform(chunk, controller) {\n        controller.enqueue(process(chunk)); // do the actual work\n      }\n    }))\n    .pipeTo(new WritableStream({\n      write(chunk) {\n        const results = document.getElementById('results');\n        results.appendChild(document.createTextNode(chunk)); // tadaa!\n      }\n    }));\n})();\n```\n\nNow you can see the first results as they come in, but your processing is blocking the main thread again!\nCan we get the best of both worlds: **process data as it comes in, but off the main thread**?\n\n## Solution\nEnter: `remote-web-streams`. With this libray, you can create pairs of readable and writable streams\nwhere you can write chunks to a writable stream inside one context, and read those chunks from a readable stream\n**inside a different context**.\nFunctionally, such a pair behaves just like an [identity transform stream][identity-transform-stream], and you can\nuse and compose them just like any other stream.\n\n## Basic setup\n\n### RemoteReadableStream\nThe basic steps for setting up a pair of linked streams are:\n1. Construct a `RemoteReadableStream`. This returns two objects:\n   * a `MessagePort` which must be used to construct the linked `WritableStream` inside the other context\n   * a `ReadableStream` which will read chunks written by the linked `WritableStream`\n```js\n// main.js\nimport { RemoteReadableStream } from 'remote-web-streams';\nconst { readable, writablePort } = new RemoteReadableStream();\n```\n2. Transfer the `writablePort` to the other context, and instantiate the linked `WritableStream` in that context\n   using `fromWritablePort`.\n```js\n// main.js\nconst worker = new Worker('./worker.js', { type: 'module' });\nworker.postMessage({ writablePort }, [writablePort]);\n\n// worker.js\nimport { fromWritablePort } from 'remote-web-streams';\nself.onmessage = (event) =\u003e {\n  const { writablePort } = event.data;\n  const writable = RemoteWebStreams.fromWritablePort(writablePort);\n}\n```\n3. Use the streams as usual! Whenever you write something to the `writable` inside one context,\n   the `readable` in the other context will receive it.\n```js\n// worker.js\nconst writer = writable.getWriter();\nwriter.write('hello');\nwriter.write('world');\nwriter.close();\n\n// main.js\n(async () =\u003e {\n  const reader = readable.getReader();\n  console.log(await reader.read()); // { done: false, value: 'hello' }\n  console.log(await reader.read()); // { done: false, value: 'world' }\n  console.log(await reader.read()); // { done: true, value: undefined }\n})();\n```\n\n### RemoteWritableStream\nYou can also create a `RemoteWritableStream`.\nThis is the complement to `RemoteReadableStream`:\n* The constructor (in the original context) returns a `WritableStream` (instead of a readable one).\n* You transfer the `readablePort` to the other context,\n  and instantiate the linked `ReadableStream` with `fromReadablePort` inside that context.\n```js\n// main.js\nimport { RemoteWritableStream } from 'remote-web-streams';\nworker.postMessage({ readablePort }, [readablePort]);\nconst writer = writable.getWriter();\n// ...\n\n// worker.js\nimport { fromReadablePort } from 'remote-web-streams';\nself.onmessage = (event) =\u003e {\n  const { readablePort } = event.data;\n  const reader = readable.getReader();\n  // ...\n}\n```\n\n## Examples\n\n### Remote transform stream\nIn the basic setup, we create one pair of streams and transfer one end to the worker.\nHowever, it's also possible to set up multiple pairs and transfer them all to a worker.\n\nThis opens up interesting possibilities. We can use a `RemoteWritableStream` to write chunks to a worker,\nlet the worker transform them using one or more `TransformStream`s, and then read those transformed chunks\nback on the main thread using a `RemoteReadableStream`.\nThis allows us to move one or more CPU-intensive `TransformStream`s off the main thread,\nand turn them into a \"remote transform stream\".\n\nTo demonstrate these \"remote transform streams\", we set one up to solve the original problem statement:\n1. Create a `RemoteReadableStream` and a `RemoteWritableStream` on the main thread.\n2. Transfer both streams to the worker. Inside the worker, connect the `readable` to the `writable` by piping it\n   through one or more `TransformStream`s.\n3. On the main thread, write data to be transformed into the `writable` and read transformed data from the `readable`.\n   Pro-tip: we can use `.pipeThrough({ readable, writable })` for this!\n\n```js\n// main.js\nimport { RemoteReadableStream, RemoteWritableStream } from 'remote-web-streams';\n(async () =\u003e {\n  const worker = new Worker('./worker.js', { type: 'module' });\n  // create a stream to send the input to the worker\n  const { writable, readablePort } = new RemoteWritableStream();\n  // create a stream to receive the output from the worker\n  const { readable, writablePort } = new RemoteReadableStream();\n  // transfer the other ends to the worker\n  worker.postMessage({ readablePort, writablePort }, [readablePort, writablePort]);\n\n  const response = await fetch('./some-data.txt');\n  await response.body\n    // send the downloaded data to the worker\n    // and receive the results back\n    .pipeThrough({ readable, writable })\n    // show the results as they come in\n    .pipeTo(new WritableStream({\n      write(chunk) {\n        const results = document.getElementById('results');\n        results.appendChild(document.createTextNode(chunk)); // tadaa!\n      }\n    }));\n})();\n\n// worker.js\nimport { fromReadablePort, fromWritablePort } from 'remote-web-streams';\nself.onmessage = async (event) =\u003e {\n  // create the input and output streams from the transferred ports\n  const { readablePort, writablePort } = event.data;\n  const readable = fromReadablePort(readablePort);\n  const writable = fromWritablePort(writablePort);\n\n  // process data\n  await readable\n    .pipeThrough(new TransformStream({\n      transform(chunk, controller) {\n        controller.enqueue(process(chunk)); // do the actual work\n      }\n    }))\n    .pipeTo(writable); // send the results back to main thread\n};\n```\nWith this set up, we achieve the desired goals:\n* Data is transformed as soon as it arrives on the main thread.\n* Transformed data is displayed on the web page as soon as it is transformed by the worker.\n* All of the data processing happens inside the worker, so it never blocks the main thread.\n\nThe results are shown as fast as possible, and your web page stays snappy. Great success! 🎉\n\n## Behind the scenes\nThe library works its magic by creating a `MessageChannel` between the `WritableStream` and the `ReadableStream`.\nThe writable end sends a message to the readable end whenever a new chunk is written,\nso the readable end can enqueue it for reading.\nSimilarly, the readable end sends a message to the writable end whenever it needs more data,\nso the writable end can release any backpressure.\n\n[streams-spec]: https://streams.spec.whatwg.org/\n[identity-transform-stream]: https://streams.spec.whatwg.org/#identity-transform-stream\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmattiasbuelens%2Fremote-web-streams","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmattiasbuelens%2Fremote-web-streams","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmattiasbuelens%2Fremote-web-streams/lists"}