{"id":13809008,"url":"https://github.com/cloudnc/observable-webworker","last_synced_at":"2025-05-14T05:32:51.378Z","repository":{"id":39420682,"uuid":"190942305","full_name":"cloudnc/observable-webworker","owner":"cloudnc","description":"Simplified API for working with Web Workers with RxJS","archived":false,"fork":false,"pushed_at":"2024-06-17T20:10:37.000Z","size":6613,"stargazers_count":230,"open_issues_count":8,"forks_count":15,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-04-14T20:56:17.421Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://cloudnc.github.io/observable-webworker","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cloudnc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-06-08T22:47:58.000Z","updated_at":"2025-04-13T22:29:08.000Z","dependencies_parsed_at":"2024-01-18T01:53:25.573Z","dependency_job_id":"6a7ef555-53a9-4a59-bc84-693dbcfab8c3","html_url":"https://github.com/cloudnc/observable-webworker","commit_stats":{"total_commits":99,"total_committers":5,"mean_commits":19.8,"dds":0.2525252525252525,"last_synced_commit":"2afe8ede43a8a328b8a3f69bb240d2a5b52b3ded"},"previous_names":[],"tags_count":28,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudnc%2Fobservable-webworker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudnc%2Fobservable-webworker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudnc%2Fobservable-webworker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudnc%2Fobservable-webworker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cloudnc","download_url":"https://codeload.github.com/cloudnc/observable-webworker/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254076979,"owners_count":22010631,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-04T01:01:57.577Z","updated_at":"2025-05-14T05:32:50.813Z","avatar_url":"https://github.com/cloudnc.png","language":"TypeScript","readme":"# Observable Webworker\n\nSimple API for using [web workers](https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Using_web_workers) with [RxJS](https://rxjs-dev.firebaseapp.com/guide/overview) observables\n\n[![Strict TypeScript Checked](https://badgen.net/badge/TS/Strict \"Strict TypeScript Checked\")](https://www.typescriptlang.org)\n[![npm version](https://badge.fury.io/js/observable-webworker.svg)](https://www.npmjs.com/package/observable-webworker)\n[![Build Status](https://github.com/cloudnc/observable-webworker/workflows/CI/badge.svg)](https://github.com/cloudnc/observable-webworker/actions)\n[![codecov](https://codecov.io/gh/cloudnc/observable-webworker/branch/master/graph/badge.svg)](https://codecov.io/gh/cloudnc/observable-webworker)\n[![Commitizen friendly](https://img.shields.io/badge/commitizen-friendly-brightgreen.svg)](https://commitizen.github.io/cz-cli/)\n[![License](https://img.shields.io/github/license/cloudnc/observable-webworker)](https://raw.githubusercontent.com/cloudnc/observable-webworker/master/LICENSE)\n![npm peer dependency version](https://img.shields.io/npm/dependency-version/observable-webworker/peer/rxjs)\n\n[![NPM](https://nodei.co/npm/observable-webworker.png?compact=true)](https://nodei.co/npm/observable-webworker/)\n\n# Features\n\n- Simple `fromWorker` function from main thread side\n- Fully RxJS interfaces allowing both main thread and worker thread streaming\n- Error handling across the thread boundaries is propagated\n  - Under the hood `materialize` and `dematerialize` is used as a robust transport of streaming errors\n- Automatic handling of worker termination on main thread unsubscription of observable\n- Framework agnostic - while the demo uses Angular, the only dependency is rxjs, so React, Vue or plain old JS is\n  completely compatible\n- Fully compatible with [Webpack worker-plugin](https://github.com/GoogleChromeLabs/worker-plugin)\n  - Therefore compatible with [Angular webworker bundling](https://angular.io/guide/web-worker) which uses this\n- Class interface based worker creation (should be familiar API for Angular developers)\n- Unopinionated on stream switching behavior, feel free to use `mergeMap`, `switchMap` or `exhaustMap` in your worker if\n  the input stream contains multiple items that generate their own stream of results\n- Built in interfaces for handling [`Transferable`](https://developer.mozilla.org/en-US/docs/Web/API/Transferable) parts\n  of message payloads so large binaries can transferred efficiently without copying - See [Transferable](#transferable)\n  section for usage\n- Automatic destruction of worker on unsubscription of output stream, this allows for smart cancelling of computation\n  using `switchMap` operator, or parallelisation of computation with `mergeMap`\n- [Worker Pool strategy](#worker-pool-strategy) - maximise the throughput of units of work by utilising all cores on the host machine\n\n## Demo\nhttps://cloudnc.github.io/observable-webworker\n\n## Tutorial\nhttps://dev.to/zakhenry/observable-webworkers-with-angular-8-4k6\n\n## Articles\n* [Observable Web Workers, a deep dive into a realistic use case](https://dev.to/zakhenry/observable-web-workers-a-deep-dive-into-a-realistic-use-case-4042)\n* [Parallel computation in the browser with observable webworkers](https://dev.to/zakhenry/parallel-computation-in-the-browser-with-observable-webworkers-hci)\n\n## Install\n\nInstall the [npm package](https://www.npmjs.com/package/observable-webworker): `observable-webworker`\n\n```sh\n# with npm\nnpm install observable-webworker\n# or with yarn\nyarn add observable-webworker\n```\n\n## Usage\n\n### Quickstart\n\n#### Main Thread\n\n💡 Take note! The webworker construction syntax differs for different version of webpack:\n\n#### Webpack \u003c 5 (deprecated)\n\n```ts\n// src/readme/hello-legacy-webpack.ts\n\nimport { fromWorker } from 'observable-webworker';\nimport { of } from 'rxjs';\n\nconst input$ = of('Hello from main thread');\n\nfromWorker\u003cstring, string\u003e(() =\u003e new Worker('./hello.worker', { type: 'module' }), input$).subscribe(message =\u003e {\n  console.log(message); // Outputs 'Hello from webworker'\n});\n\n```\n#### Webpack 5\n\n```ts\n// src/readme/hello.ts#L2-L12\n\nimport { fromWorker } from 'observable-webworker';\nimport { of } from 'rxjs';\n\nconst input$ = of('Hello from main thread');\n\nfromWorker\u003cstring, string\u003e(\n  () =\u003e new Worker(new URL('./hello.worker', import.meta.url), { type: 'module' }),\n  input$,\n).subscribe(message =\u003e {\n  console.log(message); // Outputs 'Hello from webworker'\n});\n```\n\n#### Worker Thread\n\n```ts\n// src/readme/hello.worker.ts\n\nimport { DoWork, runWorker } from 'observable-webworker';\nimport { Observable } from 'rxjs';\nimport { map } from 'rxjs/operators';\n\nexport class HelloWorker implements DoWork\u003cstring, string\u003e {\n  public work(input$: Observable\u003cstring\u003e): Observable\u003cstring\u003e {\n    return input$.pipe(\n      map(message =\u003e {\n        console.log(message); // outputs 'Hello from main thread'\n        return `Hello from webworker`;\n      }),\n    );\n  }\n}\n\nrunWorker(HelloWorker);\n\n```\n\n#### Decorator deprecation notice\n\nFuture versions of webpack (Webpack 5) minify webworkers overly aggressively, causing\nthe `@ObservableWorker()` decorator pattern to no longer function. This decorator\nhas now been deprecated, and will be removed in the next major version of this library.\n\nTo migrate from decorators, simply remove the decorator, and invoke the `runWorker`\nwith your class passed as argument (see example above).\n\nMake sure you **don't forget to remove the decorator** when you add the `runWorker(...)`\nfunction, otherwise the worker will be run twice, each acting on any message sent. \n\n## Transferable\n\nIf either your input or output (or both!) streams are passing large messages to or from the worker, it is highly\nrecommended to use message types that implement the [Transferable](https://developer.mozilla.org/en-US/docs/Web/API/Transferable)\ninterface (`ArrayBuffer`, `MessagePort`, `ImageBitmap`).\n\nBear in mind that when transferring a message to a webworker that the main thread relinquishes ownership of the data.\n\nRecommended reading:\n\n- https://developer.mozilla.org/en-US/docs/Web/API/Transferable\n- https://developer.mozilla.org/en-US/docs/Web/API/Worker/postMessage\n\nTo use `Transferable`s with observable-worker, a slightly more complex interface is provided for both sides of the\nmain/worker thread.\n\nIf the main thread is transferring `Transferable`s _to the worker_, simply add a callback to the `fromWorker` function\ncall to select which elements of the input stream are transferable.\n\n\u003c!-- prettier-ignore --\u003e\n```ts\n// src/readme/transferable.main.ts#L7-L11\n\nreturn fromWorker\u003cArrayBuffer, string\u003e(\n  () =\u003e new Worker(new URL('./transferable.worker', import.meta.url), { type: 'module' }),\n  input$,\n  input =\u003e [input],\n);\n```\n\nIf the worker is transferring `Transferable`s _to the main thread_ simply implement `DoTransferableWork`, which will\nrequire you to add an additional method `selectTransferables?(output: O): Transferable[];` which you implement to select\nwhich elements of the output object are `Transferable`.\n\nBoth strategies are compatible with each other, so if for example you're computing the hash of a large `ArrayBuffer` in\na worker, you would only need to use add the transferable selector callback in the main thread in order to mark the\n`ArrayBuffer` as being transferable in the input. The library will handle the rest, and you can just use `DoWork` in the\nworker thread, as the return type `string` is not `Transferable`.\n\n## Worker Pool Strategy\n\nIf you have a large amount of work that needs to be done, you can use the `fromWorkerPool` function to automatically \nmanage a pool of workers to allow true concurrency of work, distributed evenly across all available cores.\n\nThe worker pool strategy has the following features\n* Work can be provided as either `Observable`, `Array`, or `Iterable`\n* Concurrency is limited to `navigation.hardwareConcurrency - 1` to keep the main core free.\n  * This is a configurable option if you know you already have other workers running\n* Workers are only created when there is need for them (work is available)\n* Workers are terminated when there is no more work, freeing up threads for other processes\n  * for `Observable`, work is considered remaining while the observable is not completed\n  * for `Array`, work remains while there are items in the array\n  * for `Iterable`, work remains while the iterator is not `result.done` \n* Workers are kept running while work remains, preventing unnecessary downloading of the worker script\n* Custom observable flattening operator can be passed, allowing for custom behaviour such as correlating the output \norder with input order\n  * default operator is `mergeAll()`, which means the output from the webworker(s) is output as soon as available\n\n  \n### Example\n\nIn this simple example, we have a function that receives an array of files and returns an observable of the MD5 sum\nhashes of those files. For simplicity, we're passing the primitives back and forth, however in reality you are likely to \nwant to construct your own interface to define the messages being passed to and from the worker.\n\n#### Main Thread\n\n```ts\n// src/readme/worker-pool.main.ts\n\nimport { Observable } from 'rxjs';\nimport { fromWorkerPool } from 'observable-webworker';\n\nexport function computeHashes(files: File[]): Observable\u003cstring\u003e {\n  return fromWorkerPool\u003cFile, string\u003e(\n    () =\u003e new Worker(new URL('./worker-pool-hash.worker', import.meta.url), { type: 'module' }),\n    files,\n  );\n}\n\n```\n\n#### Worker Thread\n\n```ts\n// src/readme/worker-pool-hash.worker.ts\n\nimport * as md5 from 'js-md5';\nimport { DoWorkUnit, runWorker } from 'observable-webworker';\nimport { Observable } from 'rxjs';\nimport { map } from 'rxjs/operators';\n\nexport class WorkerPoolHashWorker implements DoWorkUnit\u003cFile, string\u003e {\n  public workUnit(input: File): Observable\u003cstring\u003e {\n    return this.readFileAsArrayBuffer(input).pipe(map(arrayBuffer =\u003e md5(arrayBuffer)));\n  }\n\n  private readFileAsArrayBuffer(blob: Blob): Observable\u003cArrayBuffer\u003e {\n    return new Observable(observer =\u003e {\n      if (!(blob instanceof Blob)) {\n        observer.error(new Error('`blob` must be an instance of File or Blob.'));\n        return;\n      }\n\n      const reader = new FileReader();\n\n      reader.onerror = err =\u003e observer.error(err);\n      reader.onload = () =\u003e observer.next(reader.result as ArrayBuffer);\n      reader.onloadend = () =\u003e observer.complete();\n\n      reader.readAsArrayBuffer(blob);\n\n      return () =\u003e reader.abort();\n    });\n  }\n}\n\nrunWorker(WorkerPoolHashWorker);\n\n```\n\nNote here that the worker class `implements DoWorkUnit\u003cFile, string\u003e`. This is different to before where we implemented\n`DoWork` which had the slightly more complex signature of inputting an observable and outputting one.\n\nIf using the `fromWorkerPool` strategy, you must only implement `DoWorkUnit` as it relies on the completion of the \nreturned observable to indicate that the unit of work is finished processing, and the next unit of work can be \ntransferred to the worker.\n\nCommonly, a worker that implements `DoWorkUnit` only needs to return a single value, so you may instead return a `Promise`\nfrom the `workUnit` method.\n\n```ts\n// src/app/doc/async-work.worker.ts#L7-L14\n\nexport class FactorizationWorker implements DoWorkUnit\u003cnumber, number[]\u003e {\n  public async workUnit(input: number): Promise\u003cnumber[]\u003e {\n    return factorize(input);\n  }\n}\n\nrunWorker(FactorizationWorker);\n\n```\n\n---\n\nAre you using observable-webworker? [Tell us about it!](https://github.com/cloudnc/observable-webworker/discussions/69) We'd love to hear about the weird and wonderful ways developers are working with streaming workers.\n\n","funding_links":[],"categories":["Underlying Technologies"],"sub_categories":["RxJS"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcloudnc%2Fobservable-webworker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcloudnc%2Fobservable-webworker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcloudnc%2Fobservable-webworker/lists"}