{"id":16839038,"url":"https://github.com/fathyb/node-parallel","last_synced_at":"2025-06-11T22:07:46.313Z","repository":{"id":75403517,"uuid":"135368108","full_name":"fathyb/node-parallel","owner":"fathyb","description":"💪 Node.js multiprocess library on steroids (WIP)","archived":false,"fork":false,"pushed_at":"2018-07-03T19:39:08.000Z","size":56,"stargazers_count":8,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-06-10T23:57:25.231Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fathyb.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2018-05-30T00:47:10.000Z","updated_at":"2024-09-04T19:59:38.000Z","dependencies_parsed_at":"2023-06-06T09:45:40.662Z","dependency_job_id":null,"html_url":"https://github.com/fathyb/node-parallel","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/fathyb/node-parallel","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fathyb%2Fnode-parallel","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fathyb%2Fnode-parallel/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fathyb%2Fnode-parallel/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fathyb%2Fnode-parallel/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fathyb","download_url":"https://codeload.github.com/fathyb/node-parallel/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fathyb%2Fnode-parallel/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259352486,"owners_count":22844738,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-13T12:28:24.670Z","updated_at":"2025-06-11T22:07:46.307Z","avatar_url":"https://github.com/fathyb.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# `node-parallel`\n\nNode.js multiprocess library on steroids.\n\n\u003e WIP: super experimental\n\n## What's in the box\n\n- A ultra-fast IPC backed by native shared memory and v8 serializer\n- A keyed shared memory `Buffer` primitive\n- A key-value data store to store anything in shared memory, with atomic operations\n\n#### Freebies\n\n- Booting a bunch of workers is faster than any Node.js API\n- TypeScript friendly, and almost entirely written in C++ for low-overhead\n- Kernel backed Copy-On-Write: it only uses physical (and reported) memory when you write it\n- Everything is managed by the garbage collector and reference counted, it's also\nreference counted based on each process lifecycle\n\n#### Downsides\n\n- It needs node.js as a static library, I'm working on scripts to fetch\nand build node.js binaries for all platforms\n- It needs to be launched from our executable :\n    - `node-workers a` instead of `node a`\n    - this executable behaves exactly like `node`\n    - you **will** get a `segfault` if you use `node`\n    - this is because we need to modify the node.js process virtual address space before booting it,\n    this could be resolved by supporting `fork` inside node.js (which `libuv` does)\n- It's currently very niche, I wrote it for use-cases like Parcel\n- Windows not supported (soon, should be trivial to implement)\n- If one of the worker doesn't die gracefully (eg. segfault) we crash everything.\nThis is done by safety to prevent any memory corruption and potential deadlocks.\n    - This may be fixed in the future by using more lock-free structures\n\n## Usage\n\n```js\nimport {createIPC, isMaster, store} from 'multiprocess'\n\nconst fibonacci = n =\u003e\n    n \u003c 1\n        ? 0\n        : n \u003c= 2\n            ? 1\n            : fibonacci(n - 1) + fibonacci(n - 2)\n\n// Create a IPC controller\nconst ipc = createIPC({\n    fibonacci(n) {\n        console.log('Computing fib(%s) in process %s', n, process.pid)\n\n        return fibonacci(n)\n    }\n})\n\n// Here we call from master but even workers can call the IPC\nif(isMaster) {\n    Promise\n        .all(\n            [10, 15, 20, 25, 30].map(number =\u003e\n                ipc.fibonacci(number)\n            )\n        )\n        .then(result =\u003e console.log('Result', result))\n        .catch(err =\u003e console.error(err))\n}\n\n// Create a cross-process virtual fs\nconst readFile = name =\u003e {\n    console.log('Fetching %s')\n\n    // Returns the value if set or sets it atomically\n    return store.getOrSet(name, () =\u003e {\n        // The reading task is distributed across processes\n        // but only done one time per file\n        console.log('Reading %s on pid %s', name, process.pid)\n\n        return fs.readFile(name, 'utf-8')\n    })\n}\n\n\nreadFile('/tmp/foo').then(foo =\u003e console.log('Foo', foo))\nreadFile('/tmp/bar').then(bar =\u003e console.log('Bar', bar))\n```\n\n```bash\n$ node-workers index.js --workers=8\n\nComputing fib(30) in process 4685\nComputing fib(10) in process 4689\nComputing fib(25) in process 4690\nComputing fib(15) in process 4686\nComputing fib(20) in process 4688\nResult [ 55, 610, 6765, 75025, 832040 ]\n```\n\n## Developement\n\nThe C++ follows the Google C++ Style Convention, I know, I know,\nme neither. This makes the code blend more naturally with v8 and Node.js APIs.\n\n### Wheels\n\nThat needed to be reinvented, somehow :\n\n- Memory allocator\n- Shared pointer\n- SpinLocks\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffathyb%2Fnode-parallel","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffathyb%2Fnode-parallel","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffathyb%2Fnode-parallel/lists"}