{"id":16103014,"url":"https://github.com/harshkhandeparkar/gpujs-hive-compute","last_synced_at":"2025-03-18T08:31:04.203Z","repository":{"id":42744236,"uuid":"281380319","full_name":"harshkhandeparkar/gpujs-hive-compute","owner":"harshkhandeparkar","description":"[PoC] Use multiple computers on a network to run a single GPU.js kernel","archived":false,"fork":false,"pushed_at":"2023-03-05T07:03:26.000Z","size":417,"stargazers_count":13,"open_issues_count":3,"forks_count":0,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-02-28T08:18:19.672Z","etag":null,"topics":["gpu","gpujs","hive-computing","parallel-computing","typescript","typescript-library"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/harshkhandeparkar.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-07-21T11:35:18.000Z","updated_at":"2021-09-13T10:41:48.000Z","dependencies_parsed_at":"2024-10-27T17:26:41.276Z","dependency_job_id":"7547ea90-7cb4-448b-8bdf-0bf9d904735a","html_url":"https://github.com/harshkhandeparkar/gpujs-hive-compute","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/harshkhandeparkar%2Fgpujs-hive-compute","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/harshkhandeparkar%2Fgpujs-hive-compute/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/harshkhandeparkar%2Fgpujs-hive-compute/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/harshkhandeparkar%2Fgpujs-hive-compute/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/harshkhandeparkar","download_url":"https://codeload.github.com/harshkhandeparkar/gpujs-hive-compute/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243910814,"owners_count":20367545,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gpu","gpujs","hive-computing","parallel-computing","typescript","typescript-library"],"created_at":"2024-10-09T18:55:18.166Z","updated_at":"2025-03-18T08:31:03.909Z","avatar_url":"https://github.com/harshkhandeparkar.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"## GPUjs Hive Compute\nUse multiple computers on a network to run a single [GPU.js](https://github.com/gpujs/gpu.js) kernel!\n\nBenchmark Results:\n\n![bench](https://files.gitter.im/5cb7663bd73408ce4fbe0054/IUTE/image.png)\n\n### Table of Contents\n- [Installation And Usage](#installation-and-usage)\n- [API](#api)\n- [Caveats](#caveats)\n- [License](LICENSE)\n\n### Installation And Usage\n#### In NodeJS\nThe package is available on npm and can be installed using `npm`/`yarn`.\n```\nnpm i gpujs-hive-compute\n```\nOR\n```\nyarn add gpujs-hive-compute\n```\n\n#### In the browser\nSee [browser-hive-compute](https://github.com/HarshKhandeparkar/browser-hive-compute).\n\n#### CLI\nThere is no default CLI for this because building one is really easy. See `examples/squares.js` and `examples/helper-cli.js`.\nYou can clone the repository, run `yarn install` and `yarn build` and run `node examples/helper-cli.js` to use a simple CLI for Helper. You can use `examples/squares.js` as a template for real CLI usage of the library or use it for testing.\n\n#### Using as a Library\n**NOTE:** This library uses Websockets for communication because they are standard, browser-compatible and easy to use.\n\nThe library has two core components, the **Helper** and the **Leader**. The *Leader* is the main device which you control and which asks the other connected devices i.e *Helper*s to build and run parts of the kernel. The Leader side code is just like writing any GPU.js kernel, the library handles all the splitting of work between devices. The Helper and Leader can communicate as long as they are on the same local network. (or if the leader's global ip and port are exposed and known)\n\nExample Leader: (This will work with typescript as well)\n```js\nconst { hiveRun } = require('gpujs-hive-compute');\nconst GPU = require('gpu.js'); // This is required to be installed separately\n\nconst gpu = new GPU(); // Instantiate\n\nhiveRun({\n  gpu: gpu, // give the GPU object\n  func: function(arg1, arg2) {\n    return arg1 + arg2; // A normal GPU.js kernel function\n  },\n  options: {\n    output: [20] // Standard GPU.js kernel settings/options\n  },\n  onWaitingForHelpers: url =\u003e console.log(url),\n  doContinueOnHelperJoin: (numHelpers) =\u003e { // This callback is fired whenever a new helper joins. Return true\n    return numHelpers \u003e 3; // If more than 3 helpers join, it will run the kernel and during this time, no new helper can join.\n  },\n  inputs: [ // Inputs for the kernel, leave blank if there are no inputs.\n    5, // arg1\n    6 // arg2\n  ],\n}).then(output =\u003e console.log(output)).catch(e =\u003e console.log(e)); // Or use async...await\n```\nSee `examples/squares.js`.\n\nExample Helper: (This will work with typescript as well)\n```js\nconst { hiveHelp } = require('gpujs-hive-compute');\nconst GPU = require('gpu.js'); // This is required to be installed separately\n\nconst gpu = new GPU(); // Instantiate\n\nhiveHelp({\n  gpu: gpu,\n  url: `ws://192.168.0.10:8782` // This URL will be logged to the console by the Leader and will differ from device to device.\n}).then(() =\u003e console.log('successfully converted')).catch(e =\u003e console.log(e)); // Or use async...await\n```\n\n### API\nThe library exports the following functions:\n\n#### `hiveRun(options) =\u003e Promise()`\nWhere options is an object with the following properties:\n1. `gpu` (GPU): Instance of a GPU.js [`GPU`](https://github.com/gpujs/gpu.js#gpu-settings) object.\n2. `func` (Function): The GPU.js [kernel](https://github.com/gpujs/gpu.js#creating-and-running-functions) function.\n3. `port` (number): The port for the websocket server. (`8782` by default)\n4. `kernelOptions` (Object): GPU.js [kernel settings/options](https://github.com/gpujs/gpu.js#gpucreatekernel-settings).\n5. `onWaitingForHelpers(url) =\u003e void` (Function): A callback that is fired when a the hive is accepting helpers, the only parameter is the join url.\n6. `doContinueOnHelperJoin(numHelpers) =\u003e boolean` (Function): This is a callback function that is fired whenever a new helper joins. The parameter `numHelpers` is the number of helpers currently active. Return `true` to run the kernel or `false` to wait for more helpers to join. No new helper can join while the kernel is running.\n7. `logFunction(...args) =\u003e void` (Function): A custom log function if you don't want console logs.  (`console.log` by default)\n8. `inputs` (Array): This is an array of [kernel inputs](https://github.com/gpujs/gpu.js#accepting-input) in the form `[arg1, arg2, arg3]`.\n\nReturns a promise with the [output](https://github.com/gpujs/gpu.js#creating-and-running-functions) or an error.\n\n#### `hiveHelp(options) =\u003e Promise()`\nWhere options is an object with the following properties:\n1. `gpu` (GPU): Instance of a GPU.js [`GPU`](https://github.com/gpujs/gpu.js#gpu-settings) object.\n2. `url` (string): The [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) URL used by the Leader and Helper to communicate. The URL will be logged to the console by the leader. e.g: `ws://192.168.0.10:8782`.\n3. `logFunction(...args) =\u003e void` (Function): A custom log function if you don't want console logs.  (`console.log` by default)\n\nReturns a promise which either rejects with an error or resolves when the whole process is complete.\n\n### Caveats\n- **3-D kernel outputs**: Will be supported soon\n- **Graphical Output**: There is no straightforward way of doing this. (Basically impossible)\n- [**Pipelining**](https://github.com/gpujs/gpu.js#pipelining): The task is distributed among multiple GPUs so there is no single texture that can be pipelined.\n- **Not All Kernel Constants are available**: Kernel constants are supported but the following names are reserved by the library: `hive_offset_x`, `hive_offset_y`, `hive_offset_z`, `hive_output_x`, `hive_output_y` and `hive_output_z`.\n- **Slightly network intensive**: The data between helpers and leaders is sent as JSON. According to a test, with a single helper, the leader and helper both received or transmitted a total of 50MB for a 1000\\*1000 matrix multiplication. This can be slow over wifi and will be much slower for larger input sizes which are quite common. At least 100Mbit/s ethernet is recommended. (Or 5GHz wifi)\n\n\n****\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fharshkhandeparkar%2Fgpujs-hive-compute","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fharshkhandeparkar%2Fgpujs-hive-compute","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fharshkhandeparkar%2Fgpujs-hive-compute/lists"}