{"id":13479978,"url":"https://github.com/facebookresearch/shumai","last_synced_at":"2025-05-15T03:07:30.942Z","repository":{"id":58191669,"uuid":"525964144","full_name":"facebookresearch/shumai","owner":"facebookresearch","description":"Fast Differentiable Tensor Library in JavaScript and TypeScript with Bun + Flashlight","archived":false,"fork":false,"pushed_at":"2024-07-23T19:43:21.000Z","size":1306,"stargazers_count":1157,"open_issues_count":18,"forks_count":27,"subscribers_count":100,"default_branch":"main","last_synced_at":"2025-05-12T01:37:47.657Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://facebookresearch.github.io/shumai","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/facebookresearch.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-08-17T21:49:58.000Z","updated_at":"2025-05-04T02:07:02.000Z","dependencies_parsed_at":"2024-01-13T19:19:57.473Z","dependency_job_id":"7eb4fbcd-7a24-4042-bbb5-3c294f70fe59","html_url":"https://github.com/facebookresearch/shumai","commit_stats":{"total_commits":303,"total_committers":11,"mean_commits":"27.545454545454547","dds":"0.43234323432343236","last_synced_commit":"8d77737fe3de18863633210f0398da7eac36ead9"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/facebookresearch%2Fshumai","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/facebookresearch%2Fshumai/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/facebookresearch%2Fshumai/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/facebookresearch%2Fshumai/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/facebookresearch","download_url":"https://codeload.github.com/facebookresearch/shumai/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254264769,"owners_count":22041794,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T17:00:29.745Z","updated_at":"2025-05-15T03:07:25.928Z","avatar_url":"https://github.com/facebookresearch.png","language":"TypeScript","funding_links":[],"categories":["3. \u003ca name='Dev'\u003e\u003c/a\u003e💻 Dev","TypeScript","Table of Contents","Frameworks","Extensions"],"sub_categories":["AI - Frameworks and Toolkits","Frameworks"],"readme":"\u003cimg width=\"1276\" alt=\"Shumai\" src=\"https://user-images.githubusercontent.com/4842908/204894855-a1e697af-ea42-4f1f-9c2e-07a5da26f582.png\"\u003e\n\n---\n\nA [fast](#benchmarks), [network-connected](https://facebookresearch.github.io/shumai/modules/network.html), differentiable tensor library for TypeScript (and JavaScript).  Built with [bun](https://bun.sh) + [flashlight](https://github.com/flashlight/flashlight) for software engineers and researchers alike.\n\n![shumai_big](https://user-images.githubusercontent.com/4842908/190880994-6e59dd90-22ef-4a6a-9129-5be04086b59a.gif)\n\n\n\n⚠️ *This is experimental software!* ⚠️\n\n[![docs](https://img.shields.io/badge/docs-available-blue)](https://facebookresearch.github.io/shumai/)\n[![build](https://github.com/facebookresearch/shumai/actions/workflows/build.yml/badge.svg)](https://github.com/facebookresearch/shumai/actions/workflows/build.yml)\n[![tests](https://circleci.com/gh/facebookresearch/shumai.svg?style=shield)](https://app.circleci.com/pipelines/github/facebookresearch/shumai)\n[![npm](https://img.shields.io/npm/v/@shumai/shumai/latest)](https://www.npmjs.com/settings/shumai/packages)\n[![Discord](https://img.shields.io/discord/1013580889940295730)](https://discord.com/channels/1013580889940295730/)\n![GitHub commit activity](https://img.shields.io/github/commit-activity/w/facebookresearch/shumai)\n[![GitHub](https://img.shields.io/github/license/facebookresearch/shumai)](https://github.com/facebookresearch/shumai/blob/main/LICENSE)\n\n---\n\n\n- [Usage](#usage)\n- [Install](#install)\n- [Build from source](#building-native-libraries-from-source)\n- [Benchmarks](#benchmarks)\n- [Contributing](#contributing)\n\n## Quickstart\nInstall [Bun](https://bun.sh/) and [ArrayFire](https://github.com/arrayfire/arrayfire/wiki/Getting-ArrayFire)\n\n\u003cdetails\u003e\u003csummary\u003e\u003cstrong\u003eFor MacOS users:\u003c/strong\u003e\u003c/summary\u003e\n  \nYou can use [Homebrew](https://brew.sh) to install ArrayFire:\n  \n```bash\ncurl https://bun.sh/install | bash\nbrew install arrayfire\n```\n  \n\u003c/details\u003e\n\n\u003cdetails\u003e\u003csummary\u003e\u003cstrong\u003eFor Linux users:\u003c/strong\u003e\u003c/summary\u003e\n  \nIf you're running Ubuntu with **x86-64**, you can use the official distribution:\n  \n```bash\ncurl https://bun.sh/install | bash\nsudo apt install -y gnupg2 ca-certificates\nsudo apt-key adv --fetch-key https://repo.arrayfire.com/GPG-PUB-KEY-ARRAYFIRE-2020.PUB\necho \"deb https://repo.arrayfire.com/debian all main\" | sudo tee /etc/apt/sources.list.d/arrayfire.list\nsudo apt update\nsudo apt install -y arrayfire-cpu3-dev arrayfire-cpu3-openblas\n```\n  \nIf you're running Ubuntu with **ARMv8**, you'll need to build from source:\n\n```bash\ncurl https://bun.sh/install | bash\nsudo apt remove libarrayfire-dev libarrayfire-cpu3 libarrayfire-cpu-dev\nsudo apt install -y libblas-dev liblapack-dev liblapacke-dev libfftw3-dev libboost-all-dev cmake make g++\ncd /tmp\nsudo rm -rf arrayfire\ngit clone https://github.com/arrayfire/arrayfire.git\ncd arrayfire\ncmake -Bbuild -DAF_BUILD_EXAMPLES=OFF -DCMAKE_BUILD_TYPE=Release -DAF_BUILD_UNIFIED=OFF -DAF_TEST_WITH_MTX_FILES=OFF -DBUILD_TESTING=OFF\nmake -j4 -Cbuild\nsudo make install -Cbuild\n```\n  \nOtherwise, see the official [ArrayFire installation guide.](https://github.com/arrayfire/arrayfire/wiki/Getting-ArrayFire)\n\u003c/details\u003e\n\nthen run:\n```\nbun install @shumai/shumai\n```\nOnly macOS and Linux are supported. Linux installs default to GPU computation with CUDA, and macOS to CPU. Detailed install instructions [below](#install).\n\n*Install is work in progress*: [**please file an issue**](https://github.com/facebookresearch/shumai/issues) if you run into problems.\n\n\n## Usage\n\nshumai will always attempt to use an attached GPU or accelerator; although CPU computation will use the ArrayFire CPU backend, which is not well-optimized.\n\nWe hope to support the ArrayFire OpenCL backend and other non-ArrayFire tensor backends soon.\n\nIf shumai seems unusually slow, please file an issue!\n\n**Standard array utilities:**\n\n\n```javascript\nimport * as sm from \"@shumai/shumai\"\n\n// create a 1024 by 1024 tensor, randomly filled with normal distribution\nlet X = sm.randn([1024, 1024])\nlet W = sm.identity(1024)\nlet Y = X.matmul(W)\nconsole.log(Y.shape)\n```\n\n**Conversion to and from JavaScript native arrays:**\n\n```javascript\nconst data : Float32Array = new Float32Array(128)\nfor (let i = 0; i \u003c 128; ++i) {\n  data[i] = Math.random()\n}\n\nconst X : Tensor = sm.tensor(data)\nconst pi = sm.scalar(3.14)\nconst Y = X.mul(pi)\n\n// tensors can be converted back to native JavaScript\nconst Y_data = Y.toFloat32Array()\n\n// scalar tensors can be converted to JavaScript numbers\nconst total : number = X.sum().toFloat32()\n```\n\n**Gradients:**\n\n```javascript\nconst W = sm.randn([128, 128])\nW.requires_grad = true\n\nconst X = sm.randn([128, 128])\nconst diff = X.sub(W)\nconst mse = diff.mul(diff).sum()\nmse.backward()\n\nW.grad // this gradient is now populated\n\n// copy W without allowing gradient updates\nconst Y = W.detach()\nY.sum().backward() // nothing changes\n\n```\n\nSome more examples can be found [here](https://github.com/facebookresearch/shumai/tree/main/examples).\n\nSupported operators can be found [here](#supported-operations).\n\n## Install\n\n**The install procedure is a work in progress!**\nIf you have any problems building or installing, we would\ngreatly appreciate filed issues. Please tell us about your platform/OS when you do.\n\n**Prerequisites**:\n- Ensure you have bun installed (https://bun.sh).\n- Install [ArrayFire](https://github.com/arrayfire/arrayfire). *macOS users should install ArrayFire's CPU backend; Linux users should install the CUDA backend^.*\n  - **macOS** --- ArrayFire can easily be installed with Homebrew:\n  ```\n  brew install arrayfire\n  ```\n- **Linux** --- instructions [can be found here](https://github.com/arrayfire/arrayfire/wiki/Getting-ArrayFire). On Ubuntu, ArrayFire can be installed via [package managers (e.g. `apt`)](https://github.com/arrayfire/arrayfire/wiki/Install-ArrayFire-From-Linux-Package-Managers).\n\nOnce `bun` and `ArrayFire` are installed, install the package and backing libs with `bun`:\n```shell\nbun install @shumai/shumai\n```\n\n### Windows Support\n\nWhile not officially supported, Windows users have been successful leveraging [Docker](https://www.docker.com/) + [WSL2](https://learn.microsoft.com/en-us/windows/wsl/install) + Linux. Including CUDA support.\n\n\n## Building Native Libraries from Source\n\n**Note:** *not required when developing TypeScript/Javascript library components locally.*\n\nFrom source build instructions for:\n- [macOS](#building-on-macos-from-source)\n- [Linux](#building-on-linux-from-source)\n\nThis process will build the dependent ffi libraries (`libflashlight` and `libflashlight_binding`) and pack them using `npm pack` to generate a `@shumai/shumai_*.tgz`\npackage. You can then use `npm install $PATH_TO_SOURCE/@shumai/shumai-*.tgz` to install the package where you'd like.\n\n### Building on macOS from Source\n\nFirst, install ArrayFire CPU with `brew install arrayfire`.\n\nBuild and install [Flashlight](https://github.com/flashlight/flashlight#building-and-installing):\n```bash\nmkdir -p $HOME/usr/ # installing flashlight here\ngit clone --recursive --depth 1 https://github.com/flashlight/flashlight.git\ncd flashlight\nmkdir -p build\ncd build\ncmake .. \\\n  -DCMAKE_BUILD_TYPE=Release \\\n  -DBUILD_SHARED_LIBS=ON  \\\n  -DCMAKE_INSTALL_PREFIX=$HOME/usr \\\n  -DFL_USE_ARRAYFIRE=ON \\\n  -DFL_ARRAYFIRE_USE_CPU=ON \\\n  -DFL_USE_ONEDNN=OFF \\\n  -DFL_BUILD_DISTRIBUTED=OFF \\\n  -DFL_BUILD_TESTS=OFF \\\n  -DFL_BUILD_EXAMPLES=OFF\nmake -j$(nproc)\nmake install\n```\n\nBuild Flashlight bindings for Shumai:\n```bash\ncd shumai\nmkdir -p build\ncd build\ncmake .. -Dflashlight_DIR=$HOME/usr/share/flashlight/cmake/\nmake -j$(nproc)\n```\n\n#### Profiling\n\nOn macOS, you can record perf with `xcrun xctrace record --template \"Time Profiler\" --launch $(which bun) train.js`.\n\n### Building on Linux from Source\n\nFirst [install ArrayFire](https://github.com/arrayfire/arrayfire/wiki/Getting-ArrayFire). The Linux build for shumai uses the CUDA backend, but from source, you can build the CPU backend as well (OpenCL support coming soon).\n\nBuild and install [Flashlight](https://github.com/flashlight/flashlight#building-and-installing):\n```bash\nmkdir -p $HOME/usr/ # installing flashlight here\ngit clone --recursive --depth 1 https://github.com/flashlight/flashlight.git\ncd flashlight\nmkdir -p build\ncd build\ncmake .. \\\n  -DCMAKE_BUILD_TYPE=RelWithDebInfo \\ # or as specified\n  -DFL_ARRAYFIRE_USE_CPU=OFF \\\n  \\ # swap with the above to build for CPU\n  -DFL_ARRAYFIRE_USE_CUDA=ON \\ \n  -DFL_BUILD_DISTRIBUTED=OFF \\\n  -DFL_USE_ONEDNN=OFF \\\n  -DFL_BUILD_TESTS=OFF \\\n  -DFL_BUILD_EXAMPLES=OFF \\\n  -DFL_BUILD_SCRIPTS=OFF \\\n  -DCMAKE_INSTALL_PREFIX=$HOME/usr/\nmake -j$(nproc)\nmake install\n```\n\nBuild bindings for shumai:\n```bash\nmkdir -p build \u0026\u0026 cd build\ncmake .. \\\n    -DBUILD_SHARED_LIBS=ON \\\n    -DCMAKE_BUILD_TYPE=RelWithDebInfo \\ # or as specified\n    -Dflashlight_DIR=${FLASHLIGHT_INSTALL_PREFIX}/share/flashlight/cmake \\\n    -DArrayFire_DIR=${ARRAYFIRE_INSTALL_PREFIX}/share/ArrayFire/cmake # if built from source, else not needed\nmake -j$(nproc)\n```\n\n\n\n## Why build this?\n\nWith Shumai, we hope to make \n\n- **Creating datasets easier**\n  - JavaScript, with native typed arrays and a JIT compiler, is perfect for twiddling with data before it can be made into big, flat GPU-compatible arrays.\n- **Training small models faster**\n  - FFI bindings in Bun are crazy fast (~3ns), so JS gets out of the way when training small models\n- **Advanced/fine-grained training/inference logic more expressive**\n  - Bun uses the JSC JIT compiler, meaning you can confidently write complex training logic without needing a native C++ implementation\n- **Building applications enoyable**\n  - JavaScript has a ~~large~~ [HUGE](https://survey.stackoverflow.co/2022/#section-most-popular-technologies-programming-scripting-and-markup-languages) ecosystem, which facilitates better application development\n  \n  \n## Benchmarks\n\nBenchmark data is collected from https://github.com/shumai-org/benchmarks\n\nOn an Apple M1 Pro:\n\n| Benchmark     | Shumai (bun)  | TF.js (node)  | Difference |\n| ------------- |---------------| --------------| -----------|\n| 32-wide addition | 624.78K iter/s | 195.627K iter/s | 3.19x |\n| 1024-wide addition | 460.008K iter/s | 94.945K iter/s | 4.84x |\n| 32768-wide addition | 57.929K iter/s | 40.484K iter/s | 1.43x |\n| 64-wide square matmul | 43 GFlop/s | 28.533 GFlop/s | 1.51x |\n| 128-wide square matmul | 518.704 GFlop/s | 58.764 GFlop/s | 8.83x |\n| 1024-wide square matmul | 2,147.771 GFlop/s | 318.826 GFlop/s | 6.74x |\n| B=64, 64-wide hidden layer + 5x pointwise |41.344K iter/s| 16.679K iter/s | 2.48x|\n| B=64, 128-wide hidden layer + 5x pointwise |24.554K iter/s| 8.563K iter/s | 2.87x|\n| B=64, 1024-wide hidden layer + 5x pointwise |2.716K iter/s| 0.969K iter/s | 2.80x|\n\nOn an Nvidia GP100:\n\n| Benchmark     | Shumai (bun)  | TF.js (node)  | Difference |\n| ------------- |---------------| --------------| -----------|\n| 32-wide addition | 243.217K iter/s | 34.539K iter/s | 7.04x |\n| 1024-wide addition | 144.771K iter/s | 18.006K iter/s | 8.04x |\n| 32768-wide addition | 71.793K iter/s | 17.071K iter/s | 4.21x |\n| 64-wide square matmul | 63.239 GFlop/s | 12.749 GFlop/s | 4.96x |\n| 128-wide square matmul | 435.565 GFlop/s | 104.885 GFlop/s | 4.15x |\n| 1024-wide square matmul | 7,165.062 GFlop/s | 6,470.793 GFlop/s | 1.11x |\n| B=64, 64-wide hidden layer + 5x pointwise |25.507K iter/s| 5.192K iter/s | 4.91x|\n| B=64, 128-wide hidden layer + 5x pointwise |22.529K iter/s| 4.861K iter/s | 4.63x|\n| B=64, 1024-wide hidden layer + 5x pointwise |11.568K iter/s| 2.854K iter/s | 4.05x|\n\n\n## Memory Usage\n\nWhile the out of the box memory management may suffice in many cases, tuning memory\nusage can greatly improve performance by reducing unnecessary overhead from the\nGarbage Collector.\n\n```\nimport { util } from '@shumai/shumai'\n\nutil.memoryOptions({\n  lowerBoundThreshold: 100e6, // 100MB\n  upperBoundThreshold: 5e9, // 5GB\n  delayBetweenGCs: 1000 // 1s\n})\n```\n\nPay special attention to `upperBoundThreshold` which if exceeded will force GC\nfor every allocated tensor, ignoring `delayBetweenGCs`. Supplying a value that\nwill fully utilize your hardware can greatly improve performance.\n\n\n## Statistics\n\n```mermaid\ngraph TD\n  OpA(Op A) --\u003e statsA{{\"stats A\"}};\n  OpB(Op B) --\u003e statsA;\n  statsA --\u003e LoggerA{{\"LoggerConsole A\"}};\n  LoggerA --\u003e Stdout((\"Stdout\"));\n  OpC(Op C) --\u003e statsA;\n  OpD(Op D) --\u003e statsA;\n  statsA --\u003e LoggerB(\"LoggerCustom B\");\n  LoggerB --\u003e Disk((\"Disk\"));\n```\n\nBasic usage of gathering statistics is as simple adding\na collector using the default `StatsLoggerConsole`.\n\n```\nimport { stats, StatsLoggerConsole, rand, matmul } from '@shumai/shumai'\n\nstats.enabled = true // all ops following will capture stats\n\n// perform ops...\n\nstats.enabled = false // all ops following will no longer capture stats\n```\n\nWhile the above examples may suffice for simple use cases, if you're\nlooking to capture stats across multiple threads, processes, and/or hosts,\n`StatsLoggerHttp` has you covered.\n\n```mermaid\ngraph TD\n  subgraph Host C\n    Processor(\"LoggerHttp Processor\")\n    style Processor stroke:#222,stroke-width:4px,stroke-dasharray:5 5\n  end\n  subgraph Host A\n    OpA(Op A) --\u003e statsA{{\"stats A\"}};\n    OpB(Op B) --\u003e statsA;\n    statsA --\u003e LoggerA{{\"LoggerHttp A\"}};\n    LoggerA --\u003e Processor;\n  end\n  subgraph Host B\n    OpC(Op C) --\u003e statsB{{\"stats B\"}};\n    OpD(Op D) --\u003e statsB;\n    statsB --\u003e LoggerB{{\"LoggerHttp B\"}};\n    LoggerB --\u003e Processor;\n  end\n```\n\n```\nimport { StatsLoggerHttp } from '@shumai/shumai'\n\nstats.logger = new StatsLoggerHttp({ url: 'http://localhost:4242' })\n```\n\nFor more custom needs you can supply your own logger:\n\n```\nimport { StatsLogger, StatsLoggerData } from '@shumai/shumai'\n\nclass CustomLogger implements StatsLogger {\n  async process(data: StatsLoggerData): Promise\u003cvoid\u003e {\n    const summary = data.collector.getSummary()\n    console.log('Collector stats:', summary)\n  }\n}\n\nstats.logger = new CustomLogger()\n```\n\nBy default stack tracing is disabled as it adds 50%+ overhead, but can be enabled via `stats.collectStacks = true`.\n\n### Scoped Statistics\n\nIf you wish to isolate stats profiling you can do this as well:\n\n```\nimport { collectStats } from '@shumai/shumai'\n\nconst scopedStats = collectStats(() =\u003e {\n  // perform ops...\n}/*, StatsCollectorOptions | StatsLogger */)\nconsole.log(scopedStats.getSummary())\n```\n\n\n## Contributing\n\nIf you'd like to make changes to the core bindings or ffi, first [build from source](#installing-from-source).\n\nAll files ending in `*.inl` or `*_gen.ts` are generated.\nThese can be modified by editing [`scripts/gen_binding.py`](https://github.com/facebookresearch/shumai/blob/main/scripts/gen_binding.py)\nand running [`./scripts/gen_all_binding.sh`](https://github.com/facebookresearch/shumai/blob/main/scripts/gen_all_binding.sh).\n\nSee the [CONTRIBUTING](CONTRIBUTING.md) file for style guidance and more info on how to help out. 😁\n\n### License\n\nshumai is MIT licensed, as found in the LICENSE file.\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffacebookresearch%2Fshumai","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffacebookresearch%2Fshumai","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffacebookresearch%2Fshumai/lists"}