{"id":15032328,"url":"https://github.com/robertknight/rten","last_synced_at":"2025-04-08T10:17:11.881Z","repository":{"id":214899563,"uuid":"531252554","full_name":"robertknight/rten","owner":"robertknight","description":"ONNX neural network inference engine","archived":false,"fork":false,"pushed_at":"2024-12-28T10:00:31.000Z","size":9121,"stargazers_count":133,"open_issues_count":35,"forks_count":9,"subscribers_count":5,"default_branch":"main","last_synced_at":"2024-12-28T11:15:12.727Z","etag":null,"topics":["machine-learning","onnx","rust"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/robertknight.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-08-31T20:43:45.000Z","updated_at":"2024-12-28T10:00:32.000Z","dependencies_parsed_at":"2024-02-28T20:28:06.345Z","dependency_job_id":"7651b441-e715-4407-84bc-b1db79a9c783","html_url":"https://github.com/robertknight/rten","commit_stats":null,"previous_names":["robertknight/rten"],"tags_count":103,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/robertknight%2Frten","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/robertknight%2Frten/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/robertknight%2Frten/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/robertknight%2Frten/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/robertknight","download_url":"https://codeload.github.com/robertknight/rten/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247819940,"owners_count":21001394,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["machine-learning","onnx","rust"],"created_at":"2024-09-24T20:18:03.007Z","updated_at":"2025-04-08T10:17:11.848Z","avatar_url":"https://github.com/robertknight.png","language":"Rust","funding_links":[],"categories":["machine-learning","Model Inference"],"sub_categories":[],"readme":"# RTen\n[![Latest Version]][crates.io] [![Documentation]][docs.rs]\n\n[Latest Version]: https://img.shields.io/crates/v/rten.svg\n[Documentation]: https://img.shields.io/docsrs/rten\n[docs.rs]: https://docs.rs/rten\n[crates.io]: https://crates.io/crates/rten\n\nRTen (the _Rust Tensor engine_) † is a runtime for machine learning models\nconverted from [ONNX](https://onnx.ai) format, which you can export from\nPyTorch and other frameworks.\n\nThe project also provides supporting libraries for common pre-processing and\npost-processing tasks in various domains. This makes RTen a more complete\ntoolkit for running models in Rust applications.\n\n† _The name is also a reference to PyTorch's ATen library._\n\n## Goals\n\n- Provide a (relatively) small and efficient neural network runtime that makes\n  it easy to take models created in frameworks such as PyTorch and run them in\n  Rust applications.\n- Be easy to compile and run on a variety of platforms, including WebAssembly\n- End-to-end Rust. This project and all of its required dependencies are\n  written in Rust.\n\n## Limitations\n\nThis project has a number of limitations to be aware of. Addressing them is\nplanned for the future:\n\n- Supports CPU inference only. There is currently no support for running models\n  on GPUs or other accelerators.\n- Not all ONNX operators are supported. See `OperatorType` in\n  [src/schema.fbs](src/schema.fbs) and [this issue](https://github.com/robertknight/rten/issues/14) for currently supported operators. For\n  implemented operators, some attributes or input shapes may not be supported.\n- Not all ONNX data types are supported. Currently supported data types for\n  tensors are: float32, int32, int64 (converted to int32), bool (converted to\n  int32), int8, uint8.\n- RTen is not as well optimized as more mature runtimes such as ONNX Runtime\n  or TensorFlow Lite. The performance difference depends on the operators used,\n  model structure, CPU architecture and platform.\n\n## Getting started\n\nThe best way to get started is to clone this repository and try running some of\nthe examples locally. The conversion scripts use popular Python machine learning\nlibraries, so you will need Python \u003e= 3.10 installed.\n\nThe examples are located in the [rten-examples/](rten-examples/) directory.\nSee the [README](rten-examples/) for descriptions of all the examples and steps\nto run them. As a quick-start, here are the steps to run the image\nclassification example:\n\n```sh\ngit clone https://github.com/robertknight/rten.git\ncd rten\n\n# Install model conversion tool\npip install -e rten-convert\n\n# Install dependencies for Python scripts\npip install -r tools/requirements.txt\n\n# Export an ONNX model. We're using resnet-50, a classic image classification model.\npython -m tools.export-timm-model timm/resnet50.a1_in1k\n\n# Convert model to this library's format\nrten-convert resnet50.a1_in1k.onnx resnet50.rten\n\n# Run image classification example. Replace `image.png` with your own image.\ncargo run -p rten-examples --release --bin imagenet mobilenet resnet50.rten image.png\n```\n\n## Converting ONNX models\n\nRTen does not load ONNX models directly. ONNX models must be run through a\nconversion tool which produces an optimized model in a\n[FlatBuffers](https://google.github.io/flatbuffers/)-based format (`.rten`) that\nthe engine can load. This is conceptually similar to the `.tflite` and `.ort`\nformats that TensorFlow Lite and ONNX Runtime use.\n\nThe conversion tool requires Python \u003e= 3.10. To convert an existing ONNX model,\nrun:\n\n```sh\npip install rten-convert\nrten-convert your-model.onnx\n```\n\nSee the [rten-convert README](rten-convert/) for more information about usage\nand version compatibility.\n\n## Usage in JavaScript\n\nTo use this library in a JavaScript application, there are two approaches:\n\n1. Prepare model inputs in JavaScript and use the rten library's built-in\n   WebAssembly API to run the model and return a tensor which will then need\n   to be post-processed in JS. This approach may be easiest for tasks where\n   the pre-processing is simple.\n\n   The [image classification](js-examples/image-classification/) example uses\n   this approach.\n\n2. Create a Rust library that uses rten and does pre-processing of inputs and\n   post-processing of outputs on the Rust side, exposing a domain-specific\n   WebAssembly API. This approach is more suitable if you have complex and/or\n   computationally intensive pre/post-processing to do.\n\nBefore running the examples, you will need to follow the steps under [\"Building\nthe WebAssembly library\"](#building-the-webassembly-library) below.\n\nThe general steps for using RTen's built-in WebAssembly API to run models in\na JavaScript project are:\n\n1.  Develop a model or find a pre-trained one that you want to run. Pre-trained\n    models in ONNX format can be obtained from the [ONNX Model Zoo](https://github.com/onnx/models)\n    or [Hugging Face](https://huggingface.co/docs/transformers/serialization).\n2.  If the model is not already in ONNX format, convert it to ONNX. PyTorch\n    users can use [torch.onnx](https://pytorch.org/docs/stable/onnx.html) for this.\n3.  Use the `rten-convert` package in this repository to convert the model\n    to the optimized format RTen uses. See the section above on converting models.\n4.  In your JavaScript code, fetch the WebAssembly binary and initialize RTen\n    using the `init` function.\n5.  Fetch the prepared `.rten` model and use it to an instantiate the `Model`\n    class from this library.\n6.  Each time you want to run the model, prepare one or more `Float32Array`s\n    containing input data in the format expected by the model, and call\n    `Model.run`. This will return a `TensorList` that provides access to the\n    shapes and data of the outputs.\n\nAfter building the library, API documentation for the `Model` and `TensorList`\nclasses is available in `dist/rten.d.ts`.\n\n## Building the WebAssembly library\n\n### Prerequisites\n\nTo build RTen for WebAssembly you will need:\n\n- A recent stable version of Rust\n- `make`\n- (Optional) The `wasm-opt` tool from [Binaryen](https://github.com/WebAssembly/binaryen)\n  can be used to optimize `.wasm` binaries for improved performance\n- (Optional) A recent version of Node for running demos\n\n### Building rten\n\n```sh\ngit clone https://github.com/robertknight/rten.git\ncd rten\nmake wasm\n```\n\nThe build created by `make wasm` requires support for WebAssembly SIMD,\navailable since Chrome 91, Firefox 89 and Safari 16.4. It is possible to\nbuild the library without WebAssembly SIMD support using `make wasm-nosimd`,\nor both using `make wasm-all`. The non-SIMD builds are significantly slower.\n\nAt runtime, you can find out which build is supported by calling the\n`binaryName()` function exported by this package.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frobertknight%2Frten","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frobertknight%2Frten","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frobertknight%2Frten/lists"}