{"id":13643245,"url":"https://github.com/nbigaouette/onnxruntime-rs","last_synced_at":"2025-04-21T01:32:08.226Z","repository":{"id":38058713,"uuid":"284863293","full_name":"nbigaouette/onnxruntime-rs","owner":"nbigaouette","description":"Rust wrapper for Microsoft's ONNX Runtime (version 1.8)","archived":false,"fork":false,"pushed_at":"2024-03-06T16:33:46.000Z","size":582,"stargazers_count":290,"open_issues_count":49,"forks_count":99,"subscribers_count":12,"default_branch":"master","last_synced_at":"2025-04-10T18:10:53.247Z","etag":null,"topics":["inference","onnx","onnx-runtime","onnxruntime","onnxruntime-sys","rust","rust-wrapper"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nbigaouette.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE-APACHE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-08-04T03:04:15.000Z","updated_at":"2025-04-09T06:58:30.000Z","dependencies_parsed_at":"2024-01-14T12:17:54.841Z","dependency_job_id":"b513bee8-aa85-418c-9ca6-c7e71e5f1db0","html_url":"https://github.com/nbigaouette/onnxruntime-rs","commit_stats":{"total_commits":403,"total_committers":9,"mean_commits":44.77777777777778,"dds":0.07444168734491319,"last_synced_commit":"31f62235e3048390fa1aba67fa6316ed90a628d7"},"previous_names":[],"tags_count":13,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nbigaouette%2Fonnxruntime-rs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nbigaouette%2Fonnxruntime-rs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nbigaouette%2Fonnxruntime-rs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nbigaouette%2Fonnxruntime-rs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nbigaouette","download_url":"https://codeload.github.com/nbigaouette/onnxruntime-rs/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249982579,"owners_count":21355724,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["inference","onnx","onnx-runtime","onnxruntime","onnxruntime-sys","rust","rust-wrapper"],"created_at":"2024-08-02T01:01:44.489Z","updated_at":"2025-04-21T01:32:07.616Z","avatar_url":"https://github.com/nbigaouette.png","language":"Rust","funding_links":[],"categories":["Lighter and Deployment Frameworks","Machine Learning","Model Inference"],"sub_categories":[],"readme":"# ONNX Runtime\n\n[\u003cimg alt=\"github\" src=\"https://img.shields.io/badge/github-nbigaouette/onnxruntime_rs-8da0cb?style=for-the-badge\u0026labelColor=555555\u0026logo=github\" height=\"20\"\u003e](https://github.com/nbigaouette/onnxruntime-rs)\n[\u003cimg alt=\"crates.io\" src=\"https://img.shields.io/crates/v/onnxruntime.svg?style=for-the-badge\u0026color=fc8d62\u0026logo=rust\" height=\"20\"\u003e](https://crates.io/crates/onnxruntime)\n[\u003cimg alt=\"docs.rs\" src=\"https://img.shields.io/badge/docs.rs-onnxruntime-66c2a5?style=for-the-badge\u0026labelColor=555555\u0026logoColor=white\u0026logo=data:image/svg+xml;base64,PHN2ZyByb2xlPSJpbWciIHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8yMDAwL3N2ZyIgdmlld0JveD0iMCAwIDUxMiA1MTIiPjxwYXRoIGZpbGw9IiNmNWY1ZjUiIGQ9Ik00ODguNiAyNTAuMkwzOTIgMjE0VjEwNS41YzAtMTUtOS4zLTI4LjQtMjMuNC0zMy43bC0xMDAtMzcuNWMtOC4xLTMuMS0xNy4xLTMuMS0yNS4zIDBsLTEwMCAzNy41Yy0xNC4xIDUuMy0yMy40IDE4LjctMjMuNCAzMy43VjIxNGwtOTYuNiAzNi4yQzkuMyAyNTUuNSAwIDI2OC45IDAgMjgzLjlWMzk0YzAgMTMuNiA3LjcgMjYuMSAxOS45IDMyLjJsMTAwIDUwYzEwLjEgNS4xIDIyLjEgNS4xIDMyLjIgMGwxMDMuOS01MiAxMDMuOSA1MmMxMC4xIDUuMSAyMi4xIDUuMSAzMi4yIDBsMTAwLTUwYzEyLjItNi4xIDE5LjktMTguNiAxOS45LTMyLjJWMjgzLjljMC0xNS05LjMtMjguNC0yMy40LTMzLjd6TTM1OCAyMTQuOGwtODUgMzEuOXYtNjguMmw4NS0zN3Y3My4zek0xNTQgMTA0LjFsMTAyLTM4LjIgMTAyIDM4LjJ2LjZsLTEwMiA0MS40LTEwMi00MS40di0uNnptODQgMjkxLjFsLTg1IDQyLjV2LTc5LjFsODUtMzguOHY3NS40em0wLTExMmwtMTAyIDQxLjQtMTAyLTQxLjR2LS42bDEwMi0zOC4yIDEwMiAzOC4ydi42em0yNDAgMTEybC04NSA0Mi41di03OS4xbDg1LTM4Ljh2NzUuNHptMC0xMTJsLTEwMiA0MS40LTEwMi00MS40di0uNmwxMDItMzguMiAxMDIgMzguMnYuNnoiPjwvcGF0aD48L3N2Zz4K\" height=\"20\"\u003e](https://docs.rs/onnxruntime)\n[\u003cimg alt=\"build status\" src=\"https://img.shields.io/github/workflow/status/nbigaouette/onnxruntime-rs/Rust/master?style=for-the-badge\" height=\"20\"\u003e](https://github.com/nbigaouette/onnxruntime-rs/actions?query=branch%3Amaster)\n[![codecov](https://codecov.io/gh/nbigaouette/onnxruntime-rs/branch/master/graph/badge.svg)](https://codecov.io/gh/nbigaouette/onnxruntime-rs)\n\nThis is an attempt at a Rust wrapper for\n[Microsoft's ONNX Runtime](https://github.com/microsoft/onnxruntime) (version 1.8).\n\nThis project consist on two crates:\n\n* [`onnxruntime-sys`](onnxruntime-sys): Low-level binding to the C API;\n* [`onnxruntime`](onnxruntime): High-level and safe API.\n\n[Changelog](CHANGELOG.md)\n\nThe `build.rs` script supports downloading pre-built versions of the Microsoft ONNX Runtime,\nwhich provides the following targets:\n\nCPU:\n\n* Linux x86_64\n* macOS x86_64\n* macOS aarch64 (no pre-built binaries, no CI testing, see [#74](https://github.com/nbigaouette/onnxruntime-rs/pull/74))\n* Windows i686\n* Windows x86_64\n\nGPU:\n\n* Linux x86_64\n* Windows x86_64\n\n---\n\n**WARNING**:\n\n* This is an experiment and work in progress; it is _not_ complete/working/safe. Help welcome!\n* Basic inference works, see [`onnxruntime/examples/sample.rs`](onnxruntime/examples/sample.rs) or [`onnxruntime/tests/integration_tests.rs`](onnxruntime/tests/integration_tests.rs)\n* ONNX Runtime has many options to control the inference process but those options are not yet exposed.\n* This was developed and tested on macOS Catalina. Other platforms should work but have not been tested.\n\n---\n\n## Setup\n\nThree different strategy to obtain the ONNX Runtime are supported by the `build.rs` script:\n\n1. Download a pre-built binary from upstream;\n2. Point to a local version already installed;\n3. Compile from source (not yet implemented).\n\nTo select which strategy to use, set the `ORT_STRATEGY` environment variable to:\n\n1. `download`: This is the default if `ORT_STRATEGY` is not set;\n2. `system`: To use a locally installed version (use `ORT_LIB_LOCATION` environment variable to point to the install path)\n3. `compile`: To compile the library\n\nThe `download` strategy supports downloading a version of ONNX that supports CUDA. To use this, set the\nenvironment variable `ORT_USE_CUDA=1` (only supports Linux or Windows).\n\nUntil the build script allow compilation of the runtime, see the [compilation notes](ONNX_Compilation_Notes.md)\nfor some details on the process.\n\n### Note on 'ORT_STRATEGY=system'\n\nWhen using `ORT_STRATEGY=system`, executing a built crate binary (for example the tests) might fail, at least on macOS,\nif the library is not installed in a system path. An error similar to the following happens:\n\n```text\ndyld: Library not loaded: @rpath/libonnxruntime.1.7.1.dylib\n  Referenced from: onnxruntime-rs.git/target/debug/deps/onnxruntime_sys-22eb0e3e89a0278c\n  Reason: image not found\n```\n\nTo fix, one can either:\n\n* Set the `LD_LIBRARY_PATH` environment variable to point to the path where the library can be found.\n* Adapt the `.cargo/config` file to contain a linker flag to provide the **full** path:\n  \n  ```toml\n  [target.aarch64-apple-darwin]\n  rustflags = [\"-C\", \"link-args=-Wl,-rpath,/full/path/to/onnxruntime/lib\"]\n  ```\n\nSee [rust-lang/cargo #5077](https://github.com/rust-lang/cargo/issues/5077) for more information.\n\n## Example\n\nThe C++ example that uses the C API\n([`C_Api_Sample.cpp`](https://github.com/microsoft/onnxruntime/blob/v1.3.1/csharp/test/Microsoft.ML.OnnxRuntime.EndToEndTests.Capi/C_Api_Sample.cpp))\nwas ported to both the low level crate (`onnxruntime-sys`) and the high level on (`onnxruntime`).\n\n### onnxruntime-sys\n\nTo run this example ([`onnxruntime-sys/examples/c_api_sample.rs`](onnxruntime-sys/examples/c_api_sample.rs)):\n\n```sh\n# Download the model (SqueezeNet 1.0, ONNX version: 1.3, Opset version: 8)\n❯ curl -LO \"https://github.com/onnx/models/raw/master/vision/classification/squeezenet/model/squeezenet1.0-8.onnx\"\n❯ cargo run --example c_api_sample\n[...]\n    Finished dev [unoptimized + debuginfo] target(s) in 1.88s\n     Running `target/debug/examples/c_api_sample`\nUsing Onnxruntime C API\n2020-08-09 09:37:41.554922 [I:onnxruntime:, inference_session.cc:174 ConstructorCommon] Creating and using per session threadpools since use_per_session_threads_ is true\n2020-08-09 09:37:41.556650 [I:onnxruntime:, inference_session.cc:830 Initialize] Initializing session.\n2020-08-09 09:37:41.556665 [I:onnxruntime:, inference_session.cc:848 Initialize] Adding default CPU execution provider.\n2020-08-09 09:37:41.556678 [I:onnxruntime:test, bfc_arena.cc:15 BFCArena] Creating BFCArena for Cpu\n2020-08-09 09:37:41.556687 [V:onnxruntime:test, bfc_arena.cc:32 BFCArena] Creating 21 bins of max chunk size 256 to 268435456\n2020-08-09 09:37:41.558313 [I:onnxruntime:, reshape_fusion.cc:37 ApplyImpl] Total fused reshape node count: 0\n2020-08-09 09:37:41.559327 [I:onnxruntime:, reshape_fusion.cc:37 ApplyImpl] Total fused reshape node count: 0\n2020-08-09 09:37:41.559476 [I:onnxruntime:, reshape_fusion.cc:37 ApplyImpl] Total fused reshape node count: 0\n2020-08-09 09:37:41.559607 [V:onnxruntime:, inference_session.cc:671 TransformGraph] Node placements\n2020-08-09 09:37:41.559615 [V:onnxruntime:, inference_session.cc:673 TransformGraph] All nodes have been placed on [CPUExecutionProvider].\n2020-08-09 09:37:41.559639 [I:onnxruntime:, session_state.cc:25 SetGraph] SaveMLValueNameIndexMapping\n2020-08-09 09:37:41.559787 [I:onnxruntime:, session_state.cc:70 SetGraph] Done saving OrtValue mappings.\n2020-08-09 09:37:41.560252 [I:onnxruntime:, session_state_initializer.cc:178 SaveInitializedTensors] Saving initialized tensors.\n2020-08-09 09:37:41.563467 [I:onnxruntime:, session_state_initializer.cc:223 SaveInitializedTensors] Done saving initialized tensors\n2020-08-09 09:37:41.563979 [I:onnxruntime:, inference_session.cc:919 Initialize] Session successfully initialized.\nNumber of inputs = 1\nInput 0 : name=data_0\nInput 0 : type=1\nInput 0 : num_dims=4\nInput 0 : dim 0=1\nInput 0 : dim 1=3\nInput 0 : dim 2=224\nInput 0 : dim 3=224\n2020-08-09 09:37:41.573127 [I:onnxruntime:, sequential_executor.cc:145 Execute] Begin execution\n2020-08-09 09:37:41.573183 [I:onnxruntime:test, bfc_arena.cc:259 AllocateRawInternal] Extending BFCArena for Cpu. bin_num:13 rounded_bytes:3154176\n2020-08-09 09:37:41.573197 [I:onnxruntime:test, bfc_arena.cc:143 Extend] Extended allocation by 4194304 bytes.\n2020-08-09 09:37:41.573203 [I:onnxruntime:test, bfc_arena.cc:147 Extend] Total allocated bytes: 9137152\n2020-08-09 09:37:41.573212 [I:onnxruntime:test, bfc_arena.cc:150 Extend] Allocated memory at 0x7fb7d6cb7000 to 0x7fb7d70b7000\n2020-08-09 09:37:41.573248 [I:onnxruntime:test, bfc_arena.cc:259 AllocateRawInternal] Extending BFCArena for Cpu. bin_num:8 rounded_bytes:65536\n2020-08-09 09:37:41.573256 [I:onnxruntime:test, bfc_arena.cc:143 Extend] Extended allocation by 4194304 bytes.\n2020-08-09 09:37:41.573262 [I:onnxruntime:test, bfc_arena.cc:147 Extend] Total allocated bytes: 13331456\n2020-08-09 09:37:41.573268 [I:onnxruntime:test, bfc_arena.cc:150 Extend] Allocated memory at 0x7fb7d70b7000 to 0x7fb7d74b7000\nScore for class [0] =  0.000045440644\nScore for class [1] =  0.0038458651\nScore for class [2] =  0.00012494653\nScore for class [3] =  0.0011804523\nScore for class [4] =  0.0013169361\nDone!\n```\n\n### onnxruntime\n\nTo run this example ([`onnxruntime/examples/sample.rs`](onnxruntime/examples/sample.rs)):\n\n```sh\n# Download the model (SqueezeNet 1.0, ONNX version: 1.3, Opset version: 8)\n❯ curl -LO \"https://github.com/onnx/models/raw/master/vision/classification/squeezenet/model/squeezenet1.0-8.onnx\"\n❯ cargo run --example sample\n[...]\n    Finished dev [unoptimized + debuginfo] target(s) in 13.62s\n     Running `target/debug/examples/sample`\nUninitialized environment found, initializing it with name \"test\".\n2020-08-09 09:34:37.395577 [I:onnxruntime:, inference_session.cc:174 ConstructorCommon] Creating and using per session threadpools since use_per_session_threads_ is true\n2020-08-09 09:34:37.399253 [I:onnxruntime:, inference_session.cc:830 Initialize] Initializing session.\n2020-08-09 09:34:37.399284 [I:onnxruntime:, inference_session.cc:848 Initialize] Adding default CPU execution provider.\n2020-08-09 09:34:37.399313 [I:onnxruntime:test, bfc_arena.cc:15 BFCArena] Creating BFCArena for Cpu\n2020-08-09 09:34:37.399335 [V:onnxruntime:test, bfc_arena.cc:32 BFCArena] Creating 21 bins of max chunk size 256 to 268435456\n2020-08-09 09:34:37.410516 [I:onnxruntime:, reshape_fusion.cc:37 ApplyImpl] Total fused reshape node count: 0\n2020-08-09 09:34:37.417478 [I:onnxruntime:, reshape_fusion.cc:37 ApplyImpl] Total fused reshape node count: 0\n2020-08-09 09:34:37.420131 [I:onnxruntime:, reshape_fusion.cc:37 ApplyImpl] Total fused reshape node count: 0\n2020-08-09 09:34:37.422623 [V:onnxruntime:, inference_session.cc:671 TransformGraph] Node placements\n2020-08-09 09:34:37.428863 [V:onnxruntime:, inference_session.cc:673 TransformGraph] All nodes have been placed on [CPUExecutionProvider].\n2020-08-09 09:34:37.428954 [I:onnxruntime:, session_state.cc:25 SetGraph] SaveMLValueNameIndexMapping\n2020-08-09 09:34:37.429079 [I:onnxruntime:, session_state.cc:70 SetGraph] Done saving OrtValue mappings.\n2020-08-09 09:34:37.429925 [I:onnxruntime:, session_state_initializer.cc:178 SaveInitializedTensors] Saving initialized tensors.\n2020-08-09 09:34:37.436300 [I:onnxruntime:, session_state_initializer.cc:223 SaveInitializedTensors] Done saving initialized tensors\n2020-08-09 09:34:37.437255 [I:onnxruntime:, inference_session.cc:919 Initialize] Session successfully initialized.\nDropping the session options.\n2020-08-09 09:34:37.448956 [I:onnxruntime:, sequential_executor.cc:145 Execute] Begin execution\n2020-08-09 09:34:37.449041 [I:onnxruntime:test, bfc_arena.cc:259 AllocateRawInternal] Extending BFCArena for Cpu. bin_num:13 rounded_bytes:3154176\n2020-08-09 09:34:37.449072 [I:onnxruntime:test, bfc_arena.cc:143 Extend] Extended allocation by 4194304 bytes.\n2020-08-09 09:34:37.449087 [I:onnxruntime:test, bfc_arena.cc:147 Extend] Total allocated bytes: 9137152\n2020-08-09 09:34:37.449104 [I:onnxruntime:test, bfc_arena.cc:150 Extend] Allocated memory at 0x7fb3b9585000 to 0x7fb3b9985000\n2020-08-09 09:34:37.449176 [I:onnxruntime:test, bfc_arena.cc:259 AllocateRawInternal] Extending BFCArena for Cpu. bin_num:8 rounded_bytes:65536\n2020-08-09 09:34:37.449196 [I:onnxruntime:test, bfc_arena.cc:143 Extend] Extended allocation by 4194304 bytes.\n2020-08-09 09:34:37.449209 [I:onnxruntime:test, bfc_arena.cc:147 Extend] Total allocated bytes: 13331456\n2020-08-09 09:34:37.449222 [I:onnxruntime:test, bfc_arena.cc:150 Extend] Allocated memory at 0x7fb3b9985000 to 0x7fb3b9d85000\nDropping Tensor.\nScore for class [0] =  0.000045440578\nScore for class [1] =  0.0038458686\nScore for class [2] =  0.0001249467\nScore for class [3] =  0.0011804511\nScore for class [4] =  0.00131694\nDropping TensorFromOrt.\nDropping the session.\nDropping the memory information.\nDropping the environment.\n```\n\nSee also the integration tests ([`onnxruntime/tests/integration_tests.rs`](onnxruntime/tests/integration_tests.rs))\nthat performs simple model download and inference, validating the results.\n\n## Bindings Generation\n\nBindings (the basis of `onnxruntime-sys`) are committed to the git repository. This means `bindgen` is not\na dependency anymore on every build (it was made optional) and thus build times are better.\n\nTo generate new bindings (for example if they don't exists for your platform or if a version bump occurred), build the crate with the `generate-bindings` feature.\n\nNOTE: Make sure to have the `rustfmt` rustup component present so that bindings are formatted:\n\n```sh\nrustup component add rustfmt\n```\n\nThen on each platform build with the proper feature flag:\n\n```sh\n❯ cd onnxruntime-sys\n❯ cargo build --features generate-bindings\n```\n\n### Generating Bindings for Linux With Docker\n\nPrepare the container:\n\n```sh\n❯ docker run -it --rm --name rustbuilder -v \"$PWD\":/usr/src/myapp -w /usr/src/myapp rust:1.50.0 /bin/bash\n❯ apt-get update\n❯ apt-get install clang\n❯ rustup component add rustfmt\n```\n\nGenerate the bindings:\n\n```sh\n❯ docker exec -it --user \"$(id -u)\":\"$(id -g)\" rustbuilder /bin/bash\n❯ cd onnxruntime-sys\n❯ cargo build --features generate-bindings\n```\n\n### Generating Bindings for Windows With Vagrant\n\nYou can use [nbigaouette/windows_vagrant_rust](https://github.com/nbigaouette/windows_vagrant_rust)\nto provision a Windows VM that can build the project and generate the bindings.\n\nWindows can build both x86 and x86_64 bindings:\n\n```sh\n❯ rustup target add i686-pc-windows-msvc x86_64-pc-windows-msvc\n❯ cd onnxruntime-sys\n❯ cargo build --features generate-bindings --target i686-pc-windows-msvc\n❯ cargo build --features generate-bindings --target x86_64-pc-windows-msvc\n```\n\n## Conduct\n\nThe [Rust Code of Conduct](https://www.rust-lang.org/conduct.html) shall be respected. For\nescalation or moderation issues please contact Nicolas (nbigaouette@gmail.com)\ninstead of the Rust moderation team.\n\n## License\n\nThis project is licensed under either of\n\n* Apache License, Version 2.0, ([LICENSE-APACHE](LICENSE-APACHE) or\n  http://www.apache.org/licenses/LICENSE-2.0)\n* MIT license ([LICENSE-MIT](LICENSE-MIT) or\n  http://opensource.org/licenses/MIT)\n\nat your option.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnbigaouette%2Fonnxruntime-rs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnbigaouette%2Fonnxruntime-rs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnbigaouette%2Fonnxruntime-rs/lists"}