{"id":51032530,"url":"https://github.com/offbit-ai/litert","last_synced_at":"2026-06-22T02:00:36.715Z","repository":{"id":351831476,"uuid":"1212534151","full_name":"offbit-ai/LiteRT","owner":"offbit-ai","description":"Rust bindings for LiteRT, successor to TensorFlow Lite. is Google's On-device framework for high-performance ML \u0026 GenAI deployment on edge platforms, via efficient conversion, runtime, and optimization","archived":false,"fork":false,"pushed_at":"2026-05-03T11:27:50.000Z","size":306,"stargazers_count":3,"open_issues_count":1,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-20T19:31:21.303Z","etag":null,"topics":["genai","litert","litert-lm","ml","tensorflow"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/offbit-ai.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":"NOTICE","maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-16T13:26:44.000Z","updated_at":"2026-05-03T11:27:54.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/offbit-ai/LiteRT","commit_stats":null,"previous_names":["offbit-ai/litert"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/offbit-ai/LiteRT","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/offbit-ai%2FLiteRT","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/offbit-ai%2FLiteRT/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/offbit-ai%2FLiteRT/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/offbit-ai%2FLiteRT/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/offbit-ai","download_url":"https://codeload.github.com/offbit-ai/LiteRT/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/offbit-ai%2FLiteRT/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34630770,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-22T02:00:06.391Z","response_time":106,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["genai","litert","litert-lm","ml","tensorflow"],"created_at":"2026-06-22T02:00:27.581Z","updated_at":"2026-06-22T02:00:36.704Z","avatar_url":"https://github.com/offbit-ai.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# LiteRT-rs\n\n[![CI](https://github.com/offbit-ai/LiteRT/actions/workflows/ci.yml/badge.svg)](https://github.com/offbit-ai/LiteRT/actions/workflows/ci.yml)\n[![crates.io](https://img.shields.io/crates/v/litert.svg?label=litert)](https://crates.io/crates/litert)\n[![crates.io](https://img.shields.io/crates/v/litertlm.svg?label=litertlm)](https://crates.io/crates/litertlm)\n[![crates.io](https://img.shields.io/crates/v/litert-sys.svg?label=litert-sys)](https://crates.io/crates/litert-sys)\n[![crates.io](https://img.shields.io/crates/v/litert-lm-sys.svg?label=litert-lm-sys)](https://crates.io/crates/litert-lm-sys)\n[![docs.rs](https://img.shields.io/docsrs/litert?label=docs.rs%2Flitert)](https://docs.rs/litert)\n[![docs.rs](https://img.shields.io/docsrs/litertlm?label=docs.rs%2Flitertlm)](https://docs.rs/litertlm)\n[![MSRV](https://img.shields.io/badge/rustc-1.75%2B-blue.svg)](https://releases.rs/docs/1.75.0/)\n[![License](https://img.shields.io/badge/license-Apache--2.0-informational.svg)](LICENSE)\n[![LiteRT](https://img.shields.io/badge/LiteRT-2.1.4-informational.svg)](https://github.com/google-ai-edge/LiteRT)\n\nSafe, zero-friction Rust bindings for [Google LiteRT] 2.x — on-device ML\ninference and LLM text generation. Add a crate to `Cargo.toml` and\n`cargo build`. No Bazel, no CMake, no `libclang` on user machines.\n\n[Google LiteRT]: https://ai.google.dev/edge/litert\n\n### ML inference (`litert`)\n\n```toml\n[dependencies]\nlitert = \"0.3\"\n```\n\n```rust\nuse litert::{CompilationOptions, CompiledModel, Environment, Model, TensorBuffer};\n\nlet env = Environment::new()?;\nlet model = Model::from_file(\"mobilenet.tflite\")?;\nlet compiled = CompiledModel::new(env, model, \u0026CompilationOptions::new()?)?;\n// ... fill input buffers, compiled.run(...), read outputs ...\n# Ok::\u003c(), litert::Error\u003e(())\n```\n\n### LLM text generation (`litertlm`)\n\n```toml\n[dependencies]\nlitertlm = \"0.3\"\n```\n\n```rust\nuse litertlm::{Backend, Engine, EngineSettings, SamplerParams};\n\nlet engine = Engine::new(\n    EngineSettings::new(\"Qwen3-0.6B.litertlm\")\n        .backend(Backend::Gpu)\n        .max_num_tokens(512),\n)?;\n\n// Streaming (token-by-token)\nlet mut conv = engine.create_conversation(SamplerParams::default().top_p(0.95))?;\nconv.send_message_stream(\"Explain Rust lifetimes\", |chunk| {\n    print!(\"{chunk}\");\n})?;\n\n// Or blocking\nlet mut session = engine.create_session(SamplerParams::default().top_p(0.95))?;\nlet response = session.generate(\"Explain Rust lifetimes\")?;\n# Ok::\u003c(), litertlm::Error\u003e(())\n```\n\n## Why\n\n| Other options                      | Friction                                                            |\n|------------------------------------|---------------------------------------------------------------------|\n| Build LiteRT from source via CMake | Bazel or CMake + protoc + flatc + abseil + Android NDK on your box  |\n| Invoke via Python (`ai-edge-litert`) | Python interpreter + wheel dependency graph                         |\n| Hand-roll FFI against TFLite C API | Maintain a sysroot per target + track header drift manually         |\n\n`litert-rs` takes the same upstream runtime binaries Google publishes, pins\neach by SHA-256, and downloads them into a user-level cache the first time\n`cargo build` runs. Your app links against that cached `libLiteRt.{so,dylib,dll}`.\n\n## Crates\n\n| Crate | What it is | crates.io |\n|-------|------------|-----------|\n| [`litert`](https://crates.io/crates/litert) | Safe ML inference wrappers (CompiledModel, TensorBuffer, GPU) | 0.3.x |\n| [`litertlm`](https://crates.io/crates/litertlm) | Safe LLM text generation (Engine, Session, Conversation streaming) | 0.3.x |\n| [`litert-sys`](https://crates.io/crates/litert-sys) | Raw FFI — LiteRT 2.x C API | 0.3.x |\n| [`litert-lm-sys`](https://crates.io/crates/litert-lm-sys) | Raw FFI — LiteRT-LM C engine API | 0.3.x |\n\n## Platform support\n\n| Rust target                     | CPU | GPU accelerator(s) shipped        | Source                         |\n|---------------------------------|-----|-----------------------------------|--------------------------------|\n| `aarch64-apple-darwin`          | ✅  | Metal, WebGPU                     | litert-lm prebuilt (Git LFS)   |\n| `x86_64-unknown-linux-gnu`      | ✅  | WebGPU                            | litert-lm prebuilt (Git LFS)   |\n| `aarch64-unknown-linux-gnu`     | ✅  | WebGPU                            | litert-lm prebuilt (Git LFS)   |\n| `x86_64-pc-windows-msvc`        | ✅  | WebGPU                            | litert-lm prebuilt (Git LFS)   |\n| `aarch64-linux-android`         | ✅  | OpenCL/GL (via `ClGlAccelerator`) | LiteRT Maven AAR               |\n| `x86_64-linux-android`          | ✅  | OpenCL/GL                         | LiteRT Maven AAR               |\n| `wasm32-unknown-emscripten`     | ✅  | — (XNNPACK only; GPU deferred)    | LiteRT-rs CMake+emcc build     |\n| `aarch64-apple-ios`             | ⏳  | —                                 | deferred (no upstream prebuilt) |\n\n`litertlm` / `litert-lm-sys` (LLM inference) are desktop/Android only this\nrelease. WASM support for the LLM stack is on the 0.4.0 roadmap — see\n`wasm-patches/litert-lm-v0.10.2/`.\n\n## Environment variables (escape hatches)\n\n| Variable              | Effect                                                                     |\n|-----------------------|----------------------------------------------------------------------------|\n| `LITERT_LIB_DIR`      | Directory containing `libLiteRt.{so,dylib,dll}`. Bypasses the downloader. |\n| `LITERT_NO_DOWNLOAD`  | Fail the build if any prebuilt is missing from cache (air-gapped CI).     |\n| `LITERT_CACHE_DIR`    | Override the cache root. Default: `$XDG_CACHE_HOME/litert-sys`.           |\n\n## macOS downstream binaries (one-time setup)\n\nThe prebuilt `libLiteRt.dylib` Google ships has `install_name=@rpath/libLiteRt.dylib`\nand wasn't linked with `-headerpad_max_install_names`, so we can't rewrite that\nidentifier to an absolute path post-download. `litert-sys`' build script emits\nan `-rpath` flag for its own tests and examples, but Cargo's `rustc-link-arg`\ndoes **not** propagate to downstream consumer binaries. Without action, the\nbinaries your crate produces on macOS will fail at launch with:\n\n    dyld: Library not loaded: @rpath/libLiteRt.dylib\n\nFix it once per downstream crate — add this tiny [build.rs](https://doc.rust-lang.org/cargo/reference/build-scripts.html)\nnext to your `Cargo.toml`:\n\n```rust\n// build.rs\nfn main() {\n    // `litert-sys` declares `links = \"LiteRt\"` and publishes its cache\n    // directory as `DEP_LITERT_LIB_DIR`. Embedding it as an rpath makes\n    // dyld find libLiteRt.dylib without DYLD_LIBRARY_PATH.\n    if let Ok(dir) = std::env::var(\"DEP_LITERT_LIB_DIR\") {\n        println!(\"cargo:rustc-link-arg=-Wl,-rpath,{dir}\");\n    }\n}\n```\n\nAlternatively, prefix individual invocations with\n`DYLD_LIBRARY_PATH=$(cargo xtask cache-dir)`, or link with\n`RUSTFLAGS=\"-C link-arg=-Wl,-rpath,/path/to/cache\"`.\n\nLinux, Windows, and Android are unaffected.\n\n## WebAssembly (browser + Node.js + wasmtime)\n\n`litert` and `litert-sys` cross-compile to `wasm32-unknown-emscripten` for\nin-browser ML inference, server-side WASM (Cloudflare Workers, wasmtime),\nor Node.js. The runtime is TFLite + XNNPACK CPU kernels, statically\nlinked into the produced `.wasm`.\n\n### Prerequisites\n\n* [emsdk](https://github.com/emscripten-core/emsdk) ≥ 5.0.7. Install via\n  `git clone https://github.com/emscripten-core/emsdk \u0026\u0026 cd emsdk \u0026\u0026 ./emsdk\n  install latest \u0026\u0026 ./emsdk activate latest`.\n* `rustup target add wasm32-unknown-emscripten`.\n* `source $EMSDK/emsdk_env.sh` before each build session.\n\n### Build\n\n```bash\nsource $EMSDK/emsdk_env.sh\n\n# NODERAWFS=1 lets the WASM module read host files (model.tflite) under\n# Node/wasmtime. ALLOW_MEMORY_GROWTH=1 lets the heap grow past 16 MB so\n# larger models load. Drop both for a browser bundle (use --preload-file\n# or fetch+MEMFS instead).\nRUSTFLAGS=\"-C link-arg=-sNODERAWFS=1 -C link-arg=-sALLOW_MEMORY_GROWTH=1\" \\\n  cargo build -p litert --example add_wasm \\\n              --target wasm32-unknown-emscripten --release\n```\n\nOutput (`target/wasm32-unknown-emscripten/release/examples/`):\n- `add_wasm.wasm` — ~5–12 MB WebAssembly module (12 MB debug, smaller in\n  release with `-Oz`).\n- `add_wasm.js` — emscripten JS shim that knows how to instantiate the\n  `.wasm`.\n\n### Run (Node.js)\n\n```bash\nnode target/wasm32-unknown-emscripten/release/examples/add_wasm.js\n# add_10x10.tflite — WASM CPU inference\n# first 5 outputs: [100.0, 102.0, 104.0, 106.0, 108.0]\n# last 5 outputs:  [290.0, 292.0, 294.0, 296.0, 298.0]\n```\n\n### Browser\n\nDrop `NODERAWFS=1` (no host filesystem in browsers). Embed the model with\nemcc's `--preload-file model.tflite` (bundles into a `.data` sidecar), or\n`fetch()` it from JS and write to MEMFS before calling `Model::from_file`,\nor use `Model::from_bytes` with a `Vec\u003cu8\u003e` you `fetch()`'d. Then drop the\n`.wasm` + `.js` into a static page:\n\n```html\n\u003cscript src=\"add_wasm.js\"\u003e\u003c/script\u003e\n```\n\n### Building from source (no published artifact yet)\n\nWhile the v0.3.0 prebuilt tarball is being staged, build the static\narchives locally and point `litert-sys` at them via `LITERT_LIB_DIR`:\n\n```bash\n# 1. Clone + patch upstream LiteRT\ngit clone --depth=1 --branch=v2.1.4 \\\n    https://github.com/google-ai-edge/LiteRT.git /tmp/litert\ncd /tmp/litert\ngit apply $LITERT_RS_DIR/wasm-patches/litert-v2.1.4/01-cmake-emscripten-support.patch\n\n# 2. Cross-compile via CMake + emcc\nemcmake cmake -S litert -B litert/build-wasm \\\n    -DCMAKE_BUILD_TYPE=Release \\\n    -DLITERT_ENABLE_GPU=OFF -DLITERT_ENABLE_NPU=OFF \\\n    -DLITERT_DISABLE_KLEIDIAI=ON -DLITERT_BUILD_TESTS=OFF \\\n    -DTFLITE_ENABLE_GPU=OFF\nemmake cmake --build litert/build-wasm \\\n    --target litert_runtime_c_api_shared_lib -j\n\n# 3. Flatten archives into a single dir for litert-sys\nmkdir -p /tmp/litert-wasm-libs\nfind litert/build-wasm \\( -name \"lib*.a\" ! -path \"*/testdata/*\" ! -name \"input.a\" \\) \\\n    -exec cp {} /tmp/litert-wasm-libs/ \\;\n\n# 4. Build\ncd $LITERT_RS_DIR\nLITERT_LIB_DIR=/tmp/litert-wasm-libs \\\nRUSTFLAGS=\"-C link-arg=-sNODERAWFS=1 -C link-arg=-sALLOW_MEMORY_GROWTH=1\" \\\n  cargo build -p litert --example add_wasm --target wasm32-unknown-emscripten\n```\n\nOnce the [`build-litert-wasm.yml`](.github/workflows/build-litert-wasm.yml)\nGitHub Actions workflow has uploaded a SHA-pinned tarball, `cargo build`\nwill download and verify it automatically — no `LITERT_LIB_DIR`, no emsdk\nsetup needed for end users on the WASM target.\n\n### Limitations (v0.3.0)\n\n- **CPU only.** WebGPU acceleration is on the 0.4.0 roadmap.\n- **No `set_global_log_severity`.** The WASM build doesn't export the\n  logger-control symbols. Returns `Error::Unsupported`; LiteRT logs at\n  default verbosity.\n- **No LLM stack.** `litertlm` / `litert-lm-sys` need separate fork patches\n  to LiteRT-LM (orchestrator + transitive C++ deps); 0.4.0 milestone.\n\n## Cross-platform development\n\nEnd users only need `cargo`. The sections below are for contributors who want\nto regenerate bindings or build for foreign targets locally.\n\n### Tooling prerequisites (contributors only)\n\n* `rustup` with stable + any target triples you want to exercise.\n* A container engine for foreign-target builds: **Docker** or **Podman**.\n  * macOS: `brew install podman \u0026\u0026 podman machine init \u0026\u0026 podman machine start`.\n  * Linux: your distro's Docker/Podman packages.\n* `cross` ≥ 0.2.5: `cargo install cross --locked`.\n* If you're using Podman: `export CROSS_CONTAINER_ENGINE=podman`.\n\nmacOS and Windows target toolchains run **natively**, not through `cross`.\n`cross` is only invoked for Linux + Android targets.\n\n### Workspace automation\n\nAll cross-target chores flow through a single `xtask` binary.\n\n```bash\ncargo xtask targets            # list every supported Rust target triple\ncargo xtask regen-bindings     # rebuild litert-sys bindings for every target\ncargo xtask regen-bindings --target aarch64-apple-darwin   # single target\ncargo xtask build-all          # cross-build the workspace for every target\n```\n\n`regen-bindings` dispatches automatically:\n\n* **Host target** → native `cargo build -p litert-sys --features generate-bindings`.\n* **Foreign target** → `cross build …`, which runs `bindgen` inside a\n  container image that already has `libclang` + the target sysroot installed\n  (see `Cross.toml`).\n\n### CI\n\n[`.github/workflows/ci.yml`](.github/workflows/ci.yml) runs three matrices on\nevery push and PR:\n\n1. **`native`** — macOS arm64 (tests + build), Linux x86_64 (tests + `fmt --check` + `clippy -D warnings`), Windows x86_64 (build).\n2. **`cross`** — Linux arm64, Android arm64, Android x86_64 (build only).\n3. **`bindings-drift`** — regenerates all 6 target binding files via `cargo xtask regen-bindings` and fails on any `git diff`. If drift is detected the regenerated files are uploaded as a build artifact so they can be inspected or accepted in a PR.\n\nDrift is the authoritative check: if the CI-generated bindings for a target\ndiffer from what's committed, that's the signal to update the committed file.\n\n### First-build download\n\nOn the first `cargo build` for a given target, `litert-sys/build.rs` emits a\none-time warning while it fetches the pinned prebuilt libraries:\n\n```\nwarning: litert-sys: downloading 4 file(s) of LiteRT prebuilt v0.10.2\n         for target `macos_arm64` into\n         /Users/you/Library/Caches/litert-sys/v0.10.2/aarch64-apple-darwin\n         (first build only)\n```\n\nSubsequent builds hit the cache (with a size check; a SHA-256 verified marker\nfile short-circuits rehashing). Deleting the cache directory or bumping the\npinned upstream version triggers a fresh download + re-verification.\n\n## Regenerating bindings for a new target\n\n```bash\n# 1. Add the triple to:\n#      Cargo.toml workspace members (if needed)\n#      Cross.toml (pre-build apt-get for libclang)\n#      litert-sys/build.rs target_spec()  — with pinned checksums\n#      xtask/src/main.rs TARGETS\n#\n# 2. Regenerate:\ncargo xtask regen-bindings --target \u003cnew-target\u003e\n#\n# 3. Commit litert-sys/src/bindings/\u003cnew-target\u003e.rs and push.\n```\n\n## Credits\n\nThis project is a binding, not a fork. The runtime that does the work —\nmodel loading, graph compilation, kernel execution, GPU/NPU delegation — is\nGoogle's [LiteRT] and [LiteRT-LM]:\n\n* **LiteRT** (Apache-2.0, © 2024–2026 Google LLC) — C API headers vendored\n  under `third_party/litert-v2.1.4/`, source:\n  \u003chttps://github.com/google-ai-edge/LiteRT\u003e.\n* **LiteRT-LM** (Apache-2.0, © 2024–2026 Google LLC) — source of the\n  prebuilt `libLiteRt.*` + accelerator plugins we download:\n  \u003chttps://github.com/google-ai-edge/litert-lm\u003e.\n* **TensorFlow Lite**, **XNNPACK**, **abseil-cpp**, **flatbuffers**,\n  **protobuf** and the rest of the transitive open-source stack that LiteRT\n  itself is built on.\n\nSee [NOTICE](NOTICE) for the full attribution and\n[third_party/litert-v2.1.4/LICENSE](third_party/litert-v2.1.4/LICENSE) for\nthe upstream LiteRT license text.\n\n[LiteRT]: https://github.com/google-ai-edge/LiteRT\n[LiteRT-LM]: https://github.com/google-ai-edge/litert-lm\n\n## License\n\nLicensed under the [Apache License, Version 2.0](LICENSE). By contributing\nyou agree that your contribution is licensed under the same terms.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foffbit-ai%2Flitert","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foffbit-ai%2Flitert","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foffbit-ai%2Flitert/lists"}