{"id":34595422,"url":"https://github.com/rdcm/triton-ng","last_synced_at":"2026-05-27T06:01:35.746Z","repository":{"id":319712094,"uuid":"1075025172","full_name":"rdcm/triton-ng","owner":"rdcm","description":"Rust SDK for writing custom backends for NVIDIA Triton Inference Server","archived":false,"fork":false,"pushed_at":"2026-04-11T01:39:30.000Z","size":157,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-11T02:22:04.955Z","etag":null,"topics":["custom-backend","infrence","nvidia","rust","triton-inference-server"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rdcm.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-10-12T23:10:04.000Z","updated_at":"2026-04-11T01:39:38.000Z","dependencies_parsed_at":null,"dependency_job_id":"c7d19107-8607-4db0-a3f9-757a4bda9f47","html_url":"https://github.com/rdcm/triton-ng","commit_stats":null,"previous_names":["rdcm/triton-custom-backend-rs"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/rdcm/triton-ng","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rdcm%2Ftriton-ng","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rdcm%2Ftriton-ng/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rdcm%2Ftriton-ng/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rdcm%2Ftriton-ng/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rdcm","download_url":"https://codeload.github.com/rdcm/triton-ng/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rdcm%2Ftriton-ng/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33553127,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-27T02:00:06.184Z","response_time":53,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["custom-backend","infrence","nvidia","rust","triton-inference-server"],"created_at":"2025-12-24T11:32:38.395Z","updated_at":"2026-05-27T06:01:35.657Z","avatar_url":"https://github.com/rdcm.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":" \u003e **WIP** — work in progress, API is unstable\n\n# triton-ng\n\nRust SDK for [NVIDIA Triton Inference Server](https://github.com/triton-inference-server/server).\n\nProvides two things:\n- A safe Rust API for writing **custom Triton backends** (compiled as `.so` and loaded by Triton)\n- A high-level async **gRPC client** for sending inference requests to a running Triton server\n\n## Crates\n\n| Crate | Description |\n|---|---|\n| `triton-ng-sys` | Raw FFI bindings generated by bindgen from `tritonbackend.h` |\n| `triton-ng` | Safe Rust wrapper over `triton-ng-sys` |\n| `triton-ng-macros` | Proc-macros for `triton-ng` |\n| `triton-ng-client` | High-level async gRPC client |\n| `example/custom-backend` | Example custom backend (MNIST, proxies to ONNX model) |\n| `example/app` | Example client application |\n\n## Writing a custom backend\n\nImplement the `Backend` trait and register it with `declare_backend!`:\n\n```rust\nuse triton_ng::backend::Backend;\nuse triton_ng::{BackendHandle, DataType, Error, InferenceRequest, Response};\n\nstruct MyBackend;\n\nimpl Backend for MyBackend {\n    fn initialize(backend: \u0026BackendHandle) -\u003e Result\u003c(), Error\u003e {\n        Ok(())\n    }\n\n    fn model_instance_execute(\n        model: triton_ng::Model,\n        requests: \u0026[triton_ng::Request],\n    ) -\u003e Result\u003c(), Error\u003e {\n        for request in requests {\n            let input = request.get_input(\"INPUT\")?;\n            let data = input.as_fp32_vec()?;\n\n            // ... run inference ...\n\n            let mut response = Response::new(request)?;\n            response\n                .create_output(\"OUTPUT\", DataType::Fp32, \u0026[1, 10])?\n                .write_fp32_vec(\u0026result)?;\n            response.send()?;\n        }\n        Ok(())\n    }\n}\n\ntriton_ng::declare_backend!(MyBackend);\n```\n\nBuild as a `cdylib`:\n\n```toml\n# Cargo.toml\n[lib]\ncrate-type = [\"cdylib\"]\n```\n\n## Using the gRPC client\n\n```rust\nuse triton_ng_client::{InferInput, InferOptions, TritonClient, TritonClientConfig};\n\n#[tokio::main]\nasync fn main() -\u003e anyhow::Result\u003c()\u003e {\n    let client = TritonClient::new(TritonClientConfig::new(\"http://localhost:8001\")).await?;\n\n    let meta = client.model_metadata(\"my_model\", None, None).await?;\n    let n: usize = meta.inputs[0].shape.iter().map(|\u0026d| d as usize).product();\n\n    let response = client\n        .infer(\n            \"my_model\",\n            None,\n            [InferInput::fp32(\"INPUT\", meta.inputs[0].shape.clone(), vec![0.0f32; n])],\n            [\"OUTPUT\"],\n            InferOptions::default(),\n        )\n        .await?;\n\n    println!(\"{:?}\", response.outputs[0].data);\n    Ok(())\n}\n```\n\nTLS:\n\n```rust\nuse triton_ng_client::{ClientTlsConfig, TritonClientConfig};\n\nlet config = TritonClientConfig::new(\"https://triton.example.com:8001\")\n    .with_tls(ClientTlsConfig::new()); // uses system roots\n```\n\n## Getting started\n\n### Prerequisites\n\n- Rust stable\n- NVIDIA driver 570+ (580+ for Blackwell / RTX 50xx)\n- [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html)\n- Docker\n\n### First run\n\n```bash\ngit submodule update --init --recursive\nmake build           # compile custom backend → target/release/libtriton_custom_backend.so\nmake download-model  # download mnist_onnx + create model version dirs\nmake docker-env-up   # start Triton (mounts .so and models/)\n```\n\n### Run the example app\n\n```bash\ncargo run --manifest-path=example/app/Cargo.toml --release\n```\n\nTriton must be running with both models in READY state.\n\n### Run integration tests\n\n```bash\nmake tests           # cargo nextest run --workspace\n```\n\nTests require a running Triton instance (`make docker-env-up`).\n\n### Rebuild after backend changes\n\n```bash\nmake build\nmake docker-env-down \u0026\u0026 make docker-env-up\n```\n\n## Features\n\n| Feature | Description |\n|---|---|\n| `cuda` | Enable GPU and pinned memory allocation in `ResponseAllocator` |\n\n```toml\ntriton-ng = { version = \"0.1\", features = [\"cuda\"] }\n```\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frdcm%2Ftriton-ng","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frdcm%2Ftriton-ng","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frdcm%2Ftriton-ng/lists"}