https://github.com/xtuc/triton-rs

Rust bindings to the Triton Inference Server
https://github.com/xtuc/triton-rs

Last synced: 12 months ago
JSON representation

Rust bindings to the Triton Inference Server

• Host: GitHub
• URL: https://github.com/xtuc/triton-rs
• Owner: xtuc
• License: mit
• Created: 2023-10-05T21:53:27.000Z (almost 3 years ago)
• Default Branch: main
• Last Pushed: 2024-03-14T09:11:21.000Z (over 2 years ago)
• Last Synced: 2024-11-03T00:12:42.104Z (almost 2 years ago)
• Language: Rust
• Homepage:
• Size: 16.6 KB
• Stars: 5
• Watchers: 2
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# Rust bindings to the Triton Inference Server 

## Triton Rust API

See [triton_rs documentation].

## Implementing a backend

```rust

use triton_rs::Backend;

struct ExampleBackend;

impl Backend for ExampleBackend {

    fn model_instance_execute(

        model: triton_rs::Model,

        requests: &[triton_rs::Request],

    ) -> Result<(), triton_rs::Error> {

        for request in requests {

            // Handle inference request here

            todo!();

        }

        Ok(())

    }

}

// Register the backend with Triton

triton_rs::declare_backend!(ExampleBackend);

```

See [example-backend] for full example.

[example-backend]: ./example-backend

[triton_rs documentation]: https://docs.rs/triton-rs