{"id":24769220,"url":"https://github.com/inference-gateway/rust-sdk","last_synced_at":"2025-10-11T19:30:49.868Z","repository":{"id":273220670,"uuid":"919030167","full_name":"inference-gateway/rust-sdk","owner":"inference-gateway","description":"An SDK written in Rust for the Inference Gateway","archived":false,"fork":false,"pushed_at":"2025-01-28T02:01:59.000Z","size":126,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-28T02:17:44.336Z","etag":null,"topics":["inference-gateway","rust-sdk","sdk"],"latest_commit_sha":null,"homepage":"https://github.com/edenreich/inference-gateway","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/inference-gateway.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-01-19T14:33:51.000Z","updated_at":"2025-01-28T02:02:02.000Z","dependencies_parsed_at":"2025-01-20T20:38:17.164Z","dependency_job_id":null,"html_url":"https://github.com/inference-gateway/rust-sdk","commit_stats":null,"previous_names":["edenreich/inference-gateway-rust-sdk","inference-gateway/rust-sdk"],"tags_count":15,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/inference-gateway%2Frust-sdk","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/inference-gateway%2Frust-sdk/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/inference-gateway%2Frust-sdk/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/inference-gateway%2Frust-sdk/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/inference-gateway","download_url":"https://codeload.github.com/inference-gateway/rust-sdk/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":236122007,"owners_count":19098256,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["inference-gateway","rust-sdk","sdk"],"created_at":"2025-01-29T02:47:54.380Z","updated_at":"2025-10-11T19:30:49.863Z","avatar_url":"https://github.com/inference-gateway.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Inference Gateway Rust SDK\n\nAn SDK written in Rust for the [Inference Gateway](https://github.com/inference-gateway/inference-gateway).\n\n- [Inference Gateway Rust SDK](#inference-gateway-rust-sdk)\n  - [Installation](#installation)\n  - [Usage](#usage)\n    - [Creating a Client](#creating-a-client)\n    - [Listing Models](#listing-models)\n    - [Listing Models from a specific provider](#listing-models-from-a-specific-provider)\n    - [Listing MCP Tools](#listing-mcp-tools)\n    - [Generating Content](#generating-content)\n    - [Streaming Content](#streaming-content)\n    - [Tool-Use](#tool-use)\n    - [Health Check](#health-check)\n  - [Supported Providers](#supported-providers)\n  - [Contributing](#contributing)\n  - [License](#license)\n\n## Installation\n\nRun `cargo add inference-gateway-sdk`.\n\n## Usage\n\n### Creating a Client\n\nHere is a full example of how to create a client and interact with the Inference Gateway API:\n\n```rust\nuse inference_gateway_sdk::{\n    CreateChatCompletionResponse,\n    GatewayError,\n    InferenceGatewayAPI,\n    InferenceGatewayClient,\n    ListModelsResponse,\n    ListToolsResponse,\n    Message,\n    Provider,\n    MessageRole\n};\nuse log::info;\nuse std::env;\n\n#[tokio::main]\nasync fn main() -\u003e Result\u003c(), GatewayError\u003e {\n    if env::var(\"RUST_LOG\").is_err() {\n        env::set_var(\"RUST_LOG\", \"info\");\n    }\n    env_logger::init();\n\n    // Create a client\n    let client = InferenceGatewayClient::new(\"http://localhost:8080/v1\");\n\n    // List all models and all providers\n    let response: ListModelsResponse = client.list_models().await?;\n    for model in response.data {\n        info!(\"Model: {:?}\", model.id);\n    }\n\n    // List models for a specific provider\n    let response: ListModelsResponse = client.list_models_by_provider(Provider::Groq).await?;\n    info!(\"Models for provider: {:?}\", response.provider);\n    for model in response.data {\n        info!(\"Model: {:?}\", model.id);\n    }\n\n    // Generate content - choose from available providers and models\n    let response: CreateChatCompletionResponse = client.generate_content(Provider::Groq, \"deepseek-r1-distill-llama-70b\", vec![\n    Message{\n        role: MessageRole::System,\n        content: \"You are an helpful assistent.\".to_string()\n    },\n    Message{\n        role: MessageRole::User,\n        content: \"Tell me a funny joke\".to_string()\n    }\n    ]).await?;\n\n    log::info!(\n        \"Generated content: {:?}\",\n        response.choices[0].message.content\n    );\n\n    Ok(())\n}\n```\n\n### Listing Models\n\nTo list all available models from all configured providers, use the `list_models` method:\n\n```rust\nuse inference_gateway_sdk::{\n    GatewayError\n    InferenceGatewayAPI,\n    InferenceGatewayClient,\n    ListModelsResponse,\n    Message,\n};\nuse log::info;\n\n#[tokio::main]\nfn main() -\u003e Result\u003c(), GatewayError\u003e {\n    // ...Create a client\n\n    // List models from all providers\n    let response: ListModelsResponse = client.list_models().await?;\n    for model in response.data {\n        info!(\"Model: {:?}\", model.id);\n    }\n\n    // ...\n}\n```\n\n### Listing Models from a specific provider\n\nTo list all available models from a specific provider, use the `list_models_by_provider` method:\n\n```rust\nuse inference_gateway_sdk::{\n    GatewayError\n    InferenceGatewayAPI,\n    InferenceGatewayClient,\n    ListModelsResponse,\n    Provider,\n};\nuse log::info;\n\n// ...Open main function\n\n// List models for a specific provider\nlet response: ListModelsResponse = client.list_models_by_provider(Provider::Groq).await?;\ninfo!(\"Models for provider: {:?}\", response.provider);\nfor model in response.data {\n    info!(\"Model: {:?}\", model.id);\n}\n\n// Example with Google provider\nlet response: ListModelsResponse = client.list_models_by_provider(Provider::Google).await?;\ninfo!(\"Google models: {:?}\", response.provider);\nfor model in response.data {\n    info!(\"Google Model: {:?}\", model.id);\n}\n\n// ...Rest of the main function\n```\n\n### Listing MCP Tools\n\nTo list all available MCP (Model Context Protocol) tools from all configured MCP servers, use the `list_tools` method:\n\n```rust\nuse inference_gateway_sdk::{\n    GatewayError,\n    InferenceGatewayAPI,\n    InferenceGatewayClient,\n    ListToolsResponse,\n};\nuse log::info;\n\n#[tokio::main]\nasync fn main() -\u003e Result\u003c(), GatewayError\u003e {\n    // ...Create a client\n\n    // List all MCP tools from all configured servers\n    let response: ListToolsResponse = client.list_tools().await?;\n    info!(\"Found {} MCP tools\", response.data.len());\n\n    for tool in response.data {\n        info!(\"Tool: {} from server: {}\", tool.name, tool.server);\n        info!(\"Description: {}\", tool.description);\n        if let Some(schema) = \u0026tool.input_schema {\n            info!(\"Input schema: {}\", schema);\n        }\n    }\n\n    Ok(())\n}\n```\n\nNote: This functionality requires that MCP servers are configured and exposed in your Inference Gateway instance. If MCP is not exposed, you'll receive a `403 Forbidden` error.\n\n### Generating Content\n\nTo generate content using a model, use the `generate_content` method:\n\n```rust\nuse inference_gateway_sdk::{\n    CreateChatCompletionResponse,\n    GatewayError,\n    InferenceGatewayAPI,\n    InferenceGatewayClient,\n    Message,\n    Provider,\n    MessageRole\n};\n\n// Generate content - choose from available providers and models\nlet response: CreateChatCompletionResponse = client.generate_content(Provider::Groq, \"deepseek-r1-distill-llama-70b\", vec![\nMessage{\n    role: MessageRole::System,\n    content: \"You are an helpful assistent.\".to_string(),\n    ..Default::default()\n},\nMessage{\n    role: MessageRole::User,\n    content: \"Tell me a funny joke\".to_string(),\n    ..Default::default()\n}\n]).await?;\n\nlog::info!(\n    \"Generated content: {:?}\",\n    response.choices[0].message.content\n);\n\n// Example with Google provider (Gemini models)\nlet response: CreateChatCompletionResponse = client.generate_content(Provider::Google, \"gemini-1.5-flash\", vec![\nMessage{\n    role: MessageRole::System,\n    content: \"You are a helpful AI assistant.\".to_string(),\n    ..Default::default()\n},\nMessage{\n    role: MessageRole::User,\n    content: \"Explain quantum computing in simple terms\".to_string(),\n    ..Default::default()\n}\n]).await?;\n\nlog::info!(\n    \"Google generated content: {:?}\",\n    response.choices[0].message.content\n);\n```\n\n### Streaming Content\n\nYou need to add the following tiny dependencies:\n\n- `futures-util` for the `StreamExt` trait\n- `serde` with feature `derive` and `serde_json` for serialization and deserialization of the response content\n\n```rust\nuse futures_util::{pin_mut, StreamExt};\nuse inference_gateway_sdk::{\n    CreateChatCompletionStreamResponse, GatewayError, InferenceGatewayAPI, InferenceGatewayClient,\n    Message, MessageRole, Provider,\n};\nuse log::info;\nuse std::env;\n\n#[tokio::main]\nasync fn main() -\u003e Result\u003c(), GatewayError\u003e {\n    if env::var(\"RUST_LOG\").is_err() {\n        env::set_var(\"RUST_LOG\", \"info\");\n    }\n    env_logger::init();\n\n    let system_message = \"You are an helpful assistent.\".to_string();\n    let model = \"deepseek-r1-distill-llama-70b\";\n\n    let client = InferenceGatewayClient::new(\"http://localhost:8080/v1\");\n    let stream = client.generate_content_stream(\n        Provider::Groq,\n        model,\n        vec![\n            Message {\n                role: MessageRole::System,\n                content: system_message,\n                ..Default::default()\n            },\n            Message {\n                role: MessageRole::User,\n                content: \"Write a poem\".to_string(),\n                ..Default::default()\n            },\n        ],\n    );\n    pin_mut!(stream);\n    // Iterate over the stream of Server Sent Events\n    while let Some(ssevent) = stream.next().await {\n        let ssevent = ssevent?;\n\n        // Deserialize the event response\n        let generate_response_stream: CreateChatCompletionStreamResponse =\n            serde_json::from_str(\u0026ssevent.data)?;\n\n        let choice = generate_response_stream.choices.get(0);\n        if choice.is_none() {\n            continue;\n        }\n        let choice = choice.unwrap();\n\n        if let Some(usage) = generate_response_stream.usage.as_ref() {\n            // Get the usage metrics from the response\n            info!(\"Usage Metrics: {:?}\", usage);\n            // Probably send them over to a metrics service\n            break;\n        }\n\n        // Print the token out as it's being sent from the server\n        if let Some(content) = choice.delta.content.as_ref() {\n            print!(\"{}\", content);\n        }\n\n        if let Some(finish_reason) = choice.finish_reason.as_ref() {\n            if finish_reason == \"stop\" {\n                info!(\"Finished generating content\");\n                break;\n            }\n        }\n    }\n\n    Ok(())\n}\n```\n\n### Tool-Use\n\nYou can pass to the generate_content function also tools, which will be available for the LLM to use:\n\n```rust\nuse inference_gateway_sdk::{\n    FunctionObject, GatewayError, InferenceGatewayAPI, InferenceGatewayClient, Message,\n    MessageRole, Provider, Tool, ToolType,\n};\nuse log::{info, warn};\nuse serde::{Deserialize, Serialize};\nuse serde_json::{json, Value};\nuse std::env;\n\n#[tokio::main]\nasync fn main() -\u003e Result\u003c(), GatewayError\u003e {\n    // Configure logging\n    if env::var(\"RUST_LOG\").is_err() {\n        env::set_var(\"RUST_LOG\", \"info\");\n    }\n    env_logger::init();\n\n    // API endpoint - store as a variable so we can reuse it\n    let api_endpoint = \"http://localhost:8080/v1\";\n\n    // Initialize the API client\n    let client = InferenceGatewayClient::new(api_endpoint);\n\n    // Define the model and provider\n    let provider = Provider::Groq;\n    let model = \"deepseek-r1-distill-llama-70b\";\n\n    // Define the weather tool\n    let tools = vec![Tool {\n        r#type: ToolType::Function,\n        function: FunctionObject {\n            name: \"get_current_weather\".to_string(),\n            description: \"Get the weather for a location\".to_string(),\n            parameters: json!({\n                \"type\": \"object\",\n                \"properties\": {\n                    \"location\": {\n                        \"type\": \"string\",\n                        \"description\": \"The city name\"\n                    }\n                },\n                \"required\": [\"location\"]\n            }),\n        },\n    }];\n\n    // Create initial conversation\n    let initial_messages = vec![\n        Message {\n            role: MessageRole::System,\n            content: \"You are a helpful assistant that can check the weather.\".to_string(),\n            ..Default::default()\n        },\n        Message {\n            role: MessageRole::User,\n            content: \"What is the current weather in Berlin?\".to_string(),\n            ..Default::default()\n        },\n    ];\n\n    // Make the initial API request\n    info!(\"Sending initial request to model\");\n    let response = client\n        .with_tools(Some(tools.clone()))\n        .generate_content(provider, model, initial_messages)\n        .await?;\n\n    info!(\"Received response from model\");\n\n    // Check if we have a response\n    let choice = match response.choices.get(0) {\n        Some(choice) =\u003e choice,\n        None =\u003e {\n            warn!(\"No choice returned\");\n            return Ok(());\n        }\n    };\n\n    // Check for tool calls in the response\n    if let Some(tool_calls) = \u0026choice.message.tool_calls {\n        // Create a new conversation starting with the initial messages\n        let mut follow_up_convo = vec![\n            Message {\n                role: MessageRole::System,\n                content: \"You are a helpful assistant that can check the weather.\".to_string(),\n                ..Default::default()\n            },\n            Message {\n                role: MessageRole::User,\n                content: \"What is the current weather in Berlin?\".to_string(),\n                ..Default::default()\n            },\n            Message {\n                role: MessageRole::Assistant,\n                content: choice.message.content.clone(),\n                tool_calls: choice.message.tool_calls.clone(),\n                ..Default::default()\n            },\n        ];\n\n        // Process each tool call\n        for tool_call in tool_calls {\n            info!(\"Tool Call Requested: {}\", tool_call.function.name);\n\n            if tool_call.function.name == \"get_current_weather\" {\n                // Parse arguments\n                let args = tool_call.function.parse_arguments()?;\n\n                // Call our function\n                let weather_result = get_current_weather(args)?;\n\n                // Add the tool response to the conversation\n                follow_up_convo.push(Message {\n                    role: MessageRole::Tool,\n                    content: weather_result,\n                    tool_call_id: Some(tool_call.id.clone()),\n                    ..Default::default()\n                });\n            }\n        }\n\n        // Send the follow-up request with the tool results\n        info!(\"Sending follow-up request with tool results\");\n\n        // Create a new client for the follow-up request\n        let follow_up_client = InferenceGatewayClient::new(api_endpoint);\n\n        let follow_up_response = follow_up_client\n            .with_tools(Some(tools))\n            .generate_content(provider, model, follow_up_convo)\n            .await?;\n\n        if let Some(choice) = follow_up_response.choices.get(0) {\n            info!(\"Final response: {}\", choice.message.content);\n        } else {\n            warn!(\"No response in follow-up\");\n        }\n    } else {\n        info!(\"No tool calls in the response\");\n        info!(\"Model response: {}\", choice.message.content);\n    }\n\n    Ok(())\n}\n\n#[derive(Debug, Deserialize, Serialize)]\nstruct Weather {\n    location: String,\n}\n\nfn get_current_weather(args: Value) -\u003e Result\u003cString, GatewayError\u003e {\n    // Parse the location from the arguments\n    let weather: Weather = serde_json::from_value(args)?;\n    info!(\n        \"Getting weather function was called for {}\",\n        weather.location\n    );\n\n    // In a real application, we would call an actual weather API here\n    // For this example, we'll just return a mock response\n    let location = weather.location;\n    Ok(format!(\n        \"The weather in {} is currently sunny with a temperature of 22°C\",\n        location\n    ))\n}\n```\n\n### Health Check\n\nTo check if the Inference Gateway is running, use the `health_check` method:\n\n```rust\n// ...rest of the imports\nuse log::info;\n\n// ...main function\nlet is_healthy = client.health_check().await?;\ninfo!(\"API is healthy: {}\", is_healthy);\n```\n\n## Supported Providers\n\nThe Inference Gateway Rust SDK supports the following providers:\n\n- **Ollama** (`Provider::Ollama`) - Local language model server\n- **Groq** (`Provider::Groq`) - High-speed inference provider\n- **OpenAI** (`Provider::OpenAI`) - GPT models and other OpenAI services\n- **Cloudflare** (`Provider::Cloudflare`) - Cloudflare Workers AI\n- **Cohere** (`Provider::Cohere`) - Cohere language models\n- **Anthropic** (`Provider::Anthropic`) - Claude models\n- **DeepSeek** (`Provider::Deepseek`) - DeepSeek models\n- **Google** (`Provider::Google`) - Google Gemini models via Generative AI API\n\nEach provider may support different models and capabilities. Use the `list_models_by_provider()` method to discover available models for each provider.\n\nExample:\n\n```rust\nuse inference_gateway_sdk::{Provider, InferenceGatewayClient, InferenceGatewayAPI};\n\nlet client = InferenceGatewayClient::new(\"http://localhost:8080/v1\");\n\n// List Google models\nlet google_models = client.list_models_by_provider(Provider::Google).await?;\nfor model in google_models.data {\n    println!(\"Google model: {}\", model.id);\n}\n```\n\n## Contributing\n\nPlease refer to the [CONTRIBUTING.md](CONTRIBUTING.md) file for information about how to get involved. We welcome issues, questions, and pull requests.\n\n## License\n\nThis SDK is distributed under the MIT License, see [LICENSE](LICENSE) for more information.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finference-gateway%2Frust-sdk","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Finference-gateway%2Frust-sdk","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finference-gateway%2Frust-sdk/lists"}