{"id":35084912,"url":"https://github.com/majiayu000/litellm-rs","last_synced_at":"2026-04-06T06:32:00.909Z","repository":{"id":315332491,"uuid":"1019953638","full_name":"majiayu000/litellm-rs","owner":"majiayu000","description":"A high-performance AI Gateway written in Rust — call 100+ LLM APIs using OpenAI format","archived":false,"fork":false,"pushed_at":"2026-03-28T10:39:00.000Z","size":9330,"stargazers_count":34,"open_issues_count":0,"forks_count":6,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-03-28T14:35:18.005Z","etag":null,"topics":["ai-gateway","anthropic","api-client","async-rust","aws-bedrock","embeddings","gemini","llm","load-balancing","multi-provider","ollama","openai","rust","streaming"],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/majiayu000.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-07-15T06:04:01.000Z","updated_at":"2026-03-28T10:39:03.000Z","dependencies_parsed_at":"2025-09-18T19:00:18.840Z","dependency_job_id":"996084ac-2429-4973-aa1e-fe5890af8586","html_url":"https://github.com/majiayu000/litellm-rs","commit_stats":null,"previous_names":["majiayu000/litellm-rs"],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/majiayu000/litellm-rs","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/majiayu000%2Flitellm-rs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/majiayu000%2Flitellm-rs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/majiayu000%2Flitellm-rs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/majiayu000%2Flitellm-rs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/majiayu000","download_url":"https://codeload.github.com/majiayu000/litellm-rs/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/majiayu000%2Flitellm-rs/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31463012,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-05T21:22:52.476Z","status":"online","status_checked_at":"2026-04-06T02:00:07.287Z","response_time":112,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-gateway","anthropic","api-client","async-rust","aws-bedrock","embeddings","gemini","llm","load-balancing","multi-provider","ollama","openai","rust","streaming"],"created_at":"2025-12-27T13:02:22.939Z","updated_at":"2026-04-06T06:32:00.904Z","avatar_url":"https://github.com/majiayu000.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# litellm-rs\n\nA high-performance Rust library and gateway for calling 100+ LLM APIs in an OpenAI-compatible format.\n\n[![Crates.io](https://img.shields.io/crates/v/litellm-rs.svg)](https://crates.io/crates/litellm-rs)\n[![Documentation](https://docs.rs/litellm-rs/badge.svg)](https://docs.rs/litellm-rs)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n\n## Features\n\n- **100+ AI Providers** - OpenAI, Anthropic, Google, Azure, AWS Bedrock, and more\n- **OpenAI-Compatible API** - Drop-in replacement for OpenAI SDK\n- **High Performance** - 10,000+ requests/second, \u003c10ms routing overhead\n- **Intelligent Routing** - Load balancing, failover, cost optimization\n- **Enterprise Ready** - Auth, rate limiting, caching, observability\n\n## Quick Start (5 Minutes, API-Only Recommended)\n\nMost users use this project as a unified API library, not as a gateway server. Start with API-only mode first.\n\n```toml\n[dependencies]\nlitellm-rs = { version = \"0.4\", default-features = false, features = [\"lite\"] }\n```\n\nFor crate users, no `make` is required.\n\n## Usage\n\n### As a Library (API Integration)\n\n```rust\nuse litellm_rs::{completion, user_message, system_message};\n\n#[tokio::main]\nasync fn main() -\u003e Result\u003c(), Box\u003cdyn std::error::Error\u003e\u003e {\n    let response = completion(\n        \"gpt-4\",\n        vec![\n            system_message(\"You are a helpful assistant.\"),\n            user_message(\"Hello!\"),\n        ],\n        None,\n    ).await?;\n\n    println!(\"{}\", response.choices[0].message.content.as_ref().unwrap());\n    Ok(())\n}\n```\n\n### As a Gateway Server\n\n#### Run from source repository\n\n```bash\ngit clone https://github.com/majiayu000/litellm-rs.git\ncd litellm-rs\ncp config/gateway.yaml.example config/gateway.yaml\ncargo run --bin gateway\n```\n\n#### Install binary and run\n\n```bash\ncargo install litellm-rs --bin gateway\nmkdir -p config\ncurl -L https://raw.githubusercontent.com/majiayu000/litellm-rs/main/config/gateway.yaml.example -o config/gateway.yaml\ngateway\n```\n\nNotes:\n\n- `gateway` and `google-gateway` binaries require `storage` feature at build time.\n- Default features include `sqlite`, so default `cargo run`/`cargo install` satisfy this requirement.\n\n## Installation\n\n```toml\n# Full gateway with SQLite + Redis (default)\n[dependencies]\nlitellm-rs = \"0.4\"\n\n# API-only - lightweight, no actix-web/argon2/aes-gcm/clap\n[dependencies]\nlitellm-rs = { version = \"0.4\", default-features = false }\n\n# API-only with metrics\n[dependencies]\nlitellm-rs = { version = \"0.4\", default-features = false, features = [\"lite\"] }\n\n# Gateway modules in library context (not standalone gateway binary runtime)\n[dependencies]\nlitellm-rs = { version = \"0.4\", default-features = false, features = [\"gateway\"] }\n```\n\n## Supported Providers\n\n| Provider | Chat | Embeddings | Images | Audio |\n|----------|------|------------|--------|-------|\n| OpenAI | ✅ | ✅ | ✅ | ✅ |\n| Anthropic | ✅ | - | - | - |\n| Google (Gemini) | ✅ | ✅ | ✅ | - |\n| Azure OpenAI | ✅ | ✅ | ✅ | ✅ |\n| AWS Bedrock | ✅ | ✅ | - | - |\n| Google Vertex AI | ✅ | ✅ | ✅ | - |\n| Groq | ✅ | - | - | ✅ |\n| DeepSeek | ✅ | - | - | - |\n| Kimi (Moonshot AI) | ✅ | - | - | - |\n| GLM (Zhipu AI) | ✅ | - | - | - |\n| MiniMax | ✅ | - | - | - |\n| Mistral | ✅ | ✅ | - | - |\n| Cohere | ✅ | ✅ | - | - |\n| OpenRouter | ✅ | - | - | - |\n| Together AI | ✅ | ✅ | - | - |\n| Fireworks AI | ✅ | ✅ | - | - |\n| Perplexity | ✅ | - | - | - |\n| Replicate | ✅ | - | ✅ | - |\n| Hugging Face | ✅ | ✅ | - | - |\n| Ollama | ✅ | ✅ | - | - |\n| And 80+ more... | | | | |\n\n## Environment Variables\n\n```bash\n# Provider API Keys\nOPENAI_API_KEY=sk-...\nANTHROPIC_API_KEY=sk-ant-...\nGOOGLE_API_KEY=...\nAZURE_OPENAI_API_KEY=...\nAWS_ACCESS_KEY_ID=...\nAWS_SECRET_ACCESS_KEY=...\nGROQ_API_KEY=...\nDEEPSEEK_API_KEY=...\nMOONSHOT_API_KEY=...\nZHIPU_API_KEY=...\nMINIMAX_API_KEY=...\n\n# Optional\nLITELLM_VERBOSE=true  # Enable verbose logging\n```\n\n## Examples\n\n### Multi-Provider Routing\n\n```rust\nuse litellm_rs::{completion, user_message};\n\n// Automatically routes to the right provider based on model name\nlet openai = completion(\"gpt-4\", vec![user_message(\"Hi\")], None).await?;\nlet anthropic = completion(\"anthropic/claude-3-opus\", vec![user_message(\"Hi\")], None).await?;\nlet google = completion(\"gemini/gemini-pro\", vec![user_message(\"Hi\")], None).await?;\nlet bedrock = completion(\n    \"bedrock/us.anthropic.claude-3-5-sonnet-20241022-v2:0\",\n    vec![user_message(\"Hi\")],\n    None,\n)\n.await?;\n```\n\n### Embeddings\n\n```rust\nuse litellm_rs::{embedding, embed_text};\n\n// Single text\nlet embedding = embed_text(\"text-embedding-3-small\", \"Hello world\").await?;\n\n// Batch\nlet embeddings = embedding(\n    \"text-embedding-3-small\",\n    vec![\"Hello\", \"World\"],\n    None,\n).await?;\n```\n\n### Streaming\n\n```rust\nuse litellm_rs::{completion_stream, user_message};\nuse futures::StreamExt;\n\nlet mut stream = completion_stream(\n    \"gpt-4\",\n    vec![user_message(\"Tell me a story\")],\n    None,\n).await?;\n\nwhile let Some(chunk) = stream.next().await {\n    if let Ok(chunk) = chunk {\n        print!(\"{}\", chunk.choices[0].delta.content.unwrap_or_default());\n    }\n}\n```\n\n## Performance\n\n- **Throughput**: 10,000+ requests/second\n- **Latency**: \u003c10ms routing overhead\n- **Memory**: ~50MB base footprint\n- **Concurrency**: Fully async with Tokio\n\n## Troubleshooting\n\n### Build/test uses too much CPU or memory\n\n- Use API-only defaults first: `cargo test --lib --tests --no-default-features --features \"lite\"`\n- Limit local parallelism when needed: `CARGO_BUILD_JOBS=4 cargo test --lib --tests --no-default-features --features \"lite\" -- --test-threads=4`\n- Avoid `--all-features` unless you are doing release/nightly validation\n\n### I only need provider API aggregation, not gateway\n\n- Prefer `default-features = false` with `features = [\"lite\"]`\n- Use gateway runtime commands only when you need HTTP server/auth/storage middleware\n\n## Documentation\n\n- [API Documentation](https://docs.rs/litellm-rs)\n- [Documentation Index](./docs/README.md)\n- [Configuration Guide](./config/gateway.yaml.example)\n- [Examples](./examples/README.md)\n\n## Contributing\n\nSee [CONTRIBUTING.md](./CONTRIBUTING.md) for development setup and guidelines.\n\n## Security\n\nSee [SECURITY.md](./SECURITY.md) for security policy and vulnerability reporting.\n\n## License\n\nMIT License - see [LICENSE](./LICENSE) for details.\n\n## Acknowledgments\n\nInspired by [LiteLLM](https://github.com/BerriAI/litellm) (Python).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmajiayu000%2Flitellm-rs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmajiayu000%2Flitellm-rs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmajiayu000%2Flitellm-rs/lists"}