{"id":47691129,"url":"https://github.com/kreuzberg-dev/liter-llm","last_synced_at":"2026-04-28T12:05:04.204Z","repository":{"id":347292817,"uuid":"1191384809","full_name":"kreuzberg-dev/liter-llm","owner":"kreuzberg-dev","description":"Universal LLM API client — 142+ providers, 11 native language bindings, powered by Rust core","archived":false,"fork":false,"pushed_at":"2026-04-24T16:31:22.000Z","size":12101,"stargazers_count":153,"open_issues_count":13,"forks_count":9,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-04-24T18:10:07.621Z","etag":null,"topics":["anthropic","api-client","llm","machine-learning","openai","polyglot","python","rust","streaming","typescript"],"latest_commit_sha":null,"homepage":"https://kreuzberg.dev","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kreuzberg-dev.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-25T07:30:37.000Z","updated_at":"2026-04-24T17:21:28.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/kreuzberg-dev/liter-llm","commit_stats":null,"previous_names":["kreuzberg-dev/liter-llm"],"tags_count":39,"template":false,"template_full_name":null,"purl":"pkg:github/kreuzberg-dev/liter-llm","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kreuzberg-dev%2Fliter-llm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kreuzberg-dev%2Fliter-llm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kreuzberg-dev%2Fliter-llm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kreuzberg-dev%2Fliter-llm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kreuzberg-dev","download_url":"https://codeload.github.com/kreuzberg-dev/liter-llm/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kreuzberg-dev%2Fliter-llm/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32379632,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-28T11:25:28.583Z","status":"ssl_error","status_checked_at":"2026-04-28T11:25:05.435Z","response_time":56,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["anthropic","api-client","llm","machine-learning","openai","polyglot","python","rust","streaming","typescript"],"created_at":"2026-04-02T15:38:36.474Z","updated_at":"2026-04-28T12:05:04.197Z","avatar_url":"https://github.com/kreuzberg-dev.png","language":"Rust","funding_links":[],"categories":["llm","Rust"],"sub_categories":[],"readme":"# liter-llm\n\n\u003cdiv align=\"center\" style=\"display: flex; flex-wrap: wrap; gap: 8px; justify-content: center; margin: 20px 0;\"\u003e\n  \u003c!-- Language Bindings --\u003e\n  \u003ca href=\"https://crates.io/crates/liter-llm\"\u003e\n    \u003cimg src=\"https://img.shields.io/crates/v/liter-llm?label=Rust\u0026color=007ec6\" alt=\"Rust\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://pypi.org/project/liter-llm/\"\u003e\n    \u003cimg src=\"https://img.shields.io/pypi/v/liter-llm?label=Python\u0026color=007ec6\" alt=\"Python\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://www.npmjs.com/package/@kreuzberg/liter-llm\"\u003e\n    \u003cimg src=\"https://img.shields.io/npm/v/@kreuzberg/liter-llm?label=Node.js\u0026color=007ec6\" alt=\"Node.js\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://www.npmjs.com/package/@kreuzberg/liter-llm-wasm\"\u003e\n    \u003cimg src=\"https://img.shields.io/npm/v/@kreuzberg/liter-llm-wasm?label=WASM\u0026color=007ec6\" alt=\"WASM\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://central.sonatype.com/artifact/dev.kreuzberg/liter-llm\"\u003e\n    \u003cimg src=\"https://img.shields.io/maven-central/v/dev.kreuzberg/liter-llm?label=Java\u0026color=007ec6\" alt=\"Java\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://github.com/kreuzberg-dev/liter-llm/tree/main/packages/go\"\u003e\n    \u003cimg src=\"https://img.shields.io/github/v/tag/kreuzberg-dev/liter-llm?label=Go\u0026color=007ec6\" alt=\"Go\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://www.nuget.org/packages/LiterLlm\"\u003e\n    \u003cimg src=\"https://img.shields.io/nuget/v/LiterLlm?label=C%23\u0026color=007ec6\" alt=\"C#\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://packagist.org/packages/kreuzberg/liter-llm\"\u003e\n    \u003cimg src=\"https://img.shields.io/packagist/v/kreuzberg/liter-llm?label=PHP\u0026color=007ec6\" alt=\"PHP\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://rubygems.org/gems/liter_llm\"\u003e\n    \u003cimg src=\"https://img.shields.io/gem/v/liter_llm?label=Ruby\u0026color=007ec6\" alt=\"Ruby\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://hex.pm/packages/liter_llm\"\u003e\n    \u003cimg src=\"https://img.shields.io/hexpm/v/liter_llm?label=Elixir\u0026color=007ec6\" alt=\"Elixir\"\u003e\n  \u003c/a\u003e\n\n  \u003ca href=\"https://github.com/kreuzberg-dev/liter-llm/tree/main/crates/liter-llm-ffi\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/C-FFI-007ec6\" alt=\"C FFI\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://github.com/kreuzberg-dev/liter-llm/pkgs/container/liter-llm\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/Docker-ghcr.io-007ec6?logo=docker\u0026logoColor=white\" alt=\"Docker\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://github.com/kreuzberg-dev/homebrew-tap/blob/main/Formula/liter-llm.rb\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/Homebrew-007ec6?logo=homebrew\u0026logoColor=white\" alt=\"Homebrew\"\u003e\n  \u003c/a\u003e\n\n  \u003c!-- Project Info --\u003e\n  \u003ca href=\"https://github.com/kreuzberg-dev/liter-llm/blob/main/LICENSE\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/License-MIT-007ec6.svg\" alt=\"License\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://docs.liter-llm.kreuzberg.dev\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/docs-kreuzberg.dev-007ec6\" alt=\"Docs\"\u003e\n  \u003c/a\u003e\n\u003c/div\u003e\n\n\u003cimg width=\"3384\" height=\"573\" alt=\"kreuzberg.dev\" src=\"https://github.com/user-attachments/assets/1b6c6ad7-3b6d-4171-b1c9-f2026cc9deb8\" /\u003e\n\n\u003cdiv align=\"center\" style=\"margin-top: 20px;\"\u003e\n  \u003ca href=\"https://discord.gg/xt9WY3GnKR\"\u003e\n    \u003cimg height=\"22\" src=\"https://img.shields.io/badge/Discord-Join%20our%20community-7289da?logo=discord\u0026logoColor=white\" alt=\"Discord\"\u003e\n  \u003c/a\u003e\n\u003c/div\u003e\n\n**A lighter, faster, safer universal LLM API client** -- one Rust core, 11 native language bindings, 143 providers.\n\n## Why liter-llm?\n\nA universal LLM API client, compiled from the ground up in Rust. No interpreter, no transitive dependency tree, no supply chain surface area. One binary, 11 native language bindings, 143 providers.\n\n- **Compiled Rust core.** No `pip install` supply chain. No `.pth` auto-execution hooks. No runtime dependency tree to compromise. The kind of [supply chain attack that hit litellm](https://www.xda-developers.com/popular-python-library-backdoor-machine/) in 2026 is structurally impossible here.\n- **Secrets stay secret.** API keys are wrapped in [`secrecy::SecretString`](https://docs.rs/secrecy/) -- zeroed on drop, redacted in logs, never serialized.\n- **Polyglot from day one.** Python, TypeScript, Go, Java, Ruby, PHP, C#, Elixir, WebAssembly, C/FFI -- all thin wrappers around the same Rust core. No reimplementation drift.\n- **Observability built in.** Production-grade [OpenTelemetry](https://opentelemetry.io/) with GenAI semantic conventions -- not an afterthought callback system.\n- **Composable middleware.** Rate limiting, caching, cost tracking, health checks, and fallback as [Tower](https://docs.rs/tower/) layers you stack like building blocks.\n\nWe give credit to [litellm](https://github.com/BerriAI/litellm) for proving the category -- our provider registry was bootstrapped from theirs. See [ATTRIBUTIONS.md](ATTRIBUTIONS.md).\n\n## Feature Comparison\n\nAn honest look at where things stand. We're newer and leaner -- litellm has breadth we haven't matched yet, and we have depth they can't easily retrofit.\n\n| | liter-llm | litellm |\n|---|---|---|\n| **Language** | Rust (compiled, memory-safe) | Python |\n| **Bindings** | 11 native (Rust, Python, TS, Go, Java, Ruby, PHP, C#, Elixir, WASM, C) | Python (+ OpenAI-compatible proxy) |\n| **Providers** | 143 (compiled at build time) | 100+ (runtime resolution) |\n| **Streaming** | SSE + AWS EventStream binary protocol | SSE + AWS EventStream |\n| **Observability** | Built-in OpenTelemetry (GenAI semconv) | 40+ callback integrations |\n| **API key safety** | `secrecy::SecretString` (zeroed, redacted) | Plain strings |\n| **Middleware** | Composable Tower stack | Built-in callback system |\n| **Proxy / Gateway** | Yes (22 OpenAI-compatible endpoints, 35MB Docker) | Yes |\n| **Guardrails** | -- | 10+ integrations, 4 execution modes (advanced: enterprise) |\n| **Semantic caching** | -- | Redis + Qdrant backends |\n| **Virtual key mgmt** | Yes (per-key model restrictions, RPM/TPM, budgets) | Yes (key rotation: enterprise) |\n| **Management API** | Config-driven (REST admin API planned) | Multi-tenant (teams, budgets, keys; tiers + reporting: enterprise) |\n| **Fine-tuning API** | -- | Enterprise only |\n| **Load balancer** | Fallback + round-robin via Tower router | Full router with strategies |\n| **Cost tracking** | Embedded pricing + OTEL spans | Per-key/team/model budgets |\n| **Rate limiting** | Per-model RPM/TPM (Tower layer) | Per-key/user/team/model |\n| **Caching** | In-memory LRU + 40+ backends via OpenDAL (S3, Redis, GCS, DynamoDB, disk, ...) | 7 backends (Redis, S3, GCS, disk, Qdrant) |\n| **Tool calling** | Parallel tools, structured output, JSON schema | Full support |\n| **Embeddings** | Yes | Yes |\n| **Batch API** | Yes | Yes |\n| **Audio / Speech** | Yes | Yes |\n| **Lifecycle hooks** | onRequest/onResponse/onError per-client | Callback integrations |\n| **Budget enforcement** | Per-model + global limits, hard/soft modes | Per-key/team budgets |\n| **Health checks** | Automatic provider probes + cooldown | -- |\n| **Custom providers** | Runtime API + TOML config file | Config + code-based |\n| **Config files** | TOML with auto-discovery (`liter-llm.toml`) | YAML proxy config |\n| **Search / OCR** | 12 search + 4 OCR providers | Yes |\n| **Image generation** | Yes | Yes |\n\n## Key Features\n\n- **143 providers** -- OpenAI, Anthropic, Google, AWS Bedrock, Groq, Mistral, Together AI, Fireworks, Perplexity, DeepSeek, Cohere, and [130+ more](schemas/providers.json)\n- **11 native bindings** -- Rust, Python, TypeScript/Node.js, Go, Java, Ruby, PHP, C#, Elixir, WebAssembly, C/FFI\n- **First-class streaming** -- SSE and AWS EventStream binary protocol with zero-copy buffers\n- **TOML configuration** -- `liter-llm.toml` with auto-discovery, custom providers, cache backends, middleware config\n- **OpenTelemetry** -- GenAI semantic conventions, cost tracking spans, HTTP-level tracing\n- **Tower middleware** -- Rate limiting, caching (40+ OpenDAL backends), cost tracking, budget enforcement, health checks, cooldowns, hooks, fallback -- all composable\n- **Search \u0026 OCR** -- Web search across 12 providers, document OCR across 4 providers\n- **Tool calling** -- Parallel tools, structured outputs, JSON schema validation\n- **Embeddings** -- Dimension selection, base64 format, multi-provider support\n- **Per-request routing** -- Automatic provider detection from model name prefix, custom provider registration at runtime\n- **Schema-driven** -- Provider registry and API types compiled from JSON schemas, no runtime lookups\n- **Local LLM support** — Run models locally with Ollama, LM Studio, vLLM, llama.cpp, LocalAI, and llamafile via OpenAI-compatible APIs\n\n## Proxy Server \u0026 CLI\n\nDrop-in replacement for litellm's proxy -- 22 OpenAI-compatible endpoints. Install the `liter-llm` CLI (which ships both the proxy server and the MCP tool server) one of three ways:\n\n```bash\n# Homebrew (macOS / Linux)\nbrew install kreuzberg-dev/tap/liter-llm\n\n# Pre-built binary (Linux x86_64/arm64, macOS arm64, Windows x86_64)\ncurl -L https://github.com/kreuzberg-dev/liter-llm/releases/latest/download/liter-llm-${VERSION}-${TARGET}.tar.gz | tar xz\n\n# Docker (35MB image)\ndocker run -p 4000:4000 -e LITER_LLM_MASTER_KEY=sk-your-key ghcr.io/kreuzberg-dev/liter-llm\n```\n\nThen call it like OpenAI:\n\n```bash\ncurl http://localhost:4000/v1/chat/completions \\\n  -H \"Authorization: Bearer sk-your-key\" \\\n  -d '{\"model\": \"openai/gpt-4o\", \"messages\": [{\"role\": \"user\", \"content\": \"Hello\"}]}'\n```\n\nOr with a TOML config file:\n\n```toml\n# liter-llm-proxy.toml\n[general]\nmaster_key = \"${LITER_LLM_MASTER_KEY}\"\n\n[[models]]\nname = \"gpt-4o\"\nprovider_model = \"openai/gpt-4o\"\napi_key = \"${OPENAI_API_KEY}\"\n\n[[models]]\nname = \"claude-sonnet\"\nprovider_model = \"anthropic/claude-sonnet-4-20250514\"\napi_key = \"${ANTHROPIC_API_KEY}\"\n\n[[keys]]\nkey = \"sk-team-a\"\nmodels = [\"gpt-4o\"]\nrpm = 100\n```\n\n**CLI:**\n\n```bash\nliter-llm api --config liter-llm-proxy.toml    # Start proxy server\nliter-llm mcp --transport stdio                 # Start MCP tool server\n```\n\n**Features:** Model routing, virtual API keys, per-key rate limiting (RPM/TPM), cost tracking, budget enforcement, response caching, SSE streaming, OpenAPI 3.1 spec at `/openapi.json`, MCP server with 22 tools, graceful shutdown.\n\n## Architecture\n\n```text\nliter-llm/\n├── crates/\n│   ├── liter-llm/           # Rust core library\n│   ├── liter-llm-py/        # Python (PyO3) core\n│   ├── liter-llm-node/      # Node.js (NAPI-RS) core\n│   ├── liter-llm-ffi/       # C-compatible FFI layer\n│   ├── liter-llm-php/       # PHP (ext-php-rs) core\n│   └── liter-llm-wasm/      # WebAssembly (wasm-bindgen) core\n├── packages/\n│   ├── python/               # Python package\n│   ├── typescript/           # TypeScript/Node.js package\n│   ├── go/                   # Go (cgo) module\n│   ├── java/                 # Java (Panama FFI) package\n│   ├── ruby/                 # Ruby (Magnus) gem\n│   ├── elixir/               # Elixir (Rustler NIF) package\n│   ├── csharp/               # .NET (P/Invoke) package\n│   └── php/                  # PHP (Composer) package\n└── schemas/                  # Provider registry and API schemas\n```\n\n## Quick Start\n\nInstall in your language of choice:\n\n| Language | Install |\n|----------|---------|\n| Python | `pip install liter-llm` |\n| Node.js | `pnpm add @kreuzberg/liter-llm` |\n| Rust | `cargo add liter-llm` |\n| Go | `go get github.com/kreuzberg-dev/liter-llm/packages/go` |\n| Java | `dev.kreuzberg:liter-llm` (Maven/Gradle) |\n| Ruby | `gem install liter_llm` |\n| PHP | `composer require kreuzberg/liter-llm` |\n| C# | `dotnet add package LiterLlm` |\n| Elixir | `{:liter_llm, \"~\u003e 1.0\"}` in mix.exs |\n| WASM | `pnpm add @kreuzberg/liter-llm-wasm` |\n| C/FFI | Build from source -- see [FFI crate](crates/liter-llm-ffi) |\n\n### Usage\n\n```python\nimport asyncio, os\nfrom liter_llm import LlmClient\n\nasync def main():\n    client = LlmClient(api_key=os.environ[\"OPENAI_API_KEY\"])\n\n    # Chat with any provider using the provider/model prefix\n    response = await client.chat(\n        model=\"openai/gpt-4o\",\n        messages=[{\"role\": \"user\", \"content\": \"Hello!\"}],\n    )\n    print(response.choices[0].message.content)\n\n    # Switch providers by changing the prefix -- no other code changes\n    client2 = LlmClient(api_key=os.environ[\"ANTHROPIC_API_KEY\"])\n    response = await client2.chat(\n        model=\"anthropic/claude-sonnet-4-20250514\",\n        messages=[{\"role\": \"user\", \"content\": \"Hello!\"}],\n    )\n    print(response.choices[0].message.content)\n\nasyncio.run(main())\n```\n\nOr use a `liter-llm.toml` config file instead of passing everything in code:\n\n```toml\napi_key = \"sk-...\"\ntimeout_secs = 120\n\n[cache]\nmax_entries = 512\nttl_seconds = 600\nbackend = \"redis\"\nbackend_config = { connection_string = \"redis://localhost:6379\" }\n\n[budget]\nglobal_limit = 50.0\nenforcement = \"hard\"\n\n[[providers]]\nname = \"my-provider\"\nbase_url = \"https://my-llm.example.com/v1\"\nmodel_prefixes = [\"my-provider/\"]\n```\n\nThe same API is available in all 11 languages -- see the language READMEs below for idiomatic examples.\n\n## Core API\n\nAll bindings expose a unified `chat()` function:\n\n| Language | Usage |\n| -------- | ----- |\n| Rust | `DefaultClient::new(config).chat(messages, options).await` |\n| Python | `LlmClient(api_key=...).chat(messages, config)` |\n| Node.js | `new LlmClient({ apiKey }).chat(messages, config)` |\n| Go | `client.Chat(ctx, messages, config)` |\n| Java | `client.chat(messages, configJson)` |\n| Ruby | `LiterLlm::LlmClient.new(api_key, config).chat(messages)` |\n| Elixir | `LiterLlm.chat(messages, config)` |\n| PHP | `LiterLlm\\LlmClient::new($apiKey)-\u003echat($messages, $config)` |\n| C# | `new LlmClient(apiKey).ChatAsync(messages, config)` |\n| WASM | `new LlmClient({ apiKey }).chat(messages, config)` |\n| C FFI | `liter_llm_chat(client, messages_json, config_json)` |\n\n## Language READMEs\n\n| Language | README | Binding |\n| -------- | ------ | ------- |\n| Python | [packages/python](packages/python/README.md) | PyO3 |\n| TypeScript / Node.js | [crates/liter-llm-node](crates/liter-llm-node/README.md) | NAPI-RS |\n| Go | [packages/go](packages/go/README.md) | cgo |\n| Java | [packages/java](packages/java/README.md) | Panama FFI |\n| Ruby | [packages/ruby](packages/ruby/README.md) | Magnus |\n| Elixir | [packages/elixir](packages/elixir/README.md) | Rustler NIF |\n| PHP | [packages/php](packages/php/README.md) | ext-php-rs |\n| .NET (C#) | [packages/csharp](packages/csharp/README.md) | P/Invoke |\n| WebAssembly | [crates/liter-llm-wasm](crates/liter-llm-wasm/README.md) | wasm-bindgen |\n| C/C++ (FFI) | [crates/liter-llm-ffi](crates/liter-llm-ffi) | C ABI |\n\n## Part of kreuzberg.dev\n\nliter-llm is built by the [kreuzberg.dev](https://kreuzberg.dev) team -- the same people behind [Kreuzberg](https://github.com/kreuzberg-dev/kreuzberg) (document extraction for 91+ formats), [tree-sitter-language-pack](https://github.com/kreuzberg-dev/tree-sitter-language-pack) (multilingual parsing), and [html-to-markdown](https://github.com/kreuzberg-dev/html-to-markdown). All our libraries share the same Rust-core, polyglot-bindings architecture. Visit [kreuzberg.dev](https://kreuzberg.dev) or find us on [GitHub](https://github.com/kreuzberg-dev).\n\n## Contributing\n\nContributions are welcome! See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.\n\nJoin our [Discord community](https://discord.gg/xt9WY3GnKR) for questions and discussion.\n\n## License\n\nMIT -- see [LICENSE](LICENSE) for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkreuzberg-dev%2Fliter-llm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkreuzberg-dev%2Fliter-llm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkreuzberg-dev%2Fliter-llm/lists"}