https://github.com/xberg-io/liter-llm
Universal LLM API client — 142+ providers, 11 native language bindings, powered by Rust core
https://github.com/xberg-io/liter-llm
anthropic api-client llm machine-learning openai polyglot python rust streaming typescript
Last synced: about 8 hours ago
JSON representation
Universal LLM API client — 142+ providers, 11 native language bindings, powered by Rust core
- Host: GitHub
- URL: https://github.com/xberg-io/liter-llm
- Owner: xberg-io
- License: mit
- Created: 2026-03-25T07:30:37.000Z (3 months ago)
- Default Branch: main
- Last Pushed: 2026-07-02T16:45:33.000Z (3 days ago)
- Last Synced: 2026-07-02T17:32:01.322Z (3 days ago)
- Topics: anthropic, api-client, llm, machine-learning, openai, polyglot, python, rust, streaming, typescript
- Language: Rust
- Homepage: https://docs.liter-llm.xberg.io
- Size: 424 MB
- Stars: 222
- Watchers: 1
- Forks: 15
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Citation: CITATION.cff
- Codeowners: .github/CODEOWNERS
- Security: SECURITY.md
Awesome Lists containing this project
- fucking-awesome-rust - xberg-io/liter-llm - llm](crates.io/crates/liter-llm)] - Universal LLM API client for 142+ providers with a unified interface, streaming, and native bindings for 11 languages. (Libraries / Artificial Intelligence)
README
# liter-llm
**A lighter, faster, safer universal LLM API client** — one Rust core, 14 native language bindings, 143 providers.
## What and Why?
liter-llm is a universal LLM API client compiled from the ground up in Rust: one core, 14 native language bindings, and 143 providers. No interpreter, no transitive dependency tree, no supply-chain surface area — and a drop-in OpenAI-compatible proxy plus an MCP server in a single 35 MB binary.
- **Compiled Rust core** — no `pip install` supply chain, no `.pth` auto-execution hooks, no runtime dependency tree to compromise.
- **Secrets stay secret** — API keys are wrapped in [`secrecy::SecretString`](https://docs.rs/secrecy/): zeroed on drop, redacted in logs, never serialized.
- **Polyglot from day one** — Python, TypeScript, Go, Java, Kotlin, C#, Ruby, PHP, Elixir, Dart, Swift, Zig, and WebAssembly, all thin wrappers over the same Rust core, plus a C/FFI surface for everything else.
- **Observability built in** — production-grade [OpenTelemetry](https://opentelemetry.io/) with GenAI semantic conventions, not an afterthought callback system.
- **Composable middleware** — rate limiting, caching, cost tracking, health checks, and fallback as [Tower](https://docs.rs/tower/) layers you stack like building blocks.
We credit [litellm](https://github.com/BerriAI/litellm) for proving the category; our provider registry was bootstrapped from theirs. See [ATTRIBUTIONS.md](ATTRIBUTIONS.md).
### Features
| Feature | Description |
| ------- | ----------- |
| **143 providers** | OpenAI, Anthropic, Google, AWS Bedrock, Groq, Mistral, Together, Fireworks, DeepSeek, Cohere, and 130+ more — compiled at build time |
| **14 native bindings** | Rust, Python, Node.js, Go, Java, Kotlin, C#, Ruby, PHP, Elixir, Dart, Swift, Zig, WebAssembly — plus a shared C/FFI surface |
| **First-class streaming** | SSE and AWS EventStream binary protocol with zero-copy buffers |
| **Proxy & MCP server** | Drop-in OpenAI-compatible proxy (22 endpoints) and MCP tool server in a 35 MB Docker image |
| **Tower middleware** | Rate limiting, caching (40+ OpenDAL backends), cost tracking, budget enforcement, health checks, and fallback — all composable |
| **Observability** | OpenTelemetry with GenAI semantic conventions, cost-tracking spans, and HTTP-level tracing |
| **Tool calling** | Parallel tools, structured outputs, and JSON-schema validation |
| **Search & OCR** | Web search across 12 providers, document OCR across 4 |
| **TOML configuration** | `liter-llm.toml` auto-discovery, custom providers, cache backends, and middleware config |
| **Local LLM support** | Ollama, LM Studio, vLLM, llama.cpp, LocalAI, and llamafile via OpenAI-compatible APIs |
⭐ Star this repo to show your support — it helps others discover liter-llm.
## Quick Start
### Language Packages
Python
```sh
pip install liter-llm
```
See [Python README](packages/python/README.md) for full documentation.
Node.js
```sh
pnpm add @xberg-io/liter-llm
```
See [Node.js README](crates/liter-llm-node/README.md) for full documentation.
Rust
```sh
cargo add liter-llm
```
See [Rust crate](crates/liter-llm) for full documentation.
Go
```sh
go get github.com/xberg-io/liter-llm/packages/go
```
See [Go README](packages/go/README.md) for full documentation.
Java
Available on Maven Central as `io.xberg.literllm:liter-llm`. See [Java README](packages/java/README.md) for the dependency snippet and current version.
Ruby
```sh
gem install liter_llm
```
See [Ruby README](packages/ruby/README.md) for full documentation.
PHP
```sh
composer require xberg-io/liter-llm
```
See [PHP README](packages/php/README.md) for full documentation.
C#
```sh
dotnet add package LiterLlm
```
See [.NET README](packages/csharp/README.md) for full documentation.
Elixir
Add `{:liter_llm, "~> 1.6"}` to your `mix.exs` dependencies. See [Elixir README](packages/elixir/README.md) for full documentation.
Dart / Flutter
```sh
dart pub add liter_llm
```
See [Dart README](packages/dart/README.md) for full documentation.
Swift
Add via Swift Package Manager. See [Swift README](packages/swift/README.md) for full documentation.
Kotlin (Android)
Available on Maven Central as `io.xberg.literllm:liter-llm-android`. See [Kotlin README](packages/kotlin-android/README.md) for the dependency snippet and current version.
Zig
See [Zig README](packages/zig/README.md) for installation and usage.
WebAssembly
```sh
pnpm add @xberg-io/liter-llm-wasm
```
See [WebAssembly README](crates/liter-llm-wasm/README.md) for full documentation.
C/C++ (FFI)
Build from source as part of this workspace. See [FFI crate](crates/liter-llm-ffi) for full documentation.
CLI, Proxy & MCP Server
The `liter-llm` CLI ships both the OpenAI-compatible proxy and the MCP tool server. Install it any of these ways:
```sh
brew install xberg-io/tap/liter-llm
cargo install liter-llm-cli # from crates.io
npx @xberg-io/liter-llm-cli --help # npm (self-installs the binary)
docker run -p 4000:4000 -e LITER_LLM_MASTER_KEY=sk-your-key ghcr.io/xberg-io/liter-llm
```
Then run the proxy or the MCP server:
```sh
liter-llm api --config liter-llm-proxy.toml # OpenAI-compatible proxy (22 endpoints)
liter-llm mcp --transport stdio # MCP tool server (stdio)
liter-llm mcp --transport http --port 3001 # MCP tool server (Streamable HTTP)
```
See the [MCP server guide](https://docs.liter-llm.xberg.io/server/mcp-server/) and the [proxy guide](https://docs.liter-llm.xberg.io/server/proxy-server/) for transports, routing, virtual keys, and budgets. To use the MCP server inside a coding agent, install the **liter-llm plugin** (below) — it auto-registers the server, no manual config required.
### AI Coding Assistants
Install the liter-llm plugin from the [`xberg-io/plugins`](https://github.com/xberg-io/plugins) marketplace. It ships the liter-llm agent skills (chat, streaming, tools, embeddings across 143 providers) and works with every major coding agent — expand your harness below.
Claude Code
```text
/plugin marketplace add xberg-io/plugins
/plugin install liter-llm@xberg
```
Codex CLI
```text
/plugins add https://github.com/xberg-io/plugins
```
Then search for `liter-llm` and select **Install Plugin**.
Cursor
Settings → Plugins → Add from URL → `https://github.com/xberg-io/plugins`, then select **liter-llm**.
Gemini CLI
```text
gemini extensions install https://github.com/xberg-io/plugins
```
Factory Droid
```text
droid plugin marketplace add https://github.com/xberg-io/plugins
droid plugin install liter-llm@xberg
```
GitHub Copilot CLI
```text
copilot plugin marketplace add https://github.com/xberg-io/plugins
copilot plugin install liter-llm@xberg
```
opencode
Not yet published as an opencode package. Install via any harness above (self-hosted marketplace); opencode support is tracked in [`xberg-io/plugins`](https://github.com/xberg-io/plugins).
## Documentation
Full guides, the unified `chat()` API for every binding, multimodal I/O, the proxy/gateway, and the complete provider list live at **[docs.liter-llm.xberg.io](https://docs.liter-llm.xberg.io)**.
## Contributing
Contributions are welcome! See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
Join our [Discord community](https://discord.gg/xt9WY3GnKR) for questions and discussion.
## Part of Xberg.io
- [Xberg](https://github.com/xberg-io/xberg) — document intelligence: text, tables, metadata from 91+ formats with optional OCR.
- [Xberg Enterprise](https://github.com/xberg-io/xberg-enterprise) — managed extraction API with SDKs, dashboards, and observability.
- [crawlberg](https://github.com/xberg-io/crawlberg) — web crawling and scraping with HTML→Markdown and headless-Chrome fallback.
- [html-to-markdown](https://github.com/xberg-io/html-to-markdown) — fast, lossless HTML→Markdown engine.
- [liter-llm](https://github.com/xberg-io/liter-llm) — universal LLM API client with native bindings for 14 languages and 143 providers.
- [tree-sitter-language-pack](https://github.com/xberg-io/tree-sitter-language-pack) — tree-sitter grammars and code-intelligence primitives.
- [alef](https://github.com/xberg-io/alef) — the polyglot binding generator that produces every per-language binding across the 5 polyglot repos.
## License
MIT — see [LICENSE](LICENSE) for details.