https://github.com/ferro-labs/ai-gateway
Unified AI Gateway for 30+ LLMs (OpenAI, Anthropic, Bedrock, Azure etc) with Caching, Guardrails, A/B test & cost controls. Go-native Fastest & Scalable AI Gateway LiteLLM & Kong AI Gateway alternative.
https://github.com/ferro-labs/ai-gateway
ai-gateway ai-infrastructure gateway guardrails kong litellm llm llm-cost llm-proxy llm-strategy llmops mcp pii-detection prompt-management semantic-cache
Last synced: 27 days ago
JSON representation
Unified AI Gateway for 30+ LLMs (OpenAI, Anthropic, Bedrock, Azure etc) with Caching, Guardrails, A/B test & cost controls. Go-native Fastest & Scalable AI Gateway LiteLLM & Kong AI Gateway alternative.
- Host: GitHub
- URL: https://github.com/ferro-labs/ai-gateway
- Owner: ferro-labs
- License: apache-2.0
- Created: 2026-02-25T07:46:16.000Z (4 months ago)
- Default Branch: main
- Last Pushed: 2026-05-23T11:22:58.000Z (27 days ago)
- Last Synced: 2026-05-23T13:19:26.115Z (27 days ago)
- Topics: ai-gateway, ai-infrastructure, gateway, guardrails, kong, litellm, llm, llm-cost, llm-proxy, llm-strategy, llmops, mcp, pii-detection, prompt-management, semantic-cache
- Language: Go
- Homepage: https://docs.ferrolabs.ai
- Size: 2.7 MB
- Stars: 95
- Watchers: 2
- Forks: 15
- Open Issues: 22
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Security: SECURITY.md
- Roadmap: ROADMAP.md
- Agents: AGENTS.md
Awesome Lists containing this project
- Awesome-LLMOps - Ferro Labs AI Gateway - native LiteLLM & Kong AI Gateway alternative.    (Inference / LLM Router)
README
Ferro Labs AI Gateway
**High-performance AI gateway in Go. Route LLM requests across 30 providers via a single OpenAI-compatible API.**
**Deploy templates**
[](https://railway.com/deploy/ferro-labs-ai-sqlite-storage?referralCode=KblxKX&utm_medium=integration&utm_source=template&utm_campaign=generic)
[](https://railway.com/deploy/ferro-labs-ai-postgresql-storage?referralCode=KblxKX&utm_medium=integration&utm_source=template&utm_campaign=generic)
[](https://render.com/deploy?repo=https://github.com/ferro-labs/ai-gateway)
[](https://go.dev)
[](https://pkg.go.dev/github.com/ferro-labs/ai-gateway)
[](https://codecov.io/gh/ferro-labs/ai-gateway)
[](LICENSE)
[](https://github.com/ferro-labs/ai-gateway/stargazers)
[](https://github.com/ferro-labs/ai-gateway/actions/workflows/ci.yml)
[](https://github.com/ferro-labs/ai-gateway/actions/workflows/code-scanning.yml)
[](https://deepwiki.com/ferro-labs/ai-gateway)
[](https://artifacthub.io/packages/search?org=ferro-labs)
[](https://discord.gg/yCAeYvJeDV)
π **30 providers, 2,500+ models β one API**
β‘ **13,925 RPS at 1,000 concurrent users**
π¦ **Single binary, zero dependencies, 32 MB base memory**
---
## Quick Start
Get from zero to first request in under 2 minutes.
### Option A β Binary (fastest)
```bash
curl -fsSL https://github.com/ferro-labs/ai-gateway/releases/download/v1.0.6/ferrogw_1.0.6_linux_amd64.tar.gz | tar xz
chmod +x ferrogw
./ferrogw init # generates config.yaml + MASTER_KEY
./ferrogw # starts the server
```
### Option B β Docker
```bash
docker pull ghcr.io/ferro-labs/ai-gateway:latest
docker run -p 8080:8080 \
-e OPENAI_API_KEY=sk-your-key \
-e MASTER_KEY=fgw_your-master-key \
ghcr.io/ferro-labs/ai-gateway:latest
```
### Option C β Go
```bash
go install github.com/ferro-labs/ai-gateway/cmd/ferrogw@latest
ferrogw init # first-run setup
ferrogw # start the server
```
### First-time setup
`ferrogw init` generates a master key and writes a minimal `config.yaml`:
```
$ ferrogw init
Master key (set as MASTER_KEY env var):
fgw_a3f2e1d4c5b6a7f8e9d0c1b2a3f4e5d6
Config written to: ./config.yaml
Next steps:
export MASTER_KEY=fgw_a3f2e1d4c5b6a7f8e9d0c1b2a3f4e5d6
export OPENAI_API_KEY=sk-...
ferrogw
```
The master key is shown once β store it in your `.env` file or secret manager. It is never written to disk.
### Minimal config
Create `config.yaml` (or use `ferrogw init`):
```yaml
strategy:
mode: fallback
targets:
- virtual_key: openai
retry:
attempts: 3
on_status_codes: [429, 502, 503]
- virtual_key: anthropic
aliases:
fast: gpt-4o-mini
smart: claude-3-5-sonnet-20241022
```
### First request
```bash
export OPENAI_API_KEY=sk-your-key
export MASTER_KEY=fgw_your-master-key # set by ferrogw init
curl http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $MASTER_KEY" \
-d '{
"model": "gpt-4o-mini",
"messages": [{"role": "user", "content": "Hello from Ferro Labs AI Gateway"}]
}' | jq
```
---
## Why Ferro Labs
Most AI gateways are Python proxies that crack under load or JavaScript services that eat memory. Ferro Labs AI Gateway is written in Go from the ground up for real-world throughput β a single binary that routes LLM requests with predictable latency and minimal resource usage.
| Feature | Ferro Labs | LiteLLM | Bifrost | Kong AI |
|:-----------------|:------------|:--------|:-----------|:------------|
| Language | Go | Python | Go | Go/Lua |
| Single binary | β
| β | β
| β |
| Providers | 30 | 100+ | 20+ | 10+ |
| MCP support | β
| β | β
| β |
| Response cache | β
| β
| β
| β (paid) |
| Guardrails | β
| β
| β | β (paid) |
| OSS license | Apache 2.0 | MIT | Apache 2.0 | Apache 2.0 |
| Managed cloud | Coming Soon | β
| β
| β
|
---
## Performance
Benchmarked against Kong OSS, Bifrost, LiteLLM, and Portkey on
**GCP n2-standard-8** (8 vCPU, 32 GB RAM) using a **60ms fixed-latency
mock upstream** β results reflect gateway overhead only.
### Ferro Labs Latency Profile
| VU | RPS | p50 | p99 | Memory |
|---:|---:|---:|---:|---:|
| 50 | 813 | 61.3ms | 64.1ms | 36 MB |
| 150 | 2,447 | 61.2ms | 63.4ms | 47 MB |
| 300 | 4,890 | 61.2ms | 64.4ms | 72 MB |
| 500 | 8,014 | 61.5ms | 72.9ms | 89 MB |
| 1,000 | 13,925 | 68.1ms | 111.9ms | 135 MB |
At 1,000 VU: **13,925 RPS**, p50 overhead **8.1ms**, memory **135 MB**.
No connection pool failures. No throughput ceiling.
### Live Upstream Overhead (OpenAI API)
Measured against **live OpenAI API** (gpt-4o-mini) using two independent methods:
the gateway's `X-Gateway-Overhead-Ms` response header (precise internal timing)
and paired direct-vs-gateway requests (external black-box validation).
| Configuration | Overhead p50 | Overhead p99 |
|:---|---:|---:|
| No plugins (bare proxy) | **0.002ms** (2 microseconds) | 0.03ms |
| With plugins (word-filter, max-token, logger, rate-limit) | **0.025ms** (25 microseconds) | 0.074ms |
The gateway adds **25 microseconds** of processing overhead per request in a typical
production configuration. LLM API calls take 500ms-2s β the gateway is 20,000x faster
than the provider it proxies.
### How to Reproduce
```bash
git clone https://github.com/ferro-labs/ai-gateway-performance-benchmarks
cd ai-gateway-performance-benchmarks
make setup && make bench
```
Full methodology, raw results, and flamegraph analysis:
[ferro-labs/ai-gateway-performance-benchmarks](https://github.com/ferro-labs/ai-gateway-performance-benchmarks)
---
## Features
### π Routing
- **8 routing strategies:** single, fallback, load balance, least latency, cost-optimized, content-based, A/B test, conditional
- Provider failover with configurable retry policies and status code filters
- Per-request model aliases (`fast β gpt-4o-mini`, `smart β claude-3-5-sonnet`)
### π Providers (30)
| OpenAI & Compatible | Anthropic & Google | Cloud & Enterprise | Open Source & Inference |
|:---|:---|:---|:---|
| OpenAI | Anthropic | AWS Bedrock | Ollama, Ollama Cloud |
| Azure OpenAI | Google Gemini | Azure Foundry | Hugging Face |
| OpenRouter | Vertex AI | Databricks | Replicate |
| DeepSeek | | Cloudflare Workers AI | Together AI |
| Perplexity | | | Fireworks |
| xAI (Grok) | | | DeepInfra |
| Mistral | | | NVIDIA NIM |
| Groq | | | SambaNova |
| Cohere | | | Novita AI |
| AI21 | | | Cerebras |
| Moonshot / Kimi | | | Qwen / DashScope |
### π‘οΈ Guardrails & Plugins
- **Word/phrase filtering** β block sensitive terms before they reach providers
- **Token and message limits** β enforce max_tokens and max_messages per request
- **Response caching** β in-memory cache with configurable TTL and entry limits
- **Rate limiting** β global RPS plus per-API-key and per-user RPM limits
- **Budget controls** β per-API-key USD spend tracking with configurable token pricing
- **Request logging** β structured logs with optional SQLite/PostgreSQL persistence
### β‘ Performance
- Per-provider HTTP connection pools with optimized settings
- `sync.Pool` for JSON marshaling buffers and streaming I/O
- Zero-allocation stream detection, async hook dispatch batching
- Single binary, ~32 MB base memory, linear scaling to 1,000+ VUs
### π€ MCP (Model Context Protocol)
- Agentic tool-call loop β the gateway drives `tool_calls` automatically
- Streamable HTTP transport (MCP 2025-11-25 spec)
- Tool filtering with `allowed_tools` and bounded `max_call_depth`
- Multiple MCP servers with cross-server tool deduplication
### π Observability
- **OpenTelemetry tracing** (v1.1.0+) β OTLP gRPC/HTTP exporter, W3C `traceparent` propagation, GenAI semantic conventions (`gen_ai.*`) plus `ferro.*` extensions for cost, routing, MCP, and stream timings; `privacy_level` enforced on error recording; configurable `shutdown_grace`
- Prometheus metrics at `/metrics`
- Deep health checks at `/health` with per-provider status
- Structured JSON request logging with SQLite/PostgreSQL persistence (trace ID unified across logs, OTel spans, and `X-Request-ID` response header)
- Admin API with usage stats, request logs, and config history/rollback
- Built-in dashboard UI at `/dashboard`
- HTTP-level connection tracing with DNS, TLS, and first-byte latency
---
## Examples
Integration examples for common use cases are in [ferro-labs/ai-gateway-examples](https://github.com/ferro-labs/ai-gateway-examples):
| Example | Description |
|:--------|:------------|
| [basic](https://github.com/ferro-labs/ai-gateway-examples/tree/main/basic) | Single chat completion to the first configured provider |
| [fallback](https://github.com/ferro-labs/ai-gateway-examples/tree/main/fallback) | Fallback strategy β try providers in order with retries |
| [loadbalance](https://github.com/ferro-labs/ai-gateway-examples/tree/main/loadbalance) | Weighted load balancing across targets (70/30 split) |
| [with-guardrails](https://github.com/ferro-labs/ai-gateway-examples/tree/main/with-guardrails) | Built-in word-filter and max-token guardrail plugins |
| [with-mcp](https://github.com/ferro-labs/ai-gateway-examples/tree/main/with-mcp) | Local MCP server with tool-calling integration |
| [embedded](https://github.com/ferro-labs/ai-gateway-examples/tree/main/embedded) | Embed the gateway as an HTTP handler inside an existing server |
---
## Configuration
Full annotated example β copy to `config.yaml` and customize:
```yaml
# Routing strategy
strategy:
mode: fallback # single | fallback | loadbalance | conditional
# least-latency | cost-optimized | content-based | ab-test
# Provider targets (tried in order for fallback mode)
targets:
- virtual_key: openai
retry:
attempts: 3
on_status_codes: [429, 502, 503]
initial_backoff_ms: 100
- virtual_key: anthropic
retry:
attempts: 2
- virtual_key: gemini
# Model aliases β resolved before routing
aliases:
fast: gpt-4o-mini
smart: claude-3-5-sonnet-20241022
cheap: gemini-1.5-flash
# Plugins β executed in order at the configured stage
plugins:
- name: word-filter
type: guardrail
stage: before_request
enabled: true
config:
blocked_words: ["password", "secret"]
case_sensitive: false
- name: max-token
type: guardrail
stage: before_request
enabled: true
config:
max_tokens: 4096
max_messages: 50
- name: rate-limit
type: guardrail
stage: before_request
enabled: true
config:
requests_per_second: 100
key_rpm: 60
- name: request-logger
type: logging
stage: before_request
enabled: true
config:
level: info
persist: true
backend: sqlite
dsn: ferrogw-requests.db
# MCP tool servers (optional)
mcp_servers:
- name: my-tools
url: https://mcp.example.com/mcp
headers:
Authorization: Bearer ${MY_TOOLS_TOKEN}
allowed_tools: [search, get_weather]
max_call_depth: 5
timeout_seconds: 30
```
See [config.example.yaml](config.example.yaml) and [config.example.json](config.example.json) for the full template with all options.
---
## Observability
Ferro Labs AI Gateway ships first-class **OpenTelemetry** support in v1.1.0+. When OTel is disabled (the default) the gateway runs with a zero-allocation no-op provider β there is no cost to leaving it off. When you set an OTLP endpoint, every request emits a `gateway.request` root span with rich GenAI semantic conventions plus Ferro-specific extensions for cost, routing, and stream timings.
### Enable in one step
Either set the standard OTel environment variable:
```bash
export OTEL_EXPORTER_OTLP_ENDPOINT=localhost:4317
ferrogw serve
```
β¦or add an `observability` block to `config.yaml`:
```yaml
observability:
tracing:
enabled: true
endpoint: localhost:4317 # or leave blank to read OTEL_EXPORTER_OTLP_ENDPOINT
protocol: grpc # grpc | http/protobuf
service_name: ferrogw
sample_ratio: 1.0
privacy_level: metadata # none | metadata | full (see below)
shutdown_grace: 10s # max time to drain OTel exports on shutdown
# headers: # OTLP export headers for authenticated backends
# dd-api-key: "${DATADOG_API_KEY}" # values support ${ENV_VAR} interpolation
# exporters wires plugin observability exporters (see "Plugin exporters" below).
# exporters:
# - name: langsmith
# enabled: true
# config:
# api_key: "${LANGSMITH_API_KEY}"
```
Standard `OTEL_*` environment variables (e.g. `OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_TRACES_SAMPLER`) always take precedence over the config file β this matches the OTel SDK convention and is required for predictable container deployments.
`observability.tracing.headers` lets you send OTLP traces to authenticated managed backends (Datadog, New Relic, Honeycomb, Grafana Cloud) by setting vendor-specific headers such as API keys. Values support `${ENV_VAR}` interpolation so secrets are never stored literally in the config file. The standard `OTEL_EXPORTER_OTLP_HEADERS` environment variable also applies per OTel convention.
The **endpoint scheme selects transport security**: an `https://` endpoint uses TLS, while an `http://` endpoint or a bare `host:port` (e.g. `localhost:4317`) connects in plaintext. Managed backends require the `https://` form.
### What gets emitted
The following attributes are **currently emitted** on the `gateway.request` root span. Attributes marked "Planned" are reserved but not yet wired.
- **`gateway.request`** root span per request (`SERVER` kind) with `gen_ai.system`, `gen_ai.operation.name`, `gen_ai.request.model`, `gen_ai.response.model`, `gen_ai.usage.{input,output}_tokens`
- **`HTTP {GET,POST}`** child span per outbound provider call (`CLIENT` kind, via `otelhttp` transport wrapping) β propagates `traceparent` to upstream providers
- **`ferro.*` emitted attributes**: `ferro.cost.{usd,input_usd,output_usd,cache_read_usd,cache_write_usd,reasoning_usd,model_found}`, `ferro.routing.{strategy,target_key}`, `ferro.stream.time_to_{first,last}_token_ms`, `ferro.gateway.trace_id`, `ferro.plugin.{name,kind,stage,outcome,reason}`, `ferro.mcp.{server,tool,latency_ms}`
- **W3C TraceContext + Baggage** propagation: inbound `traceparent` is honoured; outbound requests carry it forward
- **Unified trace ID**: the OTel `trace_id`, the `X-Request-ID` response header, and the `trace_id` field on every log line are guaranteed equal per request for all requests served through the gateway's HTTP stack. (Embedders that bypass `logging.Middleware` receive a consistent-but-independent span trace ID.)
### Try it locally with Jaeger
```bash
docker run --rm -p 16686:16686 -p 4317:4317 jaegertracing/all-in-one
OTEL_EXPORTER_OTLP_ENDPOINT=localhost:4317 ferrogw serve
# fire a request, then open http://localhost:16686
```
### Privacy levels
`privacy_level` controls how error messages are recorded on spans. No prompt or response content is exported at any level β that requires a future L3 exporter plugin.
| Level | Error recording on spans | Default |
|:------|:------|:------|
| `none` | Status and exception carry only the static string `"redacted"` β no content or internal type exposed | β |
| `metadata` | Error message is redacted (email / JWT / AWS keys replaced by tokens) before being attached | β
|
| `full` | Raw error text recorded without redaction β for trusted self-hosted debugging only | β |
Invalid values are rejected at startup by config validation.
### Plugin exporters
The `observability.exporters` config block wires plugin exporters that receive `gateway.request.completed` and `gateway.request.failed` events on every request. Exporters operate independently of whether an OTLP tracing endpoint is configured.
**No built-in exporter plugins ship in this repo.** They are provided by the `ai-gateway-plugins` repository and self-register via `observability.RegisterExporter` in their `init()`. The `observability.Exporter` contract is stable as of v1.1.0. Unrecognised or failing exporters emit a warning and are skipped β the gateway still starts.
---
## CLI
`ferrogw` is a single binary β no separate CLI tool required.
| Command | Description |
|:--------|:------------|
| `ferrogw` | Start the gateway server (default) |
| `ferrogw serve` | Start the gateway server (explicit) |
| `ferrogw init` | First-run setup β generate master key and config |
| `ferrogw validate` | Validate a config file without starting |
| `ferrogw doctor` | Check environment (API keys, config, connectivity) |
| `ferrogw status` | Show gateway health and provider status |
| `ferrogw version` | Print version, commit, and build info |
| `ferrogw admin keys list` | List API keys |
| `ferrogw admin keys create ` | Create an API key |
| `ferrogw admin logs stats` | Show request log statistics |
| `ferrogw plugins` | List registered plugins |
Global flags available on all subcommands: `--gateway-url`, `--api-key`, `--format` (table/json/yaml).
---
## Deployment
### Local development
```bash
export OPENAI_API_KEY=sk-your-key
export MASTER_KEY=fgw_your-master-key
export GATEWAY_CONFIG=./config.yaml
make build && ./bin/ferrogw
```
### Railway (SQLite)
For a fast Railway deploy with persistent SQLite storage, attach a Railway Volume at `/data` and set:
```bash
MASTER_KEY=fgw_your-master-key
OPENAI_API_KEY=sk-your-key
PORT=8080
API_KEY_STORE_BACKEND=sqlite
API_KEY_STORE_DSN=/data/keys.db
CONFIG_STORE_BACKEND=sqlite
CONFIG_STORE_DSN=/data/config.db
REQUEST_LOG_STORE_BACKEND=sqlite
REQUEST_LOG_STORE_DSN=/data/logs.db
RAILWAY_RUN_UID=0
```
### Render (PostgreSQL)
The repo includes a `render.yaml` Blueprint for a one-click Render deploy with a Docker web service and managed Postgres database. It generates `MASTER_KEY`, asks the user for `OPENAI_API_KEY`, and wires the three store DSNs to the database's internal connection string automatically.
Use the button at the top of this README, or deploy directly from:
```text
https://render.com/deploy?repo=https://github.com/ferro-labs/ai-gateway
```
### Option D β Docker Compose (dev & prod)
The repo ships three Compose files that follow the standard override pattern:
| File | Purpose |
|---|---|
| `docker-compose.yml` | Base β shared image, port mapping, all provider env var stubs |
| `docker-compose.dev.yml` | Dev β builds from source, debug logging, live config mount, Ollama host access |
| `docker-compose.prod.yml` | Prod β pinned image tag, restart policy, health check, resource limits, log rotation |
**Dev** (builds from source):
```bash
docker compose -f docker-compose.yml -f docker-compose.dev.yml up
```
**Prod** (pin to a release tag β never use `latest` in production):
```bash
IMAGE_TAG=v1.0.6 CORS_ORIGINS=https://your-domain.com \
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d
```
Provider API keys are commented out in `docker-compose.yml`. Uncomment and set the ones you need, or supply them via a `.env` file in the same directory.
---
### Docker Compose (with PostgreSQL)
```yaml
services:
ferrogw:
image: ghcr.io/ferro-labs/ai-gateway:latest
ports:
- "8080:8080"
environment:
- OPENAI_API_KEY=${OPENAI_API_KEY}
- GATEWAY_CONFIG=/etc/ferrogw/config.yaml
- CONFIG_STORE_BACKEND=postgres
- CONFIG_STORE_DSN=postgresql://ferrogw:ferrogw@db:5432/ferrogw?sslmode=disable
- API_KEY_STORE_BACKEND=postgres
- API_KEY_STORE_DSN=postgresql://ferrogw:ferrogw@db:5432/ferrogw?sslmode=disable
- REQUEST_LOG_STORE_BACKEND=postgres
- REQUEST_LOG_STORE_DSN=postgresql://ferrogw:ferrogw@db:5432/ferrogw?sslmode=disable
volumes:
- ./config.yaml:/etc/ferrogw/config.yaml:ro
depends_on:
- db
db:
image: postgres:16-alpine
environment:
POSTGRES_USER: ferrogw
POSTGRES_PASSWORD: ferrogw
POSTGRES_DB: ferrogw
volumes:
- pgdata:/var/lib/postgresql/data
volumes:
pgdata:
```
### Kubernetes via Helm
[](https://artifacthub.io/packages/search?org=ferro-labs)
```bash
helm repo add ferro-labs https://ferro-labs.github.io/helm-charts
helm repo update
helm install ferro-gw ferro-labs/ai-gateway \
--set env.OPENAI_API_KEY=sk-your-key
```
Helm charts: [github.com/ferro-labs/helm-charts](https://github.com/ferro-labs/helm-charts) | [ArtifactHub](https://artifacthub.io/packages/search?org=ferro-labs)
---
## Migrate to Ferro Labs AI Gateway
### From LiteLLM
LiteLLM users can migrate in one step. Ferro Labs AI Gateway is OpenAI-compatible β change one line in your code:
**Python (before β LiteLLM):**
```python
from litellm import completion
response = completion(
model="gpt-4o",
messages=[{"role": "user", "content": "Hello"}]
)
```
**Python (after β Ferro Labs AI Gateway):**
```python
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8080/v1",
api_key="your-ferro-api-key",
)
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "Hello"}],
)
```
**Node.js (after β Ferro Labs AI Gateway):**
```typescript
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "http://localhost:8080/v1",
apiKey: "your-ferro-api-key",
});
const response = await client.chat.completions.create({
model: "gpt-4o",
messages: [{ role: "user", content: "Hello" }],
});
```
**Why migrate from LiteLLM:**
- 14x higher throughput at 150 concurrent users (2,447 vs 175 RPS)
- 23x less memory at peak load (47 MB vs 1,124 MB under streaming)
- Single binary β no Python environment, no pip, no virtualenv
- Predictable latency β p99 stays under 65 ms at 150 VU vs LiteLLM's timeouts at the same concurrency
**Config migration:**
```
# LiteLLM config.yaml # Ferro Labs config.yaml
model_list: strategy:
- model_name: gpt-4o mode: fallback
litellm_params:
model: gpt-4o targets:
api_key: sk-... - virtual_key: openai
- model_name: claude-3-5-sonnet - virtual_key: anthropic
litellm_params:
model: claude-3-5-sonnet aliases:
api_key: sk-ant-... fast: gpt-4o
smart: claude-3-5-sonnet-20241022
```
Provider API keys are set via environment variables (`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, etc.) β not in the config file.
### From Portkey
Portkey users: Ferro Labs AI Gateway uses the standard OpenAI SDK β no custom headers required in self-hosted mode.
**Before (Portkey hosted):**
```python
from portkey_ai import Portkey
client = Portkey(api_key="portkey-key")
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "Hello"}],
)
```
**After (Ferro Labs AI Gateway self-hosted):**
```python
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8080/v1",
api_key="your-ferro-api-key",
)
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "Hello"}],
)
```
**Why migrate from Portkey:**
- Fully open source β no per-request pricing, no log limits
- Self-hosted β your data never leaves your infrastructure
- No vendor lock-in β Apache 2.0 license
- MCP support β Portkey self-hosted lacks native MCP
- FerroCloud (coming soon) for teams that want a managed service
### From OpenAI SDK directly
No gateway yet? Add Ferro Labs AI Gateway in front of your existing code with a single `base_url` change. No other code changes required.
```python
# Before β calling OpenAI directly
client = OpenAI(api_key="sk-...")
# After β routing through Ferro Labs AI Gateway
# Gains: failover, caching, rate limiting, cost tracking
client = OpenAI(
base_url="http://localhost:8080/v1",
api_key="your-ferro-api-key",
)
```
Ferro Labs AI Gateway handles provider failover automatically β if OpenAI is down, your requests fall through to Anthropic or Gemini with zero application code changes.
---
## FerroCloud
FerroCloud β the managed version of Ferro Labs AI Gateway with multi-tenancy, analytics, and cost governance β is coming soon.
π **Join the waitlist at [ferrolabs.ai](https://ferrolabs.ai)**
---
## SDKs
Official client libraries for the Ferro Labs AI Gateway:
| SDK | Install | Repository |
|:----|:--------|:-----------|
| Python | `pip install ferrolabs` | [ferro-labs/ferrolabs-python-sdk](https://github.com/ferro-labs/ferrolabs-python-sdk) |
| TypeScript | `npm install ferrolabs` | [ferro-labs/ferrolabs-typescript-sdk](https://github.com/ferro-labs/ferrolabs-typescript-sdk) |
Python
```python
from ferrolabs import FerroClient
client = FerroClient(
base_url="http://localhost:8080/v1",
api_key="your-ferro-api-key",
)
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "Hello"}],
)
```
TypeScript
```typescript
import { FerroClient } from "ferrolabs";
const client = new FerroClient({
baseURL: "http://localhost:8080/v1",
apiKey: "your-ferro-api-key",
});
const response = await client.chat.completions.create({
model: "gpt-4o",
messages: [{ role: "user", content: "Hello" }],
});
```
### OpenAI SDK Compatible
You can also use the standard OpenAI SDK directly β just change the base URL:
**Python:**
```python
from openai import OpenAI
client = OpenAI(
api_key="sk-ferro-...",
base_url="http://localhost:8080/v1",
)
```
**TypeScript:**
```typescript
import OpenAI from "openai";
const client = new OpenAI({
apiKey: "sk-ferro-...",
baseURL: "http://localhost:8080/v1",
});
```
---
## Contributing
We welcome contributions. New providers go in this OSS repo only β never in FerroCloud. See [CONTRIBUTING.md](CONTRIBUTING.md) for branch strategy, commit conventions, and PR guidelines.
---
## Community
- [GitHub Discussions](https://github.com/ferro-labs/ai-gateway/discussions)
- [Discord](https://discord.gg/yCAeYvJeDV)
- Built with Ferro Labs AI Gateway? Open a PR to add to our showcase.
---
## License
Apache 2.0 β see [LICENSE](LICENSE).