{"id":34524856,"url":"https://github.com/dpguthrie/braintrust-llm-gateway","last_synced_at":"2026-05-28T03:31:13.139Z","repository":{"id":327128037,"uuid":"1108000392","full_name":"dpguthrie/braintrust-llm-gateway","owner":"dpguthrie","description":"Example implementation of integrating Braintrust at the gateway layer","archived":false,"fork":false,"pushed_at":"2026-01-16T16:08:39.000Z","size":143,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-01-17T05:12:51.002Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dpguthrie.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-12-01T22:23:28.000Z","updated_at":"2026-01-16T16:08:45.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/dpguthrie/braintrust-llm-gateway","commit_stats":null,"previous_names":["dpguthrie/braintrust-llm-gateway"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/dpguthrie/braintrust-llm-gateway","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpguthrie%2Fbraintrust-llm-gateway","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpguthrie%2Fbraintrust-llm-gateway/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpguthrie%2Fbraintrust-llm-gateway/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpguthrie%2Fbraintrust-llm-gateway/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dpguthrie","download_url":"https://codeload.github.com/dpguthrie/braintrust-llm-gateway/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpguthrie%2Fbraintrust-llm-gateway/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33593400,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-28T02:00:06.440Z","response_time":99,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-12-24T05:06:58.188Z","updated_at":"2026-05-28T03:31:13.134Z","avatar_url":"https://github.com/dpguthrie.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# LLM Gateway with Braintrust Tracing\n\nDemo-quality LLM gateway that automatically instruments all requests with distributed Braintrust tracing.\n\n## Features\n\n- **Distributed Tracing**: Parent span propagation for complete trace hierarchy\n- **Automatic Instrumentation**: Every LLM request automatically logged to Braintrust\n- **Multi-Provider**: Support for OpenAI and Anthropic (easily extensible)\n- **OpenAI Compatible**: Works with any OpenAI SDK client\n- **Simple Auth**: Bearer token authentication\n- **Streaming**: Full support for streaming responses\n- **Modal Deployment**: Easy serverless deployment with uv\n\n## Architecture\n\n```\n[Client App] --HTTP--\u003e [LLM Gateway] --API--\u003e [LLM Provider]\n     |                      |                      |\n [BT Span]          [BT Child Span]         [Response]\n```\n\nThe gateway creates child spans that link to client application spans, maintaining complete trace hierarchy from application → gateway → LLM provider.\n\n## Local Development\n\n### Prerequisites\n\n- Python 3.13+\n- [uv](https://github.com/astral-sh/uv) for package management\n- Braintrust account and API key\n- API keys for LLM providers (OpenAI, Anthropic, etc.)\n\n### Setup\n\n1. Clone and install dependencies:\n\n```bash\ngit clone \u003crepo-url\u003e\ncd custom-llm-gateway\nuv pip install -e \".[dev]\"\n```\n\n2. Create a `.env` file (this repo does not include a `.env.example`):\n\n```bash\ncat \u003e .env \u003c\u003c'EOF'\nBRAINTRUST_API_KEY=...\nGATEWAY_AUTH_TOKEN=your-token-1,your-token-2\nOPENAI_API_KEY=...\nANTHROPIC_API_KEY=...\nEOF\n```\n\nRequired environment variables:\n- `BRAINTRUST_API_KEY`: Your Braintrust API key\n- `GATEWAY_AUTH_TOKEN`: Comma-separated bearer tokens for authentication\n- `OPENAI_API_KEY`: OpenAI API key (optional)\n- `ANTHROPIC_API_KEY`: Anthropic API key (optional)\n\n3. Run the server:\n\n```bash\nuvicorn src.llm_gateway.main:app --reload\n```\n\nThe server will start at `http://localhost:8000`.\n\n### Testing\n\nRun tests:\n\n```bash\npytest\n```\n\nRun tests with coverage:\n\n```bash\npytest --cov=src/llm_gateway --cov-report=html\n```\n\nFormat code:\n\n```bash\nruff format .\n```\n\nLint code:\n\n```bash\nruff check .\n```\n\n## Usage\n\n### Making Requests\n\nThe gateway exposes an OpenAI-compatible API at `/v1/chat/completions`.\n\n**Using curl:**\n\n```bash\ncurl http://localhost:8000/v1/chat/completions \\\n  -H \"Authorization: Bearer your-token-here\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"model\": \"gpt-4o-mini\",\n    \"messages\": [{\"role\": \"user\", \"content\": \"Hello!\"}]\n  }'\n```\n\n**Using OpenAI Python SDK:**\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(\n    base_url=\"http://localhost:8000/v1\",\n    api_key=\"your-token-here\",  # Your gateway token\n)\n\nresponse = client.chat.completions.create(\n    model=\"gpt-4o-mini\",\n    messages=[{\"role\": \"user\", \"content\": \"Hello!\"}],\n)\n\nprint(response.choices[0].message.content)\n```\n\n**Streaming:**\n\n```python\nstream = client.chat.completions.create(\n    model=\"gpt-4o-mini\",\n    messages=[{\"role\": \"user\", \"content\": \"Tell me a story\"}],\n    stream=True,\n)\n\nfor chunk in stream:\n    if chunk.choices[0].delta.content:\n        print(chunk.choices[0].delta.content, end=\"\")\n```\n\n### Supported Models\n\nThe gateway automatically routes requests based on model name:\n\n- **OpenAI**: `gpt-4o`, `gpt-4o-mini`, `gpt-4`, `gpt-3.5-turbo`, `o1-*`, etc.\n- **Anthropic**: `claude-3-5-sonnet-*`, `claude-3-opus-*`, `claude-3-haiku-*`, etc.\n\n### Distributed tracing\n\nThe gateway supports parent span propagation for distributed tracing. This allows client applications to link their spans with gateway spans, creating a complete trace hierarchy.\n\n**Without parent span (basic usage):**\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(\n    base_url=\"http://localhost:8000/v1\",\n    api_key=\"your-token-here\",\n)\n\nresponse = client.chat.completions.create(\n    model=\"gpt-4o-mini\",\n    messages=[{\"role\": \"user\", \"content\": \"Hello\"}],\n)\n# Gateway creates independent span in Braintrust\n```\n\n**With parent span (distributed tracing via `x-bt-parent`):**\n\n```python\nimport braintrust\nfrom openai import OpenAI\n\n# Initialize your application's tracing\nexperiment = braintrust.init(project=\"my-app\")\n\nwith experiment.traced(name=\"my-app-task\") as parent_span:\n    client = OpenAI(\n        base_url=\"http://localhost:8000/v1\",\n        api_key=\"your-gateway-token\",\n    )\n\n    # Pass parent span context via header (Braintrust standard)\n    response = client.chat.completions.create(\n        model=\"gpt-4o-mini\",\n        messages=[{\"role\": \"user\", \"content\": \"Hello\"}],\n        extra_headers={\n            \"x-bt-parent\": parent_span.export(),\n            # Optional JSON object (must be an object, not a string/array):\n            \"x-bt-span-metadata\": '{\"request_id\":\"req_123\",\"customer\":\"acme\"}',\n        }\n    )\n\n# Creates trace hierarchy: my-app-task → gateway-request → openai-completion\n```\n\nThe response includes a `braintrust_span` field with the gateway span context, allowing you to continue the trace chain if needed.\n\n**With W3C baggage (recommended for Java/OTEL ecosystems):**\n\nIf your client and gateway are OpenTelemetry-instrumented, you can propagate the Braintrust parent via W3C baggage:\n\n- Set `baggage: braintrust.parent=project_name:my-project` (or `experiment_id:...`)\n- Also propagate `traceparent` for full distributed tracing across services\n\nThe Python gateway will read `braintrust.parent` from the `baggage` header and create a child Braintrust span accordingly.\n\n**Example (curl)**:\n\n```bash\ncurl http://localhost:8000/v1/chat/completions \\\n  -H \"Authorization: Bearer your-token-here\" \\\n  -H \"Content-Type: application/json\" \\\n  -H \"baggage: braintrust.parent=project_name:my-project\" \\\n  -d '{\n    \"model\": \"gpt-4o-mini\",\n    \"messages\": [{\"role\": \"user\", \"content\": \"Hello!\"}]\n  }'\n```\n\nIf you also propagate `traceparent`, you’ll get full distributed tracing across services (client ↔ gateway) in any OTEL backend; Braintrust will still use `braintrust.parent` for routing spans to the right project/experiment.\n\n**Trace Hierarchy Example:**\n\n```\nRoot Span (Client: \"process-document\")\n└── Child Span (Gateway: \"llm-gateway-request\")\n    └── Grandchild Span (Provider: \"openai.chat.completions.create\")\n        ├── input: {...}\n        ├── output: {...}\n        ├── metrics: {tokens: 150}\n        └── latency: 1.2s\n```\n\n### Viewing Traces\n\nAll requests are automatically logged to Braintrust:\n\n1. Go to [braintrust.dev](https://braintrust.dev)\n2. Navigate to your project (default: \"llm-gateway\")\n3. View traces with full request/response data, token counts, and latency\n4. If using distributed tracing, see complete hierarchy from client → gateway → provider\n\n## Modal Deployment\n\n### Prerequisites\n\n- Modal account ([modal.com](https://modal.com))\n- Modal CLI: `pip install modal`\n- `uv` package manager installed\n\n### Setup\n\n1. Authenticate with Modal:\n\n```bash\nmodal setup\n```\n\n2. Create a `.env` file with your API keys:\n\n```bash\ncat \u003e .env \u003c\u003c'EOF'\nBRAINTRUST_API_KEY=...\nGATEWAY_AUTH_TOKEN=your-token-1,your-token-2\nOPENAI_API_KEY=...\nANTHROPIC_API_KEY=...\nEOF\n```\n\nRequired variables:\n```\nBRAINTRUST_API_KEY=your-braintrust-key\nGATEWAY_AUTH_TOKEN=your-token-1,your-token-2\nOPENAI_API_KEY=your-openai-key\nANTHROPIC_API_KEY=your-anthropic-key\n```\n\n3. Ensure you have a `uv.lock` file:\n\n```bash\nuv sync\n```\n\n4. Deploy:\n\n```bash\nmodal deploy modal_app.py\n```\n\nThe deployment uses:\n- `uv_sync()` to install dependencies from `pyproject.toml` and `uv.lock`\n- `Secret.from_dotenv()` to load environment variables from your `.env` file\n\n5. Get your deployment URL from the output:\n\n```\n✓ Created web function fastapi_app =\u003e https://your-app-id--llm-gateway-fastapi-app.modal.run\n```\n\n### Using Deployed Gateway\n\nUpdate your client to use the Modal URL:\n\n```python\nclient = OpenAI(\n    base_url=\"https://your-app-id--llm-gateway-fastapi-app.modal.run/v1\",\n    api_key=\"your-gateway-token\",\n)\n```\n\n## Project Structure\n\n```\n.\n├── src/llm_gateway/\n│   ├── __init__.py\n│   ├── config.py          # Configuration with Pydantic\n│   ├── auth.py            # Bearer token authentication\n│   ├── models.py          # Request/response models\n│   └── main.py            # FastAPI app + Braintrust tracing\n├── tests/\n│   ├── test_config.py\n│   ├── test_auth.py\n│   ├── test_models.py\n│   └── test_main.py\n├── docs/\n│   └── plans/             # Design and implementation docs\n├── modal_app.py           # Modal deployment config\n├── pyproject.toml         # Dependencies and config\n├── ruff.toml              # Linting/formatting config\n└── README.md\n```\n\n## How It Works\n\n### Distributed Tracing Architecture\n\nThe gateway uses Braintrust's provider-specific wrappers for automatic instrumentation with distributed tracing support:\n\n1. **On startup**:\n   - Initializes Braintrust logger for the gateway project\n   - Wraps OpenAI and Anthropic clients with `braintrust.wrap_openai()` and `braintrust.wrap_anthropic()`\n\n2. **On each request**:\n   - Extracts optional `x-bt-parent` header (parent span from client), or W3C baggage `braintrust.parent`\n   - Creates gateway span with `start_span()`, linking to parent if provided\n   - Routes to appropriate provider based on model name\n   - Wrapped client automatically creates child span under gateway span\n   - Returns response with `braintrust_span` field for continued tracing\n\n3. **Trace hierarchy created**:\n   ```\n   Client Span (optional)\n   └── Gateway Span (llm-gateway-request)\n       └── Provider Span (openai.chat.completions.create)\n   ```\n\n4. **Automatic capture**: inputs, outputs, tokens, latency, time-to-first-token, model metadata\n\nThe gateway acts as a transparent proxy while maintaining complete observability through distributed tracing.\n\n### Authentication\n\nSimple bearer token authentication:\n\n1. Client sends `Authorization: Bearer \u003ctoken\u003e` header\n2. Gateway validates token against `GATEWAY_AUTH_TOKEN` list\n3. Invalid/missing tokens return 401\n\nFor production, consider:\n- JWT tokens with expiration\n- Per-key rate limiting\n- Database-backed key management\n\n## Demo Use Cases\n\nThis gateway is designed for demos showing:\n\n1. **Distributed tracing in LLM apps**: Show complete trace hierarchy from client app through gateway to provider\n2. **How to add observability to LLM apps**: Show Braintrust traces with automatic instrumentation\n3. **Multi-provider routing**: Automatically route to OpenAI or Anthropic based on model name\n4. **OpenAI SDK compatibility**: Drop-in replacement for OpenAI API\n5. **Serverless deployment**: Deploy to Modal in minutes with uv\n\n## Limitations (By Design)\n\nThis is demo-quality code, intentionally simple:\n\n- ❌ No rate limiting per key\n- ❌ No persistent key storage\n- ❌ No advanced auth (OAuth, JWT validation)\n- ❌ No caching\n- ❌ No custom retry logic\n- ❌ No cost tracking dashboard\n\nFor production use, add these features as needed.\n\n## Troubleshooting\n\n**Import errors:**\n```bash\n# Make sure you installed in editable mode\nuv pip install -e \".[dev]\"\n```\n\n**Braintrust not logging:**\n- Check `BRAINTRUST_API_KEY` is set correctly\n- Verify project name matches your Braintrust account\n- Check Braintrust dashboard for API key permissions\n\n**LiteLLM errors:**\n- Verify provider API keys are set (OPENAI_API_KEY, etc.)\n- Check model name is correct for the provider\n- See [LiteLLM docs](https://docs.litellm.ai/docs/) for provider-specific setup\n\n**Modal deployment fails:**\n- Ensure secret `llm-gateway-secrets` exists with all required keys\n- Check Modal logs: `modal logs fastapi_app`\n- Verify you're authenticated: `modal setup`\n\n## Contributing\n\nThis is a demo project. Feel free to fork and extend for your needs!\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdpguthrie%2Fbraintrust-llm-gateway","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdpguthrie%2Fbraintrust-llm-gateway","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdpguthrie%2Fbraintrust-llm-gateway/lists"}