{"id":49057499,"url":"https://github.com/lab34-es/llm-proxy","last_synced_at":"2026-04-20T00:01:29.669Z","repository":{"id":349844194,"uuid":"1204165035","full_name":"lab34-es/llm-proxy","owner":"lab34-es","description":"LLM proxy written in go with usage \u0026 guard rails support.","archived":false,"fork":false,"pushed_at":"2026-04-07T21:10:12.000Z","size":10895,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-07T21:11:14.562Z","etag":null,"topics":["go","llm","llm-proxy"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lab34-es.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-07T18:49:34.000Z","updated_at":"2026-04-07T21:08:23.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/lab34-es/llm-proxy","commit_stats":null,"previous_names":["lab34-es/llm-proxy"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/lab34-es/llm-proxy","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lab34-es%2Fllm-proxy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lab34-es%2Fllm-proxy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lab34-es%2Fllm-proxy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lab34-es%2Fllm-proxy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lab34-es","download_url":"https://codeload.github.com/lab34-es/llm-proxy/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lab34-es%2Fllm-proxy/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32027234,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-18T20:23:30.271Z","status":"online","status_checked_at":"2026-04-19T02:00:07.110Z","response_time":55,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["go","llm","llm-proxy"],"created_at":"2026-04-20T00:00:38.032Z","updated_at":"2026-04-20T00:01:29.655Z","avatar_url":"https://github.com/lab34-es.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# LLM Proxy\n\nAn OpenAI-compatible LLM proxy that routes requests to any OpenAI-compatible backend. Written in Go. SQLite for storage.\n\n## Features\n\n- **OpenAI-compatible API** -- drop-in replacement for any OpenAI client library\n- **Streaming (SSE)** -- full support for streamed chat completions\n- **Multi-provider management** -- register any number of upstream backends\n- **Proxy API key issuance** -- `llmp-` prefixed keys so clients never see upstream credentials\n- **Per-key rate limiting** -- configurable requests-per-minute limits\n- **Token usage tracking** -- records prompt, completion, and total tokens per request\n- **Swagger UI** -- interactive API documentation at `/docs`\n- **OpenAPI 3.1 spec** -- source of truth, embedded in the binary\n- **Dashboard** -- web UI for monitoring usage and managing resources\n- **Guardrails** -- configurable safety rules to change or reject LLM interactions\n\n## Screenshots\n\n\u003ctable\u003e\n  \u003ctr\u003e\n    \u003ctd align=\"center\"\u003e\u003cstrong\u003eUsage\u003c/strong\u003e\u003cbr/\u003e\u003ca href=\"docs/screenshots/usage.png\"\u003e\u003cimg src=\"docs/screenshots/usage.png\" alt=\"Usage\" width=\"300\" /\u003e\u003c/a\u003e\u003cbr/\u003eToken usage stats and request metrics\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e\u003cstrong\u003eProviders\u003c/strong\u003e\u003cbr/\u003e\u003ca href=\"docs/screenshots/providers.png\"\u003e\u003cimg src=\"docs/screenshots/providers.png\" alt=\"Providers\" width=\"300\" /\u003e\u003c/a\u003e\u003cbr/\u003eUpstream LLM backend configuration\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e\u003cstrong\u003eAPI Keys\u003c/strong\u003e\u003cbr/\u003e\u003ca href=\"docs/screenshots/api_keys.png\"\u003e\u003cimg src=\"docs/screenshots/api_keys.png\" alt=\"API Keys\" width=\"300\" /\u003e\u003c/a\u003e\u003cbr/\u003eAPI key issuance and management\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd align=\"center\"\u003e\u003cstrong\u003eGuardrails\u003c/strong\u003e\u003cbr/\u003e\u003ca href=\"docs/screenshots/guardrails.png\"\u003e\u003cimg src=\"docs/screenshots/guardrails.png\" alt=\"Guardrails\" width=\"300\" /\u003e\u003c/a\u003e\u003cbr/\u003eSafety rules and content filtering\u003c/td\u003e\n    \u003ctd\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n## Quick start\n\n```bash\n# Build\ngo build -o llm-proxy .\n\n# Run (ADMIN_TOKEN is required)\nADMIN_TOKEN=my-secret-admin-token ./llm-proxy\n```\n\nThe server starts on `:8080` by default. Open http://localhost:8080/docs for the Swagger UI. The dashboard is available at http://localhost:8080/dashboard.\n\n## Configuration\n\nAll configuration is via environment variables:\n\n| Variable | Default | Description |\n|---|---|---|\n| `ADDR` | `:8080` | Server listen address |\n| `DSN` | `llm-proxy.db` | SQLite database file path |\n| `ADMIN_TOKEN` | *(required)* | Bearer token for `/admin/*` endpoints and dashboard login |\n\n## Development\n\nRun both the Go backend and the React (Vite) frontend in a single command:\n\n```bash\n./development.sh\n```\n\nThis starts the Go backend on `:8080` and the Vite dev server on `:5173` with hot reload. The dashboard is available at http://localhost:5173/dashboard.\n\nThe `ADMIN_TOKEN` defaults to `admin` in development. Override it with:\n\n```bash\nADMIN_TOKEN=my-secret ./development.sh\n```\n\n**Prerequisites:** Go 1.25+, Node.js 22+.\n\n### Production build\n\n```bash\ncd internal/web/frontend \u0026\u0026 npm ci \u0026\u0026 npm run build \u0026\u0026 cd -\ngo build -trimpath -ldflags=\"-s -w\" -o llm-proxy .\n```\n\nThis produces a single binary with the React dashboard embedded.\n\n## Project structure\n\n```\nllm-proxy/\n├── openapi.yaml                    # OpenAPI 3.1 spec (embedded at compile time)\n├── main.go                         # Entrypoint\n├── development.sh                  # Start frontend + backend for local dev\n├── Dockerfile                      # Multi-stage build (Node + Go)\n├── docker-compose.yml              # Local dev / deployment\n└── internal/\n    ├── config/config.go            # Environment-based configuration\n    ├── db/\n    │   ├── db.go                   # SQLite connection (WAL mode)\n    │   └── migrations.go           # Auto-migrating schema\n    ├── models/models.go            # Domain types\n    ├── store/\n    │   ├── provider.go             # Provider CRUD\n    │   ├── apikey.go               # API key create/lookup/revoke\n    │   ├── usage.go                # Usage recording and queries\n    │   ├── guardrail.go            # Guardrail rule CRUD\n    │   └── guardrail_event.go      # Guardrail event logging\n    ├── middleware/\n    │   ├── auth.go                 # Admin + proxy key authentication\n    │   └── ratelimit.go            # Per-key rate limiting\n    ├── handler/\n    │   ├── admin.go                # Admin API handlers\n    │   ├── proxy.go                # Proxy API handlers\n    │   └── docs.go                 # Swagger UI + spec serving\n    ├── proxy/forwarder.go          # Upstream request forwarding + guardrail enforcement\n    └── web/\n        ├── spa.go                  # SPA handler (embeds frontend/dist/)\n        └── frontend/               # React app (Vite + TypeScript + MUI Joy)\n            ├── src/\n            │   ├── api/client.ts   # API client for /admin/* endpoints\n            │   ├── context/        # Auth context (localStorage token)\n            │   ├── components/     # Layout (sidebar), DismissibleAlert\n            │   └── pages/          # Providers, Keys, Usage, Guardrails, Playground\n            ├── vite.config.ts\n            └── package.json\n```\n\n## Benchmarks\n\nBenchmarks run against a realistic environment with **100 providers**, **100 API keys**, and **100 guardrail rules** loaded in the database. The mock upstream responds instantly, so results isolate proxy overhead.\n\n```bash\ngo test -bench=. -benchmem -benchtime=3s ./internal/proxy/ -run='^$'\n```\n\nResults on Apple M3 Max (arm64):\n\n| Benchmark | ns/op | B/op | allocs/op |\n|---|--:|--:|--:|\n| ForwardChatCompletion_NonStreaming | 484,511 | 517,372 | 3,894 |\n| ForwardChatCompletion_Streaming | 538,411 | 574,631 | 3,888 |\n| APIKeyLookup | 7,806 | 1,576 | 43 |\n| GuardrailEvaluation | 480,464 | 461,046 | 3,659 |\n| UsageRecording | 23,312 | 720 | 17 |\n| AdminListProviders | 251,850 | 133,757 | 2,447 |\n\n**Key takeaways:**\n\n- **API key lookup** (SHA-256 hash + SQLite query among 100 keys) completes in ~8 us.\n- **Usage recording** (SQLite INSERT) takes ~23 us per request.\n- **Guardrail evaluation** with 100 compiled regex patterns is the dominant cost in the proxy path (~480 us). This cost is per-request since patterns are compiled on each evaluation.\n- **Full chat completion** round-trip (parse, guardrails, upstream call, response copy, usage write) stays under 0.5 ms for non-streaming and 0.54 ms for streaming, excluding actual upstream latency.\n\n## License\n\nMIT -- see [LICENSE](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flab34-es%2Fllm-proxy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flab34-es%2Fllm-proxy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flab34-es%2Fllm-proxy/lists"}