{"id":50621119,"url":"https://github.com/ictup/llm-serving-gateway-vllm","last_synced_at":"2026-06-06T12:02:00.204Z","repository":{"id":358911763,"uuid":"1242229071","full_name":"ictup/llm-serving-gateway-vllm","owner":"ictup","description":"OpenAI-compatible LLM serving gateway with vLLM, FastAPI, Redis quotas, Prometheus/Grafana, GPU metrics, benchmarks, Docker, Kubernetes, Helm, GitOps, Terraform, and CI/CD.","archived":false,"fork":false,"pushed_at":"2026-05-19T23:19:06.000Z","size":295,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-05-20T02:32:45.602Z","etag":null,"topics":["ai-gateway","ai-infrastructure","argocd","benchmarking","docker","fastapi","gitops","grafana","helm","kubernetes","llm","llm-gateway","llmops","model-serving","openai-compatible","openai-proxy","prometheus","redis","terraform","vllm"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ictup.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"docs/security.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-18T08:34:08.000Z","updated_at":"2026-05-19T23:16:13.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/ictup/llm-serving-gateway-vllm","commit_stats":null,"previous_names":["ictup/mini_llm_serving_platform","ictup/llm-serving-gateway-vllm"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/ictup/llm-serving-gateway-vllm","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ictup%2Fllm-serving-gateway-vllm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ictup%2Fllm-serving-gateway-vllm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ictup%2Fllm-serving-gateway-vllm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ictup%2Fllm-serving-gateway-vllm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ictup","download_url":"https://codeload.github.com/ictup/llm-serving-gateway-vllm/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ictup%2Fllm-serving-gateway-vllm/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33981125,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-06T02:00:07.033Z","response_time":107,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-gateway","ai-infrastructure","argocd","benchmarking","docker","fastapi","gitops","grafana","helm","kubernetes","llm","llm-gateway","llmops","model-serving","openai-compatible","openai-proxy","prometheus","redis","terraform","vllm"],"created_at":"2026-06-06T12:01:58.766Z","updated_at":"2026-06-06T12:02:00.196Z","avatar_url":"https://github.com/ictup.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# OpenAI-Compatible LLM Serving Gateway\n\n[![CI](https://github.com/ictup/llm-serving-gateway-vllm/actions/workflows/ci.yml/badge.svg)](https://github.com/ictup/llm-serving-gateway-vllm/actions/workflows/ci.yml)\n[![Security](https://github.com/ictup/llm-serving-gateway-vllm/actions/workflows/security.yml/badge.svg)](https://github.com/ictup/llm-serving-gateway-vllm/actions/workflows/security.yml)\n[![Release](https://github.com/ictup/llm-serving-gateway-vllm/actions/workflows/release.yml/badge.svg)](https://github.com/ictup/llm-serving-gateway-vllm/actions/workflows/release.yml)\n[![Latest Release](https://img.shields.io/github/v/release/ictup/llm-serving-gateway-vllm?sort=semver)](https://github.com/ictup/llm-serving-gateway-vllm/releases)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)\n[![Python 3.11+](https://img.shields.io/badge/Python-3.11%2B-3776AB?logo=python\u0026logoColor=white)](pyproject.toml)\n[![FastAPI](https://img.shields.io/badge/FastAPI-OpenAI--compatible-009688?logo=fastapi\u0026logoColor=white)](gateway/app/main.py)\n[![vLLM](https://img.shields.io/badge/vLLM-GPU%20serving-6A5ACD)](docker-compose.gpu.yml)\n[![Kubernetes + Helm](https://img.shields.io/badge/Kubernetes%20%2B%20Helm-ready-326CE5?logo=kubernetes\u0026logoColor=white)](deploy/helm)\n[![Observability](https://img.shields.io/badge/Observability-Prometheus%20%2B%20Grafana-orange)](monitoring)\n\nProduction-style FastAPI gateway for OpenAI-compatible LLM serving. It sits in\nfront of mock or vLLM backends and adds the platform layer that a raw model\nserver does not own: API keys, request IDs, model aliases, weighted routing,\nRedis-backed RPM/TPM/concurrency limits, streaming metrics, Prometheus,\nGrafana, GPU telemetry, Docker, Kubernetes, Helm, GitOps, Terraform,\nsupply-chain checks, release automation, and repeatable direct-vs-gateway\nbenchmarks.\n\nThis is a portfolio-grade AI infrastructure project. It is designed to show\nhow an LLM serving gateway is built, operated, benchmarked, secured, and\nreleased without pretending to be a full enterprise GPU scheduler.\n\n## Why This Project Exists\n\nvLLM already exposes an OpenAI-compatible server. This project answers the\nnext platform question: what do you put around that server when teams need a\nstable API contract, quotas, routing, metrics, deployment automation, and\nrelease discipline?\n\nThe Gateway keeps those concerns outside model execution:\n\n- Clients call one OpenAI-compatible `/v1` API.\n- vLLM or the mock backend handles model responses.\n- Redis enforces RPM, TPM, and concurrent request limits.\n- Prometheus and Grafana expose request, streaming, vLLM, and GPU behavior.\n- Docker, Kubernetes, Helm, Argo CD, and Terraform describe deployment paths.\n- CI, security scans, SBOM/provenance, GHCR publishing, and SemVer releases\n  make the repository behave like a maintained production project.\n\n## Recruiter Snapshot\n\n| Signal | Evidence |\n| --- | --- |\n| AI platform engineering | OpenAI-compatible Gateway, vLLM backend, streaming SSE, model routing, token-aware quotas |\n| Production operations | Redis rate limits, readiness/warmup, structured logs, Prometheus metrics, Grafana dashboards, alert rules |\n| Performance discipline | Direct-vs-Gateway benchmark runner, TTFT, ITL, TPOT, p95/p99, error-rate, tokenizer-level output token metrics |\n| Deployment maturity | Docker Compose, K8s overlays, Helm chart, Argo CD examples, Terraform entry point |\n| Delivery hygiene | CI, GHCR image publishing, Trivy/pip-audit, SBOM/provenance, Dependabot, SemVer release workflow |\n\nIn short: this is not only an inference demo. It shows benchmarking,\nobservability, and the platform scope recruiters expect:\nGitOps, Terraform, supply-chain checks, release automation.\n\n## Feature Overview\n\n| Area | Implemented |\n| --- | --- |\n| API surface | `/v1/models`, `/v1/chat/completions`, OpenAI-compatible request/response schemas |\n| Streaming | Server-Sent Events proxying with time-to-first-token measurement |\n| Backends | No-GPU mock backend for CI and local demos, vLLM OpenAI server for CUDA serving |\n| Routing | Model aliases, backend model mapping, weighted canary routes, fallback targets |\n| Auth and safety | API key auth, request IDs, request body limits, chat message limits, normalized errors |\n| Rate limiting | Redis-backed RPM, tokenizer-aware TPM, and concurrent request limits |\n| Observability | Prometheus metrics, structured JSON logs, Grafana dashboards, alert rules |\n| GPU telemetry | DCGM exporter wiring for GPU utilization and framebuffer memory |\n| Benchmarking | Async direct-vLLM vs Gateway runs with RPS, latency, TTFT, ITL, TPOT, output tokens/sec, p95/p99, error rate |\n| Deployment | Docker Compose, Kubernetes base and GPU overlays, Helm, Argo CD, Terraform skeleton |\n| Supply chain | CI, Trivy, pip-audit, SBOM, provenance, Dependabot, GHCR image publishing |\n| Release engineering | SemVer validation, changelog, release workflow, versioned GitHub Releases |\n\n## Architecture\n\n```mermaid\nflowchart LR\n    Client[\"OpenAI SDK\u003cbr/\u003ecurl\u003cbr/\u003eRAG app\u003cbr/\u003eagent runtime\"] --\u003e Gateway[\"FastAPI Gateway\u003cbr/\u003eOpenAI-compatible /v1 API\"]\n    Gateway --\u003e Platform[\"Platform controls\u003cbr/\u003eauth, request ID\u003cbr/\u003elimits, logs, metrics\"]\n    Platform --\u003e Router[\"Model router\u003cbr/\u003ealiases, weights, fallback\"]\n    Router --\u003e Mock[\"Mock backend\u003cbr/\u003eCI and no-GPU demo\"]\n    Router --\u003e VLLM[\"vLLM OpenAI server\u003cbr/\u003eCUDA serving\"]\n    Platform --\u003e Redis[\"Redis\u003cbr/\u003eRPM, TPM, concurrency\"]\n    Platform --\u003e Prometheus[\"Prometheus\u003cbr/\u003eGateway and vLLM metrics\"]\n    VLLM --\u003e DCGM[\"DCGM exporter\u003cbr/\u003eGPU util and memory\"]\n    DCGM --\u003e Prometheus\n    Prometheus --\u003e Grafana[\"Grafana dashboards\u003cbr/\u003eGateway, vLLM, GPU\"]\n```\n\nThe important design choice is separation of concerns. vLLM executes the model;\nthe Gateway owns client-facing policy, routing, limits, observability, and the\noperational contract.\n\n## Quick Start: No GPU\n\nThe mock backend makes the platform reproducible on a laptop and in CI.\n\nPrint the portfolio demo walkthrough:\n\n```bash\nuv run python scripts/demo_portfolio.py\n```\n\nRun the no-GPU live demo:\n\n```bash\nuv run python scripts/demo_portfolio.py --execute-local\n```\n\n```bash\nuv sync --frozen --all-groups\nuv run python scripts/local_e2e.py\n```\n\nRun the quality gate:\n\n```bash\nuv run ruff check .\nuv run pytest\n```\n\nStart the full no-GPU local stack:\n\n```bash\ndocker compose up --build\n```\n\n| Service | URL |\n| --- | --- |\n| Gateway | http://localhost:8080 |\n| Mock backend | http://localhost:9000 |\n| Prometheus | http://localhost:9090 |\n| Grafana | http://localhost:3000 |\n| Redis | `localhost:6379` |\n\nGrafana defaults to `admin` / `admin`.\n\n## Quick Start: GPU vLLM\n\nUse the GPU override when Docker can access an NVIDIA runtime:\n\n```powershell\n$env:VLLM_MODEL=\"Qwen/Qwen2.5-0.5B-Instruct\"\n$env:VLLM_IMAGE_TAG=\"v0.8.5.post1\"\ndocker compose -f docker-compose.yml -f docker-compose.gpu.yml up --build\n```\n\nWarm up the Gateway and run an OpenAI SDK smoke test:\n\n```powershell\nuv run python scripts/warmup_gateway.py --model qwen-small\n\n$env:OPENAI_BASE_URL=\"http://localhost:8080/v1\"\n$env:OPENAI_API_KEY=\"dev-key\"\n$env:LLM_MODEL=\"qwen-small\"\nuv run python benchmark/client_smoke_test.py\n```\n\nThe default GPU model is intentionally small because it has been validated on\nan 8GB RTX 4060 Laptop GPU. Larger models can be selected by overriding\n`VLLM_MODEL` on machines with enough free GPU memory.\n\n## API Example\n\n```bash\ncurl http://localhost:8080/v1/chat/completions \\\n  -H \"Authorization: Bearer dev-key\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"model\": \"mock\",\n    \"messages\": [\n      {\n        \"role\": \"user\",\n        \"content\": \"Explain TTFT in one sentence.\"\n      }\n    ],\n    \"stream\": false\n  }'\n```\n\nFor streaming examples, model listing, error shapes, and health checks, see\n[docs/api_usage.md](docs/api_usage.md).\n\n## Benchmark Snapshot\n\nPortfolio profile on a local RTX 4060 Laptop GPU with 100 measured streaming\nrequests per concurrency level:\n\n| Concurrency | Direct RPS | Gateway RPS | Direct P95 Latency | Gateway P95 Latency | Gateway P50 TTFT |\n| ---: | ---: | ---: | ---: | ---: | ---: |\n| 1 | 1.73 | 1.98 | 1043.08 ms | 896.34 ms | 39.27 ms |\n| 4 | 5.71 | 6.31 | 1256.95 ms | 1119.99 ms | 50.63 ms |\n| 8 | 9.45 | 10.14 | 1474.63 ms | 1368.80 ms | 54.08 ms |\n| 16 | 13.58 | 14.19 | 2004.51 ms | 1902.34 ms | 70.16 ms |\n| 32 | 17.87 | 16.11 | 3121.88 ms | 3529.36 ms | 169.88 ms |\n\nBoth direct and Gateway paths completed with zero errors. Gateway-faster rows\nshould be read as local run variance and \"no obvious Gateway bottleneck\", not\nas proof that the Gateway accelerates vLLM.\n\nFull report:\n[docs/gateway_overhead_report.md](docs/gateway_overhead_report.md)\n\n## Run Direct vs Gateway Benchmarks\n\nRun direct vLLM:\n\n```bash\nuv run python benchmark/run_benchmark.py \\\n  --profile portfolio \\\n  --base-url http://localhost:8000/v1 \\\n  --api-key local-vllm-key \\\n  --model Qwen/Qwen2.5-0.5B-Instruct \\\n  --prompts benchmark/prompts/short_prompts.jsonl \\\n  --output-tokenizer-path D:/models/qwen-tokenizer.json \\\n  --timeout-seconds 120 \\\n  --stream true\n```\n\nFor Gateway serving-capacity runs, raise the demo quota before starting the\nDocker stack:\n\n```powershell\n$env:RATE_LIMIT_RPM=\"10000\"\n$env:RATE_LIMIT_TPM=\"2000000\"\n$env:RATE_LIMIT_CONCURRENT_REQUESTS=\"64\"\ndocker compose -f docker-compose.yml -f docker-compose.gpu.yml up --build\n```\n\nRun through the Gateway:\n\n```bash\nuv run python benchmark/run_benchmark.py \\\n  --profile portfolio \\\n  --base-url http://localhost:8080/v1 \\\n  --api-key dev-key \\\n  --model qwen-small \\\n  --prompts benchmark/prompts/short_prompts.jsonl \\\n  --output-tokenizer-path D:/models/qwen-tokenizer.json \\\n  --timeout-seconds 120 \\\n  --stream true\n```\n\nGenerate the comparison report:\n\n```bash\nuv run python benchmark/compare_results.py \\\n  --direct-result benchmark/results/\u003cdirect-result\u003e.json \\\n  --gateway-result benchmark/results/\u003cgateway-result\u003e.json \\\n  --prometheus-snapshot benchmark/results/\u003cprometheus-snapshot\u003e.json \\\n  --prometheus-timeseries benchmark/results/\u003cprometheus-timeseries\u003e.json \\\n  --output docs/gateway_overhead_report.md\n```\n\nThe `portfolio` profile runs concurrency `1, 4, 8, 16, 32` with 100 measured\nrequests per level and 10 warmup requests. Supplying\n`--output-tokenizer-path` adds tokenizer-level output tokens/sec and TPOT. Use\n`--profile stress` for 1000 requests per level after the local GPU path is\nstable.\n\nSee [docs/performance_benchmarking.md](docs/performance_benchmarking.md).\n\n## Observability\n\nThe local and deployment assets expose three dashboard layers:\n\n| Dashboard | What it shows |\n| --- | --- |\n| Gateway Overview | request rate, latency, errors, rejections, streaming TTFT, streaming duration |\n| vLLM Engine Overview | running requests, waiting requests, KV cache pressure, prompt/generation tokens/sec |\n| GPU Overview | DCGM GPU utilization and framebuffer memory usage |\n\nPrometheus alert examples cover elevated Gateway error rate, p95 latency,\nstreaming TTFT, rejection rate, vLLM queued requests, and vLLM KV cache usage.\n\n## Deployment Paths\n\n| Target | Entry point | Purpose |\n| --- | --- | --- |\n| Docker Compose, no GPU | `docker-compose.yml` | Reproducible local demo |\n| Docker Compose, vLLM | `docker-compose.gpu.yml` | Local CUDA-backed serving plus DCGM GPU metrics |\n| Kubernetes base | `deploy/k8s` | Gateway, mock backend, Redis, Prometheus |\n| Kubernetes GPU overlay | `deploy/k8s-gpu` | vLLM backend, vLLM metrics, DCGM scraping |\n| Helm | `deploy/helm` | Parameterized mock or vLLM deployment |\n| GitOps / Argo CD | `deploy/gitops` | Continuous sync examples for Helm releases |\n| Terraform IaC | `deploy/terraform` | Namespace, Secret boundary, Argo CD Application entry point |\n\nValidate manifests:\n\n```bash\nkubectl kustomize deploy/k8s\nkubectl kustomize deploy/k8s-gpu\n\nhelm lint deploy/helm\nhelm template mini-llm deploy/helm --namespace mini-llm-serving\nhelm template mini-llm deploy/helm \\\n  --namespace mini-llm-serving \\\n  --set vllm.enabled=true \\\n  --set mockBackend.enabled=false \\\n  --set dcgmExporter.enabled=true\n```\n\n## Repository Guide\n\n| Path | Purpose |\n| --- | --- |\n| `gateway/app` | FastAPI Gateway, auth, rate limiting, proxying, metrics |\n| `serving/mock_backend` | OpenAI-compatible mock backend |\n| `benchmark` | SDK smoke tests, async benchmark runner, report tools |\n| `monitoring` | Prometheus config, alert rules, Grafana dashboards |\n| `deploy/k8s` | No-GPU Kubernetes manifests |\n| `deploy/k8s-gpu` | vLLM and DCGM Kubernetes overlay |\n| `deploy/helm` | Helm chart for mock and vLLM modes |\n| `deploy/gitops` | Argo CD Applications for mock and vLLM modes |\n| `deploy/terraform` | Terraform root module for GitOps cluster entry points |\n| `docs` | API, configuration, design decisions, operations, reports |\n\n## Verified State\n\n| Area | Status |\n| --- | --- |\n| No-GPU local path | Verified with mock backend and SDK smoke test |\n| GPU path | Verified locally with Docker Desktop and NVIDIA GPU |\n| CI | Python lint, tests, Helm lint, Helm template rendering |\n| Kubernetes | Base and GPU overlays render with Kustomize |\n| Helm | Mock and vLLM modes render successfully |\n| Release | `v0.1.1` published with CI, security, release, and container workflows passing |\n| External RAG app wiring | Intentionally excluded from this completion |\n\nGPU validation snapshot from May 19, 2026:\n\n| Item | Value |\n| --- | --- |\n| GPU | NVIDIA GeForce RTX 4060 Laptop GPU, 8GB VRAM |\n| vLLM image | `vllm/vllm-openai:v0.8.5.post1` |\n| Served model | `Qwen/Qwen2.5-0.5B-Instruct` |\n| Gateway alias | `qwen-small` |\n| Result | Direct vLLM and Gateway streaming benchmarks completed with zero errors |\n\n## Documentation\n\n- [API usage](docs/api_usage.md)\n- [Configuration matrix](docs/configuration.md)\n- [Design decisions](docs/design_decisions.md)\n- [Failure analysis](docs/failure_analysis.md)\n- [Production hardening notes](docs/production_hardening.md)\n- [GitOps deployment guide](docs/gitops_deployment.md)\n- [Terraform IaC guide](deploy/terraform/README.md)\n- [Security and supply chain](docs/security.md)\n- [Portfolio demo playbook](docs/demo.md)\n- [Release process](docs/release_process.md)\n- [Gateway overhead report](docs/gateway_overhead_report.md)\n- [Performance benchmarking guide](docs/performance_benchmarking.md)\n- [Project status and acceptance checklist](docs/project_status.md)\n- [Portfolio summary](docs/portfolio_summary.md)\n- [RAG integration guide](docs/rag_integration.md)\n- [Recommended GitHub repository metadata](docs/repository_metadata.md)\n\n## License\n\nThis project is released under the [MIT License](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fictup%2Fllm-serving-gateway-vllm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fictup%2Fllm-serving-gateway-vllm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fictup%2Fllm-serving-gateway-vllm/lists"}