{"id":38860724,"url":"https://github.com/haitwang-cloud/vllm-istio-stack","last_synced_at":"2026-01-17T14:21:44.292Z","repository":{"id":314984973,"uuid":"1057620354","full_name":"haitwang-cloud/vllm-istio-stack","owner":"haitwang-cloud","description":null,"archived":false,"fork":false,"pushed_at":"2025-09-16T02:46:20.000Z","size":24,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-16T03:33:15.127Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/haitwang-cloud.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-16T01:40:00.000Z","updated_at":"2025-09-16T02:46:24.000Z","dependencies_parsed_at":"2025-09-16T03:33:17.296Z","dependency_job_id":"4acc0acc-e6a6-4fd7-ab80-70876546fb49","html_url":"https://github.com/haitwang-cloud/vllm-istio-stack","commit_stats":null,"previous_names":["haitwang-cloud/vllm-istio-stack"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/haitwang-cloud/vllm-istio-stack","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/haitwang-cloud%2Fvllm-istio-stack","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/haitwang-cloud%2Fvllm-istio-stack/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/haitwang-cloud%2Fvllm-istio-stack/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/haitwang-cloud%2Fvllm-istio-stack/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/haitwang-cloud","download_url":"https://codeload.github.com/haitwang-cloud/vllm-istio-stack/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/haitwang-cloud%2Fvllm-istio-stack/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28509945,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-17T13:38:16.342Z","status":"ssl_error","status_checked_at":"2026-01-17T13:37:44.060Z","response_time":85,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-17T14:21:44.228Z","updated_at":"2026-01-17T14:21:44.279Z","avatar_url":"https://github.com/haitwang-cloud.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# vLLM-Istio Stack\n\nA comprehensive solution for integrating [vLLM](https://github.com/vllm-project/vllm) with [Istio](https://istio.io/) for production-grade Large Language Model (LLM) hosting in Kubernetes.\n\n## Overview\n\nThis repository provides everything you need to deploy vLLM as a scalable, production-ready LLM serving framework using Istio service mesh for advanced traffic management, security, and observability.\n\n## Features\n\n- 🚀 **Production-ready vLLM deployment** with Kubernetes manifests\n- 🌐 **Istio integration** for advanced traffic management and load balancing\n- 🔒 **Security configurations** including mTLS and authentication policies\n- 📊 **Observability** with metrics, logging, and distributed tracing\n- 🎛️ **Traffic management** with request routing and circuit breaking\n- ⚡ **Auto-scaling** configurations for high availability\n- 🐳 **Docker configurations** for custom vLLM images\n\n## Prerequisites\n\n- Kubernetes cluster (v1.20+)\n- Istio installed and configured (v1.15+)\n- kubectl configured to access your cluster\n- Docker (for building custom images)\n\n## Quick Start\n\n1. **Clone this repository:**\n   ```bash\n   git clone https://github.com/haitwang-cloud/vllm-istio-stack.git\n   cd vllm-istio-stack\n   ```\n\n2. **Deploy vLLM with Istio:**\n   ```bash\n   # Deploy the vLLM service\n   kubectl apply -f k8s/\n   \n   # Configure Istio traffic management\n   kubectl apply -f istio/\n   ```\n\n3. **Access your LLM endpoint:**\n   ```bash\n   # Get the Istio gateway external IP\n   kubectl get svc istio-ingressgateway -n istio-system\n   \n   # Test the endpoint\n   curl -X POST http://\u003cEXTERNAL-IP\u003e/v1/completions \\\n     -H \"Content-Type: application/json\" \\\n     -d '{\"model\": \"gpt2\", \"prompt\": \"Hello world\", \"max_tokens\": 50}'\n   ```\n\n## Architecture\n\n```\n┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐\n│   Client        │───▶│  Istio Gateway  │───▶│   vLLM Service  │\n└─────────────────┘    └─────────────────┘    └─────────────────┘\n                              │\n                              ▼\n                       ┌─────────────────┐\n                       │ Traffic Policies│\n                       │ • Load Balancing│\n                       │ • Circuit Break │\n                       │ • Rate Limiting │\n                       │ • mTLS          │\n                       └─────────────────┘\n```\n\n## Directory Structure\n\n```\nvllm-istio-stack/\n├── README.md                 # This file\n├── docker/                   # Docker configurations\n│   ├── Dockerfile           # Custom vLLM image\n│   └── requirements.txt     # Python dependencies\n├── k8s/                     # Kubernetes manifests\n│   ├── namespace.yaml       # Namespace configuration\n│   ├── deployment.yaml      # vLLM deployment\n│   ├── service.yaml         # Kubernetes service\n│   ├── configmap.yaml       # Configuration files\n│   └── hpa.yaml            # Horizontal Pod Autoscaler\n├── istio/                   # Istio configurations\n│   ├── gateway.yaml         # Istio Gateway\n│   ├── virtualservice.yaml  # Traffic routing rules\n│   ├── destinationrule.yaml # Load balancing policies\n│   ├── peerauthentication.yaml # mTLS configuration\n│   └── authorizationpolicy.yaml # Access control\n├── monitoring/              # Observability configs\n│   ├── servicemonitor.yaml  # Prometheus monitoring\n│   ├── grafana-dashboard.json # Grafana dashboard\n│   └── jaeger.yaml         # Distributed tracing\n├── scripts/                 # Deployment scripts\n│   ├── deploy.sh           # Full deployment script\n│   ├── cleanup.sh          # Cleanup script\n│   └── test.sh             # Testing script\n└── examples/               # Example configurations\n    ├── llama2/             # Llama 2 specific configs\n    ├── codellama/          # Code Llama configs\n    └── mixtral/            # Mixtral configs\n```\n\n## Configuration\n\n### Model Configuration\n\nEdit `k8s/configmap.yaml` to configure your model:\n\n```yaml\napiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: vllm-config\ndata:\n  model: \"meta-llama/Llama-2-7b-chat-hf\"  # Change this to your model\n  tensor_parallel_size: \"1\"\n  max_model_len: \"4096\"\n```\n\n### Scaling Configuration\n\nAdjust `k8s/hpa.yaml` for auto-scaling:\n\n```yaml\nspec:\n  minReplicas: 1\n  maxReplicas: 10\n  targetCPUUtilizationPercentage: 70\n```\n\n### Traffic Management\n\nConfigure load balancing in `istio/destinationrule.yaml`:\n\n```yaml\nspec:\n  trafficPolicy:\n    loadBalancer:\n      simple: LEAST_CONN  # Options: ROUND_ROBIN, LEAST_CONN, RANDOM\n```\n\n## Advanced Features\n\n### Circuit Breaker\n\nEnable circuit breaking to prevent cascade failures:\n\n```yaml\n# In destinationrule.yaml\nspec:\n  trafficPolicy:\n    outlierDetection:\n      consecutiveErrors: 3\n      interval: 30s\n      baseEjectionTime: 30s\n```\n\n### Rate Limiting\n\nConfigure rate limiting for API protection:\n\n```yaml\n# Custom rate limit configuration\napiVersion: networking.istio.io/v1alpha3\nkind: EnvoyFilter\nmetadata:\n  name: rate-limit-filter\nspec:\n  configPatches:\n  - applyTo: HTTP_FILTER\n    match:\n      listener:\n        filterChain:\n          filter:\n            name: \"envoy.filters.network.http_connection_manager\"\n    patch:\n      operation: INSERT_BEFORE\n      value:\n        name: envoy.filters.http.local_ratelimit\n```\n\n### GPU Support\n\nFor GPU-enabled deployments, update the deployment manifest:\n\n```yaml\nspec:\n  template:\n    spec:\n      containers:\n      - name: vllm\n        resources:\n          limits:\n            nvidia.com/gpu: \"1\"\n          requests:\n            nvidia.com/gpu: \"1\"\n```\n\n## Monitoring\n\nAccess monitoring dashboards:\n\n1. **Prometheus**: `http://\u003cCLUSTER-IP\u003e/prometheus`\n2. **Grafana**: `http://\u003cCLUSTER-IP\u003e/grafana`\n3. **Jaeger**: `http://\u003cCLUSTER-IP\u003e/jaeger`\n4. **Kiali**: `http://\u003cCLUSTER-IP\u003e/kiali`\n\n## Troubleshooting\n\n### Common Issues\n\n1. **Pod not starting**: Check GPU availability and model download\n   ```bash\n   kubectl describe pod -l app=vllm -n vllm-system\n   ```\n\n2. **Traffic not routing**: Verify Istio gateway configuration\n   ```bash\n   kubectl get gateway,vs,dr -n vllm-system\n   ```\n\n3. **High latency**: Check resource allocation and scaling policies\n   ```bash\n   kubectl top pods -n vllm-system\n   ```\n\n### Logs\n\nView logs for debugging:\n\n```bash\n# vLLM service logs\nkubectl logs -f deployment/vllm -n vllm-system\n\n# Istio proxy logs\nkubectl logs -f deployment/vllm -c istio-proxy -n vllm-system\n```\n\n## Contributing\n\n1. Fork the repository\n2. Create a feature branch: `git checkout -b feature/amazing-feature`\n3. Commit your changes: `git commit -m 'Add amazing feature'`\n4. Push to the branch: `git push origin feature/amazing-feature`\n5. Open a Pull Request\n\n## License\n\nThis project is licensed under the Apache License 2.0 - see the [LICENSE](LICENSE) file for details.\n\n## Support\n\n- 📖 [Documentation](https://github.com/haitwang-cloud/vllm-istio-stack/wiki)\n- 🐛 [Report Issues](https://github.com/haitwang-cloud/vllm-istio-stack/issues)\n- 💬 [Discussions](https://github.com/haitwang-cloud/vllm-istio-stack/discussions)\n\n## Acknowledgments\n\n- [vLLM Project](https://github.com/vllm-project/vllm) for the excellent LLM serving framework\n- [Istio Community](https://istio.io/) for the powerful service mesh platform\n- [Kubernetes](https://kubernetes.io/) for container orchestration","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhaitwang-cloud%2Fvllm-istio-stack","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhaitwang-cloud%2Fvllm-istio-stack","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhaitwang-cloud%2Fvllm-istio-stack/lists"}