{"id":33373606,"url":"https://github.com/frankbesch/nim-gke","last_synced_at":"2026-05-16T07:35:41.131Z","repository":{"id":320795714,"uuid":"1083371152","full_name":"frankbesch/nim-gke","owner":"frankbesch","description":"Production-grade NVIDIA NIM inference on GKE with L4 GPUs, autoscaling, and operational runbooks.","archived":false,"fork":false,"pushed_at":"2025-10-25T22:36:24.000Z","size":135,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-10-26T00:08:41.639Z","etag":null,"topics":["ai","cloud","gke","gpu","inference","kubernetes","llm","min","ml","nvidia","tensorrt","vllm"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/frankbesch.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-10-25T21:55:43.000Z","updated_at":"2025-10-25T22:36:27.000Z","dependencies_parsed_at":"2025-10-26T00:08:43.170Z","dependency_job_id":"c5ec9255-7240-4296-a665-ed00851db042","html_url":"https://github.com/frankbesch/nim-gke","commit_stats":null,"previous_names":["frankbesch/nim-gke"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/frankbesch/nim-gke","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/frankbesch%2Fnim-gke","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/frankbesch%2Fnim-gke/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/frankbesch%2Fnim-gke/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/frankbesch%2Fnim-gke/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/frankbesch","download_url":"https://codeload.github.com/frankbesch/nim-gke/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/frankbesch%2Fnim-gke/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":285873538,"owners_count":27246054,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-11-22T02:00:05.934Z","response_time":64,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","cloud","gke","gpu","inference","kubernetes","llm","min","ml","nvidia","tensorrt","vllm"],"created_at":"2025-11-22T23:00:33.228Z","updated_at":"2025-11-22T23:01:05.389Z","avatar_url":"https://github.com/frankbesch.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# nim-gke\n\n**GPU-accelerated NVIDIA NIM inference on Google Kubernetes Engine**\n\nProduction-grade reference implementation for deploying NVIDIA NIM microservices on GKE with L4 GPUs, autoscaling, and cost optimization.\n\n**Based on**: [Google Codelabs - Deploy an AI model on GKE with NVIDIA NIM](https://codelabs.developers.google.com/codelabs/nvidia-nim-google-cloud)\n\n---\n\n## What This Adds to the Tutorial\n\nThis repository extends the [official Google Codelabs tutorial](https://codelabs.developers.google.com/codelabs/nvidia-nim-google-cloud) with production-grade enhancements:\n\n**Operational Excellence**:\n- Comprehensive error handling (`set -euo pipefail` in all scripts)\n- Idempotent operations (safe to run multiple times)\n- 60-minute deployment monitoring script\n- Troubleshooting runbook (465 lines, 6 failure modes)\n- Cost tracking and optimization strategies\n\n**Automation \u0026 Testing**:\n- Environment validation script (prerequisites, quotas, NGC key)\n- Integration test suite with load testing\n- CI/CD validation (shellcheck, yamllint, security scanning)\n- Automated cleanup with verification\n\n**Production Features**:\n- Autoscaling GPU node pool (0-2 nodes)\n- Cost optimization ($1.36/hour vs. tutorial's fixed deployment)\n- Persistent volume for model caching (faster restarts)\n- Resource limits and requests defined\n- Production Helm values configuration\n\n**Documentation**:\n- Architecture deep-dive (517 lines: GPU memory layout, autoscaling mechanics)\n- Interview preparation guide (357 lines: design decisions, talking points)\n- Operational runbooks (troubleshooting, monitoring, incident response)\n- Quick reference guide (one-page ops commands)\n- Script documentation (usage, security, examples)\n\n**Developer Experience**:\n- Structured repository (charts, scripts, docs, runbooks separated)\n- GitHub templates (PR, issues)\n- Contributing guidelines\n- Verification script (validates complete setup)\n\n**Tutorial Compatibility**: All core deployment steps from the [Google Codelabs tutorial](https://codelabs.developers.google.com/codelabs/nvidia-nim-google-cloud) are preserved and enhanced, not replaced.\n\n---\n\n## Architecture\n\nNIM container → TensorRT-LLM → vLLM backend → L4 GPU → GKE node pool\n\n**Components**:\n- **Model**: Meta Llama 3 8B Instruct\n- **Runtime**: NVIDIA NIM 1.0.0 (TensorRT-LLM + vLLM)\n- **Orchestration**: Kubernetes StatefulSet + Helm\n- **Compute**: GKE with g2-standard-4 nodes (L4 GPU, 24GB VRAM)\n- **API**: OpenAI-compatible REST (`/v1/chat/completions`)\n\n**Autoscaling**: GPU node pool scales 0→2 based on pod requests.\n\n**Cost**: ~$1.36/hour when active. $0/hour when scaled to zero.\n\n---\n\n## Prerequisites\n\n| Requirement | Version | Purpose |\n|-------------|---------|---------|\n| `gcloud` CLI | Latest | GCP authentication, cluster management |\n| `kubectl` | 1.28+ | Kubernetes operations |\n| `helm` | 3.0+ | Chart deployment |\n| NGC API Key | — | NIM image registry auth |\n| GCP Project | — | Billing enabled |\n| GPU Quota | 1× L4 | us-central1 or compatible region |\n\n**GPU quota approval**: Required before deployment. See `/docs/GPU_QUOTA_GUIDE.md`.\n\n---\n\n## Deployment\n\n### Quick Start\n\n```bash\n# 1. Set NGC API key\nexport NGC_CLI_API_KEY='your-key-here'\n\n# 2. Configure project\nexport PROJECT_ID=\"your-gcp-project\"\nexport REGION=\"us-central1\"\nexport ZONE=\"us-central1-a\"\n\n# 3. Deploy\n./scripts/deploy_nim_gke.sh\n\n# 4. Verify\nkubectl get pods -n nim\nkubectl port-forward service/my-nim-nim-llm 8000:8000 -n nim\n```\n\n### Production Deployment\n\n```bash\n# Validate environment\n./scripts/setup_environment.sh\n\n# Deploy with production values\n./scripts/deploy_nim_production.sh\n\n# Run integration tests\n./scripts/test_nim_production.sh\n```\n\n**Expected duration**: 25-35 minutes (cluster creation + model loading).\n\n---\n\n## Verify\n\n```bash\n# Health check\ncurl http://localhost:8000/v1/health/ready\n\n# List models\ncurl http://localhost:8000/v1/models\n\n# Inference test\ncurl -X POST http://localhost:8000/v1/chat/completions \\\n  -H 'Content-Type: application/json' \\\n  -d '{\n    \"messages\": [{\"role\": \"user\", \"content\": \"What is TensorRT?\"}],\n    \"model\": \"meta/llama3-8b-instruct\",\n    \"max_tokens\": 100\n  }'\n```\n\n**Expected response time**: 3-6 seconds.\n\n---\n\n## Operate\n\n### Monitor\n\n```bash\n# Pod status\nkubectl get pods -n nim -w\n\n# Logs\nkubectl logs -f my-nim-nim-llm-0 -n nim\n\n# GPU utilization\nkubectl exec -n nim my-nim-nim-llm-0 -- nvidia-smi\n\n# Resource usage\nkubectl top pod -n nim\n```\n\n### Scale\n\n```bash\n# Manual scale (StatefulSet)\nkubectl scale statefulset my-nim-nim-llm --replicas=2 -n nim\n\n# GPU node pool resize\ngcloud container node-pools resize gpupool \\\n  --cluster=nim-demo \\\n  --zone=us-central1-a \\\n  --num-nodes=2\n```\n\n### Cost Control\n\n```bash\n# Remove deployment (keep cluster)\nhelm uninstall my-nim -n nim\n# GPU nodes auto-scale to 0\n\n# Delete cluster (stop all costs)\n./scripts/cleanup.sh\n```\n\n---\n\n## Troubleshoot\n\n**Pod stuck in Pending**:\n```bash\nkubectl describe pod -n nim my-nim-nim-llm-0\n# Check: GPU availability, node readiness, quotas\n```\n\n**ImagePullBackOff**:\n```bash\n# Verify NGC secret\nkubectl get secret ngc-api -n nim -o yaml\n# Recreate if needed\nkubectl delete secret ngc-api -n nim\nkubectl create secret generic ngc-api \\\n  --from-literal=NGC_API_KEY=$NGC_CLI_API_KEY \\\n  -n nim\n```\n\n**Model loading slow**:\n- Expected: 10-15 minutes on first deployment\n- Monitor: `kubectl logs -f my-nim-nim-llm-0 -n nim`\n\nSee `/runbooks/troubleshooting.md` for complete procedures.\n\n---\n\n## Repository Structure\n\n```\nnim-gke/\n├── charts/                     # Helm charts and values\n│   ├── nim-llm-1.3.0.tgz      # NVIDIA NIM chart\n│   └── values-production.yaml  # Production config\n├── scripts/                    # Deployment and ops scripts\n│   ├── deploy_nim_gke.sh      # Main deployment\n│   ├── setup_environment.sh    # Prerequisite validation\n│   ├── test_nim_production.sh  # Integration tests\n│   ├── cleanup.sh             # Resource deletion\n│   └── monitor_deployment.sh   # Status monitoring\n├── docs/                       # Documentation\n│   ├── DEPLOYMENT_SUCCESS.md   # Deployment guide\n│   ├── PRODUCTION_GUIDE.md     # Operations manual\n│   ├── GPU_QUOTA_GUIDE.md      # Quota request process\n│   └── interview/              # Interview preparation materials\n├── runbooks/                   # Operational procedures\n│   └── troubleshooting.md      # Incident response\n├── examples/                   # Configuration templates\n│   └── set_ngc_key.sh.template # NGC key setup\n└── README.md                   # This file\n```\n\n---\n\n## Configuration\n\n### Helm Values\n\nEdit `charts/values-production.yaml`:\n\n```yaml\nimage:\n  repository: \"nvcr.io/nim/meta/llama3-8b-instruct\"\n  tag: \"1.0.0\"\n\nresources:\n  limits:\n    nvidia.com/gpu: 1\n  requests:\n    nvidia.com/gpu: 1\n\npersistence:\n  enabled: true\n  size: 50Gi\n```\n\n### Environment Variables\n\n| Variable | Default | Purpose |\n|----------|---------|---------|\n| `PROJECT_ID` | `your-gcp-project` | GCP project |\n| `REGION` | `us-central1` | GCP region |\n| `ZONE` | `us-central1-a` | GKE zone |\n| `CLUSTER_NAME` | `nim-demo` | Cluster identifier |\n| `GPU_TYPE` | `nvidia-l4` | GPU accelerator type |\n| `NODE_POOL_MACHINE_TYPE` | `g2-standard-4` | Node instance type |\n\n---\n\n## Performance\n\n| Metric | Value | Notes |\n|--------|-------|-------|\n| **First token latency** | 2-3s | Cold start |\n| **Throughput** | 15-20 tokens/s | L4 GPU, FP16 |\n| **Batch size** | Dynamic | vLLM continuous batching |\n| **Context length** | 8192 tokens | Llama 3 limit |\n| **GPU memory** | ~12GB used | Of 24GB available |\n\n---\n\n## Cost\n\n**Baseline** (no load):\n- Control plane: $0.13/hour\n- **Total**: $0.13/hour\n\n**Active** (1 GPU node):\n- Control plane: $0.13/hour\n- GPU node (g2-standard-4): $0.50/hour\n- L4 GPU: $0.73/hour\n- **Total**: $1.36/hour (~$980/month)\n\n**Optimization strategies**:\n1. Autoscaling to zero when idle\n2. Preemptible nodes (-80% cost, accepts interruption)\n3. Committed use discounts (-37% for 3-year)\n4. Regional vs. zonal deployment tradeoffs\n\n---\n\n## Security\n\n- ✅ NGC API key stored as Kubernetes Secret\n- ✅ Image pull secrets for nvcr.io registry\n- ✅ Service exposed via ClusterIP (internal only)\n- ✅ TLS for production (configure Ingress + cert-manager)\n- ⚠️ Authentication: Implement API gateway for production workloads\n\n---\n\n## Limitations\n\n- **Single GPU**: Multi-GPU tensor parallelism requires code changes\n- **Model size**: Llama 3 8B fits L4. Larger models need A100/H100\n- **Persistence**: Model cached on PV. Deletion triggers re-download\n- **Regional availability**: L4 not in all GCP zones\n\n---\n\n## References\n\n### Primary Sources\n\n- **[Google Codelabs - Deploy AI on GKE with NVIDIA NIM](https://codelabs.developers.google.com/codelabs/nvidia-nim-google-cloud)** - Original tutorial this repository is based on\n- **[NVIDIA NIM Documentation](https://docs.nvidia.com/nim/)** - Official NIM microservices documentation\n- **[GKE GPU Guide](https://cloud.google.com/kubernetes-engine/docs/how-to/gpus)** - Google Cloud GPU setup and configuration\n\n### Core Technologies\n\n- **[TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM)** - NVIDIA's optimized inference engine (FP16 precision, fused kernels)\n- **[vLLM](https://github.com/vllm-project/vllm)** - High-throughput LLM serving framework (continuous batching, PagedAttention)\n- **[Kubernetes](https://kubernetes.io/docs/)** - Container orchestration platform\n- **[Helm](https://helm.sh/docs/)** - Kubernetes package manager\n\n### Additional Resources\n\n- **[NVIDIA AI Enterprise](https://www.nvidia.com/en-us/data-center/products/ai-enterprise/)** - Enterprise AI software platform\n- **[GCP GPU Regions](https://cloud.google.com/compute/docs/gpus/gpu-regions-zones)** - GPU availability by region\n- **[Llama 3 Model Card](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct)** - Model documentation\n\n---\n\n## License\n\nProvided as-is for educational and reference purposes. NVIDIA NIM requires acceptance of NVIDIA AI Enterprise EULA.\n\n---\n\n**Status**: Production-ready ✅  \n**Last validated**: October 2025  \n**GKE version**: 1.34+  \n**NIM version**: 1.0.0\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffrankbesch%2Fnim-gke","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffrankbesch%2Fnim-gke","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffrankbesch%2Fnim-gke/lists"}