{"id":48103741,"url":"https://github.com/kholcomb/gateway_eks","last_synced_at":"2026-04-04T15:48:51.618Z","repository":{"id":328509072,"uuid":"1110816054","full_name":"kholcomb/gateway_eks","owner":"kholcomb","description":"[PoC] EKS deployment of AI-Governance stack","archived":false,"fork":false,"pushed_at":"2026-03-26T01:12:03.000Z","size":324,"stargazers_count":0,"open_issues_count":7,"forks_count":0,"subscribers_count":0,"default_branch":"dev","last_synced_at":"2026-03-26T23:19:12.054Z","etag":null,"topics":["ai","aws","bedrock","ec2","eks","grafana","litellm","openwebui","prometheus","rds","redis","secretsmanager","terraform"],"latest_commit_sha":null,"homepage":"","language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kholcomb.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":"security/ARCHITECTURE.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-12-05T19:07:11.000Z","updated_at":"2026-01-09T18:05:57.000Z","dependencies_parsed_at":"2026-02-23T11:06:40.019Z","dependency_job_id":null,"html_url":"https://github.com/kholcomb/gateway_eks","commit_stats":null,"previous_names":["kholcomb/gateway_eks"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/kholcomb/gateway_eks","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kholcomb%2Fgateway_eks","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kholcomb%2Fgateway_eks/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kholcomb%2Fgateway_eks/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kholcomb%2Fgateway_eks/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kholcomb","download_url":"https://codeload.github.com/kholcomb/gateway_eks/tar.gz/refs/heads/dev","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kholcomb%2Fgateway_eks/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31404014,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-04T10:20:44.708Z","status":"ssl_error","status_checked_at":"2026-04-04T10:20:06.846Z","response_time":60,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","aws","bedrock","ec2","eks","grafana","litellm","openwebui","prometheus","rds","redis","secretsmanager","terraform"],"created_at":"2026-04-04T15:48:51.102Z","updated_at":"2026-04-04T15:48:51.610Z","avatar_url":"https://github.com/kholcomb.png","language":"HCL","funding_links":[],"categories":[],"sub_categories":[],"readme":"# LiteLLM + OpenWebUI EKS Deployment\n\n**Production-ready AI/LLM infrastructure on Amazon EKS** with comprehensive observability, security, and authentication.\n\n## Quick Links\n\n| Guide | Description |\n|-------|-------------|\n| 🚀 [Quick Start](#quick-start) | Get started in 20-35 minutes |\n| 📖 [Deployment Guide](docs/DEPLOYMENT_GUIDE.md) | Complete step-by-step walkthrough |\n| 🔐 [JWT Setup](docs/JWT_AUTHENTICATION_SETUP.md) | Configure Okta OIDC authentication |\n| 📦 [ECR Setup](docs/ECR_SETUP.md) | Container registry configuration |\n| 🤖 [MCP Deployment](docs/MCP_DEPLOYMENT.md) | Deploy Model Context Protocol servers |\n| 🏗️ [MCP Operator](docs/MCP_OPERATOR_ARCHITECTURE.md) | Kubernetes operator for MCP servers |\n\n---\n\n## Architecture\n\n```mermaid\ngraph TB\n    subgraph \"Users\"\n        User[Users/Clients]\n    end\n\n    subgraph \"External Services\"\n        Bedrock[AWS Bedrock\u003cbr/\u003eClaude, Llama, Mistral]\n        RDS[Amazon RDS\u003cbr/\u003ePostgreSQL]\n        SecretsManager[AWS Secrets Manager]\n        Okta[Okta OIDC]\n    end\n\n    subgraph \"EKS Cluster\"\n        subgraph \"Application\"\n            OpenWebUI[OpenWebUI\u003cbr/\u003eChat Frontend]\n            LiteLLM[LiteLLM Proxy\u003cbr/\u003eJWT Auth + Routing]\n            Redis[Redis HA\u003cbr/\u003eCaching]\n        end\n\n        subgraph \"Observability\"\n            Prometheus[Prometheus]\n            Grafana[Grafana]\n            Jaeger[Jaeger]\n        end\n\n        subgraph \"Security\"\n            ESO[External Secrets]\n            OPA[OPA Gatekeeper]\n        end\n    end\n\n    User --\u003e|HTTPS| OpenWebUI\n    OpenWebUI --\u003e|API + JWT| LiteLLM\n    LiteLLM --\u003e|Model Requests| Bedrock\n    LiteLLM --\u003e|Cache| Redis\n    OpenWebUI --\u003e|Session Data| RDS\n    LiteLLM --\u003e|Metrics| Prometheus\n    Prometheus --\u003e|Visualize| Grafana\n    ESO --\u003e|Sync Secrets| SecretsManager\n\n    style LiteLLM fill:#326CE5,color:#fff\n    style OpenWebUI fill:#61DAFB\n```\n\n## Components\n\n| Component | Purpose |\n|-----------|---------|\n| **LiteLLM** | API gateway to AWS Bedrock models |\n| **OpenWebUI** | Chat frontend with Okta authentication |\n| **Redis** | Caching |\n| **Prometheus/Grafana** | Metrics collection \u0026 visualization |\n| **Jaeger** | Distributed tracing |\n| **External Secrets** | AWS Secrets Manager integration |\n| **OPA Gatekeeper** |Policy enforcement |\n\n---\n\n## Prerequisites\n\n### AWS Account Setup\n- AWS account with appropriate permissions\n- AWS CLI v2 configured (`aws configure`)\n- EKS cluster permissions\n\n### Local Tools\n```bash\n# macOS\nbrew install awscli kubectl helm\n\n# Verify installations\naws --version      # AWS CLI 2.x\nkubectl version    # v1.28+\nhelm version       # v3.0+\n```\n\n---\n\n## Quick Start\n\n### 1. Deploy Infrastructure\n\n```bash\n# Set environment variables\nexport AWS_REGION=us-east-1\nexport EKS_CLUSTER_NAME=litellm-eks\nexport AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)\n\n# Choose deployment method\ncd scripts\n./deploy.sh infrastructure\n# You'll be prompted to choose: [T]erraform or [E]ksctl\n```\n\n### 2. Configure kubectl\n\n```bash\naws eks update-kubeconfig --name $EKS_CLUSTER_NAME --region $AWS_REGION\nkubectl cluster-info  # Verify connection\n```\n\n### 3. Create Required Secrets\n\nCreate Okta secrets in AWS Secrets Manager ([detailed guide](docs/JWT_AUTHENTICATION_SETUP.md)):\n\n```bash\n# LiteLLM: JWT public key URL\naws secretsmanager create-secret \\\n  --name litellm/jwt-public-key-url \\\n  --secret-string \"https://\u003cyour-okta-domain\u003e/oauth2/default/v1/keys\" \\\n  --region $AWS_REGION\n\n# OpenWebUI: Session encryption, Okta client ID/secret, admin email\n# See JWT_AUTHENTICATION_SETUP.md for complete secret creation steps\n```\n\n**Note:** Database URL secret should already exist from Terraform/eksctl setup.\n\n### 4. Deploy Applications\n\n```bash\ncd scripts\n./deploy.sh all\n```\n\nThis deploys:\n- ✅ External Secrets Operator\n- ✅ OPA Gatekeeper + policies\n- ✅ Prometheus/Grafana monitoring\n- ✅ Jaeger distributed tracing\n- ✅ Redis cluster\n- ✅ LiteLLM proxy with JWT authentication\n- ✅ OpenWebUI with Okta OIDC\n\n### 5. Verify Deployment\n\n```bash\nkubectl get pods -A | grep -E 'litellm|open-webui|monitoring|redis'\nkubectl get externalsecret -A  # Verify secrets synced\n```\n\n### 6. Access Applications\n\n**Option A: From bastion host**\n```bash\n./scripts/setup-bastion.sh create\n./scripts/setup-bastion.sh connect\n\n# Inside bastion:\nllm-ui          # OpenWebUI → http://localhost:8080\nllm-grafana     # Grafana → http://localhost:3000\n```\n\n**Option B: Port-forward from local machine**\n```bash\n# OpenWebUI\nkubectl port-forward -n open-webui svc/open-webui 8080:80\n\n# Grafana (default: admin / prom-operator)\nkubectl port-forward -n monitoring svc/kube-prometheus-grafana 3000:80\n```\n\n---\n\n## Deployment Options\n\nThe `deploy.sh` script supports granular deployment:\n\n```bash\n# Full deployment\n./deploy.sh all\n\n# Infrastructure only\n./deploy.sh terraform    # or: ./deploy.sh eksctl\n\n# Individual components\n./deploy.sh irsa                 # Create IAM roles\n./deploy.sh secrets              # Create AWS secrets\n./deploy.sh external-secrets     # Deploy External Secrets Operator\n./deploy.sh redis                # Deploy Redis HA\n./deploy.sh litellm              # Deploy LiteLLM\n./deploy.sh openwebui            # Deploy OpenWebUI\n./deploy.sh monitoring           # Deploy Prometheus/Grafana\n./deploy.sh jaeger               # Deploy Jaeger\n./deploy.sh gatekeeper           # Deploy OPA Gatekeeper\n./deploy.sh verify               # Verify deployment\n\n# Complete teardown\n./deploy.sh infrastructure-destroy\n```\n\n**Deployment Modes:**\n- **Interactive** (default): Prompts before updating existing resources\n- **Non-interactive**: `INTERACTIVE_MODE=false ./deploy.sh all`\n\n📖 **See [Deployment Guide](docs/DEPLOYMENT_GUIDE.md)** for detailed deployment workflows.\n\n---\n\n## Configuration\n\n### LiteLLM Models\n\n**Customize models:** Edit `helm-values/litellm-values.yaml`\n\n### Required AWS Secrets\n\n| Secret Name | Description | Created By |\n|-------------|-------------|------------|\n| `litellm/database-url` | PostgreSQL connection string | Manual |\n| `litellm/jwt-public-key-url` | Okta JWKS endpoint | Manual |\n| `litellm/master-key` | LiteLLM admin key | deploy.sh |\n| `litellm/salt-key` | DB encryption salt (immutable) | deploy.sh |\n| `litellm/redis-password` | Redis password | deploy.sh |\n| `openwebui/webui-secret-key` | Session encryption | Manual |\n| `openwebui/okta-openid-url` | Okta OpenID discovery URL | Manual |\n| `openwebui/okta-client-id` | Okta app client ID | Manual |\n| `openwebui/okta-client-secret` | Okta app client secret | Manual |\n| `openwebui/admin-email` | Admin user emails | Manual |\n\n📖 **See [JWT Authentication Setup](docs/JWT_AUTHENTICATION_SETUP.md)** for detailed secret creation.\n\n---\n\n## Monitoring \u0026 Observability\n\n### Grafana Dashboards\n\nAccess: `kubectl port-forward -n monitoring svc/kube-prometheus-grafana 3000:80`\n\n### Prometheus Metrics\n\nAccess: `kubectl port-forward -n monitoring svc/kube-prometheus-kube-prome-prometheus 9090:9090`\n\n### Jaeger Tracing\n\nAccess: `kubectl port-forward -n monitoring svc/jaeger-query 16686:16686`\n\n---\n\n## Advanced Features\n\n### Model Context Protocol (MCP) Servers\n\n📖 **See [MCP Deployment Guide](docs/MCP_DEPLOYMENT.md)** for deployment patterns and examples.\n\n📖 **See [MCP Operator Architecture](docs/MCP_OPERATOR_ARCHITECTURE.md)** for Kubernetes operator design.\n\n### Container Registry (ECR)\n\n📖 **See [ECR Setup Guide](docs/ECR_SETUP.md)** for detailed configuration.\n\n### OPA Gatekeeper Policies\n\nSecurity policies automatically enforced:\n\n- ✅ Approved container registries only\n- ✅ No `:latest` image tags\n- ✅ Container resource limits required\n- ✅ Non-root containers only\n- ✅ Required labels and probes\n\n[View policies](/manifests/opa-policies/)\n\n---\n\n## Troubleshooting\n\n### Quick Diagnostics\n\n```bash\n# Check all pods\nkubectl get pods -A | grep -E 'litellm|open-webui|monitoring|redis'\n\n# Check External Secrets sync\nkubectl get externalsecret -A\nkubectl describe externalsecret litellm-secrets -n litellm\n\n# Check LiteLLM logs\nkubectl logs -n litellm -l app.kubernetes.io/name=litellm --tail=100\n\n# Check OpenWebUI logs\nkubectl logs -n open-webui -l app.kubernetes.io/name=open-webui --tail=100\n```\n\n📖 **See [Deployment Guide](docs/DEPLOYMENT_GUIDE.md#troubleshooting)** for comprehensive troubleshooting.\n\n---\n\n## Cleanup\n\n```bash\n# Delete bastion host\n./scripts/setup-bastion.sh cleanup\n\n# Delete applications\nhelm uninstall open-webui -n open-webui\nhelm uninstall litellm -n litellm\nhelm uninstall redis -n litellm\nhelm uninstall jaeger -n monitoring\nhelm uninstall kube-prometheus -n monitoring\nhelm uninstall external-secrets -n external-secrets\n\n# Delete infrastructure\n./scripts/deploy.sh infrastructure-destroy\n```\n\n---\n\n## Contributing\n\n### Infrastructure\n- **[terraform/README.md](terraform/README.md)** - Terraform deployment guide\nSee [CONTRIBUTING.md](CONTRIBUTING.md) for git workflow and contribution guidelines.\n\n## Additional Resources\n\n### External Documentation\n- [LiteLLM Documentation](https://docs.litellm.ai/)\n- [OpenWebUI Documentation](https://docs.openwebui.com/)\n- [AWS Bedrock Models](https://aws.amazon.com/bedrock/claude/)\n- [OPA Gatekeeper](https://open-policy-agent.github.io/gatekeeper/)\n- [External Secrets Operator](https://external-secrets.io/)\n\n### Related Guides\n- [Script Usage](scripts/README.md)\n- [Security Architecture](security/ARCHITECTURE.md)\n- [OPA Policies](manifests/opa-policies/README.md)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkholcomb%2Fgateway_eks","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkholcomb%2Fgateway_eks","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkholcomb%2Fgateway_eks/lists"}