{"id":51049604,"url":"https://github.com/mrjhonyvidal/eks-ai-ops-toolkit","last_synced_at":"2026-06-22T16:02:40.931Z","repository":{"id":356946943,"uuid":"1234265479","full_name":"mrjhonyvidal/eks-ai-ops-toolkit","owner":"mrjhonyvidal","description":"AI-powered Ops Toolkit for EKS featuring proactive incident resolution and an interactive Slack chatbot with MCP/Strands tool calling.","archived":false,"fork":false,"pushed_at":"2026-05-10T16:07:28.000Z","size":96,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-10T16:34:29.240Z","etag":null,"topics":["aws","eks","mcp","strands-agents"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mrjhonyvidal.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-10T00:40:43.000Z","updated_at":"2026-05-10T16:07:32.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/mrjhonyvidal/eks-ai-ops-toolkit","commit_stats":null,"previous_names":["mrjhonyvidal/eks-sre-agent"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/mrjhonyvidal/eks-ai-ops-toolkit","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrjhonyvidal%2Feks-ai-ops-toolkit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrjhonyvidal%2Feks-ai-ops-toolkit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrjhonyvidal%2Feks-ai-ops-toolkit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrjhonyvidal%2Feks-ai-ops-toolkit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mrjhonyvidal","download_url":"https://codeload.github.com/mrjhonyvidal/eks-ai-ops-toolkit/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrjhonyvidal%2Feks-ai-ops-toolkit/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34655724,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-22T02:00:06.391Z","response_time":106,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","eks","mcp","strands-agents"],"created_at":"2026-06-22T16:02:40.241Z","updated_at":"2026-06-22T16:02:40.925Z","avatar_url":"https://github.com/mrjhonyvidal.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# EKS AI Ops Toolkit\n\nAI-powered Ops Toolkit for EKS featuring proactive incident resolution and an interactive Slack chatbot with MCP/Strands tool calling.\n\n![Slack Bot Screenshot](./example-slack-bot.png)\n\nServerless AI operations toolkit for EKS with two reusable capabilities in one repository:\n\n1. **Proactive AI monitoring**: automatic EKS issue detection, analysis, Slack notification, optional GitHub auto-fix PR.\n\n\n2. **Interactive Slack bot**: Slack Q\u0026A for EKS troubleshooting using an orchestrator + specialist + MCP-backed tools.\n\n## Architecture\n\n### Proactive flow\n\n```mermaid\ngraph LR\n    A[EventBridge / CloudWatch] --\u003e B(Proactive Handler)\n    B --\u003e C[EKS AI Toolkit]\n    C --\u003e D[(DynamoDB Record)]\n    C --\u003e E[Slack Update]\n    C --\u003e F[GitHub PR Auto-fix]\n```\n\n### Interactive flow (re:Invent-style semantics)\n\n```mermaid\ngraph LR\n    A[Slack Interface] --\u003e B(K8S Orchestrator Agent)\n    B --\u003e C{Intent?}\n    C --\u003e|Non-K8s| D[Early Exit]\n    C --\u003e|K8s Troubleshooting| E[K8S Specialist Agent]\n    E \u003c--\u003e F[Amazon EKS Hosted MCP Tools]\n```\n\n### Concrete code mapping\n\n- Proactive Lambda entrypoint: `eks_ai_ops.proactive.handler.handler`\n- Proactive orchestration: `eks_ai_ops.proactive.flow.ProactiveIncidentFlow`\n- Interactive Lambda entrypoint: `eks_ai_ops.interactive.handler.handler`\n- **K8S Orchestrator Agent**: `eks_ai_ops.interactive.orchestrator.K8sOrchestratorAgent`\n- Intent classifier: `eks_ai_ops.interactive.orchestrator.K8sIntentClassifier`\n- **K8S Specialist Agent**: `eks_ai_ops.interactive.strands_agent.InteractiveEKSAgent`\n- MCP adapter: `eks_ai_ops.interactive.mcp_tools`\n- Backward-compatible wrappers: `eks_ai_ops.handler`, `eks_ai_ops.bot_handler`\n\n## Repository structure\n\n```text\nsrc/eks_ai_ops/\n  proactive/\n    handler.py\n    flow.py\n  interactive/\n    handler.py\n    orchestrator.py\n    strands_agent.py\n    mcp_tools.py\n  shared/\n    config.py\n    prompts.py\n```\n\n## Prerequisites\n\n- Python 3.11+\n- AWS account with permissions for Lambda, EventBridge, CloudWatch, DynamoDB, IAM, API Gateway\n- Slack workspace where you can create/install apps\n- Optional:\n  - Bedrock access for Nova/Claude models\n  - Strands SDK (`pip install -e \".[strands]\"`)\n  - MCP gateway that exposes EKS tools\n\n## Installation\n\n```bash\ncp .env.template .env\npip install -e \".[dev]\"\n```\n\nOptional specialist backend extras:\n\n```bash\npip install -e \".[strands]\"\n```\n\n## Environment variables\n\nUse `.env.template`.\n\n### Shared\n\n- `AWS_REGION` (default: `us-east-1`)\n- `LLM_PROVIDER` (`bedrock` or `anthropic`)\n- `BEDROCK_MODEL_ID`\n- `CLUSTER_NAME`\n- `INCIDENT_TABLE`\n- `DEPLOY_TABLE`\n\n### Slack\n\n- `SLACK_BOT_TOKEN`\n- `SLACK_CHANNEL`\n- `SLACK_SIGNING_SECRET`\n\n### Proactive flow\n\n- `GITHUB_TOKEN`\n- `GITHUB_REPO`\n- `GITHUB_BASE_BRANCH`\n- `KUBECTL_LAMBDA`\n\n### Interactive flow\n\n- `INTERACTIVE_AGENT_BACKEND` (default `strands`)\n- `EKS_MCP_SERVER` (default `eks`)\n- `MCP_GATEWAY_URL`\n- `MCP_GATEWAY_API_KEY` (optional)\n- `INTENT_USE_LLM` (`false` by default)\n- `INTENT_MODEL_ID` (default: `us.amazon.nova-micro-v1:0`)\n\n## MCP gateway (interactive bot)\n\nThe interactive specialist calls Kubernetes tools through an HTTP MCP gateway.\nSet `MCP_GATEWAY_URL` to a service that accepts:\n\n```text\nPOST {MCP_GATEWAY_URL}/tools/call\n{ \"server\": \"eks\", \"tool\": \"list_pods\", \"arguments\": { \"namespace\": \"api\" } }\n```\n\nTools dispatched today: `list_pods`, `describe_resource`, `get_pod_logs`\n(see `src/eks_ai_ops/interactive/mcp_tools.py`).\n\nIf `MCP_GATEWAY_URL` is empty, the bot still answers via the LLM-only fallback —\nuseful for a first run without standing up a gateway. Add an API key with\n`MCP_GATEWAY_API_KEY` if your gateway requires bearer auth.\n\n### Picking a gateway\n\n`MCP_GATEWAY_URL` is **not** an AWS-managed endpoint — it's whatever HTTP\nservice speaks the contract above. Options, easiest first:\n\n1. **AWS Labs `eks-mcp-server`** (recommended) — official open-source MCP\n   server with EKS-aware tools. It speaks MCP-over-stdio out of the box, so\n   you wrap it behind a thin HTTP shim that translates `POST /tools/call` →\n   MCP `tools/call`. Repo: \u003chttps://github.com/awslabs/mcp\u003e (see\n   `src/eks-mcp-server/`).\n2. **Self-host the shim on AWS:**\n   - **App Runner** — simplest. Push a container, get an HTTPS URL, point\n     `MCP_GATEWAY_URL` at it.\n   - **ECS Fargate behind ALB** — if you need VPC peering with EKS.\n   - **Lambda + Function URL** — fine for low traffic; cold starts add latency.\n3. **kubectl in-VPC** — `KubectlHelperFunction` already runs inside the EKS\n   VPC and can reach the cluster API. The MCP gateway can delegate to it for\n   real `kubectl` calls (this is how the AWS Labs server talks to the cluster\n   too).\n\nAuth: store any API key in SSM as `/eks-ai-ops-toolkit/mcp-gateway-api-key`\n(SecureString) and add a corresponding parameter mapping to\n`infrastructure/template.yaml` if you want it injected automatically. Until\nthen `MCP_GATEWAY_API_KEY` is read from the Lambda environment.\n\n\u003e **Quickest path:** skip the gateway. The Slack bot will answer Kubernetes\n\u003e questions from Bedrock alone. Add the gateway later when you want the bot\n\u003e to actually run `kubectl` against a cluster.\n\n## Kubectl helper VPC (SSM parameters)\n\nThe `KubectlHelperFunction` Lambda runs inside the EKS VPC so it can reach the\ncluster API. Before `make deploy` you must publish three SSM parameters:\n\n```bash\naws ssm put-parameter --name /eks-ai-ops-toolkit/eks-vpc-id \\\n  --value vpc-xxxxxxxx --type String --overwrite\naws ssm put-parameter --name /eks-ai-ops-toolkit/eks-private-subnet-1 \\\n  --value subnet-aaaaaaaa --type String --overwrite\naws ssm put-parameter --name /eks-ai-ops-toolkit/eks-private-subnet-2 \\\n  --value subnet-bbbbbbbb --type String --overwrite\n```\n\nUse private subnets that have a route to the EKS control plane and to NAT for\npulling Bedrock/STS endpoints.\n\n## Engineer journey 1: Greenfield (AWS + Slack only)\n\n### Step 1: Create/configure Slack app\n\n1. Create app in Slack API console.\n2. Add bot scopes:\n   - `chat:write`\n   - `app_mentions:read`\n   - `channels:history`\n3. Enable Event Subscriptions.\n4. Subscribe to `app_mention`.\n5. Enable Interactivity.\n6. Install app into workspace and invite bot to your target channel.\n\n### Step 2: Configure AWS secrets (SSM)\n\nUse helper target:\n\n```bash\nmake setup-ssm\n```\n\n### Step 3: Build and deploy with SAM\n\n```bash\nmake build\nmake deploy\n```\n\nAfter deploy, capture `SlackBotEndpoint` from stack outputs and set it in:\n- Slack Event Subscriptions Request URL\n- Slack Interactivity Request URL\n\n### Step 4: Verify end-to-end\n\n- Trigger/emit an alarm event and confirm proactive Slack alert appears.\n- Mention bot in thread with EKS question and verify orchestrator route behavior.\n\n## Engineer journey 2: Existing EKS project wants SlackBot + AI monitoring\n\n### Step 1: Keep current EKS; integrate this stack\n\n- Reuse existing cluster name by passing `ClusterName` parameter on deploy.\n- Reuse existing alarm conventions or route your existing events into EventBridge patterns expected by template.\n\n### Step 2: Connect to existing ops repositories\n\n- Set `GITHUB_REPO` to infra repo where you want auto-fix PRs.\n- Keep PR auto-merge disabled; human review remains required.\n\n### Step 3: Integrate MCP gateway for interactive troubleshooting\n\n- Point `MCP_GATEWAY_URL` to your existing EKS MCP service.\n- Set `EKS_MCP_SERVER` if your MCP server identifier differs from default.\n\n### Step 4: Roll out safely\n\n- Deploy to a non-prod Slack channel first.\n- Validate non-K8s questions are exited by orchestrator.\n- Validate K8s troubleshooting questions route to specialist and use MCP tools.\n\n## GitHub Actions (CI/CD)\n\nThe repository includes GitHub Actions workflows for continuous integration (`ci.yml`) and automated deployment (`deploy.yml`).\n\n- **Continuous Integration (`ci.yml`):** Runs automatically on every push and pull request to `main` to run linting and tests.\n- **Automated Deployment (`deploy.yml`):** By default, this workflow is set to `workflow_dispatch` (disabled from running automatically) to prevent unexpected AWS deployments if you are just evaluating the toolkit locally.\n\n### How to Enable Automated Deployments\n\nIf you want the pipeline to automatically deploy your infrastructure on merge:\n1. **Enable Triggers:** Edit `.github/workflows/deploy.yml`. Change `on: workflow_dispatch:` to:\n   ```yaml\n   on:\n     push:\n       branches: [ main ]\n   ```\n2. **Set Secrets in GitHub:** Go to your repository **Settings \u003e Secrets and variables \u003e Actions**.\n   - Add a repository variable `AWS_REGION` (e.g., `us-east-1`).\n   - Add a repository variable `EKS_CLUSTER_NAME`.\n   - Add a repository variable `SLACK_CHANNEL`.\n   - Add a repository variable `GITHUB_REPO`.\n   - Add a repository secret `AWS_DEPLOY_ROLE_ARN` (An IAM Role ARN configured for GitHub OIDC to deploy the SAM stack).\n\nOnce configured, the CI pipeline will run Ruff/Pytest on every PR, and the Deploy pipeline will build and update your AWS SAM stack whenever code is merged to `main`.\n\n## Local run and testing\n\n### Run checks\n\n```bash\nruff check src tests\npython -m compileall src tests\npytest --no-cov\n```\n\n### Run focused architecture tests\n\n```bash\npytest --no-cov tests/test_orchestrator.py tests/test_bot_handler.py\n```\n\n### Local proactive invoke example\n\n```bash\npython -c \"\nfrom eks_ai_ops.proactive.handler import handler\nevent = {\n  'source': 'aws.cloudwatch',\n  'detail': {\n    'alarmName': 'sre-prod-api-checkout-error-rate-high',\n    'state': {'value': 'ALARM', 'reason': 'Threshold crossed'},\n    'previousState': {'value': 'OK'},\n    'configuration': {}\n  }\n}\nprint(handler(event, None))\n\"\n```\n\n### Local observation tip (k9s)\n\nWhile testing in a real cluster, use `k9s` to watch pod logs and events in parallel with Slack interactions.\nThis makes it easier to verify whether orchestrator routing and specialist troubleshooting outputs match live cluster state.\n\n## Deploy commands summary\n\n```bash\nmake install\nmake test-fast\nmake build\nmake deploy\n```\n\nIf already configured and iterating:\n\n```bash\nmake deploy-fast\n```\n\n## Cleanup and Teardown\n\nWhen you're done using the toolkit, remove all AWS resources to avoid unnecessary charges:\n\n### Basic Cleanup\n\nDelete the main CloudFormation stack and associated resources:\n\n```bash\nmake destroy-confirm\n```\n\nThis removes:\n- Lambda functions\n- DynamoDB tables (marked for retention)\n- CloudWatch log groups\n- SSM parameters\n- CloudWatch alarms\n\n### Complete Cleanup\n\nTo also delete EKS cluster and all EC2 instances (if deployed with eksctl):\n\n```bash\nmake destroy-all\n```\n\n### Verify Cleanup\n\nCheck that all resources have been deleted:\n\n```bash\nmake verify-cleanup\n```\n\n### Detailed Cleanup Guide\n\nFor comprehensive instructions, troubleshooting, and step-by-step cleanup of individual resources, see [CLEANUP.md](./CLEANUP.md).\n\n### Cost Considerations\n\n- EKS clusters charge hourly (plus data transfer costs)\n- EC2 instances charge by the second\n- NAT gateways charge hourly\n- Data transfer across AZs incurs charges\n\n**Recommendation**: Run `make destroy-confirm` or `make destroy-all` when not actively developing to avoid unexpected AWS bills.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmrjhonyvidal%2Feks-ai-ops-toolkit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmrjhonyvidal%2Feks-ai-ops-toolkit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmrjhonyvidal%2Feks-ai-ops-toolkit/lists"}