{"id":46405151,"url":"https://github.com/upbound/configuration-aws-database-ai","last_synced_at":"2026-03-05T12:02:03.952Z","repository":{"id":309580120,"uuid":"1029066443","full_name":"upbound/configuration-aws-database-ai","owner":"upbound","description":"Intelligent Database abstraction","archived":false,"fork":false,"pushed_at":"2026-02-16T10:23:21.000Z","size":200,"stargazers_count":3,"open_issues_count":2,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-02-16T18:55:13.407Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"KCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/upbound.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-07-30T13:23:54.000Z","updated_at":"2026-02-04T18:02:15.000Z","dependencies_parsed_at":"2025-08-12T18:23:11.800Z","dependency_job_id":"63acbcab-2ae1-4b38-b3bf-1c7c1967327e","html_url":"https://github.com/upbound/configuration-aws-database-ai","commit_stats":null,"previous_names":["upbound/configuration-aws-database-ai"],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/upbound/configuration-aws-database-ai","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/upbound%2Fconfiguration-aws-database-ai","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/upbound%2Fconfiguration-aws-database-ai/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/upbound%2Fconfiguration-aws-database-ai/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/upbound%2Fconfiguration-aws-database-ai/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/upbound","download_url":"https://codeload.github.com/upbound/configuration-aws-database-ai/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/upbound%2Fconfiguration-aws-database-ai/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30123729,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-05T11:11:57.947Z","status":"ssl_error","status_checked_at":"2026-03-05T11:11:29.001Z","response_time":93,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-03-05T12:02:01.778Z","updated_at":"2026-03-05T12:02:03.943Z","avatar_url":"https://github.com/upbound.png","language":"KCL","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AWS Database Configuration\n\nThis repository contains an Upbound project, tailored for users establishing their initial control plane with [Upbound](https://cloud.upbound.io). This configuration deploys fully managed AWS database instances.\n\n## Overview\n\nThe core components of a custom API in [Upbound Project](https://docs.upbound.io/learn/control-plane-project/) include:\n\n- **CompositeResourceDefinition (XRD):** Defines the API's structure.\n- **Composition(s):** Configures the Functions Pipeline\n- **Embedded Function(s):** Encapsulates the Composition logic and implementation within a self-contained, reusable unit\n\nIn this specific configuration, the API contains:\n\n- **an [AWS Database](/apis/definition.yaml) custom resource type (v2 namespaced).**\n- **Composition:** Configured in [/apis/composition-rds-metrics.yaml](/apis/composition-rds-metrics.yaml)\n- **Embedded Function:** The Composition logic is encapsulated within [embedded function](/functions/sqlinstance/main.k)\n\n## Intelligent RDS Scaling with Crossplane Operations\n\n### Architecture\n**Pipeline Flow**: `RDS Instance → CloudWatch Metrics → Status → Operations (Scheduled AI Analysis)`\n\n```\n┌─────────────────┐    ┌──────────────────┐\n│   RDS Instance  │    │ function-rds-    │\n│                 │───▶│    metrics       │───▶ status.performanceMetrics\n│ (CloudWatch)    │    │                  │\n└─────────────────┘    └──────────────────┘\n                                                      │\n                                                      ▼\n                                            ┌─────────────────────┐\n                                            │ CronOperation       │\n                                            │ (Every 10 min)      │\n                                            │                     │\n                                            │ function-claude     │\n                                            │ (Claude AI)         │\n                                            └─────────────────────┘\n                                                      │\n                                                      ▼\n                                            ┌─────────────────┐\n                                            │ Scaling Actions │\n                                            │ + Audit Trail   │\n                                            └─────────────────┘\n```\n\n### Component Integration\n\n**1. Composition Pipeline**\n- **function-rds-metrics**: Fetches CloudWatch metrics (CPU, memory, IOPS, connections, storage)\n- Writes metrics to `status.performanceMetrics` for Operations consumption\n- **function-auto-ready**: Marks composition ready\n\n**2. CronOperation (Scheduled Scaling)**\n- **function-claude**: Analyzes metrics from XR status every minute (demo frequency)\n- Makes scaling decisions using **Claude Haiku 4.5** (85%+ cost reduction vs Sonnet)\n- Only scales resources with `scale: me` label\n- Rate-limited to prevent scaling thrash\n- **Requires**: function-claude v0.4.0+ for reliable markdown handling\n\n### Key Technical Challenges Solved\n\n**1. Large YAML Handling**\n- **Problem**: Claude failed with empty tool parameters for complex observed resources\n- **Solution**: Increased `maxTokens` from 1024 → 8192 and refined prompt engineering\n\n**2. Pipeline Context Flow**\n- **Problem**: function-claude wasn't receiving metrics from previous pipeline step\n- **Solution**: Extended input schema with `contextFields` parameter for generic context access\n\n**3. Resource State Management**\n- **Problem**: First reconciliation had empty observed resources\n- **Solution**: Fallback logic to use desired resources when observed is empty\n\n**4. Schema Validation**\n- **Problem**: Returning full observed state caused resourceRefs uid schema errors\n- **Solution**: Prompt engineering to return only clean desired spec excluding system fields\n\n### Decision Making Process\n\n**Input**: Real CloudWatch metrics\n```yaml\nperformanceMetrics:\n  metrics:\n    CPUUtilization: { value: 1.98, unit: \"Percent\" }\n    DatabaseConnections: { value: 0, unit: \"Count\" }\n    FreeStorageSpace: { value: 2775404544, unit: \"Bytes\" }\n    # ... other metrics\n```\n\n**AI Analysis**: Claude evaluates against thresholds\n- CPU \u003e80% → scale up instance class\n- Memory \u003c20% free → memory-optimized instance  \n- IOPS \u003e80% → increase storage/instance\n- Connections \u003e80% → larger instance\n\n**Output**: Structured decision with audit trail\n```yaml\nstatus:\n  claudeDecision:\n    reasoning: \"CPU utilization at 1.98% is well below 80% threshold...\"\n    timestamp: \"2025-07-29T22:05:18Z\"\n```\n\n### Cost Expectations (Haiku Model)\n\n**AI inference costs at 1-minute schedule (demo)**:\n- ~$9.52/day per database\n- ~$285/month per database\n- ~$3,470/year per database\n\n**Production schedules** (cost scales with frequency):\n- 5-minute: ~$695/year per database\n- 10-minute: ~$347/year per database\n- 30-minute: ~$116/year per database\n\n*Note: Costs include AI analysis only, not AWS infrastructure*\n\n### Production Results\n- ✅ **End-to-end pipeline**: All components synced and ready\n- ✅ **Real-time decisions**: ~2-5 second analysis latency\n- ✅ **Cost-efficient AI**: ~$0.02-0.04 per scaling decision\n- ✅ **Audit compliance**: Full reasoning captured in XR status\n- ✅ **Safety**: Only scales up, prevents accidental downsizing\n\n### Usage\nDeploy an intelligent scaling database:\n```yaml\napiVersion: aws.platform.upbound.io/v1alpha1\nkind: SQLInstance\nmetadata:\n  name: my-intelligent-db\n  namespace: default\n  labels:\n    scale: me  # Enable CronOperation scaling\nspec:\n  crossplane:\n    compositionSelector:\n      matchLabels:\n        type: rds-metrics\n  parameters:\n    engine: mariadb\n    engineVersion: \"10.11\"\n    storageGB: 20\n    region: us-west-2\n    networkRef:\n      id: my-network\n    passwordSecretRef:\n      name: my-db-password\n      key: password\n    # ... other parameters\n```\n\n### Prerequisites\n1. **Claude API Secret**: Store Anthropic API key in control plane\n   ```bash\n   kubectl create secret generic claude \\\n     --from-literal=ANTHROPIC_API_KEY=your-api-key \\\n     -n crossplane-system\n   ```\n\n2. **AWS Credentials**: Ensure CloudWatch metrics access for RDS monitoring\n\n**Current Status**: **Experimental** - Successfully proven concept with real infrastructure, ready for production validation and monitoring.\n\n## Performance Testing (perftest)\n\n### How to Run Performance Tests\n\nThis repository includes automated scripts to validate the intelligent scaling system and trigger AI-driven scaling decisions.\n\n#### Automated Load Test with Monitoring\n\nUse `perf-scale-demo.sh` for a complete end-to-end scaling demonstration:\n\n```bash\n# Run with defaults (rds-metrics-database-ai-mysql in database-team namespace)\n./perf-scale-demo.sh\n\n# Custom SQLInstance name\n./perf-scale-demo.sh my-database\n\n# Custom SQLInstance and namespace\n./perf-scale-demo.sh my-database default\n```\n\n**What it does**:\n- Automatically extracts connection details from the SQLInstance's connection secret\n- Launches 30 concurrent MySQL processes with intensive CPU load (BENCHMARK operations)\n- Monitors CPU utilization, instance class, and scaling decisions every 20 seconds\n- Automatically stops when scaling is detected or after 5 minutes\n- Cleans up all background processes on completion\n\n**Expected timeline**:\n1. 30-60 seconds: CPU reaches 50%+ utilization\n2. 1-2 minutes: CloudWatch metrics update\n3. Next CronOperation run: Claude analyzes metrics and triggers scaling\n4. 5-10 minutes: Instance class upgrade completes\n\n#### Continuous Monitoring\n\nUse `simple-demo-scaling-monitor.sh` to watch scaling decisions in real-time:\n\n```bash\n# Monitor with defaults (45-second refresh interval)\n./simple-demo-scaling-monitor.sh\n\n# Custom SQLInstance name\n./simple-demo-scaling-monitor.sh my-database\n\n# Custom refresh interval (30 seconds)\n./simple-demo-scaling-monitor.sh my-database default 30\n```\n\n**What it displays**:\n- CPU utilization, free memory, and database connections\n- Current instance class\n- All `intelligent-scaling/*` annotations (last-scaled-decision, last-scaled-at, cooldown-until)\n- Refreshes continuously until stopped (Ctrl+C)\n\n#### Expected Behavior\n1. **CPU Spike**: Database CPU reaches 60%+ utilization (threshold for demo scaling)\n2. **Metrics Collection**: function-rds-metrics captures CloudWatch data in XR status\n3. **AI Analysis**: Claude Haiku evaluates metrics against thresholds\n4. **Scaling Decision**: Instance class upgraded (e.g., db.t3.micro → db.t3.small → db.t3.medium)\n5. **Cooldown Period**: 5-minute rate limit prevents scaling thrash\n6. **Performance Recovery**: CPU utilization drops after scaling completes\n\n## Testing\n\nThe configuration can be tested using:\n\n- `up composition render --xrd=apis/definition.yaml apis/composition-rds-metrics.yaml examples/mariadb-xr-rds-metrics.yaml --function-credentials=tests/test-credentials.yaml` to render the composition\n- `up test run tests/test-sqlinstance/` to run composition tests\n- `up test run tests/e2etest-sqlinstance/ --e2e` to run end-to-end tests\n\n## Crossplane v2.0 Operations\n\nThis configuration includes **Crossplane Operations** for automated intelligent scaling using both scheduled and reactive patterns. Operations are automatically deployed with `up project run`.\n\n### CronOperation: Scheduled Scaling Analysis\n- **Schedule**: Every minute (`* * * * *`) for demo - adjust for production\n- **Target**: SQLInstance resources with `scale: me` label\n- **Purpose**: Regular scheduled analysis for predictable workloads\n- **Rate Limiting**: 5-minute cooldown between scaling actions\n- **Recommended production**: 5-10 minute intervals for cost optimization\n\n### Operation Features\n- **AI-Powered Decision Making**: Uses `upbound-function-claude` for intelligent scaling decisions\n- **Conservative Thresholds**: CPU \u003e85%, Memory \u003c15%, Connections \u003e85%\n- **Instance Progression**: `db.t3.micro → db.t3.small → db.t3.medium → db.t3.large`\n- **Rate Limiting**: Annotations-based cooldown to prevent excessive scaling\n- **Audit Trail**: Full reasoning captured in resource annotations\n\n### Quick Start with Operations\n1. **Setup Claude API Secret**:\n   ```bash\n   kubectl create secret generic claude \\\n     --from-literal=ANTHROPIC_API_KEY=your-api-key \\\n     -n crossplane-system\n   ```\n\n2. **Deploy Configuration and Operations**:\n   ```bash\n   up project run  # Automatically includes operations/\n   ```\n\n3. **Deploy Example with Scaling Labels**:\n   ```bash\n   # Network dependency\n   kubectl apply -f examples/network-rds-metrics.yaml\n   \n   # Database with scaling enabled\n   kubectl apply -f examples/mariadb-xr-rds-metrics.yaml\n   ```\n\n4. **Monitor Operations**:\n   ```bash\n   # Check operation status\n   kubectl get cronoperation,watchoperation -n crossplane-system\n   \n   # Monitor operation executions\n   kubectl get operation -n crossplane-system --watch\n   ```\n\n### Example Resources for Operations Testing\nThe configuration includes dedicated examples for operations testing:\n- `examples/network-rds-metrics.yaml`: Network setup for RDS metrics testing\n- `examples/mariadb-xr-rds-metrics.yaml`: MariaDB SQLInstance with `scale: me` label for operation targeting\n- `examples/postgres-xr-rds-metrics.yaml`: PostgreSQL SQLInstance for testing\n\n## Deployment\n\n- Execute `up project run` (automatically includes operations)\n- Alternatively, install the Configuration from the [Upbound Marketplace](https://marketplace.upbound.io/configurations/upbound/configuration-aws-database)\n- Check [examples](/examples/) for example XR(Composite Resource)\n\n## Next steps\n\nThis repository serves as a foundational step. To enhance the configuration, consider:\n\n1. create new API definitions in this same repo\n2. editing the existing API definition to your needs\n\nTo learn more about how to build APIs for your managed control planes in Upbound, read the guide on [Upbound's docs](https://docs.upbound.io/).","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fupbound%2Fconfiguration-aws-database-ai","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fupbound%2Fconfiguration-aws-database-ai","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fupbound%2Fconfiguration-aws-database-ai/lists"}