{"id":32760502,"url":"https://github.com/redis/rdi-cloud-automation","last_synced_at":"2026-06-18T20:31:58.807Z","repository":{"id":322321046,"uuid":"1060197608","full_name":"redis/rdi-cloud-automation","owner":"redis","description":null,"archived":false,"fork":false,"pushed_at":"2026-04-15T15:37:05.000Z","size":96,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-15T17:29:34.871Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/redis.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":".github/SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-19T14:24:26.000Z","updated_at":"2026-04-15T15:37:10.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/redis/rdi-cloud-automation","commit_stats":null,"previous_names":["redis/rdi-cloud-automation"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/redis/rdi-cloud-automation","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/redis%2Frdi-cloud-automation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/redis%2Frdi-cloud-automation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/redis%2Frdi-cloud-automation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/redis%2Frdi-cloud-automation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/redis","download_url":"https://codeload.github.com/redis/rdi-cloud-automation/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/redis%2Frdi-cloud-automation/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34507155,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-18T02:00:06.871Z","response_time":128,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-11-04T03:01:01.586Z","updated_at":"2026-06-18T20:31:58.801Z","avatar_url":"https://github.com/redis.png","language":"HCL","funding_links":[],"categories":[],"sub_categories":[],"readme":"# rdi-cloud-automation\n\n[![Secret Scanning](https://github.com/redis/rdi-cloud-automation/actions/workflows/secret-scan.yml/badge.svg?branch=main)](https://github.com/redis/rdi-cloud-automation/actions/workflows/secret-scan.yml)\n\nTerraform modules to configure producer databases and network connectivity for Redis Data Integration (RDI).\n\n## 🚀 Overview\n\nThis repository provides production-ready Terraform modules to deploy and configure source databases for **Redis Data Integration (RDI)** with secure AWS PrivateLink connectivity. It supports multiple database engines with automatic CDC (Change Data Capture) user provisioning and optional sample data loading.\n\n### Supported Database Engines\n\n| Database | Engine | CDC Method | Auto User Creation | High Availability |\n|----------|--------|------------|-------------------|-------------------|\n| **PostgreSQL** | Aurora PostgreSQL | Logical Replication | ❌ (uses admin) | ✅ Multi-AZ |\n| **MySQL** | Aurora MySQL 8.0 | Debezium (binlog) | ✅ `debezium` user | ✅ Multi-AZ |\n| **SQL Server** | RDS SQL Server SE | Change Tracking | ✅ `rdi_user` | ✅ Multi-AZ |\n\n## 📋 Prerequisites\n\n- [Terraform](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli) \u003e= 1.5.7\n- [AWS CLI](https://aws.amazon.com/cli/) configured with credentials\n- [Redis Cloud](https://redis.com/try-free/) account with RDI enabled\n- Database client tools (optional, for testing):\n  - `psql` for PostgreSQL\n  - `mysql` for MySQL\n  - `sqlcmd` for SQL Server\n\n## 📚 Examples\n\nThe `examples` directory contains complete, ready-to-deploy examples:\n\n### aws-ec2-privatelink\n\nCreates a PostgreSQL database on EC2 exposed with PrivateLink. This example creates a VPC and can be used to try RDI quickly with no existing resources.\n\n**Use case:** Quick testing and development with PostgreSQL\n\n### aws-rds-privatelink-failover\n\nCreates production-ready RDS databases (PostgreSQL, MySQL, or SQL Server) with automatic failover support via AWS PrivateLink.\n\n**Features:**\n- ✅ **Multi-engine support:** Choose PostgreSQL, MySQL, or SQL Server\n- ✅ **Automatic CDC user creation:** MySQL and SQL Server users created automatically\n- ✅ **High availability:** Multi-AZ deployment with automatic failover\n- ✅ **Lambda-based failover:** Automatically updates NLB targets during RDS failover\n- ✅ **Optional sample data:** Chinook database for testing\n- ✅ **Secure connectivity:** AWS PrivateLink for private VPC-to-VPC connections\n\n**Quick Start:**\n```bash\ncd examples/aws-rds-privatelink-failover\n\n# For PostgreSQL\nterraform apply -var-file example-postgres.tfvars\n\n# For MySQL\nterraform apply -var-file example-mysql.tfvars\n\n# For SQL Server\nterraform apply -var-file example-sqlserver.tfvars\n```\n\nSee [examples/aws-rds-privatelink-failover/README.md](examples/aws-rds-privatelink-failover/README.md) for detailed documentation.\n\n## 🧩 Modules\n\nThe `modules` directory contains reusable Terraform modules which can be composed together to build custom database infrastructure.\n\n### Database Modules\n\n| Module | Description | Database Type | Use Case |\n|--------|-------------|---------------|----------|\n| **aws-rdi-quickstart-postgres** | VPC and EC2 instance with PostgreSQL | PostgreSQL on EC2 | Quick testing and development |\n| **aws-rds-chinook** | Aurora PostgreSQL RDS cluster | Aurora PostgreSQL | Production PostgreSQL with HA |\n| **aws-rds-mysql-chinook** | Aurora MySQL RDS cluster | Aurora MySQL 8.0 | Production MySQL with HA |\n| **aws-rds-sqlserver-chinook** | RDS SQL Server instance | SQL Server SE 2022 | Production SQL Server with HA |\n\n### Infrastructure Modules\n\n| Module | Description | Purpose |\n|--------|-------------|---------|\n| **aws-privatelink** | Network Load Balancer + PrivateLink | Secure VPC-to-VPC connectivity |\n| **aws-rds-lambda** | Lambda function for RDS event handling | Automatic failover detection and NLB updates |\n| **aws-secret-manager** | KMS Key + Secrets Manager | Secure credential storage for RDI |\n\n### Key Features by Module\n\n**Database Modules:**\n- ✅ VPC with public, private, and database subnets\n- ✅ Security groups with self-referencing rules\n- ✅ Multi-AZ deployment for high availability\n- ✅ CDC-enabled parameter groups\n- ✅ Automatic engine version selection (MySQL)\n\n**aws-privatelink:**\n- ✅ Network Load Balancer (internal or public)\n- ✅ Cross-zone load balancing\n- ✅ PrivateLink VPC Endpoint Service\n- ✅ Automatic principal whitelisting\n\n**aws-rds-lambda:**\n- ✅ SNS topic for RDS events\n- ✅ Lambda function to detect writer changes\n- ✅ Automatic NLB target group updates\n- ✅ CloudWatch logging\n\n**aws-secret-manager:**\n- ✅ KMS encryption for secrets\n- ✅ IAM policy for Redis Cloud access\n- ✅ Automatic credential rotation support\n\n## 🔐 CDC User Management\n\nThe Terraform modules automatically create CDC (Change Data Capture) users with appropriate permissions for MySQL and SQL Server. PostgreSQL uses the admin user directly.\n\n### Automatic User Creation\n\n| Database | User | Password | Permissions | Created By |\n|----------|------|----------|-------------|------------|\n| **PostgreSQL** | `postgres` | Admin password | Superuser (logical replication) | RDS (admin user) |\n| **MySQL** | `debezium` | Auto-generated | SELECT, RELOAD, REPLICATION SLAVE, REPLICATION CLIENT | Terraform `null_resource` |\n| **SQL Server** | `rdi_user` | Auto-generated | dbcreator, VIEW SERVER STATE, VIEW ANY DEFINITION | Terraform `null_resource` |\n\n### How It Works\n\n**MySQL:**\n```bash\n# Automatically runs during terraform apply\nmysql -h \u003cnlb_hostname\u003e -u admin -p \u003c\u003cSQL\nCREATE USER IF NOT EXISTS 'debezium'@'%' IDENTIFIED BY '\u003cauto-generated\u003e';\nGRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT, LOCK TABLES ON *.* TO 'debezium'@'%';\nFLUSH PRIVILEGES;\nSQL\n```\n\n**SQL Server:**\n```sql\n-- Automatically runs during terraform apply\nCREATE LOGIN rdi_user WITH PASSWORD = '\u003cauto-generated\u003e';\nCREATE USER rdi_user FOR LOGIN rdi_user;\nALTER SERVER ROLE [dbcreator] ADD MEMBER rdi_user;\nGRANT VIEW SERVER STATE TO rdi_user;\nGRANT VIEW ANY DEFINITION TO rdi_user;\n```\n\n### Credentials Storage\n\nAll credentials are automatically stored in **AWS Secrets Manager** with:\n- ✅ KMS encryption at rest\n- ✅ IAM policy allowing Redis Cloud access\n- ✅ Automatic rotation support (optional)\n\nAccess credentials via Terraform outputs:\n```bash\nterraform output rdi_username  # CDC username for RDI\nterraform output rdi_password  # CDC password (sensitive)\nterraform output secret_arn    # Secrets Manager ARN for Redis Cloud\n```\n\n## 🏗️ Architecture\n\n### Network Flow\n\n```\nRedis Cloud RDI\n    ↓\nAWS PrivateLink (VPC Endpoint)\n    ↓\nNetwork Load Balancer (NLB)\n    ↓\nRDS Database (Multi-AZ)\n```\n\n### Failover Handling\n\nFor Aurora clusters (PostgreSQL and MySQL):\n1. RDS emits failover event to SNS\n2. Lambda function detects writer instance change\n3. Lambda updates NLB target group with new writer IP\n4. RDI connections automatically route to new writer\n5. Zero configuration changes needed in Redis Cloud\n\n### Security Architecture\n\n- ✅ **Private connectivity:** AWS PrivateLink (no internet exposure)\n- ✅ **Encryption at rest:** RDS storage encryption enabled\n- ✅ **Encryption in transit:** TLS support (optional)\n- ✅ **Credential management:** AWS Secrets Manager with KMS\n- ✅ **Network isolation:** VPC with private subnets for databases\n- ✅ **Least privilege:** IAM policies scoped to specific resources\n\n## 💡 Common Use Cases\n\n### 1. Quick Testing with Sample Data\n\nDeploy PostgreSQL and manually load Chinook sample database:\n\n```bash\ncd examples/aws-rds-privatelink-failover\n\n# Deploy infrastructure\nterraform apply -var-file example-postgres.tfvars\n\n# Load sample data (requires network access to RDS)\n# Option A: If nlb_internal = false (public NLB)\n./psql.sh  # Then manually load Chinook SQL\n\n# Option B: From bastion host or VPN connection\n# See example README for detailed setup instructions\n```\n\n### 2. Production MySQL with CDC\n\nDeploy Aurora MySQL with automatic debezium user creation:\n\n```bash\ncd examples/aws-rds-privatelink-failover\nterraform apply -var-file example-mysql.tfvars\n# debezium user is automatically created with CDC permissions\n# Credentials stored in AWS Secrets Manager\n```\n\n### 3. SQL Server with Change Tracking\n\nDeploy SQL Server with automatic rdi_user creation:\n\n```bash\ncd examples/aws-rds-privatelink-failover\nterraform apply -var-file example-sqlserver.tfvars\n# rdi_user is automatically created with CDC permissions\n# Ready for SQL Server Change Tracking\n```\n\n### 4. Multi-Region Deployment\n\nDeploy databases in multiple AWS regions for disaster recovery:\n\n```bash\n# Region 1 (us-east-1)\ncd examples/aws-rds-privatelink-failover\nterraform workspace new us-east-1\nterraform apply -var-file example-postgres.tfvars -var region=us-east-1\n\n# Region 2 (us-west-2)\nterraform workspace new us-west-2\nterraform apply -var-file example-postgres.tfvars -var region=us-west-2\n```\n\n## 🔧 Troubleshooting\n\n### CDC User Creation Issues\n\n**Problem:** MySQL debezium user creation fails\n\n**Solution:**\n- Ensure `mysql` client is installed locally\n- Check network connectivity to NLB (set `nlb_internal = false` for testing)\n- Verify admin password is correct\n- Check CloudWatch logs for detailed error messages\n\n**Problem:** SQL Server rdi_user creation fails\n\n**Solution:**\n- Ensure `sqlcmd` is installed locally ([installation guide](https://learn.microsoft.com/en-us/sql/linux/sql-server-linux-setup-tools))\n- Check network connectivity to NLB\n- Verify sa password is correct\n- SQL Server may take 60+ seconds to be ready after creation\n\n### Connection Issues\n\n**Problem:** Cannot connect to database from laptop\n\n**Solution:**\n- Set `nlb_internal = false` in tfvars to make NLB public\n- Ensure security group allows your IP address\n- Use the correct port (5432 for PostgreSQL, 3306 for MySQL, 1433 for SQL Server)\n- Get connection details: `terraform output`\n\n**Problem:** RDI cannot connect via PrivateLink\n\n**Solution:**\n- Verify PrivateLink service is whitelisted for Redis Cloud principal\n- Check that the correct secret ARN is configured in Redis Cloud RDI\n- Ensure the secret contains the correct username and password\n- Test connectivity using `./connect.sh` script\n\n### Failover Issues\n\n**Problem:** Lambda not updating NLB after failover\n\n**Solution:**\n- Check Lambda CloudWatch logs for errors\n- Verify SNS topic is subscribed to RDS events\n- Ensure Lambda has IAM permissions to modify NLB target group\n- Test failover manually: `aws rds failover-db-cluster --db-cluster-identifier \u003ccluster-id\u003e`\n\n## 🔒 Security\n\nThis repository uses automated secret scanning to prevent accidental credential leaks:\n\n- **Gitleaks** - Fast regex-based secret detection\n- **TruffleHog** - High-entropy string detection with verification\n- **detect-secrets** - Baseline-based secret scanning\n\nSecret scanning runs automatically on:\n- Every push to main branches\n- Every pull request\n- Weekly scheduled scans\n\nFor more information, see:\n- [Security Policy](.github/SECURITY.md)\n- [Secret Scanning Guide](.github/SECRET_SCANNING.md)\n\n### Quick Start - Local Scanning\n\n```bash\n# Install Gitleaks\nbrew install gitleaks  # macOS\n\n# Scan before committing\ngitleaks detect --no-git\n\n# Install pre-commit hook\ncurl -sSfL https://raw.githubusercontent.com/gitleaks/gitleaks/master/scripts/pre-commit.py -o .git/hooks/pre-commit\nchmod +x .git/hooks/pre-commit\n```\n\n## 🤝 Contributing\n\nContributions are welcome! Please follow these guidelines:\n\n1. **Fork the repository** and create a feature branch\n2. **Test your changes** thoroughly with `terraform plan` and `terraform apply`\n3. **Run security scans** before committing:\n   ```bash\n   gitleaks detect --no-git\n   ```\n4. **Update documentation** if you add new features or modules\n5. **Submit a pull request** with a clear description of changes\n\n### Development Setup\n\n```bash\n# Clone the repository\ngit clone https://github.com/redis/rdi-cloud-automation.git\ncd rdi-cloud-automation\n\n# Install pre-commit hooks\ncurl -sSfL https://raw.githubusercontent.com/gitleaks/gitleaks/master/scripts/pre-commit.py -o .git/hooks/pre-commit\nchmod +x .git/hooks/pre-commit\n\n# Initialize Terraform\ncd examples/aws-rds-privatelink-failover\nterraform init\n```\n\n## 📄 License\n\nThis project is licensed under the terms specified in the repository.\n\n## 🆘 Support\n\nFor issues, questions, or contributions:\n- **GitHub Issues:** [Report a bug or request a feature](https://github.com/redis/rdi-cloud-automation/issues)\n- **Redis Documentation:** [Redis Data Integration (RDI) Docs](https://redis.io/docs/latest/integrate/redis-data-integration/)\n\n## 📚 Additional Resources\n\n- [AWS PrivateLink Documentation](https://docs.aws.amazon.com/vpc/latest/privatelink/)\n- [Redis Data Integration (RDI) Overview](https://redis.io/docs/latest/operate/rc/databases/rdi/)\n- [Terraform AWS Provider Documentation](https://registry.terraform.io/providers/hashicorp/aws/latest/docs)\n- [Debezium Documentation](https://debezium.io/documentation/)\n- [AWS RDS Best Practices](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_BestPractices.html)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fredis%2Frdi-cloud-automation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fredis%2Frdi-cloud-automation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fredis%2Frdi-cloud-automation/lists"}