{"id":22351925,"url":"https://github.com/screwdriver-cd/aws-producer-scripts","last_synced_at":"2026-01-05T17:47:41.730Z","repository":{"id":45869098,"uuid":"409269872","full_name":"screwdriver-cd/aws-producer-scripts","owner":"screwdriver-cd","description":"Infrastructure-as-code for Screwdriver AWS Integration Queue","archived":false,"fork":false,"pushed_at":"2023-03-20T15:13:00.000Z","size":40,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":9,"default_branch":"main","last_synced_at":"2025-01-31T13:15:13.761Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://cd.screwdriver.cd/pipelines/8110/events","language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/screwdriver-cd.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-09-22T16:04:40.000Z","updated_at":"2021-12-28T23:48:07.000Z","dependencies_parsed_at":"2022-09-05T06:51:15.207Z","dependency_job_id":null,"html_url":"https://github.com/screwdriver-cd/aws-producer-scripts","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/screwdriver-cd%2Faws-producer-scripts","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/screwdriver-cd%2Faws-producer-scripts/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/screwdriver-cd%2Faws-producer-scripts/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/screwdriver-cd%2Faws-producer-scripts/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/screwdriver-cd","download_url":"https://codeload.github.com/screwdriver-cd/aws-producer-scripts/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245650496,"owners_count":20650105,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-04T12:16:33.765Z","updated_at":"2026-01-05T17:47:36.684Z","avatar_url":"https://github.com/screwdriver-cd.png","language":"HCL","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Screwdriver AWS Integration Producer Scripts\nInfrastructure-as-code script for creating a Managed Kafka Service and VPC for Screwdriver AWS Integration\n\n## Introduction\n\nThis repository is meant to serve as an install/uninstall/update script to provision necessary cloud infrastructure resources required for Screwdriver AWS Integration. The following are the resources created by the installation script by default:\n- 1 AWS Managed Kafka Cluster\n- 3 AWS VPC Endpoint Service  (1 for each availability zone of MSK broker endpoint)\n- 3 Network Load Balancers (1 for each endpoint service)\n- 3 Target Groups (1 for each load balancer)\n- 1 Security Group For AWS MSK\n- 1 Customer managed KMS Key for the MSK cluster\nThe following resources will be crated with new creation:\n- 1 VPC based on the provided cidr block\n- Private subnets\n- Public subnets\n- NAT Gateway\n- Internet Gateway\n- Route Table\n\nIf you opt for installation with an existing vpc, it will skip the vpc infrastructure creation\n\nThis script uses open source tool [terraform](https://www.terraform.io/) to provision all the resources\n\n### Dependencies\n\nThe followings are the external dependencies required to run this onboarding script:\n\n- [terraform](https://github.com/hashicorp/terraform/releases/latest)\n- [aws cli](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html)\n\nAll of these tools can be installed via Homebrew on Mac OS X.\n\n## Prerequisite\nScrewdriver API's must be deployed in the same AWS account and region which will connect to the producer service and queue.\nAlso a Amazon Secret Manager secret needs to be created manually (for now) with secret values that will be used in the consumer service. \n\n## Instructions\n\nTo get started, update the var file with the required details. Please refer to [`env.tfvars.json`](./env.tfvars.json.tmpl) for the variables list. Remove `.tmpl` extension from file.\nExample var file for prod is at [`example-prod`](./example-prod.tfvars.json.tmpl).\n\nSecond, configure the AWS CLI by running `aws configure` with your AWS credentials and select profile for the desired account.\n```\nexport AWS_PROFILE=\u003cprofile_name\u003e\nexport AWS_REGION=\u003cregion_name\u003e\n```\n\nNext, to begin the infrastructure provisioning process:\n\n### install\n```sh\n# by default, setup.sh will try to find \"env.tfvars.json\"\n./setup.sh \n```\n\n`./setup.sh` will first validate env.tfvars.json for all variables and use default for the ones not found, it will then run terraform init, followed by plan and apply to provision infrastructure.\n\nFor step by step installation, you can use the following options\n```sh\n# -i flag will run terraform init and verify backend infrastructure\n./setup.sh -i\n# -p flag will run terraform plan and create a tf plan\n./setup.sh -p\n# -a flag will run terraform apply and create the resources\n./setup.sh -a\n```\n\nYou can also run validation to check for errors before running plan and after running apply by using\nthe -v flag\n```sh\n./setup.sh -v\n```\n\nAlternatively, to uninstall all infrastructure\n\n```sh\n./setup.sh -d\n```\n### Considerations for VPC setup\n\nThe the number of resources in the infrastructure will be created based on the VPC configuration. There are 2 scenarios\n\n- [Consumer Resources with Existing VPC](#consumer-svc-with-existing-vpc)\n- [Consumer Resources with New VPC](#consumer-svc-with-new-vpc)\n\n#### Consumer Resources with Existing VPC\n\nFor existing VPC and subnets, all we need are the resource ID of the VPC and the cidr's of the private subnets. If using existing VPC it needs to have both private and public subnets as the resources will be created in private subnets. Also the private subnets should have outbound access to the internet. Therefore, we highly recommend reviewing your existing VPC to see if it fits or a new one should be created instead. Additionally, you can update the other variables like vpc name and consumer function name.\n\nExample configuration for exiting vpc:\n```yaml\naws_region=\"us-west-2\"\ntf_backend_bucket=\"sd-aws-consumer-tf-backend-11111111\"\nprivate_subnets   = [\"10.10.104.0/25\", \"10.10.104.128/25\", \"10.10.105.0/25\", \"10.10.105.128/25\"]\nvpc_id            =\"vpc-1234\"\nmsk_cluster_name  = \"beta-sd-msk\"\nmsk_ebs_vol       = 100\nmsk_instance_type = \"kafka.t3.small\"\nmsk_nodes_count   = 3\nmsk_secret_arn    = \"arn:someExampleSecret\"\nmsk_secret_name   = \"AmazonMSK_BETA_SD_SECRET\"\ntags              = {PRODUCT:\"SCREWDRIVER\",ENVIRONMENT:\"beta\",SERVICE:\"sd/producer\"}\ndeploy_env        = \"beta\"\n```\n#### Consumer Resources with New VPC\n\nIn this case a VPC will be created and consumer svc will be provisioned in the new vpc. The required configuration needed for a new VPC setup are the VPC CIDR, the list of private and public subnet cidrs and the availability zones. The VPC CIDR prefix must be between `/16` and `/24`. Additionally, you can update the other variables like vpc name and consumer function name.\n\nExample configuration is new vpc:\n```yaml\naws_region=\"us-west-2\"\ntf_backend_bucket=\"sd-aws-consumer-tf-backend-11111111\"\ncidr_block        = \"10.10.104.0/22\"\nprivate_subnets   = [\"10.10.104.0/25\", \"10.10.104.128/25\", \"10.10.105.0/25\", \"10.10.105.128/25\"]\npublic_subnets    = [\"10.10.106.0/25\", \"10.10.106.128/25\", \"10.10.107.0/25\", \"10.10.107.128/25\"]\nazs               = [\"us-west-2a\", \"us-west-2b\", \"us-west-2c\", \"us-west-2d\"]\nvpc_name          =\"sd-producer\"\nmsk_cluster_name  = \"beta-sd-msk\"\nmsk_ebs_vol       = 100\nmsk_instance_type = \"kafka.t3.small\"\nmsk_nodes_count   = 3\nmsk_secret_arn    = \"arn:someExampleSecret\"\nmsk_secret_name   = \"AmazonMSK_BETA_SD_SECRET\"\ntags              = {PRODUCT:\"SCREWDRIVER\",ENVIRONMENT:\"beta\",SERVICE:\"sd/producer\"}\ndeploy_env        = \"beta\"\n```\n## Configurations\n\nThe config variables are all part of .tfvars file. These variables will be used in creating the resources.\n### Config Definitions\n\nThe following table describes all the configurable variables defined in `env.tfvars.json.json`\n\n| Name | Type | Description |\n| - | - | - |\n| aws_region \u003csup\u003e*\u003c/sup\u003e | String | AWS Region where resources will be provisioned |\n| tf_backend_bucket \u003csup\u003e*\u003c/sup\u003e | String | Terraform backend S3 bucket for storing tf state |\n| msk_cluster_name \u003csup\u003e*\u003c/sup\u003e | String | Screwdriver MSK cluster name |\n| msk_ebs_vol \u003csup\u003e*\u003c/sup\u003e | Integer | EBS volume size for MSK cluster |\n| msk_instance_type \u003csup\u003e*\u003c/sup\u003e | String | Type of ec2 instance for kafka cluster  |\n| msk_nodes_count \u003csup\u003e*\u003c/sup\u003e | Integer | Number of nodes for Kafka Cluster |\n| msk_secret_name \u003csup\u003e*\u003c/sup\u003e | String | Name of the MSK secret for authentication |\n| vpc_id \u003csup\u003e*\u003c/sup\u003e | String | User VPC Id  |\n| private_subnets \u003csup\u003e*\u003c/sup\u003e | List | List of private subnets |\n| public_subnets \u003csup\u003e#\u003c/sup\u003e | List | List of public subnets |\n| cidr_block \u003csup\u003e#\u003c/sup\u003e | String | CIDR block for the user vpc |\n| vpc_name \u003csup\u003e#\u003c/sup\u003e | String | Name of the user vpc |\n| azs \u003csup\u003e#\u003c/sup\u003e | List | List of availability zones |\n| tags \u003csup\u003e*\u003c/sup\u003e | Map | Map of tags to be used for resource creation |\n| deploy_env \u003csup\u003e*\u003c/sup\u003e | String | The environment prefix where resources will be deployed |\n\n\u003ci\u003e\u003csup\u003e*\u003c/sup\u003e required config\u003c/i\u003e\n\n\u003ci\u003e\u003csup\u003e#\u003c/sup\u003e required config when creating new vpc\u003c/i\u003e\n\n### Provider config vars\n```aws_region=\"us-west-2\"\ntf_backend_bucket=\"sd-aws-producer-tf-backend-\u003caccountId\u003e\" #replace accountId\n```\n### Msk cluster config vars\n```msk_cluster_name=\"example-sd-msk\"\nmsk_ebs_vol=100\nmsk_instance_type=\"\"kafka.t3.small\"\"\nmsk_nodes_count=3\nmsk_secret_name=\"AmazonMSK_EXAMPLE_SD_SECRET\"\n```\n### Config for VPC (existing or new)\n```\nvpc_id=null\nprivate_subnets=[\"10.10.106.0/25\", \"10.10.106.128/25\", \"10.10.107.0/25\", \"10.10.107.128/25\"]\ncidr_block=\"10.10.104.0/22\"\npublic_subnets=[\"10.10.104.0/25\", \"10.10.104.128/25\", \"10.10.105.0/25\", \"10.10.105.128/25\"]\nazs=[\"us-west-2a\", \"us-west-2b\", \"us-west-2c\", \"us-west-2d\"]\nvpc_name=\"screwdriver-producer\"\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscrewdriver-cd%2Faws-producer-scripts","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fscrewdriver-cd%2Faws-producer-scripts","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscrewdriver-cd%2Faws-producer-scripts/lists"}