{"id":21202447,"url":"https://github.com/infrablocks/terraform-aws-msk","last_synced_at":"2025-07-10T06:32:44.033Z","repository":{"id":39866178,"uuid":"326660415","full_name":"infrablocks/terraform-aws-msk","owner":"infrablocks","description":" Terraform module for building AWS Managed Streaming for Kafka cluster","archived":false,"fork":false,"pushed_at":"2025-04-03T19:50:56.000Z","size":261,"stargazers_count":2,"open_issues_count":4,"forks_count":3,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-04-05T11:34:32.964Z","etag":null,"topics":["aws","infrastructure","kafka","msk","terraform","terraform-modules"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/infrablocks.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE.txt","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"security_group.tf","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["tobyclemson"]}},"created_at":"2021-01-04T11:22:01.000Z","updated_at":"2025-02-12T10:04:07.000Z","dependencies_parsed_at":"2023-02-18T10:16:07.243Z","dependency_job_id":"e2bea933-8c47-443c-af35-b05cf3d2ff5c","html_url":"https://github.com/infrablocks/terraform-aws-msk","commit_stats":null,"previous_names":[],"tags_count":18,"template":false,"template_full_name":null,"purl":"pkg:github/infrablocks/terraform-aws-msk","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/infrablocks%2Fterraform-aws-msk","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/infrablocks%2Fterraform-aws-msk/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/infrablocks%2Fterraform-aws-msk/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/infrablocks%2Fterraform-aws-msk/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/infrablocks","download_url":"https://codeload.github.com/infrablocks/terraform-aws-msk/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/infrablocks%2Fterraform-aws-msk/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264538584,"owners_count":23624436,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","infrastructure","kafka","msk","terraform","terraform-modules"],"created_at":"2024-11-20T20:16:01.326Z","updated_at":"2025-07-10T06:32:43.687Z","avatar_url":"https://github.com/infrablocks.png","language":"Ruby","funding_links":["https://github.com/sponsors/tobyclemson"],"categories":[],"sub_categories":[],"readme":"Terraform AWS MSK\n=================\n\n[![CircleCI](https://circleci.com/gh/infrablocks/terraform-aws-msk.svg?style=svg)](https://circleci.com/gh/infrablocks/terraform-aws-msk)\n\nA Terraform module for building an MSK cluster in AWS.\n\nThe MSK cluster requires:\n* An existing VPC\n* Some existing subnets\n \nThe MSK cluster consists of:\n* A Kafka Cluster\n* Security groups\n* KMS key\n\nUsage\n-----\n\nTo use the module, include something like the following in your Terraform\nconfiguration:\n\n```hcl-terraform\nmodule \"msk\" {\n  source = \"infrablocks/msk/aws\"\n  component = \"kafka-cluster\"\n  deployment_identifier = \"dev\"\n\n  tags = {}\n\n  cluster_name = \"my-cluster\"\n  vpc_id = \"vpc-id\"\n  egress_cidrs = \"0.0.0.0/0\"\n  allowed_cidrs = [\"10.1.0.0/16\"]\n  \n  number_of_broker_nodes = 2\n  client_subnets = [\"subnet-id\"]\n  private_subnet_ids = [\"private-subnet-id\"]\n\n  s3_logs_prefix = \"\"\n  s3_logs_enabled = false\n  s3_logs_bucket_name = \"\"\n}\n```\n\nAs mentioned above, the MSK cluster deploys into an existing base network. \nWhilst the base network can be created using any mechanism you like, the \n[AWS Base Networking](https://github.com/tobyclemson/terraform-aws-base-networking)\nmodule will create everything you need. See the \n[docs](https://github.com/tobyclemson/terraform-aws-base-networking/blob/main/README.md)\nfor usage instructions.\n\nSee the \n[Terraform registry entry](https://registry.terraform.io/modules/infrablocks/msk/aws/latest) \nfor more details.\n\n### Inputs\n\n| Name                                       | Description                                                                                                      | Default            | Required                                 |\n|--------------------------------------------|------------------------------------------------------------------------------------------------------------------|:------------------:|:----------------------------------------:|\n| region                                     | The region into which to deploy the cluster                                                                      | -                  | yes                                      |\n| vpc_id                                     | The ID of the VPC into which to deploy the cluster                                                               | -                  | yes                                      |\n| subnet_ids                                 | The IDs of the subnets for container instances                                                                   | -                  | yes                                      |\n| component                                  | The component this cluster will contain                                                                          | -                  | yes                                      |\n| deployment_identifier                      | An identifier for this instantiation                                                                             | -                  | yes                                      |\n|vpc_id|\n|region|\n|include_default_egress_rule|\n|egress_cidrs|\n|include_default_ingress_rule|\n|allowed_cidrs|\n|cluster_name|\n|kafka_version|\n|number_of_broker_nodes|\n|instance_type|\n|ebs_volume_size|\n|s3_logs_prefix|\n|client_subnets|\n|cloud_watch_logs_enabled|\n|log_group_name|\n|firehose_enabled|\n|firehose_stream_name|\n|s3_logs_enabled|\n|s3_logs_bucket_name|\n|tags|\n\n### Outputs\n\n| Name                      | Description                                                                      |\n|---------------------------|----------------------------------------------------------------------------------|\n| cluster_id                | The ID of the created MSK cluster                                                |\n| cluster_name              | The name of the created MSK cluster                                              |\n| cluster_arn               | The ARN of the created MSK cluster                                               |\n|security_group_id|\n\n### Compatibility\n\nThis module is compatible with Terraform versions greater than or equal to \nTerraform 1.0.\n\n### Required Permissions\n\n* logs:CreateLogGroup\n* logs:DescribeLogGroups\n* logs:ListTagsLogGroup\n* logs:DeleteLogGroup\n\n\nDevelopment\n-----------\n\n### Machine Requirements\n\nIn order for the build to run correctly, a few tools will need to be installed\non your development machine:\n\n* Ruby (3.1.1)\n* Bundler\n* git\n* git-crypt\n* gnupg\n* direnv\n* aws-vault\n\n#### Mac OS X Setup\n\nInstalling the required tools is best managed by [homebrew](http://brew.sh).\n\nTo install homebrew:\n\n```\nruby -e \"$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)\"\n```\n\nThen, to install the required tools:\n\n```\n# ruby\nbrew install rbenv\nbrew install ruby-build\necho 'eval \"$(rbenv init - bash)\"' \u003e\u003e ~/.bash_profile\necho 'eval \"$(rbenv init - zsh)\"' \u003e\u003e ~/.zshrc\neval \"$(rbenv init -)\"\nrbenv install 3.1.1\nrbenv rehash\nrbenv local 3.1.1\ngem install bundler\n\n# git, git-crypt, gnupg\nbrew install git\nbrew install git-crypt\nbrew install gnupg\n\n# aws-vault\nbrew cask install\n\n# direnv\nbrew install direnv\necho \"$(direnv hook bash)\" \u003e\u003e ~/.bash_profile\necho \"$(direnv hook zsh)\" \u003e\u003e ~/.zshrc\neval \"$(direnv hook $SHELL)\"\n\ndirenv allow \u003crepository-directory\u003e\n```\n\n### Running the build\n\nRunning the build requires an AWS account and AWS credentials. You are free to\nconfigure credentials however you like as long as an access key ID and secret\naccess key are available. These instructions utilise\n[aws-vault](https://github.com/99designs/aws-vault) which makes credential\nmanagement easy and secure.\n\nTo provision module infrastructure, run tests and then destroy that\ninfrastructure, execute:\n\n```bash\naws-vault exec \u003cprofile\u003e -- ./go\n```\n\nTo provision the module prerequisites:\n\n```bash\naws-vault exec \u003cprofile\u003e -- ./go deployment:prerequisites:provision[\u003cdeployment_identifier\u003e]\n```\n\nTo provision the module contents:\n\n```bash\naws-vault exec \u003cprofile\u003e -- ./go deployment:root:provision[\u003cdeployment_identifier\u003e]\n```\n\nTo destroy the module contents:\n\n```bash\naws-vault exec \u003cprofile\u003e -- ./go deployment:root:destroy[\u003cdeployment_identifier\u003e]\n```\n\nTo destroy the module prerequisites:\n\n```bash\naws-vault exec \u003cprofile\u003e -- ./go deployment:prerequisites:destroy[\u003cdeployment_identifier\u003e]\n```\n\nConfiguration parameters can be overridden via environment variables:\n\n```bash\nDEPLOYMENT_IDENTIFIER=testing aws-vault exec \u003cprofile\u003e -- ./go\n```\n\nWhen a deployment identifier is provided via an environment variable,\ninfrastructure will not be destroyed at the end of test execution. This can\nbe useful during development to avoid lengthy provision and destroy cycles.\n\nBy default, providers will be downloaded for each terraform execution. To\ncache providers between calls:\n\n```bash\nTF_PLUGIN_CACHE_DIR=\"$HOME/.terraform.d/plugin-cache\" aws-vault exec \u003cprofile\u003e -- ./go\n```\n\n### Common Tasks\n\n#### Generating an SSH key pair\n\nTo generate an SSH key pair:\n\n```\nssh-keygen -m PEM -t rsa -b 4096 -C integration-test@example.com -N '' -f config/secrets/keys/bastion/ssh\n```\n\n#### Generating a self-signed certificate\n\nTo generate a self signed certificate:\n\n```\nopenssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365\n```\n\nTo decrypt the resulting key:\n\n```\nopenssl rsa -in key.pem -out ssl.key\n```\n\n#### Managing CircleCI keys\n\nTo encrypt a GPG key for use by CircleCI:\n\n```bash\nopenssl aes-256-cbc \\\n  -e \\\n  -md sha1 \\\n  -in ./config/secrets/ci/gpg.private \\\n  -out ./.circleci/gpg.private.enc \\\n  -k \"\u003cpassphrase\u003e\"\n```\n\nTo check decryption is working correctly:\n\n```bash\nopenssl aes-256-cbc \\\n  -d \\\n  -md sha1 \\\n  -in ./.circleci/gpg.private.enc \\\n  -k \"\u003cpassphrase\u003e\"\n```\n\nContributing\n------------\n\nBug reports and pull requests are welcome on GitHub at \nhttps://github.com/infrablocks/terraform-aws-msk. This project is \nintended to be a safe, welcoming space for collaboration, and contributors are \nexpected to adhere to the \n[Contributor Covenant](http://contributor-covenant.org) code of conduct.\n\nLicense\n-------\n\nThe library is available as open source under the terms of the\n[MIT License](http://opensource.org/licenses/MIT).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finfrablocks%2Fterraform-aws-msk","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Finfrablocks%2Fterraform-aws-msk","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finfrablocks%2Fterraform-aws-msk/lists"}