{"id":15167118,"url":"https://github.com/igwtcode/aws-multimaster-saltstack","last_synced_at":"2025-10-01T00:31:01.720Z","repository":{"id":162460124,"uuid":"586205739","full_name":"igwtcode/aws-multimaster-saltstack","owner":"igwtcode","description":"infrastructure in aws using cdk in typescript for a secure high available and scalable multi master SaltStack setup","archived":true,"fork":false,"pushed_at":"2023-01-07T11:33:17.000Z","size":283,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-09-22T20:01:41.736Z","etag":null,"topics":["aws","aws-cdk","cdk","cdk-examples","cloudformation","saltstack"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/igwtcode.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-01-07T10:12:57.000Z","updated_at":"2024-06-22T09:38:12.000Z","dependencies_parsed_at":null,"dependency_job_id":"0f7cb255-0ab5-4c02-a1c0-16296d9b7a5b","html_url":"https://github.com/igwtcode/aws-multimaster-saltstack","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/igwtcode%2Faws-multimaster-saltstack","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/igwtcode%2Faws-multimaster-saltstack/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/igwtcode%2Faws-multimaster-saltstack/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/igwtcode%2Faws-multimaster-saltstack/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/igwtcode","download_url":"https://codeload.github.com/igwtcode/aws-multimaster-saltstack/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":219875009,"owners_count":16554636,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","aws-cdk","cdk","cdk-examples","cloudformation","saltstack"],"created_at":"2024-09-27T05:40:14.641Z","updated_at":"2025-10-01T00:31:01.339Z","avatar_url":"https://github.com/igwtcode.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# aws-multimaster-saltstack\n\n## Intro\n\nThis is a demo project, creating an infrastructure in [aws (amazon web services)](https://aws.amazon.com/) using [CDK (cloud development kit) v2](https://docs.aws.amazon.com/cdk/v2/guide/home.html) in [Typescript](https://www.typescriptlang.org/) to achieve the objective described below\n\n## Objective\n\nsecure high available and scalable multi master [SaltStack](https://saltproject.io/) master/minion setup\n\n### Features\n\n- master instances should register/deregister themselves automatically in all minions\n- minion instances should register/deregister themselves automatically in all masters\n- all master instances are also minions\n- `salt-api` should be accessible via an application load balancer isolated in vpc, connecting to all masters\n- dynamodb table to collect ec2 instances (master and minion) attributes such as: _minion_id, instance_id, ip address, instance tier, etc..._\n- api gateway to access salt-api and dynamodb table data with apikey and api usage plan\n\n\u003e **Security**:\n\u003e\n\u003e - iam policies and permissions\n\u003e - data encryption at rest and in transit (kms key, acm tls certificate)\n\u003e - security groups\n\u003e - private and isolated subnets with _nat_, _s3_ and _dynamodb_ gateway\n\u003e - _custom header_ for application load balancer\n\u003e - apikey and usage plan for api gateway\n\u003e\n\u003e **High Availability and Scale**: auto scaling groups with application load balancer using multiple subnets across multiple availability zones (at least 3) with an event driven architecture to automatically register/deregister nodes\n\n## aws services and tools\n\nhere is a brief overview of aws services and tools used in this app\n\n- budgets: _budget alarm_\n- vpc: _vpc, subnet, security group, nat gateway, s3 gateway, dynamodb gateway_\n- ec2 image builder: _pipeline_ with custom recipe\n- iam: _policy, permissions_\n- kms: _kms key_ for data encryption at rest _(s3 bucket, dynamodb table, efs)_\n- acm: _tls certificate_ for data encryption in transit _(api gateway and application load balancer)_\n- dynamodb: _table, ttl, autoscaling_\n- ec2: _instance, launch template, ami, target group, autoscaling group, application load balancer_\n- ssm: _run command, parameter store, ssm agent (ec2 instance)_\n- eventbridge: _event_\n- sns: _topic_\n- s3: _bucket, bucket policy, lifecycle policy, replication rule_\n- route53: _public and private hosted zone, alias record_\n- lambda: _layer, function, alias, autoscaling, policy and permissions, integration with: vpc, eventbridge event, sns notification, efs, dynamodb, application load balancer, ssm run command, ssm parameter store, ec2 ami, snapshot, volume, launch template and autoscaling_\n- cloudwatch: _cloudwatch agent (ec2 instance)_\n\n## Architecture\n\n![image](docs/architecture-1.png)\n\n## Requirements\n\n### Knowledge\n\n- [SaltStack](https://saltproject.io/)\n- aws [CDK](https://docs.aws.amazon.com/cdk/v2/guide/home.html) and [Cloudformation](https://aws.amazon.com/cloudformation/)\n- [Typescript](https://www.typescriptlang.org/) language\n- [Bash](https://www.gnu.org/software/bash/)\n\n### Tools\n\n- code editor or IDE. ([vscode](https://code.visualstudio.com/))\n\n- [aws account](https://portal.aws.amazon.com/billing/signup?refid=d97c9d89-00ee-48c6-84a2-4f1d2dd976da\u0026redirect_url=https%3A%2F%2Faws.amazon.com%2Fregistration-confirmation#/start/email)\n\n- [aws cli](https://aws.amazon.com/cli/)\n\n- aws profile (aws iam user with admin permissions)\n\n```bash\n# check if aws cli is installed\naws --version\n\n# configure profile\naws configure\n```\n\n## Deploy\n\n### create `.env` file in project root directory with following entries:\n\n```bash\n# required, ex: 123456789012\nACCOUNT_ID=\u003caws_account_id\u003e\n\n# required, ex: example.com\nDOMAIN=\u003cregistered_domain_name_associated_with_aws_route53_public_hosted_zone\u003e\n\n# required, ex: iuhfskjbbe87ggjgsd765ajhf765afHJGFSDtasewr76HFJ\nAPI_KEY=\u003capikey_for_app_api_gateway\u003e\n\n# required, ex: admin\nCDK_IAM_USER=\u003caws_iam_user_with_admin_permissions\u003e\n\n# required, ex: example@example.com\nEMAIL=\u003cemail_address_for_sns_notifications_and_budget_alarm\u003e\n\n# optional, default: us-east-1\nREGION=\u003caws_region_to_deploy_cdk_app\u003e\n\n# optional, default: saltuser\nSALT_API_USER=\u003cusername_for_saltapi_user_account\u003e\n\n# optional, default: saltPassword\nSALT_API_PASSWORD=\u003cpassword_for_saltapi_user_account\u003e\n```\n\n### install npm packages\n\n```bash\nnpm install\n```\n\n### install aws cdk\n\n```bash\nsudo npm install -g aws-cdk\n```\n\n### bootstrap cdk\n\n\u003e this is only required, if it's not done before\n\n```bash\ncdk bootstrap\n```\n\n### synthesize app _(optional)_\n\n\u003e to create and review cloudformation template\n\n```bash\ncdk synth \u003e template.yml\n```\n\n### deploy app to aws account\n\n```bash\ncdk deploy\n```\n\n\u003e ```bash\n\u003e # make scripts executable\n\u003e chmod +x src/scripts/*\n\u003e ```\n\n\u003e **NOTE** execute all scripts from project root directory\n\n### copy files to s3 buckets\n\nafter app deployment is successfully finished, several files should be copied to s3 buckets. this could be done by executing a script\n\n\u003e **NOTE** check the bucket names and data directory paths in `src/scripts/s3_sync.sh`\n\u003e bucket names should be automatically set in the script file during synth process\n\n```bash\nbash src/scripts/s3_sync.sh\n```\n\n### execute/run ec2 image builder pipelines\n\nwhen files are successfully copied to s3 buckets, execute the ec2 image builder pipelines to build _salt-master_ and _salt-minion_ ami. this could be also done by executing the following script.\n\n\u003e **NOTE** check the image builder pipeline arn in `src/scripts/imagebuilder_pipeline_exec.sh`\n\u003e they should be automatically set in the script file during synth process\n\n```bash\nbash src/scripts/imagebuilder_pipeline_exec.sh\n```\n\nafter about 30-45 minutes image builder pipelines should be done. they will automatically:\n\n- put the created ami id in ssm parameter store\n- create a new launch template version with the new ami id and set it as default\n- modify the autoscaling groups to use the latest launch template version\n- update autoscaling groups instances with new ami\n- delete older ami _(images, snapshots, volumes)_\n\n### start salt-master cluster\n\nwhen deploying the cdk infrastructure app, 2 autoscaling groups for salt-master and salt-minion would be created having the _amazon linux 2_ as ami set in launch template with (min:0, max: 3) capacity, so there should be no instance running from these autoscaling groups at first\n\n\u003e **NOTE** make sure that new ec2 images (ami) are created for both salt-master and salt-minion by ec2 image builder from previous step\n\nchange/increase desired or minimum capacity for salt-master autoscaling group to create ec2 instances. in [aws management console](https://signin.aws.amazon.com/)\n\n### start salt-minion cluster\n\n\u003e **NOTE** make sure, that at least 1 salt-master instance is running to accept minions\n\ndo the same steps as above for salt-minion autoscaling group\n\n### app api\n\nfollowing 2 api endpoints should be available, which could be tested by executing following script\n\n\u003e **NOTE** `X-API-Key` Header with _apiKey value_ must be set in request, which is done in script\n\n- **GET** `https://\u003cAPI_DOMAIN_NAME\u003e/instances`\n  returns the dynamodb table data containing information about ec2 instances\n- **GET** `https://\u003cAPI_DOMAIN_NAME\u003e/salt`\n  returns ping result from `salt-api` of all registered minions\n  \u003e **NOTE** it takes some time (about 2-5 minutes) after an ec2 instance comes up or goes down, for the changes to be reflected in masters and minions (register/deregister), namely accepting or deleting the minion key or adding new salt-master ip to salt-minion master ip list\n\n\u003e **NOTE** check the _api domain name_ and _api key_ in `src/scripts/api.sh`\n\u003e they should be automatically set in the script file during synth process.\n\u003e `curl` package is used in script\n\n```bash\nbash src/scripts/api.sh\n```\n\n## Destroy/Delete\n\n```bash\ncdk destroy\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Figwtcode%2Faws-multimaster-saltstack","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Figwtcode%2Faws-multimaster-saltstack","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Figwtcode%2Faws-multimaster-saltstack/lists"}