{"id":20207835,"url":"https://github.com/exasol/ci-isolation-aws","last_synced_at":"2025-04-10T12:51:21.950Z","repository":{"id":38614911,"uuid":"377727785","full_name":"exasol/ci-isolation-aws","owner":"exasol","description":"AWS account setup for isolating CI builds","archived":false,"fork":false,"pushed_at":"2025-03-06T11:53:36.000Z","size":129,"stargazers_count":2,"open_issues_count":0,"forks_count":2,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-03-24T11:38:24.029Z","etag":null,"topics":["aws","exasol-integration","integration-testing"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/exasol.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-06-17T06:27:47.000Z","updated_at":"2025-03-06T11:47:52.000Z","dependencies_parsed_at":"2024-04-08T08:52:32.373Z","dependency_job_id":"ec40b98f-1b38-43a5-aebb-262bcb051f86","html_url":"https://github.com/exasol/ci-isolation-aws","commit_stats":null,"previous_names":[],"tags_count":14,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/exasol%2Fci-isolation-aws","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/exasol%2Fci-isolation-aws/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/exasol%2Fci-isolation-aws/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/exasol%2Fci-isolation-aws/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/exasol","download_url":"https://codeload.github.com/exasol/ci-isolation-aws/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248220001,"owners_count":21067216,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","exasol-integration","integration-testing"],"created_at":"2024-11-14T05:32:34.612Z","updated_at":"2025-04-10T12:51:21.934Z","avatar_url":"https://github.com/exasol.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AWS CI Account Setup\n\n[![Build Status](https://github.com/exasol/ci-isolation-aws/actions/workflows/ci-build.yml/badge.svg)](https://github.com/exasol/ci-isolation-aws/actions/workflows/ci-build.yml)\n[![Maven Central \u0026ndash; CI Isolation AWS](https://img.shields.io/maven-central/v/com.exasol/ci-isolation-aws)](https://search.maven.org/artifact/com.exasol/ci-isolation-aws)\n\n[![Quality Gate Status](https://sonarcloud.io/api/project_badges/measure?project=com.exasol%3Aci-isolation-aws\u0026metric=alert_status)](https://sonarcloud.io/dashboard?id=com.exasol%3Aci-isolation-aws)\n\n[![Security Rating](https://sonarcloud.io/api/project_badges/measure?project=com.exasol%3Aci-isolation-aws\u0026metric=security_rating)](https://sonarcloud.io/dashboard?id=com.exasol%3Aci-isolation-aws)\n[![Reliability Rating](https://sonarcloud.io/api/project_badges/measure?project=com.exasol%3Aci-isolation-aws\u0026metric=reliability_rating)](https://sonarcloud.io/dashboard?id=com.exasol%3Aci-isolation-aws)\n[![Maintainability Rating](https://sonarcloud.io/api/project_badges/measure?project=com.exasol%3Aci-isolation-aws\u0026metric=sqale_rating)](https://sonarcloud.io/dashboard?id=com.exasol%3Aci-isolation-aws)\n[![Technical Debt](https://sonarcloud.io/api/project_badges/measure?project=com.exasol%3Aci-isolation-aws\u0026metric=sqale_index)](https://sonarcloud.io/dashboard?id=com.exasol%3Aci-isolation-aws)\n\n[![Code Smells](https://sonarcloud.io/api/project_badges/measure?project=com.exasol%3Aci-isolation-aws\u0026metric=code_smells)](https://sonarcloud.io/dashboard?id=com.exasol%3Aci-isolation-aws)\n[![Coverage](https://sonarcloud.io/api/project_badges/measure?project=com.exasol%3Aci-isolation-aws\u0026metric=coverage)](https://sonarcloud.io/dashboard?id=com.exasol%3Aci-isolation-aws)\n[![Duplicated Lines (%)](https://sonarcloud.io/api/project_badges/measure?project=com.exasol%3Aci-isolation-aws\u0026metric=duplicated_lines_density)](https://sonarcloud.io/dashboard?id=com.exasol%3Aci-isolation-aws)\n[![Lines of Code](https://sonarcloud.io/api/project_badges/measure?project=com.exasol%3Aci-isolation-aws\u0026metric=ncloc)](https://sonarcloud.io/dashboard?id=com.exasol%3Aci-isolation-aws)\n\nThis repository contains a setup for provisioning an AWS sub-account for Continuous Integration (CI) testing.\n\n[Features](doc/features.md)\n\n![CI isolation cloud architecture diagram](doc/diagrams/ci-isolation-aws.svg)\n\nAs shown in the picture the repository consists of two parts: cleanup-stack and ci-user-stack. The cleanup-stack is only required once per account. The ci-user-stack for each project that you want to test.\n\nThe cleanup stack deletes all resources in the AWS account except:\n\n* Some resources prefixed with `protected-`: These are the resources of the ci-isolation itself. For example the code-build jobs that deletes everything. The ci-users are not allowed to create such resources.\n* S3 buckets and prefixed with `persistent-`. You can create such buckets to store data that is not deleted. For example for long-term-caching.\n\n## Usage\n\nFirst deploy the ci-user-stack from this repository:\n\n```shell\ncdk --profile \u003cAWS-PROFILE\u003e deploy --parameters exaOwner=\u003cYOUR E-MAIL\u003e\n```\n\n### Adding a CI-User for a new Project\n\nNow you can add a ci user for the project you want to test:\n\n* In the project repository create a new CDK stack:\n\n    ```shell\n    cdk init app --language=java\n    ```\n* Now remove the autogenerated stack (for example `rm src/main/java/com/myorg/TestStack.java`. The exact stack name depends on the folder name).\n* Add this project as a maven dependency.\n* Add a policy document to the resources of the CDK project (see next section)\n* In the App add:\n    ```java\n    final CiUserStack stack = new CiUserStack(app, CiUserStack.CiUserStackProps.builder().projectName(\"\u003cYOUR PROJECT_NAME\u003e\")\n                    .addRequiredPermissions(new PolicyReader().readPolicyFromResources(\"\u003cYOUR POLICY DOCUMENT\u003e\")).build());\n    ```\n* Deploy the stack\n\n## Update Minimal Permissions\n\nIn order to keep the impact of a hacking attack low, we want to only grant the required permissions to the CI user. In this section we describe an approach for detecting the required permissions of your CI build.\n\nDetermining the minimal set of permissions by hand is quite a lot of work. To make our lives easier we use the tool [iamlive](https://github.com/iann0036/iamlive). This tool spies on the local AWS CLI and Terraform, and reports the used permissions. These permissions are not complete in case Terraform uses CloudFormation under the hood, but it's a lot better than having nothing.\n\nTo extract the minimal permissions do the following steps:\n\n* Set your AWS credential environment variables for a user with broader permissions (usually using `. aws_get_session_token`).\n* Run `iamlive`:\n\n  ```shell\n  ./iamlive --set-ini --mode proxy --force-wildcard-resource\n  ```\n* In another terminal run:\n\n  ```shell\n  export HTTP_PROXY=http://127.0.0.1:10080\n  export HTTPS_PROXY=http://127.0.0.1:10080\n  ```\n* Run everything your CI runs in the 2nd terminal. Typically:\n    * Create infrastructure (e.g. using Terraform)\n    * Run tests\n    * Destroy infrastructure\n* Copy the last output from the `iamlive` command to a file. Now you can use this file as permissions in the setup.\n\n### Exasol Cloud Formation Template\n\nThe Exasol database is usually created using a CloudFormation template. The steps from this template are not recorded by iamlive since the CloudFormation template is evaluated in the cloud and by that, the request don't pass the proxy.\n\nSo we have to find out the required permissions by hand. Luckily Exasol offers a [policy for running an Exasol cluster](https://s3.eu-central-1.amazonaws.com/cloudtools.exasol.com/iam_policy.json). We simply downloaded this one and added it as resource. In the future it might be required to update this resource.\n\n## Additional Tasks for Setting up a CI Account\n\n* Visit https://aws.amazon.com/marketplace/pp?sku=ctqmztsepbuk7e9f2ks9nlwj1 and accept license (subscribe)\n* Configure account alias\n    * Open AWS Console\n    * Go to `IAM` / `Dashboard`\n    * On the top of the page edit the alias\n\n## Testing\n\nThis repository contains the integration tests (`PermissionBoundaryIT`) that tests that the CI user only has the expected privileges.\n\nIn order to use this test you need an AWS user with admin privileges. For that reason we do not run this in the CI. To run it locally create a file named `test_config.yaml` in this directory:\n\n```yaml\nowner: \u003cyour e-mail; used as exa:owner tag\u003e\nprofile: \u003cthe aws-profile for the admin user\u003e\n```\n\nBefore running the tests, deploy the stack using the CDK.\n\n## Additional Information\n\n* [Dependencies](dependencies.md)\n* [Changelog](doc/changes/changelog.md)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fexasol%2Fci-isolation-aws","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fexasol%2Fci-isolation-aws","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fexasol%2Fci-isolation-aws/lists"}