{"id":16179146,"url":"https://github.com/jonashackt/pulumi-python-aws-ansible","last_synced_at":"2025-03-16T10:31:35.920Z","repository":{"id":36045660,"uuid":"211320940","full_name":"jonashackt/pulumi-python-aws-ansible","owner":"jonashackt","description":"Example project showing how to use Pulumi locally \u0026 with TravisCI to create Infrastructure on AWS - and then how to use \u0026 integrate Pulumi with Ansible to install Docker on the EC2 instance \u0026 continuously test it with Testinfra \u0026 pytest (TDD)","archived":false,"fork":false,"pushed_at":"2024-10-23T22:59:05.000Z","size":3826,"stargazers_count":38,"open_issues_count":15,"forks_count":13,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-10-24T11:10:56.311Z","etag":null,"topics":["ansible","ansible-role","aws-cli","aws-pulumi","aws-python","docker","ec2-instance","pulumi-ansible","pulumi-python","pulumi-stacks","pytest","testinfra","travis","travis-ci","ubuntu"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jonashackt.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-09-27T13:04:06.000Z","updated_at":"2024-05-14T03:48:17.000Z","dependencies_parsed_at":"2023-12-26T23:25:18.470Z","dependency_job_id":"7e9da701-8faa-4ab5-be41-77a1ae168898","html_url":"https://github.com/jonashackt/pulumi-python-aws-ansible","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonashackt%2Fpulumi-python-aws-ansible","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonashackt%2Fpulumi-python-aws-ansible/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonashackt%2Fpulumi-python-aws-ansible/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonashackt%2Fpulumi-python-aws-ansible/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jonashackt","download_url":"https://codeload.github.com/jonashackt/pulumi-python-aws-ansible/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243810862,"owners_count":20351602,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ansible","ansible-role","aws-cli","aws-pulumi","aws-python","docker","ec2-instance","pulumi-ansible","pulumi-python","pulumi-stacks","pytest","testinfra","travis","travis-ci","ubuntu"],"created_at":"2024-10-10T05:25:36.954Z","updated_at":"2025-03-16T10:31:35.473Z","avatar_url":"https://github.com/jonashackt.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pulumi-python-aws-ansible\n[![Build Status](https://github.com/jonashackt/pulumi-python-aws-ansible/workflows/ec2-pulumi-ansible/badge.svg)](https://github.com/jonashackt/pulumi-python-aws-ansible/actions)\n[![License](http://img.shields.io/:license-mit-blue.svg)](https://github.com/jonashackt/spring-boot-vuejs/blob/master/LICENSE)\n[![renovateenabled](https://img.shields.io/badge/renovate-enabled-yellow)](https://renovatebot.com)\n[![versionpulumi](https://img.shields.io/github/pipenv/locked/dependency-version/jonashackt/pulumi-python-aws-ansible/pulumi?color=brightgreen)](https://www.pulumi.com/)\n[![versionawscli](https://img.shields.io/github/pipenv/locked/dependency-version/jonashackt/pulumi-python-aws-ansible/awscli?color=brightgreen)](https://aws.amazon.com/cli/)\n[![versionansible](https://img.shields.io/github/pipenv/locked/dependency-version/jonashackt/pulumi-python-aws-ansible/ansible?color=brightgreen)](https://docs.ansible.com/ansible/latest/index.html)\n[![versiontestinfra](https://img.shields.io/github/pipenv/locked/dependency-version/jonashackt/pulumi-python-aws-ansible/testinfra?color=brightgreen)](https://testinfra.readthedocs.io/en/latest/index.html)\n[![versionpytest](https://img.shields.io/github/pipenv/locked/dependency-version/jonashackt/pulumi-python-aws-ansible/pytest?color=brightgreen)](https://docs.pytest.org/en/latest/)\n\nExample project showing how to use Pulumi locally \u0026 with TravisCI to create Infrastructure on AWS \n\n[![asciicast](https://asciinema.org/a/273687.svg)](https://asciinema.org/a/273687)\n\nSee https://github.com/jonashackt/pulumi-talk#what-is-pulumi for more info on \"What is Pulumi?\".\n\n## Table of Contents  \n* [Prerequisites](#prerequisites)\n  * [Install Pulumi](#install-pulumi)\n  * [Configure AWS credentials](#configure-aws-credentials)\n* [An example Project with AWS \u0026 Python](#an-example-project-with-aws--python)\n  * [A clean Python environment with virtualenv](#a-clean-python-environment-with-virtualenv)\n  * [Pulumi first run](#pulumi-first-run)\n* [A comparable Use Case](#a-comparable-use-case)\n  * [Build a common ground: Create an EC2 instance with SSH access](#build-a-common-ground-create-an-ec2-instance-with-ssh-access)\n* [Run Pulumi with Travis](#run-pulumi-with-travis)\n  * [Configure TravisCI to connect to AWS \u0026 app.pulumi.com](#configure-travisci-to-connect-to-aws--apppulumicom)\n  * [Install \u0026 configure Pulumi on TravisCI](#install--configure-pulumi-on-travisci)\n  * [Fire up Pulumi on TravisCI](#fire-up-pulumi-on-travisci)\n* [Install Docker on EC2 instance](#install-docker-on-ec2-instance)\n  * [Creating an EC2 keypair with Ansible \u0026 firing up an EC2 instance with Pulumi](#creating-an-ec2-keypair-with-ansible--firing-up-an-ec2-instance-with-pulumi)\n  * [SSH connection to the Pulumi created EC2 instance](#ssh-connection-to-the-pulumi-created-ec2-instance)\n  * [Configure outgoing traffic for apt with a second Security group rule (\"egress\")](#configure-outgoing-traffic-for-apt-with-a-second-security-group-rule-egress)\n  * [Reuse Ansible role 'docker' with ansible-galaxy CLI \u0026 requirements.yml](#reuse-ansible-role-docker-with-ansible-galaxy-cli--requirementsyml)\n* [Replace direct usage of virtualenv and pip with pipenv](#replace-direct-usage-of-virtualenv-and-pip-with-pipenv)\n  * [Configure renovate and shields.io badges to use our pipenv managed versions](#configure-renovate-and-shieldsio-badges-to-use-our-pipenv-managed-versions)\n  * [Using pipenv on TravisCI](#using-pipenv-on-travisci)\n* [Test-driven Development with Pulumi](#test-driven-development-with-pulumi)\n  * [Using Testinfra for Testing Python based Pulumi code](#using-testinfra-for-testing-python-based-pulumi-code)\n  * [Configure the Pulumi created EC2 instance in Testinfra/pytest](#configure-the-pulumi-created-ec2-instance-in-testinfrapytest)\n  * [Continuous Cloud infrastructure with Pulumi, Ansible \u0026 Testinfra on TravisCI](#continuous-cloud-infrastructure-with-pulumi-ansible--testinfra-on-travisci)\n* [Run Pulumi on GitHub Actions](#run-pulumi-on-github-actions)\n\n\n\n## Prerequisites\n\n#### Install Pulumi\n\nhttps://www.pulumi.com/docs/get-started/install/\n\nInstall Pulumi SDK on MacOS:\n\n`brew install pulumi`\n\nor on Linux:\n\n`curl -fsSL https://get.pulumi.com | sh`\n\n\nIf you choose to use Python as your preferred language for Pulumi, you should also install Python:\n\n`brew install python`\n\n\n#### Configure AWS credentials \n\nLike already described in https://github.com/jonashackt/molecule-ansible-docker-vagrant#install--configure-aws-cli, we need to do the following:\n\n\nFirst we need to sure to have the [AWS CLI installed](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html). We can do this via Python pip package manager with (or any other system package manager):\n \n```\npip3 install awscli\n```\n \nNow we should check, if AWS CLI was successfully installed. The `aws --version` command should print out sometime like:\n\n```\n$ aws --version\naws-cli/1.16.255 Python/3.7.4 Darwin/18.7.0 botocore/1.12.245\n```\n\nNow configure the AWS CLI to use the correct credentials. [According to the AWS docs](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html#cli-quick-configuration), the fastest way to accomplish that is to run `aws configure`:\n\n```\n$ aws configure\nAWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE\nAWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\nDefault region name [None]: eu-central-1\nDefault output format [None]: json\n```\n\n\n## An example Project with AWS \u0026 Python\n\nLet's create an Pulumi example project using Python and AWS. Therefore create an empty directory:\n\n```\nmkdir pulumi-aws-python-example\ncd pulumi-aws-python-example\n```\n\nNow create a Pulumi project with: `pulumi new aws-python`. Then you're promted to login to Pulumi (for more info about that, visit https://www.pulumi.com/docs/troubleshooting/faq/#how-does-pulumi-depend-on-pulumi-com), if you run `pulumi new` for the first time:\n\n```\nManage your Pulumi stacks by logging in.\nRun `pulumi login --help` for alternative login options.\nEnter your access token from https://app.pulumi.com/account/tokens\n    or hit \u003cENTER\u003e to log in using your browser                   :\n```\n\nI used GitHub to authorize Pulumi Cloud in my Browser:\n\n![pulumi-cli-login](screenshots/pulumi-cli-login.png)\n\nNow the console needs our attention again - Pulumi want's to know about __project name__, __description__, __stack name__:\n\n```\n  Welcome to Pulumi!\n\n  Pulumi  helps you create, deploy, and manage infrastructure on any cloud using\n  your favorite language. You can get started today with Pulumi at:\n\n      https://www.pulumi.com/docs/get-started/\n\nThis command will walk you through creating a new Pulumi project.\n\nEnter a value or leave blank to accept the (default), and press \u003cENTER\u003e.\nPress ^C at any time to quit.\n\nproject name: (pulumi-aws) pulumi-example-aws-python\nproject description: (A minimal AWS Python Pulumi program)\nCreated project 'pulumi-example-aws-python'\n\nPlease enter your desired stack name.\nTo create a stack in an organization, use the format \u003corg-name\u003e/\u003cstack-name\u003e (e.g. `acmecorp/dev`).\nstack name: (dev) jonashackt/dev\nCreated stack 'dev'\n```\n\nThen in case of AWS Pulumi needs a region to be used. For me I use `eu-central-1`:\n\n```\naws:region: The AWS region to deploy into: (us-east-1) eu-central-1\nSaved config\n```\n\nNow we're already finished:\n\n```\nYour new project is ready to go! ✨\n\nTo perform an initial deployment, run the following commands:\n\n   1. virtualenv -p python3 venv\n   2. source venv/bin/activate\n   3. pip3 install -r requirements.txt\n\nThen, run 'pulumi up'\n```\n\nNow you can also login to pulumi.com - I use `jonashackt` as the organisation, so you'll find my projects under: https://app.pulumi.com/jonashackt\n\n![pulumi-com-project-overview](screenshots/pulumi-com-project-overview.png)\n\n\n\n#### A clean Python environment with virtualenv\n\nBefore running `pulumi up`, you should install [virtualenv](https://virtualenv.pypa.io/en/latest/installation/), which is basically a project local dependency management for pip packages (this is also referred to as a good Python package dependency user style). If you're a Maven, Gradle, NPM user - think of a project local directory, where the all your dependecies are downloaded:\n\n```\n# On MacOS or Windows, simply do a\npip install virtualenv\n\n# be more careful on Linux distros, e.g. use your distro's package manager for installation or use pip install --user virtualenv\n```\n\nNow we have to configure `virtualenv` to create the virtual pip environment - this will create a new directory `venv` inside your project folder:\n\n```\nvirtualenv -p python3 venv\n```\n\nThe `-p python3` option tells virtualenv to use the Python interpreter with version 3. Now we also need to `activate` the new isolated Python environment - and virtualenv provides us with a script to do so:\n\n```\nsource venv/bin/activate\n```\n\n`source` is just a synonym for `.`, which simply [executes something on a shell](https://superuser.com/a/46146/497608). To run a check, if you're using the isolated virtualenv's Python now, run a `pip3 -V`. It should contain the correct path to your project:\n\n```\n$ pip3 -V\npip 19.2.3 from /Users/jonashecht/dev/pulumi-aws/venv/lib/python3.7/site-packages/pip (python 3.7)\n\n# whereas the non-isolated Python would give something like this (on MacOS using brew):\npip 19.1.1 from /usr/local/lib/python3.7/site-packages/pip (python 3.7)\n# then something would be missing - try some research on https://virtualenv.pypa.io/en/latest/userguide/\n```\n\nFinally we are now installing this Pulumi project's dependencies with:\n\n```\npip3 install -r requirements.txt\n```\n\n\n#### Pulumi first run\n\nNow we should have everything prepared to run Pulumi with `pulumi up`:\n\n```\n(venv)  jonashecht  ~/dev/pulumi-aws   master ●  pulumi up\nPlease choose a stack, or create a new one: dev\nPreviewing update (dev):\n\n     Type                 Name                           Plan\n +   pulumi:pulumi:Stack  pulumi-example-aws-python-dev  create\n +   └─ aws:s3:Bucket     my-bucket                      create\n\nResources:\n    + 2 to create\n\nDo you want to perform this update? details\n+ pulumi:pulumi:Stack: (create)\n    [urn=urn:pulumi:dev::pulumi-example-aws-python::pulumi:pulumi:Stack::pulumi-example-aws-python-dev]\n    + aws:s3/bucket:Bucket: (create)\n        [urn=urn:pulumi:dev::pulumi-example-aws-python::aws:s3/bucket:Bucket::my-bucket]\n        [provider=urn:pulumi:dev::pulumi-example-aws-python::pulumi:providers:aws::default_1_5_0::04da6b54-80e4-46f7-96ec-b56ff0331ba9]\n        acl         : \"private\"\n        bucket      : \"my-bucket-58790f7\"\n        forceDestroy: false\n\nDo you want to perform this update?\n```\n\nAfter choosing a stack to perform the action on, Pulumi outlines everything it will do when we choose `yes` to perform the update. Open the `details` to see everything in much more depth.\n\nThen choose `yes` and Pulumi runs the update:\n\n```\nUpdating (dev):\n\n     Type                 Name                           Status\n +   pulumi:pulumi:Stack  pulumi-example-aws-python-dev  created\n +   └─ aws:s3:Bucket     my-bucket                      created\n\nOutputs:\n    bucket_name: \"my-bucket-33cba3e\"\n\nResources:\n    + 2 created\n\nDuration: 9s\n\nPermalink: https://app.pulumi.com/jonashackt/pulumi-example-aws-python/dev/updates/1\n```\n\nYou can have a look at the link provided and you'll see your Pulumi console:\n\n![pulumi-first-run](screenshots/pulumi-first-run.png)\n\nThe example project creates a S3 bucket, which you could also check out by clicking onto the `Open in AWS console` button:\n\n![aws-s3-console](screenshots/aws-s3-console.png)\n\nThat's it: this is our first Cloud resource created by Pulumi!\n\n\n\n## A comparable Use Case\n\nAs we want to compare Pulumi (in terms of apples vs. bananas ;P ) with other Infrastructure-as-Code tools like Ansible. Therefore we should pick a use case like the one in https://github.com/jonashackt/molecule-ansible-docker-vagrant - which is \"Installing Docker on an EC2 Ubuntu box\".\n\nLet's delete the initial stack and S3 bucket first. To destroy an existing stack and its resources, simply run:\n\n```\npulumi destroy\n```\n\n#### Build a common ground: Create an EC2 instance with SSH access\n\nLet's have a look into the tutorials: https://www.pulumi.com/docs/tutorials/aws/ec2-webserver/\n\nand the Pulumi API reference -\u003e https://www.pulumi.com/docs/reference/pkg/python/pulumi_aws/\n\nYeah, now we should be able to setup our first EC2 instance with Pulumi. Open [__main__.py](__main__.py) in your IDE and add some code:\n\n```\nimport pulumi\nimport pulumi_aws as aws\nfrom pulumi_aws import ec2\n\n# AMI image configuration\nec2_image_id = 'ami-0cc0a36f626a4fdf5'\nec2_image_owner = '099720109477'\nec2_instance_size = 't2.micro'\nec2_instance_name = 'aws-ec2-ubuntu'\n\n# Lets use Pulumi to get the AMI image\npulumi_ami = aws.get_ami(\n                    filters = [{ \"name\": \"image-id\", \"values\": [ec2_image_id]}],\n                    owners  = [ec2_image_owner])\n\n# Create a EC2 security group\nssh_port = 22\n\npulumi_security_group = ec2.SecurityGroup(\n                            'pulumi-secgrp',\n                            description = 'Enable HTTP access',\n                            ingress = [\n                                { 'protocol': 'tcp', 'from_port': ssh_port, 'to_port': ssh_port, 'cidr_blocks': ['0.0.0.0/0'] }\n                            ]\n)\n\n# Create EC2 instance\nec2_instance = ec2.Instance(\n                    ec2_instance_name,\n                    instance_type = ec2_instance_size,\n                    security_groups = [pulumi_security_group.name],\n                    ami = pulumi_ami.id\n)\n\npulumi.export('publicIp', ec2_instance.public_ip)\npulumi.export('publicHostName', ec2_instance.public_dns)\n\n\n```\n\n\u003e Maybe you find yourself lost in the Pulumi documentation like me: Many things are simply not available right now. E.g. the Pulumi AWS provider is derived from the Terraform AWS provider - so have a look there: https://www.terraform.io/docs/providers/aws/index.html!\n\u003e If you look for aws.get_ami special `owners` parameter, have a look at https://www.terraform.io/docs/providers/aws/d/ami.html\n\nLet's first choose an AMI as described in [Choosing an Ubuntu 18.04 AMI](https://github.com/jonashackt/molecule-ansible-docker-vagrant#choosing-an-ubuntu-1804-ami) using the [ubuntu Amazon EC2 AMI Locator](https://cloud-images.ubuntu.com/locator/ec2/). Combine your AWS region and the desired Ubuntu version and type this into the search box:\n\n```\neu-central-1 18.04 LTS\n```\n\nNow choose the latest AMI id with the Instance Type `hvm:ebs-ssd` like this: `ami-0cc0a36f626a4fdf5`\n\n\u003e Preventing the \"Exception: invoke of aws:index/getAmi:getAmi failed: \"owners\": required field is not set ()\"\n\nIf you're also just starting with Pulumi, you may also wonder if we __really need__ the `owners` parameter in the module `aws.get_ami`.\n\nWell - this one is required - have a look at https://www.terraform.io/docs/providers/aws/d/ami.html#owners. So to prevent the mentioned Exception we need to also provide the valid `owners` id for the Ubuntu image. Having a look at the AWS docs, these ids could be found occasionally: https://docs.aws.amazon.com/de_de/AWSEC2/latest/UserGuide/finding-an-ami.html#finding-quick-start-ami\n\nThe Ubuntu owners id is `099720109477` for example, the RedHat AMIs need the `309956199498` and so on. Amazon images simply need `amazon` or `aws-marketplace`, Microsoft images have `microsoft` also.\n\nNow give your Pulumi code a try and fire it up with `pulumi up`. This should give something like:\n\n```\n$ pulumi up\nPreviewing update (dev):\n\n     Type                      Name                           Plan\n +   pulumi:pulumi:Stack       pulumi-example-aws-python-dev  create\n +   ├─ aws:ec2:SecurityGroup  pulumi-secgrp                  create\n +   └─ aws:ec2:Instance       aws-ec2-ubuntu                 create\n\nResources:\n    + 3 to create\n\nDo you want to perform this update? yes\nUpdating (dev):\n\n     Type                      Name                           Status\n +   pulumi:pulumi:Stack       pulumi-example-aws-python-dev  created\n +   ├─ aws:ec2:SecurityGroup  pulumi-secgrp                  created\n +   └─ aws:ec2:Instance       aws-ec2-ubuntu                 created\n\nOutputs:\n    publicHostName: \"ec2-3-120-32-194.eu-central-1.compute.amazonaws.com\"\n    publicIp      : \"3.120.32.194\"\n\nResources:\n    + 3 created\n\nDuration: 33s\n\nPermalink: https://app.pulumi.com/jonashackt/pulumi-example-aws-python/dev/updates/3\n```\n\nHave a look into your AWS management console and you should see the new instance:\n\n![first-ec2-instance-running](screenshots/first-ec2-instance-running.png)\n\n\nOn console you can also [access the variables exported](https://www.pulumi.com/docs/reference/pkg/python/pulumi/#stack-exports%C2%B6) with `pulumi.export` like:\n\n```\n$ pulumi stack output publicHostName\nec2-3-120-235-112.eu-central-1.compute.amazonaws.com\n\n# or\n\n$ pulumi stack output publicIp\n3.120.235.112\n```\n\n\n## Run Pulumi with Travis\n\nNow that we're able to run our Pulumi code against AWS, we should also configure Travis to do the job for us every time we push our code.\n\nMost of the needed parts on how to execute a Python library on Travis to connect to AWS has been already described here: https://github.com/jonashackt/molecule-ansible-docker-vagrant#use-travisci-to-execute-molecule-with-ec2-infrastructure\n\n\n#### Configure TravisCI to connect to AWS \u0026 app.pulumi.com\n\nSo let's create a [.travis.yml](.travis.yml), that doesn't contain so much in the first place:\n\n```yaml\nsudo: false\nlanguage: python\n\nenv:\n- BOTO_CONFIG=\"/dev/null\"\n```\n\nAfter that, we should activate TravisCI for our project at https://travis-ci.org/account/repositories. \n\nAnd we should already now think about the needed Pulumi login to `app.pulumi.com`, which could be done by defining the `PULUMI_ACCESS_TOKEN` variable, that needs to contain a correct access token. To create a Pulumi access token, head over to https://app.pulumi.com/yourUserNameHere/settings/tokens and click on `NEW ACCESS TOKEN`:\n\n![app-pulumi-com-access-token](screenshots/app-pulumi-com-access-token.png)\n\nNow we can switch over to the settings page of our Travis configuration at https://travis-ci.org/jonashackt/pulumi-example-aws-python/settings and create the needed environment variables `AWS_ACCESS_KEY` \u0026 `AWS_SECRET_KEY` for the `aws configure set` command and the `PULUMI_ACCESS_TOKEN` variable:\n\n![travis-env-aws-pulumi-access-vars](screenshots/travis-env-aws-pulumi-access-vars.png)\n\n\nInside the `install` section of our `.travis.yml` we are now able to install and configure the boto packages \u0026 the AWS CLI, which Pulumi later uses to communicate with AWS:\n\n```yaml\ninstall:\n# install AWS related packages\n- pip install boto boto3\n- pip install --upgrade awscli\n# configure AWS CLI\n- aws configure set aws_access_key_id $AWS_ACCESS_KEY\n- aws configure set aws_secret_access_key $AWS_SECRET_KEY\n- aws configure set default.region eu-central-1\n# show AWS CLI config\n- aws configure list\n```\n\nTo avoid [the knows problems with boto (the AWS python client) on Travis](aus dem Weg gehen), we already defined `sudo: false` and configured the `BOTO_CONFIG=\"/dev/null\"` environment variable directly inside our [.travis.yml](.travis.yml). With that our AWS communication should work like a charm.\n\n\n#### Install \u0026 configure Pulumi on TravisCI\n\nNow we should also install the Pulumi SDK - as we already did locally. This time, we use Python's pip to do that for us: \n\n```\nscript:\n  # Install Pulumi SDK with the installation script from https://www.pulumi.com/docs/get-started/install/#installation-script\n  - curl -fsSL https://get.pulumi.com | sh\n  # Add Pulumi to Travis' PATH so the executable could be found\n  - export PATH=$PATH:/home/travis/.pulumi/bin\n  - pulumi version\n  # login to app.pulumi.com with the predefined PULUMI_ACCESS_TOKEN\n  - pulumi login\n```\n\nAs you see, we add the Pulumi executable to the Travis build environments' PATH, otherwise we run into `pulumi: command not found` errors. We also log in to app.pulumi.com.\n\n\n#### Fire up Pulumi on TravisCI\n\nNow we're nearly there! But as we're running on TravisCI, we should skip the virtualenv usage - since TravisCI's Python environment is already based on a virtualenv configuration (see https://docs.travis-ci.com/user/languages/python/#travis-ci-uses-isolated-virtualenvs).\nTherefore we only need to install the libraries needed via `pip install -r requirements.txt`:\n\n```yaml\nscript:\n...\n  # skip virtualenv in virtualenv Travis inception (see https://docs.travis-ci.com/user/languages/python/#travis-ci-uses-isolated-virtualenvs)\n  # and simply install pip libraries directly (otherwise pulumi: command not found error will come after us again)\n  - pip install -r requirements.txt\n\n  # Select your Pulumi projects' stack\n  - pulumi stack select dev\n```\n\nBefore running `pulumi up`, we have to select the stack Pulumi should use. Otherwise we'll run into `error: no stack selected;` errors. You can list your stacks on the command line with `pulumi stack ls` or have a look into the Pulumi online portal at https://app.pulumi.com/yourUserNameHere.\n\nThe final step then is to fire up Pulumi (and destroy the infrastructure again after it was created to prevent unnecessary costs):\n\n```yaml\nscript:\n...\n  # Run Pulumi unattended\n  - pulumi up --yes\n  # After everything has been created, we should also destroy the infrastructure again\n  - pulumi destroy --yes\n```\n\nMind the `--yes` switch at the end of the commands to automatically approve and perform the update/destruction after previewing it \n\nYou may now have a look into your TravisCI build ([like this](https://travis-ci.org/jonashackt/pulumi-example-aws-python/builds/596168913)). It should show an output similar to this at the end of the log:\n\n```\n$ \n\nThe command \"\" exited with 0.\n\n0.69s$ pulumi stack select dev\n\nThe command \"pulumi stack select dev\" exited with 0.\n\n49.93s$ pulumi up --yes\n\nPreviewing update (dev):\n\n +  pulumi:pulumi:Stack pulumi-example-aws-python-dev create \n\n +  aws:ec2:SecurityGroup pulumi-secgrp create \n\n +  aws:ec2:Instance aws-ec2-ubuntu create \n\n +  pulumi:pulumi:Stack pulumi-example-aws-python-dev create \n\n \n\nResources:\n\n    + 3 to create\n\nUpdating (dev):\n\n +  pulumi:pulumi:Stack pulumi-example-aws-python-dev creating \n\n +  aws:ec2:SecurityGroup pulumi-secgrp creating \n\n +  aws:ec2:SecurityGroup pulumi-secgrp created \n\n +  aws:ec2:Instance aws-ec2-ubuntu creating \n\n@ Updating.....\n\n +  aws:ec2:Instance aws-ec2-ubuntu created \n\n +  pulumi:pulumi:Stack pulumi-example-aws-python-dev created \n\n \n\nOutputs:\n\n    publicHostName: \"ec2-18-195-71-41.eu-central-1.compute.amazonaws.com\"\n\n    publicIp      : \"18.195.71.41\"\n\nResources:\n\n    + 3 created\n\nDuration: 45s\n\nPermalink: https://app.pulumi.com/jonashackt/pulumi-example-aws-python/dev/updates/11\n\nThe command \"pulumi up --yes\" exited with 0.\n\n58.06s$ pulumi destroy --yes\n\nPreviewing destroy (dev):\n\n -  aws:ec2:Instance aws-ec2-ubuntu delete \n\n -  aws:ec2:SecurityGroup pulumi-secgrp delete \n\n -  pulumi:pulumi:Stack pulumi-example-aws-python-dev delete \n\n -  pulumi:pulumi:Stack pulumi-example-aws-python-dev delete \n\n \n\nOutputs:\n\n  - publicHostName: \"ec2-18-195-71-41.eu-central-1.compute.amazonaws.com\"\n\n  - publicIp      : \"18.195.71.41\"\n\nResources:\n\n    - 3 to delete\n\nDestroying (dev):\n\n -  aws:ec2:Instance aws-ec2-ubuntu deleting \n\n@ Destroying.....\n\n -  aws:ec2:Instance aws-ec2-ubuntu deleted \n\n -  aws:ec2:SecurityGroup pulumi-secgrp deleting \n\n -  aws:ec2:SecurityGroup pulumi-secgrp deleted \n\n -  pulumi:pulumi:Stack pulumi-example-aws-python-dev deleting \n\n -  pulumi:pulumi:Stack pulumi-example-aws-python-dev deleted \n\n \n\nOutputs:\n\n  - publicHostName: \"ec2-18-195-71-41.eu-central-1.compute.amazonaws.com\"\n\n  - publicIp      : \"18.195.71.41\"\n\nResources:\n\n    - 3 deleted\n\nDuration: 56s\n\nPermalink: https://app.pulumi.com/jonashackt/pulumi-example-aws-python/dev/updates/12\n\nThe resources in the stack have been deleted, but the history and configuration associated with the stack are still maintained. \n\nIf you want to remove the stack completely, run 'pulumi stack rm dev'.\n\nThe command \"pulumi destroy --yes\" exited with 0.\n\nDone. Your build exited with 0.\n```\n\nYou can also have a look into the AWS management console to see the EC2 instance beeing created and then destroyed again.\n\n\n\n## Install Docker on EC2 instance\n\nSo EC2's running, now we want to install Docker on it. But is there a way on how to issue shell commands and the like with Pulumi?\n\nThe answer is: NO! There's currently no way to do this (see https://github.com/pulumi/pulumi/issues/99).  \n\nIf you want to know more about that topic, have a look into: https://github.com/jonashackt/pulumi-talk#provision-yes-configure-no\n\nAt the end, [Pulumi website tells us](https://www.pulumi.com/docs/intro/vs/chef_puppet_etc/) how to deal with such a problem:\n \n\u003e Simply use Chef, Puppet, Ansible or Salt here!\n\nOk, since Ansible leads the pack, we'll use that right away!\n\n\n#### Creating an EC2 keypair with Ansible \u0026 firing up an EC2 instance with Pulumi\n\nSo let's use the Pulumi AWS documentation (sorry, we need to use the JavaScript docs here, since the Python docs aren't quite good right now): https://www.pulumi.com/docs/reference/pkg/nodejs/pulumi/aws/\n\nAnd inside the [aws.ec2.Instance](https://www.pulumi.com/docs/reference/pkg/nodejs/pulumi/aws/ec2/#Instance) there's a parameter [keyName](https://www.pulumi.com/docs/reference/pkg/nodejs/pulumi/aws/ec2/#Instance-keyName) which uses the type [aws.ec2.KeyPair](https://www.pulumi.com/docs/reference/pkg/nodejs/pulumi/aws/ec2/#KeyPair), which is sadly not linked in the docs - but you can find it looking into the ec2 docs again searching for `keypair`: https://www.pulumi.com/docs/reference/pkg/nodejs/pulumi/aws/ec2/. But the docs also state, that\n\n\u003e Currently this resource requires an existing user-supplied key pair. This key pair’s public key will be registered with AWS to allow logging-in to EC2 instances.\n\nSo it seems that we have to provide our own keypair to use this module :(\n\nCompared to the Ansible [ec2_key module](https://docs.ansible.com/ansible/latest/modules/ec2_key_module.html), __this is rather astonishing__. So we could only use the Pulumi [aws.ec2.KeyPair module](https://www.pulumi.com/docs/reference/pkg/nodejs/pulumi/aws/ec2/#KeyPair) as a wrapper around an EC2 keypair, that has to be generated elsewhere.\n\nSo let's do that also with Ansible. We simply create a [keypair.yml](keypair.yml), that creates a new EC2 keypair, if there's no local private key already. The generated private key is then saved to `.ec2ssh/pulumi_key` inside the projects directory:\n\n```yaml\n- name: Create EC2 Keypair\n  hosts: localhost\n  gather_facts: false\n  vars:\n    keypair_name: pulumi_key\n    keypair_path: \".ec2ssh//{{ keypair_name }}\"\n  tasks:\n    - name: Be sure to have .ec2ssh as local directory present\n      file:\n        path: \".ec2ssh\"\n        state: directory\n\n    - name: Delete remote keypair\n      ec2_key:\n        name: \"{{ keypair_name }}\"\n        state: absent\n\n    - name: Create keypair\n      ec2_key:\n        name: \"{{ keypair_name }}\"\n      register: keypair\n\n    - name: Persist the keypair\n      copy:\n        dest: \"{{ keypair_path }}\"\n        content: \"{{ keypair.key.private_key }}\"\n        mode: 0600\n```\n\nTo run the playbook you should have Ansible (and boto3 for AWS accessibility) installed on our system:\n\n`pip install ansible boto3`\n\nRun the Ansible playbook now to generate the EC2 keypair:\n\n```\nansible-playbook keypair.yml\n```\n\nNow a file called `pulumi_key` should be generated. This is the private key of our new EC2 key pair. You can find the key pair also in the AWS EC2 management console:\n\n![aws-ec2-keypair](screenshots/aws-ec2-keypair.png)\n\nThe last part is to tell Pulumi, that this new key pair should be used at the EC2 instance creation time. Therefore we change our Pulumi Python code in [__main__.py](__main__.py) and add the `key_name` parameter to the `ec2.Instance()` call:\n\n```python\n...\nec2_keypair_name = 'pulumi_key'\n...\n# Create EC2 instance\nec2_instance = ec2.Instance(\n                    ec2_instance_name,\n                    key_name = ec2_keypair_name,\n                    instance_type = ec2_instance_size,\n                    security_groups = [pulumi_security_group.name],\n                    ami = pulumi_ami.id\n)\n...\n```\n\nNow we should create a new Pulumi stack (and therefore destroy the old one first):\n\n```\n$ pulumi destroy --yes\n$ pulumi up --yes\n```\n\n\n#### SSH connection to the Pulumi created EC2 instance\n\nWith that [the key pair's public key is configured inside our new EC2 instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html) and is ready to be used. Ansible should now be able to connect to that machine via SSH, so we need a playbook to read the `public IP` from the Pulumi stack and connect to the EC2 instance. Let's open our [playbook.yml](playbook.yml):\n\n```yaml\n- name: Connect to Pulumi created EC2 instance and install Docker\n  hosts: localhost\n  vars:\n    keypair_name: pulumi_key\n    keypair_path: \".ec2ssh//{{ keypair_name }}\"\n    # this is really important to configure, since Ansible will use this user for SSH connection\n    ssh_user: ubuntu\n  tasks:\n    - name: Gather Pulumi created EC2 instance public IP\n      shell: pulumi stack output publicIp\n      register: pulumi_stack_output\n\n    - set_fact:\n        ec2_public_ip: \"{{ pulumi_stack_output.stdout }}\"\n\n    - debug:\n        msg: \"The public IP of the Pulumi created EC2 instance is: {{ ec2_public_ip }}\"\n\n    - name: Wait 300 seconds for port 22 to become open and contain \"SSH\" - then the SSH connection should work afterwards\n      wait_for:\n        port: 22\n        host: \"{{ ec2_public_ip }}\"\n        search_regex: SSH\n        delay: 10\n        timeout: 320\n```\n\nTo prevent waiting command prompts like this:\n\n```\nThe authenticity of host '3.120.32.99 (3.120.32.99)' can't be established.\nECDSA key fingerprint is SHA256:+1Pb+VPlnUntX1E6bnegvpdU7qEPbxAsdXN6mFDgtZY.\nAre you sure you want to continue connecting (yes/no)\n```\n\nWe should also create a proper [ansible.cfg](ansible.cfg) containing `ssh_args = -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no`.\n\nIf nothing seems to work, you can debug the SSH connection with direct usage of Ansible ping module like this (insert the public IP retrieved by `pulumi stack output publicIp` and append a `,`):\n\n```\nansible -i $(pulumi stack output publicIp), -m ping all --user=ubuntu --private-key=.ec2ssh/pulumi_key -vvv\n```\n\n#### Configure outgoing traffic for apt with a second Security group rule (\"egress\")\n\nNow we finally want to use the ansible-galaxy provided Ansible role 'docker' to install Docker on our Pulumic created EC2 instance!\n\nFirst thing: we should prevent outgoing connection errors like this:\n\n```bash\n        \"0% [Connecting to eu-central-1.ec2.archive.ubuntu.com (52.59.228.109)] [Connect\\u001b[0m\",\n        \"                                                                               \",\n        \"Err:2 http://eu-central-1.ec2.archive.ubuntu.com/ubuntu bionic InRelease\",\n        \"  Could not connect to eu-central-1.ec2.archive.ubuntu.com:80 (52.59.244.233)\n```\n\nTherefore we need to create an EC2 __egress__ Security role also inside our Pulumi program [__main__.py](__main__.py) (have a look into the Pulumi [ec2.Security resource docs](https://www.pulumi.com/docs/reference/pkg/nodejs/pulumi/aws/ec2/#SecurityGroup)))\n\n```python\n# Create a EC2 security group\npulumi_security_group = ec2.SecurityGroup(\n                            'pulumi-secgrp',\n                            description = 'Enable HTTP access',\n                            ingress = [\n                                { 'protocol': 'tcp', 'from_port': ec2_ssh_port, 'to_port': ec2_ssh_port, 'cidr_blocks': ['0.0.0.0/0'] }\n                            ],\n                            egress = [\n                                { 'protocol': '-1', 'from_port': 0, 'to_port': 0, 'cidr_blocks': ['0.0.0.0/0'] }\n                            ]\n)\n```\n\nA crucial point here is to set the __egress__ `protocol` to `-1`, since this is semantically equivalent to `\"all\"`, which is needed for successful outgoing communication of the package manager `apt-get` we need to use for the Docker installation. Inside the AWS EC2 management console, the `Outbound` configuration has to look like this:\n\n![aws-security-group-egress-outgoing-all-traffic](screenshots/aws-security-group-egress-outgoing-all-traffic.png)\n \n__Don't try to look that up inside the Pulumi docs, it's not there!__ But follow the link to [the Terraform docs, there you'll find this specific parameter configuration](https://github.com/terraform-providers/terraform-provider-aws/blob/master/website/docs/r/security_group.html.markdown#argument-reference).\n\nNow apt should be able to talk to the outside world finally.\n\n\n#### Reuse Ansible role 'docker' with ansible-galaxy CLI \u0026 requirements.yml\n\nNow we need a Ansible role that installs Docker on Ubuntu for us. That problem is already solved for us (again) in https://github.com/jonashackt/molecule-ansible-docker-vagrant \nBut since we don't want to copy the role, we use Ansible-Galaxy CLI to do that for us. [There's a nice not so much advertised feature](https://docs.ansible.com/ansible/latest/reference_appendices/galaxy.html#installing-multiple-roles-from-a-file) of `ansible-galaxy`, where you only have to provide a `requirements.yml` with the needed role with their git urls. So let's create our own [requirements.yml](requirements.yml):\n\n```yaml\n# dependency to docker role\n- src: https://github.com/jonashackt/molecule-ansible-docker-vagrant\n  name: docker\n```\n\nNow use `ansible-galaxy` CLI to download the role into the standard `/roles` directory, where your play/role could use it:\n\n```\nansible-galaxy install -r requirements.yml -p roles/\n```\n\nNow the Ansible role should reside at [roles/docker/tasks/main.yml](roles/docker/tasks/main.yml) and is ready to be used!\n\n\n##### Use our role to install Docker on Ubuntu\n\nNow we finally want to use the ansible-galaxy provided Ansible role 'docker' to install Docker on our Pulumic created EC2 instance!\n\nLet's open our [playbook.yml](playbook.yml) again:\n\n```yaml\n    # Since the docker role needs facts like ansible_distribution \u0026 ansible_lsb.codename, we need to gather those facts beforehand\n    # from our delegate_to host (EC2 instance)\n    - name: Gather facts of Pulumi created EC2 instance for later role execution\n      setup:\n      delegate_to: \"{{ ec2_public_ip }}\"\n      vars:\n        ansible_ssh_private_key_file: \"{{ keypair_path }}\"\n        ansible_user: ubuntu\n\n    - name: Now use the ansible-galaxy prepared docker role to install Docker on our EC2 instance\n      import_role:\n        name: docker\n      delegate_to: \"{{ ec2_public_ip }}\"\n      become: true\n      vars:\n        ansible_ssh_private_key_file: \"{{ keypair_path }}\"\n        ansible_user: ubuntu\n```\n\nWith the first usage of Ansible's [setup module](https://docs.ansible.com/ansible/latest/modules/setup_module.html) we're able to gather facts based dynamically on our Pulumi created host. In order to use this dynamic host, we use Ansible's [Delegation feature](https://docs.ansible.com/ansible/latest/user_guide/playbooks_delegation.html#delegation) with the keyword `delegate_to`.\n\nWe also need to set the `ansible_ssh_private_key_file` and `ansible_user` SSH connection properties correctly to the keypair we used to create our EC2 instance using Pulumi.\n\nIn the last step we finally use our role `docker` to install Docker on our Pulumi created EC2 instance. Simply run our playbook with: \n\n```\nansible-playbook playbook.yml\n```\n\nI also created a shell script [create_all.sh](create_all.sh) that contains all neccessary steps in the right order:\n\n```shell script\n#!/usr/bin/env bash\nset -euo pipefail\n\necho \"destroy pre-created Pulumi instances\"\npulumi destroy --yes\n\necho \"generate EC2 keypair and save private key locally (since Pulumi isn't able to do that now)\"\nansible-playbook keypair.yml\n\necho \"execute Pulumi to create EC2 instances\"\npulumi up --yes\n\necho \"Downloading the Ansible role 'docker' with ansible-galaxy\"\nansible-galaxy install -r requirements.yml -p roles/\n\necho \"run Ansible role to install Docker on Ubuntu\"\nansible-playbook playbook.yml\n```\n\nRun it with `./create_all.sh`!\n\nAnd here's also a recording of the whole thing:\n\n[![asciicast](https://asciinema.org/a/273830.svg)](https://asciinema.org/a/273830)\n\n\n## Replace direct usage of virtualenv and pip with pipenv\n\nWhile `virtualenv` does a great job to seperate Python build dependencies from different projects (and your system's pip packages), the `requirements.txt` doesn't lead to fully deterministic builds, because it doesn't manage transient dependencies - and Python packages tend to use unpinned dependencies -\u003e so different environments could lead to different build outputs (see this article for more details: https://realpython.com/pipenv-guide/).\n\nThe solution for this is the use of [pipenv](https://github.com/pypa/pipenv), which effectively replaces the usage of pip and virtualenv and manages both for you. It also introduces two new files, which replace the requirements.txt: [Pipfile](https://github.com/pypa/pipfile) and `Pipfile.lock` to ensure deterministic builds.\n\nTo install `pipenv`, simply run\n\n```\npip3 install pipenv\n```\n\nNow create a new virtual environment with\n\n```\npipenv shell --python 3.7\n```\n\nHere, `--python 3.7` ensures a current Python 3 installation. \n\nNow we can use `pipenv` to install all required dependencies with \n\n```\npipenv install\n```\n\nWe can now also add Ansible as a requirement, since our project depends on it, incl. boto3, which Ansible needs to work with AWS correctly (remember, we don't use `pip` command any more, only `pipenv` to install packages):\n\n```\npipenv install ansible boto3\n```\n\n#### Configure renovate and shields.io badges to use our pipenv managed versions\n\nAs this project is kept up-to-date by [renovate](https://github.com/renovatebot/renovate), we should configure the [renovate.json](renovate.json) configuration file to tell renovate about pipenv ([which is currently a beta feature](https://docs.renovatebot.com/configuration-options/#pipenv)):\n\n```json\n{\n  \"extends\": [\n    \"config:base\"\n  ],\n  \"groupName\": \"all\",\n  \"pipenv\": {\n    \"enabled\": true\n  }\n}\n```\n\nAnd as our project uses [shields.io badges](https://shields.io/category/platform-support) to show used dependency versions at the top of the [README.md](README.md), we should also configure them to be dynamically read from the [Pipfile](Pipfile).\n\nThere was a recent update to the dynamic shields.io endpoint (have a look into https://github.com/badges/shields/issues/2259#issuecomment-537699182 for more info), so we could upgrade our badges like that:\n\n```\n# old\n[![versionansible](https://img.shields.io/badge/ansible-2.8.5-brightgreen.svg)](https://docs.ansible.com/ansible/latest/index.html)\n\n# new\n[![versionansible](https://img.shields.io/github/pipenv/locked/dependency-version/jonashackt/pulumi-python-aws-ansible/ansible?color=brightgreen)](https://docs.ansible.com/ansible/latest/index.html)\n```\n\nNow every badge stays up-to-date with every renovate update - like this one here: [![versionpulumi](https://img.shields.io/github/pipenv/locked/dependency-version/jonashackt/pulumi-python-aws-ansible/pulumi?color=brightgreen)](https://www.pulumi.com/)\n\n\n#### Using pipenv on TravisCI\n\nAs we're now using `pipenv` for dependecy management, we could also rework our TravisCI configuration. [As this post states](https://medium.com/@dirk.avery/quirks-of-pipenv-on-travis-ci-and-appveyor-10d6adb6c55b) and we already learned, TravisCI uses `virtualenv` environment per default. Now `pipenv` is luckily smart enough to detect that, so all we have to do is the following inside our [.travis.yml](.travis.yml):\n\n```yaml\nsudo: false\nlanguage: python\n\nenv:\n  - BOTO_CONFIG=\"/dev/null\"\n\ninstall:\n  # First: install Pulumi SDK with the installation script from https://www.pulumi.com/docs/get-started/install/#installation-script\n  - curl -fsSL https://get.pulumi.com | sh\n  # Add Pulumi to Travis' PATH so the executable could be found\n  - export PATH=$PATH:/home/travis/.pulumi/bin\n  - pulumi version\n\n  # Second: Install pulumi-aws dependency (among others like awscli \u0026 Ansible) via pipenv dependency manager\n  - pip install pipenv\n  # Install required (and locked) dependecies from Pipfile.lock (especially pulumi-aws, otherwise we run into \"error: no resource plugin 'aws' found in the workspace or on your $PATH\"\n  # pipenv is smart enough to recognise the existing virtualenv without a prior pipenv shell command (see https://medium.com/@dirk.avery/quirks-of-pipenv-on-travis-ci-and-appveyor-10d6adb6c55b)\n  - pipenv install\n\n  # Third: Check, if Pulumi aws plugin was installed correctly\n  - pulumi plugin ls\n\n  # Forth: Configure AWS CLI\n  - aws configure set aws_access_key_id $AWS_ACCESS_KEY\n  - aws configure set aws_secret_access_key $AWS_SECRET_KEY\n  - aws configure set default.region eu-central-1\n  # show AWS CLI config\n  - aws configure list\n\n\nscript:\n  # login to app.pulumi.com with the predefined PULUMI_ACCESS_TOKEN\n  - pulumi login\n\n  # Select your Pulumi projects' stack\n  - pulumi stack select dev\n\n  # Run Pulumi unattended\n  - pulumi up --yes\n  # After everything has been created, we should also destroy the infrastructure again\n  - pulumi destroy --yes\n``` \n\n__There's one big thing to take care of:__ Pulumi needs to be installed first like it is described on the official docs: https://www.pulumi.com/docs/get-started/install/#installation-script\n\nAnd __then__ install the dependencies containing `pulumi-aws` __afterwards__. Otherwise, you'll run into the error `error: no resource plugin 'aws' found in the workspace or on your $PATH` (see https://github.com/pulumi/pulumi/issues/2097 and [this build log](https://travis-ci.org/jonashackt/pulumi-python-aws-ansible/builds/603558895?utm_medium=notification\u0026utm_source=github_status)).\n\nYou can double check if your Pulumi AWS installation is correct with `pulumi plugin ls` - this has to contain the Pulumi aws resource plugin like that:\n\n```\n$ pulumi plugin ls\nNAME  KIND      VERSION  SIZE    INSTALLED    LAST USED\naws   resource  1.7.0    220 MB  2 hours ago  2 hours ago\n```\n\nIf this is empty, AWS connectivity won't work!\n\nAfter that's safe, the `aws CLI` has to be configured as usual. Now the TravisCI build is ready for it's script phase! After a `pulumi login` based on the correctly set `PULUMI_ACCESS_TOKEN` and the selection of the wanted Pulumi stack via `pulumi stack select dev`, you can start using Pulumi!\n\n\n## Test-driven Development with Pulumi\n\nhttps://www.pulumi.com/blog/testing-your-infrastructure-as-code-with-pulumi/\n\nhttps://www.pulumi.com/blog/tag/testing/ --\u003e 1 article! :(\n\nhttps://www.pulumi.com/blog/unit-testing-infrastructure-in-nodejs-and-mocha/\n\nDiscussion to test harnesses ended in this https://github.com/pulumi/pulumi/issues/1902\n\nThere's a 1 star plugin for [Chef's TDD harness tool kitchenCI](https://kitchen.ci/): https://github.com/jacoblearned/kitchen-pulumi\n\n\n### Using Testinfra for Testing Python based Pulumi code\n\nSo there are currently no docs/articles available on how to test Python based Pulumi setups. But there are [great tools around for doing testing in the Python world](https://blog.codecentric.de/en/2018/12/test-driven-infrastructure-ansible-molecule/)) - like pytest and [Testinfra](https://testinfra.readthedocs.io/en/latest/). So why not use them?!\n\nTherefore let's install Testinfra with pipenv:\n\n```\npipenv install testinfra\n```\n\nNow we need some test code. Since we're re-using the use case of installing Docker on an AWS EC2 instance from this project https://github.com/jonashackt/molecule-ansible-docker-vagrant, we could also use the existing test code from the [test_docker.py](https://github.com/jonashackt/molecule-ansible-docker-vagrant/blob/master/docker/molecule/tests/test_docker.py) file:\n\n```\nimport os\n\n\ndef test_is_docker_installed(host):\n    package_docker = host.package('docker-ce')\n\n    assert package_docker.is_installed\n\n\ndef test_vagrant_user_is_part_of_group_docker(host):\n    user_vagrant = host.user('vagrant')\n\n    assert 'docker' in user_vagrant.groups\n\n\ndef test_run_hello_world_container_successfully(host):\n    hello_world_ran = host.run(\"sudo docker run hello-world\")\n\n    assert 'Hello from Docker!' in hello_world_ran.stdout\n```\n\nAll we have to change here, is the way how Testinfra gets to know about the Pulumi created EC2 instances, since we don't have a Molecule inventory file here anymore. So let's delete the parts containing `import testinfra.utils.ansible_runner` and `MOLECULE_INVENTORY_FILE` and try to somehow configure the Testinfra hosts in another way.\n\n\n### Configure the Pulumi created EC2 instance in Testinfra/pytest\n\nWe should now be able to finally run our Testinfra test code with pytest?! Well, we need to tweak the standard `py.test -v tests/test_docker.py` command a bit before!\n\nFirst, we should use the [pytest ssh backend](https://testinfra.readthedocs.io/en/latest/backends.html#ssh) to connect to our EC2 instance. Why? Because we need to be able to configure our Ansible generated EC2 keypair for the connection and the ssh backend has a nice `--ssh-identity-file=/path/to/key` configuration option.\n\nWe also need to dynamically use the Pulumi created EC2 instance' host IP. Remember the `pulumi stack output publicIp` already used inside our Ansible playbook to gather the IP address? So let's [use this inline](https://stackoverflow.com/a/8663623/4964553) inside our pytest command with the help of the `$()` nesting feature of our console together with the `--hosts='ssh://server'` notation (otherwise, pytest tries to use paramiko connection backend):\n\n```\n --hosts='ssh://'$(pulumi stack output publicIp)\n```\n\nAdditionally, we also need to prevent such ssh prompts for later usage inside our CI/CD pipelines:\n\n```\ntests/test_docker.py::test_is_docker_installed[ssh://18.194.233.114] The authenticity of host '18.194.233.114 (18.194.233.114)' can't be established.\nECDSA key fingerprint is SHA256:gtQ84TwZZUqsgS/rJSn2rvNuqnzt9Xp1z0f9gFnfzOk.\nAre you sure you want to continue connecting (yes/no)? yes\n```\n\nTherefore, we need to configure `pytest` in a way it tell's the underlying ssh connection to use those ssh configuration args: `-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no`. As there's no way to do that with the help of the commandline, we need to use the `--ssh-config=tests/pytest_ssh_config` parameter and configure them with an ssh_config file called [tests/pytest_ssh_config](tests/pytest_ssh_config):\n\n```\nHost *\n   User ubuntu\n   StrictHostKeyChecking no\n   UserKnownHostsFile=/dev/null\n```\n\nAlso we should configure the `ssh user` inside this file: `User ubuntu`, since otherwise the connection won't work (it would use root, but our EC2 keypair is configured for `ubuntu` user).\n\n\nFinally everything should be prepared so that we can execute pytest and Testinfra on our setup:\n\n```\npy.test -v tests/test_docker.py --ssh-identity-file=.ec2ssh/pulumi_key --ssh-config=tests/pytest_ssh_config --hosts='ssh://'$(pulumi stack output publicIp)\n```\n\nIf everything went well, the output should look somehow like this:\n\n![pytest-testinfra-successful-passed](screenshots/pytest-testinfra-successful-passed.png)\n\n\n### Continuous Cloud infrastructure with Pulumi, Ansible \u0026 Testinfra on TravisCI\n\nNow we should have everything prepared to let this setup run not only once - but continuously! Quoting [this blog post](https://blog.codecentric.de/en/2018/12/continuous-infrastructure-ansible-molecule-travisci/):\n\n\u003e The benefits of Test-driven development (TDD) for infrastructure code are undeniable. But we shouldn´t settle there! What about executing these tests automatically and based on a regular schedule? Applying Continuous Integration to infrastructure code should be the next step.\n\nSo let's configure TravisCI to do the magic continuously and therefore add all those steps from [create_all.sh](create_all.sh) to our [.travis.yml](.travis.yml):\n\n```\nscript:\n  # login to app.pulumi.com with the predefined PULUMI_ACCESS_TOKEN\n  - pulumi login\n\n  # Select your Pulumi projects' stack\n  - pulumi stack select dev\n\n  # destroy pre-created Pulumi instances\n  - pulumi destroy --yes\n\n  # generate EC2 keypair and save private key locally (since Pulumi isn't able to do that now)\n  - ansible-playbook keypair.yml\n\n  # execute Pulumi to create EC2 instances\n  - pulumi up --yes\n\n  # Downloading the Ansible role 'docker' with ansible-galaxy\n  - ansible-galaxy install -r requirements.yml -p roles/\n\n  # run Ansible role to install Docker on Ubuntu\n  - ansible-playbook playbook.yml\n\n  # use Testinfra with Pytest to execute our tests\n  - py.test -v tests/test_docker.py --ssh-identity-file=.ec2ssh/pulumi_key --ssh-config=tests/pytest_ssh_config --hosts='ssh://'$(pulumi stack output publicIp)\n\n  # destroy Pulumi instances after successful tests\n  - pulumi destroy --yes\n```\n\n\n## Run Pulumi on GitHub Actions\n\nAs we're forced to move away from TravisCI (see https://blog.codecentric.de/en/2021/02/github-actions-pipeline/), we also need to move our setup to GitHub Actions.\n\nTherefore let's create a [.github/workflows/ec2-pulumi-ansible.yml](.github/workflows/ec2-pulumi-ansible.yml) workflow file.\n\nI already used the great Pulumi GitHub Action at https://github.com/jonashackt/azure-training-pulumi/blob/main/.github/workflows/preview-and-up.yml and I really loved it.\n\nThe documentation also provides us with an example to use Pulumi with AWS https://www.pulumi.com/docs/guides/continuous-delivery/github-actions/ which would reduce the code needed with Travis I guess.\n\nIn order to use [the pulumi GitHub Action](https://github.com/pulumi/actions) with AWS we need to create the two repository secrets `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` first. We also need to create the `PULUMI_ACCESS_TOKEN` containing a Pulumi access token we need to freshly create in our Pulumi console at https://app.pulumi.com/jonashackt/settings/tokens. Create all three secrets in `Settings/Secrets` of your repository:\n\n![github-actions-repo-secrets](screenshots/github-actions-repo-secrets.png)\n\nNow let's have a look into the [.github/workflows/ec2-pulumi-ansible.yml](.github/workflows/ec2-pulumi-ansible.yml):\n\n\n\nIn my first attempts I used [the pulumi GitHub Action](https://github.com/pulumi/actions) which is really really great to get you started fast. If you have a look at the examples like in https://www.pulumi.com/docs/guides/continuous-delivery/github-actions/ you may tend to repeatedly use the Pulumi Action incl. repeating the environment variables needed.\n\nSo you end up with something like this:\n\n```yaml\n name: ec2-pulumi-ansible\n\n on: [push]\n\n jobs:\n   ec2-pulumi-ansible:\n     runs-on: ubuntu-latest\n\n     steps:\n       - uses: actions/checkout@v2\n\n       - name: Cache pipenv virtualenvs incl. all pip packages\n         uses: actions/cache@v2\n         with:\n           path: ~/.local/share/virtualenvs\n           key: ${{ runner.os }}-pipenv-${{ hashFiles('**/Pipfile.lock') }}\n           restore-keys: |\n             ${{ runner.os }}-pipenv-\n\n       - uses: actions/setup-python@v2\n         with:\n           python-version: '3.9'\n\n       - name: Install required dependencies with pipenv\n         run: |\n           pip install pipenv\n           pipenv install\n\n       - name: Destroy pre-created Pulumi instances\n         uses: pulumi/actions@v1\n         with:\n           command: destroy\n         env:\n           AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}\n           AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}\n           PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}\n\n       - name: Generate EC2 keypair and save private key locally (since Pulumi isn't able to do that now)\n         run: pipenv run ansible-playbook keypair.yml\n         env:\n           AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}\n           AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}\n           AWS_REGION: eu-central-1\n\n       - name: Execute Pulumi to create EC2 instances\n         uses: pulumi/actions@v1\n         with:\n           command: up\n         env:\n           AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}\n           AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}\n           PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}\n\n       - name: Downloading the Ansible role 'docker' with ansible-galaxy \u0026 run Ansible role to install Docker on Ubuntu\n         run: |\n           pipenv run ansible-galaxy install -r requirements.yml -p roles/\n           echo \"Select your Pulumi projects stack\"\n           pulumi stack select dev\n           pipenv run ansible-playbook playbook.yml\n         env:\n           AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}\n           AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}\n           AWS_REGION: eu-central-1\n           PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}\n\n       - name: Use Testinfra with Pytest to execute our tests\n         run: pipenv run py.test -v tests/test_docker.py --ssh-identity-file=.ec2ssh/pulumi_key --ssh-config=tests/pytest_ssh_config --hosts='ssh://'$(pulumi stack output publicIp)\n         env:\n           PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}\n\n       - name: Destroy Pulumi instances after successful tests\n         uses: pulumi/actions@v1\n         with:\n           command: destroy\n         env:\n           AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}\n           AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}\n           PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}\n```\n\nThat's not bad at all! We even do caching using [the GHA cache action](https://github.com/actions/cache)! Especially the `key` definition with `key: ${{ runner.os }}-pipenv-${{ hashFiles('**/Pipfile.lock') }}` is something to keep an eye on.\n\n[The `hashFiles` function](https://docs.github.com/en/actions/reference/context-and-expression-syntax-for-github-actions#hashfiles) is really cool, since it will invalidate the cache only, if our `Pipfile.lock` (which could reside somewhere in the repository) changes.\n\nTaking the next step in optimizing our GHA workflow, we can use GHA's ability [to condense environment variables on the step or even job level](https://docs.github.com/en/actions/reference/environment-variables). This leads to a much leaner workflow file:\n\n```yaml\nname: ec2-pulumi-ansible\n\non: [push]\n\njobs:\n  ec2-pulumi-ansible:\n    runs-on: ubuntu-latest\n    env:\n      AWS_REGION: eu-central-1\n      AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}\n      AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}\n      PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}\n    steps:\n    - uses: actions/checkout@v2\n\n    - name: Cache pipenv virtualenvs incl. all pip packages\n      uses: actions/cache@v2\n      with:\n        path: ~/.local/share/virtualenvs\n        key: ${{ runner.os }}-pipenv-${{ hashFiles('**/Pipfile.lock') }}\n        restore-keys: |\n          ${{ runner.os }}-pipenv-\n\n    - uses: actions/setup-python@v2\n      with:\n        python-version: '3.9'\n\n    - name: Install required dependencies with pipenv\n      run: |\n        pip install pipenv\n        pipenv install\n\n    - name: Destroy pre-created Pulumi instances\n      uses: pulumi/actions@v1\n      with:\n        command: destroy\n\n    - name: Generate EC2 keypair and save private key locally (since Pulumi isn't able to do that now)\n      run: pipenv run ansible-playbook keypair.yml\n\n    - name: Execute Pulumi to create EC2 instances\n      uses: pulumi/actions@v1\n      with:\n        command: up\n\n    - name: Downloading the Ansible role 'docker' with ansible-galaxy \u0026 run Ansible role to install Docker on Ubuntu\n      run: |\n        pipenv run ansible-galaxy install -r requirements.yml -p roles/\n        echo \"Select your Pulumi projects stack\"\n        pulumi stack select dev\n        pipenv run ansible-playbook playbook.yml\n\n    - name: Use Testinfra with Pytest to execute our tests\n      run: pipenv run py.test -v tests/test_docker.py --ssh-identity-file=.ec2ssh/pulumi_key --ssh-config=tests/pytest_ssh_config --hosts='ssh://'$(pulumi stack output publicIp)\n\n    - name: Destroy Pulumi instances after successful tests\n      uses: pulumi/actions@v1\n      with:\n        command: destroy\n```\n\nHaving a look [at the build log](https://github.com/jonashackt/pulumi-python-aws-ansible/runs/1976853168?check_suite_focus=true), I got aware of another point we could optimize. As you can see [the pulumi GitHub Action](https://github.com/pulumi/actions) introduces a step called `Build pulumi/action@v1` which seems to pull and prepare a container to run Pulumi in the steps later.\n\nThis step takes 2 1/2 minutes in every build. And it's only there since we use the Pulumi GHA. Additionally every Pulumi GHA step collects every Python package needed to run Pulumi again and again - it simply doesn't recognize our setup here using https://docs.pipenv.org/\n\nSo wouldn't it be better if we could craft a workflow, that is `pipenv` aware? Yes we can: We simply need to switch from the Pulumi GHA to only use the much simpler [install-pulumi-cli GitHub Action](https://github.com/pulumi/action-install-pulumi-cli). Now we can (and must) rely solely on our `pipenv` environment and our jobs will only need half of the execution time: \n\nSo finally our [.github/workflows/ec2-pulumi-ansible.yml](.github/workflows/ec2-pulumi-ansible.yml) looks like this:\n\n```yaml\nname: ec2-pulumi-ansible\n\non: [push]\n\njobs:\n  ec2-pulumi-ansible:\n    runs-on: ubuntu-latest\n    env:\n      AWS_REGION: eu-central-1\n      AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}\n      AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}\n      PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}\n    steps:\n      - uses: actions/checkout@v2\n\n      - name: Cache pipenv virtualenvs incl. all pip packages\n        uses: actions/cache@v2\n        with:\n          path: ~/.local/share/virtualenvs\n          key: ${{ runner.os }}-pipenv-${{ hashFiles('**/Pipfile.lock') }}\n          restore-keys: |\n            ${{ runner.os }}-pipenv-\n\n      - uses: actions/setup-python@v2\n        with:\n          python-version: '3.9'\n\n      - name: Install Pulumi CLI\n        uses: pulumi/action-install-pulumi-cli@v1.0.1\n\n      - name: Install required dependencies with pipenv \u0026 login to Pulumi app \u0026 select stack\n        run: |\n          pip install pipenv\n          pipenv install\n\n          echo \"login to app.pulumi.com with the predefined PULUMI_ACCESS_TOKEN\"\n          pulumi login\n\n          echo \"Select your Pulumi projects stack\"\n          pulumi stack select dev\n\n      - name: Destroy pre-created Pulumi instances\n        run: pipenv run pulumi destroy --yes\n\n      - name: Generate EC2 keypair and save private key locally (since Pulumi isn't able to do that now)\n        run: pipenv run ansible-playbook keypair.yml\n\n      - name: Execute Pulumi to create EC2 instances\n        run: pipenv run pulumi up --yes\n\n      - name: Downloading the Ansible role 'docker' with ansible-galaxy \u0026 run Ansible role to install Docker on Ubuntu\n        run: |\n          pipenv run ansible-galaxy install -r requirements.yml -p roles/\n          pipenv run ansible-playbook playbook.yml\n\n      - name: Use Testinfra with Pytest to execute our tests\n        run: pipenv run py.test -v tests/test_docker.py --ssh-identity-file=.ec2ssh/pulumi_key --ssh-config=tests/pytest_ssh_config --hosts='ssh://'$(pulumi stack output publicIp)\n\n      - name: Destroy Pulumi instances after successful tests\n        run: pipenv run pulumi destroy --yes\n```\n\n\n\n\n\n## Links\n\nSee example projects https://github.com/pulumi/examples\n\nhttps://www.pulumi.com/docs/tutorials/aws/ec2-webserver/\n\nhttps://blog.scottlowe.org/2019/05/05/a-sandbox-for-learning-pulumi/\n\nSome also use Pulumi + Ansible already: https://blog.wallaroolabs.com/2018/10/spinning-up-a-wallaroo-cluster-is-easy/\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjonashackt%2Fpulumi-python-aws-ansible","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjonashackt%2Fpulumi-python-aws-ansible","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjonashackt%2Fpulumi-python-aws-ansible/lists"}