{"id":14984455,"url":"https://github.com/jonashackt/molecule-ansible-docker-aws","last_synced_at":"2025-04-07T06:10:07.745Z","repository":{"id":35199092,"uuid":"158685708","full_name":"jonashackt/molecule-ansible-docker-aws","owner":"jonashackt","description":"Example project showing how to test Ansible roles with Molecule using Testinfra and a multiscenario approach with Docker, Vagrant \u0026 AWS EC2 as infrastructure providers","archived":false,"fork":false,"pushed_at":"2025-03-28T23:41:00.000Z","size":5912,"stargazers_count":89,"open_issues_count":20,"forks_count":11,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-03-29T00:27:13.739Z","etag":null,"topics":["amazon-web-services","ansible","ansible-role","ansible-test","aws","aws-ec2","circleci","circleci2","docker","docker-installation","molecule","pytest","python","testinfra","travis-ci","vagrant","virtualbox"],"latest_commit_sha":null,"homepage":"https://blog.codecentric.de/en/2018/12/test-driven-infrastructure-ansible-molecule/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jonashackt.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-11-22T10:57:41.000Z","updated_at":"2023-11-22T15:20:46.000Z","dependencies_parsed_at":"2024-06-04T00:59:12.901Z","dependency_job_id":"2e2e1b4f-5397-4f28-b33f-6eb988a5c3c9","html_url":"https://github.com/jonashackt/molecule-ansible-docker-aws","commit_stats":{"total_commits":377,"total_committers":2,"mean_commits":188.5,"dds":"0.48275862068965514","last_synced_commit":"2b6ae3b2206d79849c65777ad9932c98234e39cd"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonashackt%2Fmolecule-ansible-docker-aws","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonashackt%2Fmolecule-ansible-docker-aws/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonashackt%2Fmolecule-ansible-docker-aws/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonashackt%2Fmolecule-ansible-docker-aws/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jonashackt","download_url":"https://codeload.github.com/jonashackt/molecule-ansible-docker-aws/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247601448,"owners_count":20964864,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["amazon-web-services","ansible","ansible-role","ansible-test","aws","aws-ec2","circleci","circleci2","docker","docker-installation","molecule","pytest","python","testinfra","travis-ci","vagrant","virtualbox"],"created_at":"2024-09-24T14:09:05.466Z","updated_at":"2025-04-07T06:10:07.718Z","avatar_url":"https://github.com/jonashackt.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# molecule-ansible-docker-aws\n[![Build Status](https://github.com/jonashackt/molecule-ansible-docker-aws/workflows/docker/badge.svg)](https://github.com/jonashackt/molecule-ansible-docker-aws/actions)\n[![Build Status](https://github.com/jonashackt/molecule-ansible-docker-aws/workflows/vagrant/badge.svg)](https://github.com/jonashackt/molecule-ansible-docker-aws/actions)\n[![renovateenabled](https://img.shields.io/badge/renovate-enabled-yellow)](https://renovatebot.com)\n[![versionansible](https://img.shields.io/github/pipenv/locked/dependency-version/jonashackt/molecule-ansible-docker-aws/ansible?color=brightgreen)](https://docs.ansible.com/ansible/latest/index.html)\n[![versionmolecule](https://img.shields.io/github/pipenv/locked/dependency-version/jonashackt/molecule-ansible-docker-aws/molecule?color=brightgreen)](https://molecule.readthedocs.io/en/latest/)\n[![versiontestinfra](https://img.shields.io/github/pipenv/locked/dependency-version/jonashackt/molecule-ansible-docker-aws/testinfra?color=brightgreen)](https://testinfra.readthedocs.io/en/latest/)\n[![versionmolecule-vagrant](https://img.shields.io/github/pipenv/locked/dependency-version/jonashackt/molecule-ansible-docker-aws/molecule-vagrant?color=brightgreen)](https://github.com/ansible-community/molecule-vagrant)\n[![versionmolecule-docker](https://img.shields.io/github/pipenv/locked/dependency-version/jonashackt/molecule-ansible-docker-aws/molecule-docker?color=brightgreen)](https://github.com/ansible-community/molecule-docker)\n[![versionmolecule-ec2](https://img.shields.io/github/pipenv/locked/dependency-version/jonashackt/molecule-ansible-docker-aws/molecule-ec2?color=brightgreen)](https://github.com/ansible-community/molecule-ec2)\n[![versionawscli](https://img.shields.io/github/pipenv/locked/dependency-version/jonashackt/molecule-ansible-docker-aws/awscli?color=brightgreen)](https://aws.amazon.com/cli/)\n\nExample project showing how to test Ansible roles with Molecule using Testinfra and a multiscenario approach with Vagrant, Docker \u0026 AWS EC2 as the infrastructure under test. \n\n[![asciicast](https://asciinema.org/a/214914.svg)](https://asciinema.org/a/214914)\n\nThere are already two blog posts complementing this repository:\n* [Test-driven infrastructure development with Ansible \u0026 Molecule](https://blog.codecentric.de/en/2018/12/test-driven-infrastructure-ansible-molecule/)\n* [Continuous Infrastructure with Ansible, Molecule \u0026 TravisCI](https://blog.codecentric.de/en/2018/12/test-driven-infrastructure-ansible-molecule/)\n* [Continuous cloud infrastructure with Ansible, Molecule \u0026 TravisCI on AWS](https://blog.codecentric.de/en/2019/01/ansible-molecule-travisci-aws/)\n\n...and some articles in the german iX Magazin:\n\n[![ix-2019-04](screenshots/ix-2019-04.png)](https://www.heise.de/select/ix/2019/4/)\n[![ix-2019-09](screenshots/ix-2019-09.png)](https://www.heise.de/select/ix/2019/9/)\n\n## Table of Contents  \n* [TDD for Infrastructure code with Molecule!](#tdd-for-infrastructure-code-with-molecule)\n* [Prerequisites](#prerequisites)\n* [Project structure](#project-structure)\n* [Molecule configuration](#molecule-configuration)\n* [Execute Molecule](#execute-molecule)\n* [Multi-Scenario Molecule setup](#multi-scenario-molecule-setup)\n* [Ubuntu based Docker-in-Docker builds](#ubuntu-based-docker-in-docker-builds)\n  * [Docker-in-Docker with ubuntu:bionic](#docker-in-docker-with-ubuntubionic)\n  * [Testing the Docker-in-Docker installation](#testing-the-docker-in-docker-installation)\n* [Molecule with AWS EC2 as the infrastructure provider](#molecule-with-aws-ec2-as-the-infrastructure-provider)\n  * [Configure Molecule to use EC2](#configure-molecule-to-use-ec2)\n  * [Having a look into the create.yml](#having-a-look-into-the-createyml)\n  * [Install \u0026 configure AWS CLI](#install--configure-aws-cli)\n  * [Configure Region \u0026 VPC subnet id](#configure-region--vpc-subnet-id)\n  * [Choosing an Ubuntu 18.04 AMI](#choosing-an-ubuntu-1804-ami)\n  * [Creating a EC2 instance with Molecule](#creating-a-ec2-instance-with-molecule)\n  * [Run a first Test on EC2 with Molecule](#run-a-first-test-on-ec2-with-molecule)\n  * [Cleaning up](#cleaning-up)\n  * [Final check: molecule test](#final-check-molecule-test)\n* [Use TravisCI to execute Molecule with EC2 infrastructure](#use-travisci-to-execute-molecule-with-ec2-infrastructure)\n  * [Problems with boto on Travis](#problems-with-boto-on-travis)\n  * [Use pipenv with TravisCI](#use-pipenv-with-travisci-1)\n* [Use CircleCI to execute Molecule with EC2 infrastructure](#use-circleci-to-execute-molecule-with-ec2-infrastructure)\n  * [Use pipenv with CircleCI](#use-pipenv-with-circleci)\n  * [Schedule regular CircleCI builds with workflow triggers \u0026 cron](#schedule-regular-circleci-builds-with-workflow-triggers--cron)\n* [Upgrade to Molecule v3](#upgrade-to-molecule-v3)\n* [Use Vagrant on TravisCI to execute Molecule](#use-vagrant-on-travisci-to-execute-molecule)\n  * [Using VirtualBox locally furthermore - but switching to libvirt on TravisCI](#using-virtualbox-locally-furthermore---but-switching-to-libvirt-on-travisci)\n  * [Install Python 3 for sudo access \u0026 pipenv based on Python 3](#install-python-3-for-sudo-access--pipenv-based-on-python-3)\n  * [Run our Molecule Vagrant Scenario](#run-our-molecule-vagrant-scenario)\n\n## TDD for Infrastructure code with Molecule!\n\n[Molecule](https://molecule.readthedocs.io/en/latest/#) seems to be a pretty neat TDD framework for testing Infrastructur-as-Code using Ansible. As previously announced on September 26 2018 [Ansible treats Molecule as a first class citizen](https://groups.google.com/forum/#!topic/ansible-project/ehrb6AEptzA) from now on - backed by Redhat also.\n\nMolecule executes the following steps:\n\n![tdd-for-iac](https://yuml.me/diagram/scruffy/class/[Given:%20Spin%20up%20Infrastructure%20with%20Docker%20or%20Vagrant%20or%20Other]-\u0026gt;[When:%20Execute%20Ansible%20Playbooks/Roles],[When:%20Execute%20Ansible%20Playbooks/Roles]-\u0026gt;[Then:%20Assert%20with%20Testinfra],[Then:%20Assert%20with%20Testinfra]-\u0026gt;[Cleanup%20Infrastructure])\n\nIn the `Then` phase Molecule executes different [Verifiers](https://molecule.readthedocs.io/en/latest/configuration.html#verifier), one of them is [Testinfra](https://testinfra.readthedocs.io/en/latest/), where you can write Unittest with a Python DSL. \n\n\u003e Beware of old links to v1 of Molecule! As I followed blog posts that were just from a year or two ago, I walked into the trap of using Molecule v1. This is especially problematic if you try to install Molecule via `homebrew` where you currently also get the v1 version. So always double check, if there´s no `/v1/` in the docs´ url:\n\n![current-docs-url](screenshots/current-docs-url.png)\n\nJust start here: [Molecule docs](https://molecule.readthedocs.io/en/latest/configuration.html)\n\n\n## Prerequisites\n\n* `brew cask install virtualbox`\n* `brew cask install vagrant`\n\n\u003e Please don´t install Ansible and Molecule with homebrew on Mac, but always with pip3 since you only get old versions and need to manually install testinfra, ansible, flake8 and other packages\n\nAnd as we learned with [pulumi-python-aws-ansible](https://github.com/jonashackt/pulumi-python-aws-ansible) and escpecially in [Replace direct usage of virtualenv and pip with pipenv](https://github.com/jonashackt/pulumi-python-aws-ansible#replace-direct-usage-of-virtualenv-and-pip-with-pipenv), we should use a great Python dependency management tool like [pipenv](https://github.com/pypa/pipenv) instead of no or non-pinned (`requirements.txt`) dependencies.\n\nTherefore let's install `pipenv`:\n\n```\npip3 install pipenv\n``` \n\nThen create a virtual environment for the current project (pinned to Python 3.9+):\n\n```\npipenv shell --python 3.9\n```\n\nAnd finally install needed pip packages with:\n\n```\npipenv install\n```\n\nThis will install all needed dependencies which are listed incl. their versions in the [Pipfile](Pipfile).\n\nThis project uses the following molecule modules:\n\n* [molecule-docker](https://github.com/ansible-community/molecule-docker) (needs `molecule-docker` and `docker` pip packages)\n* [molecule-ec2](https://github.com/ansible-community/molecule-ec2) (needs `molecule-ec2` and `awscli` and `boto3` pip packages)\n* [molecule-vagrant](https://github.com/ansible-community/molecule-vagrant) for Vagrant support (needs `molecule-vagrant` and `python-vagrant` pip packages)\n\n\n\n## Project structure\n\nTo initialize a new Molecule powered Ansible role named `docker` with the Vagrant driver and the Testinfra verifier you have to execute the following command:\n\n`pipenv run molecule init role docker --verifier-name testinfra --driver-name docker`\n\nThis will give:\n\n`--\u003e Initializing role docker...\n Successfully initialized new role in /Users/jonashecht/dev/molecule-ansible-vagrant/docker.`\n\nMolecule introduces a well known project structure (at least for a Java developer like me):\n\n![projectstructure](screenshots/projectstructure.png)\n\nAs you may notice the role standard directory `tasks` is now accompanied by a `tests` directory inside the `molecule` folder where the Testinfra testcases reside.\n\nThis repository uses [a Ansible role](tasks/main.yml) that installs Docker into an Ubuntu Box:\n\n```yaml\n- name: add Docker apt key\n  apt_key:\n    url: https://download.docker.com/linux/ubuntu/gpg\n    id: 9DC858229FC7DD38854AE2D88D81803C0EBFCD88\n    state: present\n  ignore_errors: true\n\n- name: add docker apt repo\n  apt_repository:\n    repo: \"deb [arch=amd64] https://download.docker.com/linux/ubuntu {{ ansible_lsb.codename }} stable\"\n    update_cache: yes\n  become: true\n\n- name: install Docker apt package\n  apt:\n    pkg: docker-ce\n    state: latest\n    update_cache: yes\n  become: true\n\n- name: add vagrant user to docker group.\n  user:\n    name: vagrant\n    groups: docker\n    append: yes\n  become: true\n```\n\nWith Testinfra we can assert on things we want to achieve with our Ansible role: Install the `docker` package and add the user `vagrant` to the group `docker`. Testinfra uses [pytest](https://docs.pytest.org/en/latest/example/index.html) to execute the tests. Our testcases could be found in [test_docker.py](molecule/tests/test_docker.py):\n\n```python\nimport testinfra.utils.ansible_runner\n\ntestinfra_hosts = testinfra.utils.ansible_runner.AnsibleRunner(\n    '.molecule/ansible_inventory').get_hosts('all')\n\n\ndef test_is_docker_installed(host):\n    package_docker = host.package('docker-ce')\n\n    assert package_docker.is_installed\n\n\ndef test_vagrant_user_is_part_of_group_docker(host):\n    user_vagrant = host.user('vagrant')\n\n    assert 'docker' in user_vagrant.groups\n    \n```\n\n__More Testinfra code examples:__\n\nAs you´re not an in-depth Python hacker (like me), you´ll be maybe also interested in example code. Have a look at:\n\nhttps://github.com/philpep/testinfra#quick-start\n\nhttps://github.com/openmicroscopy/ansible-role-prometheus/blob/0.2.0/tests/test_default.py\n\nhttps://github.com/mongrelion/ansible-role-docker/blob/master/molecule/default/tests/test_default.py\n\nhttps://blog.octo.com/test-your-infrastructure-topology-on-aws/\n\n\n## Molecule configuration\n\nThe [molecule.yml](molecule/vagrant-ubuntu/molecule.yml) configures Molecule:\n\n```\nscenario:\n  name: vagrant-ubuntu\n\ndriver:\n  name: vagrant\n  provider:\n    name: virtualbox\nplatforms:\n  - name: vagrant-ubuntu\n    box: ubuntu/bionic64\n    memory: 512\n    cpus: 1\n\nprovisioner:\n  name: ansible\n  lint:\n    name: ansible-lint\n    enabled: false\n  playbooks:\n    converge: playbook.yml\n\nlint:\n  name: yamllint\n  enabled: false\nverifier:\n  name: testinfra\n  directory: ../tests/\n  env:\n    # get rid of the DeprecationWarning messages of third-party libs,\n    # see https://docs.pytest.org/en/latest/warnings.html#deprecationwarning-and-pendingdeprecationwarning\n    PYTHONWARNINGS: \"ignore:.*U.*mode is deprecated:DeprecationWarning\"\n  lint:\n    name: flake8\n  options:\n    # show which tests where executed in test output\n    v: 1\n\n```\n\n\nThere's a special `PYTHONWARNINGS: \"ignore:.*U.*mode is deprecated:DeprecationWarning\"` environment variable definition. If you don´t configure this you´ll end up with bloated test logs like this:\n\n![verify-with-deprecation-warnings](screenshots/verify-with-deprecation-warnings.png)\n\nIf you use the `PYTHONWARNINGS` environment variable you gather beautiful and __green__ test executions:\n\n![verify-with-deprecation-warnings-ignored](screenshots/verify-with-deprecation-warnings-ignored.png)\n\n\nIn case everything runs green you may notice that there´s is no hint which tests were executed. But I think that´s rather a pity since we want to see our whole test suite executed. That was the whole point why we even started to use a testing framework like Molecule!\n\nBut luckily there´s a way to get those tests shown inside our output. As Molecule uses [Testinfra](https://testinfra.readthedocs.io/en/latest/invocation.html) which itself leverages [pytest](https://docs.pytest.org/en/latest/) to execute our test cases and Testinfra is able to invoke pytest with additional properties. And pytest has [many options we can experiment with](https://docs.pytest.org/en/latest/reference.html#configuration-options). To configure a more verbose output for our tests in Molecule, add the following to the `verifier` section of your `molecule.yml`:\n\n```\nverifier:\n  name: testinfra\n...\n  options:\n    # show which tests where executed in test output\n    v: 1\n...\n```\n\nIf we now execute a `molecule verify` we should see a much nicer overview of which test cases where executed:\n\n![verify-with-pytest-verbose](screenshots/verify-with-pytest-verbose.png)\n\n\n## Execute Molecule\n \nNow we´re able to run our first test. Go into `docker` directory and run:\n\n`molecule test`\n\nAs Molecule has different phases, you can also explicitely run `molecule converge` or `molecule verify` - the commands will recognice required upstream phases like `create` and skips them if they where already run before (e.g. if the Vagrant Box is running already).\n\n\n\n# Multi-Scenario Molecule setup\n\nWith Molecule we could not only test our Ansible roles against one infrastructure setup - but we can use multiple of them! We only need to leverage the power of [Molecule scenarios](https://molecule.readthedocs.io/en/latest/configuration.html#scenario).\n\nTo get an idea on how this works I sligthly restructured the repository. We started out with the Vagrant driver / scenario. Now after also implementing a Docker driver in this repository on it´s own: https://github.com/jonashackt/molecule-ansible-docker I integrated it into this repository.\n\nAnd because Docker is the default Molecule driver for testing Ansible roles I changed the name of the Vagrant scenario to `vagrant-ubuntu`. Don´t forget to install `docker`:\n\n```\npip3 install molecule-docker docker\n```\n\nUsing `molecule test` as we´re used to will now execute the Docker (e.g. `default`) scenario. This change results in the following project structure:\n\n![multi-scenario-projectstructure](screenshots/multi-scenario-projectstructure.png)\n\nTo execute the `vagrant-ubuntu` Scenario you have to explicitely call it´s name:\n\n```\nmolecule test --scenario-name vagrant-ubuntu\n```\n\n[![asciicast](https://asciinema.org/a/213352.svg)](https://asciinema.org/a/213352)\n\nAll the files which belong only to a certain scenario are placed inside the scenarios directory. For example in the Docker scenario there's only the `molecule.yml` and in Vagrant one this is an additional `prepare.yml`. Also the `molecule.yml` files have to access the `playbook.yml` and the testfiles differently since they are now separate from the scenario directory to be [able to reuse them over all scenarios](https://molecule.readthedocs.io/en/latest/examples.html#sharing-across-scenarios):\n\n```\n...\nprovisioner:\n  name: ansible\n...\n  playbooks:\n    converge: ../playbook.yml\n...\nverifier:\n  name: testinfra\n  directory: ../tests/\n...\n```\n\nNow we can also integrate \u0026 use TravisCI in this repository since the [default scenario Docker is supported on Travis](https://molecule.readthedocs.io/en/latest/testing.html#travis-ci)! :)\n\n![travisci-molecule-executing-testinfra-tests](screenshots/travisci-molecule-executing-testinfra-tests.png)\n\n\n\n## Ubuntu based Docker-in-Docker builds\n\nAs we don´t have a Vagrant environment in a Cloud CI system available for us right now (see https://github.com/jonashackt/vagrant-ansible-on-appveyor), we should be enable us somehow to test our Ansible role only with the Docker driver.\n\nThe standard Docker-in-Docker image provided by Docker Inc is based on Alpine Linux (see https://stackoverflow.com/a/53459483/4964553 \u0026 the `dind` tags in https://hub.docker.com/_/docker/). But our role is designed for Ubuntu and thus uses the `apt` package manager instead of `apk`. So we can´t use the standard Docker-in-Docker image.\n\nBut there should be a way to do Docker-in-Docker installation with a Ubuntu base image like `ubuntu:bionic`! And there is :) \n\nLet´s assume the [standard Ubuntu Docker installation](https://docs.docker.com/install/linux/docker-ce/ubuntu/#install-docker-ce-1) our starting point. Everything described there is done inside our Ansible role under test __docker__ inside the playbook [docker/tasks/main.yml](tasks/main.yml).\n\nAll the extra steps needed to install Docker inside a Ubuntu Docker container will be handled inside the `prepare` step. Therefore we´ll use [Molecules´ prepare.yml playbook](https://molecule.readthedocs.io/en/latest/configuration.html#id12):\n\n\u003e The prepare playbook executes actions which bring the system to a given state prior to converge. It is executed after create, and only once for the duration of the instances life. This can be used to bring instances into a particular state, prior to testing.\n\n\n### Configure a custom prepare step\n \nThe Docker-in-Docker build is only used ([and should only be used](https://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/)) inside our CI pipeline. The `prepare` step inside our Molecule test suites´s `default` Docker scenario will be only executed for testing purposes.\n \nSo let´s configure our `default` Docker scenario to use a `prepare.yml` which could be done inside the `molecule.yml`:\n\n```yaml\n...\n  playbooks:\n    prepare: prepare-docker-in-docker.yml\n    converge: ../playbook.yml\n...\n```\n\n\n### Docker-in-Docker with ubuntu:bionic\n\nNow we should have a look into the [prepare-docker-in-docker.yml](molecule/prepare-docker-in-docker.yml):\n\n```yaml\n# Prepare things only necessary in Ubuntu Docker-in-Docker scenario\n- name: Prepare\n  hosts: all\n  tasks:\n    # We need to anticipate the installation of Docker before the role execution...\n  - name: use our role to install Docker\n    include_tasks: ../../tasks/main.yml\n\n  - name: create /etc/docker\n    file:\n      state: directory\n      path: /etc/docker\n\n  - name: set storage-driver to vfs via daemon.json\n    copy:\n      content: |\n        {\n          \"storage-driver\": \"vfs\"\n        }\n      dest: /etc/docker/daemon.json\n\n  # ...since we need to start Docker in a complete different way\n  - name: start Docker daemon inside container see https://stackoverflow.com/a/43088716/4964553\n    shell: \"/usr/bin/dockerd -H unix:///var/run/docker.sock \u003e dockerd.log 2\u003e\u00261 \u0026\"\n```\n\nFirst the Docker installation has to be executed just in the same way as on a virtual machine using Vagrant. So we simply re-use the existing Docker role here - so we´re not forced to copy code!\n\nThen really being able to run Docker-in-Docker we need to do three things:\n\n1. Run Docker with `--priviledged` (which should really only be used inside our CI environment, because it grant´s full access to the host environment (see https://hub.docker.com/_/docker/))\n2. Use the [storage-driver `vfs`](https://docs.docker.com/storage/storagedriver/vfs-driver/#configure-docker-with-the-vfs-storage-driver), which is slow \u0026 inefficient but is the only one guaranteed to work regardless of underlying filesystems\n3. Start the Docker daemon with `/usr/bin/dockerd -H unix:///var/run/docker.sock \u003e dockerd.log 2\u003e\u00261 \u0026`, or otherwise you´ll run into errors like `Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?` (see https://stackoverflow.com/a/43088716/4964553)\n\nYou may noticed that __2.__ \u0026 __3.__ are handled by our `prepare-docker-in-docker.yml` already. To enable the __1.__ `--priviledged` mode we need to configure Molecules´ Docker driver inside the `molecule.yml`:\n\n```yaml\n...\ndriver:\n  name: docker\nplatforms:\n  - name: docker-ubuntu\n    image: ubuntu:bionic\n    privileged: true\n...\n```\n\n### Testing the Docker-in-Docker installation\n\nThe last step - or the first, if you leverage the power of Test-Driven-Development (TDD) - was to create a suitable testcase. This test should verify whether the Docker-in-Docker installation was successful.\n\nTherefore we can use the [hello-world](https://hub.docker.com/_/hello-world/) Docker image. Let´s execute a `docker run hello-world` straight inside our test case `test_run_hello_world_container_successfully` in our test suite [test_docker.py](molecule/tests/test_docker.py):\n\n```\ndef test_run_hello_world_container_successfully(host):\n    hello_world_ran = host.run(\"docker run hello-world\")\n\n    assert 'Hello from Docker!' in hello_world_ran.stdout\n```\n\nThis will verify that\n\n1. the Docker client is able to contact the Docker daemon\n2. the Docker daemon successfully pulled the image `hello-world` from the Docker Hub\n3. the Docker daemon created a new container from that image and runs the executable inside\n4. the Docker daemon streamed the executables output containing `Hello from Docker!` to the Docker client, which send it to the terminal\n\n\n# Molecule with AWS EC2 as the infrastructure provider\n\nNow that we successfully used Vagrant \u0026 Docker as infrastructure providers for Molecule, we should now start to use a cloud provider like AWS. \n\nWe should be able to use [Molecule's EC2 driver](https://github.com/ansible-community/molecule-ec2), which itself uses [Ansible's ec2 module](http://docs.ansible.com/ansible/latest/ec2_module.html) to interact with AWS.\n\nFirst we need to install the [Boto Python packages](https://pypi.org/project/boto3/). They will provide interfaces to Amazon Web Services:\n\n```\npip3 install molecule-ec2 awscli boto3\n```\n\n\n### Configure Molecule to use EC2\n\nThen we just init a new Molecule scenario inside our existing multi scenario project called `aws-ec2-ubuntu`. Therefore we can leverage Molecule's `molecule init scenario` command:\n\n```\ncd molecule-ansible-docker-aws\n\nmolecule init scenario --driver-name ec2 --role-name docker --scenario-name aws-ec2-ubuntu\n```\n\nThat should create a new directory `aws-ec2-ubuntu` inside the `molecule` folder.  We'll integrate the results into our multi scenario project in a second.\n\nNow let's dig into the generated `molecule.yml`:\n\n```yaml\nscenario:\n  name: aws-ec2-ubuntu\n\ndriver:\n  name: ec2\nplatforms:\n  - name: instance\n    image: ami-a5b196c0\n    instance_type: t2.micro\n    vpc_subnet_id: subnet-6456fd1f\n\nprovisioner:\n  name: ansible\n  lint:\n    name: ansible-lint\n    enabled: false\n  playbooks:\n    converge: ../playbook.yml\n\nlint:\n  name: yamllint\n  enabled: false\n\nverifier:\n  name: testinfra\n  directory: ../tests/\n  env:\n    # get rid of the DeprecationWarning messages of third-party libs,\n    # see https://docs.pytest.org/en/latest/warnings.html#deprecationwarning-and-pendingdeprecationwarning\n    PYTHONWARNINGS: \"ignore:.*U.*mode is deprecated:DeprecationWarning\"\n  lint:\n    name: flake8\n  options:\n    # show which tests where executed in test output\n    v: 1\n```\n\nAs we already tuned the `molecule.yml` files for our other scenarios like `docker` and `vagrant-ubuntu`, we know what to change here. `provisioner.playbook.converge` needs to be configured, so the one `playbook.yml` could be found.\n\nAlso the `verifier` section has to be enhanced to gain all the described advantages like supressed deprecation warnings and the better test result overview.\n\nAs you may noticed, the driver now uses `ec2` and the platform is already pre-configured with a concrete Amazon Machine Image (AMI) and `instance_type` etc. I just tuned the instance name to `aws-ec2-ubuntu`, like we did in our Docker and Vagrant scenarios.\n\n\n### Having a look into the create.yml\n\nThe generated `create.yml` and `destroy.yml` Ansible playbooks look rather stunning - especially to AWS newbees. Let's have a look into the `create.yml`:\n\n```yaml\n- name: Create\n  hosts: localhost\n  connection: local\n  gather_facts: false\n  no_log: \"{{ not (lookup('env', 'MOLECULE_DEBUG') | bool or molecule_yml.provisioner.log|default(false) | bool) }}\"\n  vars:\n    ssh_user: ubuntu\n    ssh_port: 22\n\n    security_group_name: molecule\n    security_group_description: Security group for testing Molecule\n    security_group_rules:\n      - proto: tcp\n        from_port: \"{{ ssh_port }}\"\n        to_port: \"{{ ssh_port }}\"\n        cidr_ip: '0.0.0.0/0'\n      - proto: icmp\n        from_port: 8\n        to_port: -1\n        cidr_ip: '0.0.0.0/0'\n    security_group_rules_egress:\n      - proto: -1\n        from_port: 0\n        to_port: 0\n        cidr_ip: '0.0.0.0/0'\n\n    keypair_name: molecule_key\n    keypair_path: \"{{ lookup('env', 'MOLECULE_EPHEMERAL_DIRECTORY') }}/ssh_key\"\n  tasks:\n    - name: Create security group\n      ec2_group:\n        name: \"{{ security_group_name }}\"\n        description: \"{{ security_group_name }}\"\n        rules: \"{{ security_group_rules }}\"\n        rules_egress: \"{{ security_group_rules_egress }}\"\n\n    - name: Test for presence of local keypair\n      stat:\n        path: \"{{ keypair_path }}\"\n      register: keypair_local\n\n    - name: Delete remote keypair\n      ec2_key:\n        name: \"{{ keypair_name }}\"\n        state: absent\n      when: not keypair_local.stat.exists\n\n    - name: Create keypair\n      ec2_key:\n        name: \"{{ keypair_name }}\"\n      register: keypair\n\n    - name: Persist the keypair\n      copy:\n        dest: \"{{ keypair_path }}\"\n        content: \"{{ keypair.key.private_key }}\"\n        mode: 0600\n      when: keypair.changed\n\n    - name: Create molecule instance(s)\n      ec2:\n        key_name: \"{{ keypair_name }}\"\n        image: \"{{ item.image }}\"\n        instance_type: \"{{ item.instance_type }}\"\n        vpc_subnet_id: \"{{ item.vpc_subnet_id }}\"\n        group: \"{{ security_group_name }}\"\n        instance_tags:\n          instance: \"{{ item.name }}\"\n        wait: true\n        assign_public_ip: true\n        exact_count: 1\n        count_tag:\n          instance: \"{{ item.name }}\"\n      register: server\n      with_items: \"{{ molecule_yml.platforms }}\"\n      async: 7200\n      poll: 0\n\n    - name: Wait for instance(s) creation to complete\n      async_status:\n        jid: \"{{ item.ansible_job_id }}\"\n      register: ec2_jobs\n      until: ec2_jobs.finished\n      retries: 300\n      with_items: \"{{ server.results }}\"\n\n    # Mandatory configuration for Molecule to function.\n\n    - name: Populate instance config dict\n      set_fact:\n        instance_conf_dict: {\n          'instance': \"{{ item.instances[0].tags.instance }}\",\n          'address': \"{{ item.instances[0].public_ip }}\",\n          'user': \"{{ ssh_user }}\",\n          'port': \"{{ ssh_port }}\",\n          'identity_file': \"{{ keypair_path }}\",\n          'instance_ids': \"{{ item.instance_ids }}\", }\n      with_items: \"{{ ec2_jobs.results }}\"\n      register: instance_config_dict\n      when: server.changed | bool\n\n    - name: Convert instance config dict to a list\n      set_fact:\n        instance_conf: \"{{ instance_config_dict.results | map(attribute='ansible_facts.instance_conf_dict') | list }}\"\n      when: server.changed | bool\n\n    - name: Dump instance config\n      copy:\n        content: \"{{ instance_conf | to_json | from_json | molecule_to_yaml | molecule_header }}\"\n        dest: \"{{ molecule_instance_config }}\"\n      when: server.changed | bool\n\n    - name: Wait for SSH\n      wait_for:\n        port: \"{{ ssh_port }}\"\n        host: \"{{ item.address }}\"\n        search_regex: SSH\n        delay: 10\n        timeout: 320\n      with_items: \"{{ lookup('file', molecule_instance_config) | molecule_from_yaml }}\"\n\n    - name: Wait for boot process to finish\n      pause:\n        minutes: 2\n\n``` \n\nIf we skim over the code, we notice the usage of Ansible's [ec2_group module](https://docs.ansible.com/ansible/latest/modules/ec2_group_module.html). It is used to maintains AWS EC2 security groups. Using the parameters `rules` and `rules_egress`, it configures appropriate firewall inbound and outbound rules.\n\nThereafter Ansible's [ec2_key module](https://docs.ansible.com/ansible/latest/modules/ec2_key_module.html) is used to create a new EC2 key pair \u0026 store it locally, if non exists on your local machine already.\n\nAnd then the [ec2 module](https://docs.ansible.com/ansible/latest/modules/ec2_module.html) takes over, which is able to create or terminate AWS EC2 instances without the need to hassle with the GUI.\n\nWaiting for the EC2 instance to come up, the `create.yml` playbook uses the [async_status module](https://docs.ansible.com/ansible/latest/modules/async_status_module.html) on the pre-registered variable `server.results` from the `ec2` module.\n\nThe following Ansible module are solely used to create an inline Ansible inventory, which is finally used to connect into the EC2 instance via SSH.\n\nThe generated `destroy.yml` playbook is just the opposite to the `create.yml` playbook and tears the created EC2 instance down.\n\n\n### Install \u0026 configure AWS CLI\n\nNow let's give our configuration a shot. Just be sure to meet some prerequisites. \n\nFirst we need to sure to have the [AWS CLI installed](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html). We can also do this via pip package manager with:\n \n```\npipenv install awscli\n```\n \nNow we should check, if AWS CLI was successfully installed. The `aws --version` command should print out sometime like:\n\n```\n$ aws --version\naws-cli/1.16.80 Python/3.7.2 Darwin/18.2.0 botocore/1.12.70\n```\n\nNow configure the AWS CLI to use the correct credentials. [According to the AWS docs](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html#cli-quick-configuration), the fastest way to accomplish that is to run `aws configure`:\n\n```\n$ aws configure\nAWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE\nAWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\nDefault region name [None]: eu-central-1\nDefault output format [None]: json\n```\n\n### Configure Region \u0026 VPC subnet id \n\nAs there currently is a discrepancy in the UX of Molecule, where every configuration parameter is generated correctly by a `molecule init --driver-name ec2` command and picked up from `~/.aws/credentials`. __Except__ the `region` configuration from `~/.aws/config`!\n\nRunning a Molecule test without setting the region correctly as environment variable, currently results [in the (already documented) error](https://github.com/ansible/molecule/issues/1570):\n\n```\n\"msg\": \"Either region or ec2_url must be specified\",\n```\n\nSo for now we need to set the region manually before running our Molecule tests against EC2:\n\n```\nexport AWS_REGION=eu-central-1\n```\n\nAnd there's another thing to do: We need to configure the correct `vpc_subnet_id` inside our `molecule.yml` - if not, we get an error like this:\n\n```\n    \"    raise self.ResponseError(response.status, response.reason, body)\",\n    \"boto.exception.EC2ResponseError: EC2ResponseError: 400 Bad Request\",\n    \"\u003c?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?\u003e\",\n    \"\u003cResponse\u003e\u003cErrors\u003e\u003cError\u003e\u003cCode\u003eInvalidSubnetID.NotFound\u003c/Code\u003e\u003cMessage\u003eThe subnet ID 'subnet-6456fd1f' does not exist\u003c/Message\u003e\u003c/Error\u003e\u003c/Errors\u003e\u003cRequestID\u003e1c971ba2-0d86-4335-9445-d989e988afce\u003c/RequestID\u003e\u003c/Response\u003e\"\n]\n```\n\nWe need to somehow figure out the correct VPC subnet id of our region. Therefore, you can simply use Ansible to gather that for you by using the [ec2_vpc_subnet_facts module](https://docs.ansible.com/ansible/latest/modules/ec2_vpc_subnet_facts_module.html):\n\n```\n    - name: Gather facts about all VPC subnets\n      ec2_vpc_subnet_facts:\n```\n\nThis will give the correct String inside the field `subnets.id`:\n\n```\n    ok: [localhost] =\u003e {\n        \"changed\": false,\n        \"invocation\": {\n            \"module_args\": {\n                \"aws_access_key\": null,\n                ...\n            }\n        },\n        \"subnets\": [\n            {\n                \"assign_ipv6_address_on_creation\": false,\n                \"availability_zone\": \"eu-central-1b\",\n                \"availability_zone_id\": \"euc1-az3\",\n                ...\n                \"id\": \"subnet-a2efa1d9\",\n                ...\n            },\n\n```\n\nAs an alternative you can use AWS CLI with `aws ec2 describe-subnets` to find the correct ID inside the field `Subnets.SubnetId`:\n\n```\n$ aws ec2 describe-subnets\n\n{\n    \"Subnets\": [\n        {\n            \"AvailabilityZone\": \"eu-central-1b\",\n            \"AvailabilityZoneId\": \"euc1-az3\",\n            ...\n            \"SubnetId\": \"subnet-a2efa1d9\",\n            ...\n        },\n        {\n            \"AvailabilityZone\": \"eu-central-1c\",\n            ...\n        },\n        {\n            \"AvailabilityZone\": \"eu-central-1a\",\n            ...\n        }\n    ]\n}\n```\n\nNow head over to your [molecule.yml](molecule/aws-ec2-ubuntu/molecule.yml) and edit the `vpc_subnet_id` to contain the correct value:\n\n```\ndriver:\n  name: ec2\n  \nplatforms:\n  - name: aws-ec2-ubuntu\n    image_owner: 099720109477\n    image_name: ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-*\n    instance_type: t2.micro\n    vpc_subnet_id: subnet-a2efa1d9\n    instance_tags:\n      - Name: molecule-aws-ec2-ubuntu\n...\n```\n\n\n### Choosing an Ubuntu 18.04 AMI\n\n[As stated here[(https://askubuntu.com/a/1031156/451114), Amazon has a \"official\" Ubuntu 18.04 AMI - but this one isn't available under the free tier right now. \n\nBut the official image is just the same as from Canonical, which could be found with the [ubuntu Amazon EC2 AMI Locator](https://cloud-images.ubuntu.com/locator/ec2/):\n\nFill in your region - e.g. for me this is `eu-central-1` - together with the desired Ubuntu version - like `18.04 LTS` - and the site should filter all available AMI images in this region:\n\n![ubuntu-Amazon-EC2-AMI-Locator](screenshots/ubuntu-Amazon-EC2-AMI-Locator.png)\n\nNow choose the AMI id with the __Instance Type__ `hvm:ebs-ssd`, which means that our EC2 instance will use Amazon Elastic Block Storage memory. Only this instance type is eligible for the free tier. In our region the the correct AMI id for Ubuntu 18.04 is `ami-080d06f90eb293a27`. We need to configure this inside our [molecule.yml](molecule/aws-ec2-ubuntu/molecule.yml):\n\n```\nscenario:\n  name: aws-ec2-ubuntu\n\ndriver:\n  name: ec2\nplatforms:\n  - name: aws-ec2-ubuntu\n    image: ami-080d06f90eb293a27\n    instance_type: t2.micro\n    vpc_subnet_id: subnet-a2efa1d9\n...\n```\n\n\n### Creating a EC2 instance with Molecule\n\n\nNow we should have everything prepared. Let's try to run our first Molecule test on AWS EC2 (including `--debug` so that we see what's going on):\n\n```\nmolecule --debug create --scenario-name aws-ec2-ubuntu\n```\n\n\u003e Right now we have an issue with the new community module\n\n```shell\nTASK [Create molecule instance(s)] *********************************************\nchanged: [localhost] =\u003e (item={'image_name': 'ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-*', 'image_owner': '099720109477', 'instance_tags': [{'Name': 'molecule-aws-ec2-ubuntu'}], 'instance_type': 't2.micro', 'name': 'aws-ec2-ubuntu', 'vpc_subnet_id': 'subnet-a2efa1d9'})\n\nTASK [Wait for instance(s) creation to complete] *******************************\nFAILED - RETRYING: Wait for instance(s) creation to complete (300 retries left).\nok: [localhost] =\u003e (item={'started': 1, 'finished': 0, 'ansible_job_id': '397490056884.47351', 'results_file': '/Users/jonashecht/.ansible_async/397490056884.47351', 'changed': True, 'failed': False, 'item': {'image_name': 'ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-*', 'image_owner': '099720109477', 'instance_tags': [{'Name': 'molecule-aws-ec2-ubuntu'}], 'instance_type': 't2.micro', 'name': 'aws-ec2-ubuntu', 'vpc_subnet_id': 'subnet-a2efa1d9'}, 'ansible_loop_var': 'item', 'index': 0, 'ansible_index_var': 'index'})\n\nTASK [Populate instance config dict] *******************************************\nfatal: [localhost]: FAILED! =\u003e {\"msg\": \"The task includes an option with an undefined variable. The error was: list object has no element 0\\n\\nThe error appears to be in '/Users/jonashecht/dev/molecule-ansible-docker-aws/molecule/aws-ec2-ubuntu/create.yml': line 112, column 7, but may\\nbe elsewhere in the file depending on the exact syntax problem.\\n\\nThe offending line appears to be:\\n\\n\\n    - name: Populate instance config dict\\n      ^ here\\n\"}\n```\n\nSee https://github.com/ansible-community/molecule/discussions/2946, https://github.com/ansible-community/molecule/pull/2869/files, https://github.com/ansible-community/molecule-ec2/pull/34 (only merged 3 days ago)\n\n\n\n\nWe could have a sneak peak into our Ansible EC2 Management console under https://eu-central-1.console.aws.amazon.com/ec2/v2/home?region=eu-central-1. We should see our EC2 instance starting up:\n\n![aws-ec2-management-console-instance-startup](screenshots/aws-ec2-management-console-instance-startup.png)\n\nIf everything went fine, the `molecule create` command should succeed and your EC2 console should show the running EC2 instance:\n\n![aws-ec2-management-console-instance-running](screenshots/aws-ec2-management-console-instance-running.png)\n\n\n\n### Run a first Test on EC2 with Molecule\n\nNow let's try to run our Ansible role against our new EC2 instance with Molecule:\n\n```\nmolecule converge --scenario-name aws-ec2-ubuntu\n```\n\nThat should just work fine. We then could head over to the verify-phase:\n\n```\nmolecule verify --scenario-name aws-ec2-ubuntu\n```\n\nThis should successfully execute our Testinfra test suite described in [test_docker.py](molecule/tests/test_docker.py) onto our EC2 instance. If everything ran fine, it should give something like:\n\n![aws-molecule-verify-success](screenshots/aws-molecule-verify-success.png)\n\nIf the last test function fails with an error like:\n```\ndocker: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock\n```\nyou maybe have to add `sudo ` to the `docker run hello-world` statement:\n\n```\ndef test_run_hello_world_container_successfully(host):\n    hello_world_ran = host.run(\"sudo docker run hello-world\")\n\n    assert 'Hello from Docker!' in hello_world_ran.stdout\n```\n\n\n### Cleaning up\n\nFinally it's time to clean up. Let's run a `molecule destroy` command to tear down or EC2 instance:\n\n```\nmolecule --debug destroy --scenario-name aws-ec2-ubuntu\n```\n\nIf that command went well, we can have a look into our AWS EC2 management console again:\n\n![aws-ec2-management-console-instance-terminated](screenshots/aws-ec2-management-console-instance-terminated.png)\n\nOur instance should have reached the state `terminated`, which is simmilar to `stopped` state - just for instances that used EBS storage (see the [AWS EC2 Instance Lifecycle chart](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-lifecycle.html) for more info).\n\nHere's a full asciinema cast of all the steps:\n\n[![asciicast](https://asciinema.org/a/220434.svg)](https://asciinema.org/a/220434)\n\n\n### Final check: molecule test\n\nAs you may remember, Molecule provides the consecutive list of steps that run one by one. For introductory reasons we choosed to execute each step manually.\n\nBut having everything configured and in place right now, we should be able to run the command that summarizes all the others: `molecule test`:\n\n```\nmolecule test --scenario-name aws-ec2-ubuntu\n```\n\n\n\n\n\n## Use TravisCI to execute Molecule with EC2 infrastructure\n\nMy ultimate goal of the whole Molecule journey was to be able to let TravisCI create Cloud environments and execute Ansible roles on them.\n\nSo we should bring all the things learned togehter now: \n\n\u003e Using Molecule to develop and test an Ansible role - togehter with the infrastructure provider AWS EC2 - automatically executed by TravisCI after commits or regularly with TravisCI cron jobs.\n\nSo let's do it! First we need to configure TravisCI. Therefore we need to enhance our [.travis.yml](.travis.yml). As we need the same python package additions as locally, we need to install `boto`, `boto3` and `awscli`:\n\n```yaml\nsudo: required\nlanguage: python\n\nenv:\n- EC2_REGION=eu-central-1\n\nservices:\n- docker\n\ninstall:\n- pip install molecule docker\n\n# install AWS related packages\n- pip install boto boto3\n- pip install --upgrade awscli\n# configure AWS CLI\n- aws configure set aws_access_key_id $AWS_ACCESS_KEY\n- aws configure set aws_secret_access_key $AWS_SECRET_KEY\n- aws configure set default.region $DEPLOYMENT_REGION\n# show AWS CLI config\n- aws configure list\n\nscript:\n# Molecule Testing Travis-locally with Docker\n- molecule test\n# Molecule Testing on AWS EC2\n- molecule create --scenario-name aws-ec2-ubuntu\n- molecule converge --scenario-name aws-ec2-ubuntu\n- molecule verify --scenario-name aws-ec2-ubuntu\n- molecule destroy --scenario-name aws-ec2-ubuntu\n```\n\nAfter that, we need to configure our AWS CLI to use the correct credentials and AWS region. This can be achieved by usind the `aws configure set` command. Then we need to head over to the settings tab of our TravisCI project (for the current project this can be found at https://travis-ci.org/jonashackt/molecule-ansible-docker-vagrant/settings) and insert the three environment variables `AWS_ACCESS_KEY`, `AWS_SECRET_KEY` and `DEPLOYMENT_REGION`:\n\n![travisci-aws-settings-env-variables](screenshots/travisci-aws-settings-env-variables.png)\n\nThe last part is to add the molecule commands to the `script` section of the `.travis.yml`. \n\n\n### Problems with boto on Travis\n\nAs https://github.com/boto/boto/issues/3717 suggests, there are currently problems with the AWS connection library boto on TravisCI. They result in errors like:\n\n```\n        \"msg\": \"Traceback (most recent call last):\\n  File \\\"/home/travis/.ansible/tmp/ansible-tmp-1547040777.3-69673074739502/async_wrapper.py\\\", line 147, in _run_module\\n    (filtered_outdata, json_warnings) = _filter_non_json_lines(outdata)\\n  File \\\"/home/travis/.ansible/tmp/ansible-tmp-1547040777.3-69673074739502/async_wrapper.py\\\", line 88, in _filter_non_json_lines\\n    raise ValueError('No start of json char found')\\nValueError: No start of json char found\\n\", \n        \"stderr\": \"Traceback (most recent call last):\\n  File \\\"/home/travis/.ansible/tmp/ansible-tmp-1547040777.3-69673074739502/AnsiballZ_ec2.py\\\", line 113, in \u003cmodule\u003e\\n    _ansiballz_main()\\n  File \\\"/home/travis/.ansible/tmp/ansible-tmp-1547040777.3-69673074739502/AnsiballZ_ec2.py\\\", line 105, in _ansiballz_main\\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\\n  File \\\"/home/travis/.ansible/tmp/ansible-tmp-1547040777.3-69673074739502/AnsiballZ_ec2.py\\\", line 48, in invoke_module\\n    imp.load_module('__main__', mod, module, MOD_DESC)\\n  File \\\"/tmp/ansible_ec2_payload_y3lfWH/__main__.py\\\", line 552, in \u003cmodule\u003e\\n  File \\\"/home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages/boto/__init__.py\\\", line 1216, in \u003cmodule\u003e\\n    boto.plugin.load_plugins(config)\\nAttributeError: 'module' object has no attribute 'plugin'\\n\", \n        \"stderr_lines\": [\n            \"Traceback (most recent call last):\", \n            \"  File \\\"/home/travis/.ansible/tmp/ansible-tmp-1547040777.3-69673074739502/AnsiballZ_ec2.py\\\", line 113, in \u003cmodule\u003e\", \n            \"    _ansiballz_main()\", \n            \"  File \\\"/home/travis/.ansible/tmp/ansible-tmp-1547040777.3-69673074739502/AnsiballZ_ec2.py\\\", line 105, in _ansiballz_main\", \n            \"    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\", \n            \"  File \\\"/home/travis/.ansible/tmp/ansible-tmp-1547040777.3-69673074739502/AnsiballZ_ec2.py\\\", line 48, in invoke_module\", \n            \"    imp.load_module('__main__', mod, module, MOD_DESC)\", \n            \"  File \\\"/tmp/ansible_ec2_payload_y3lfWH/__main__.py\\\", line 552, in \u003cmodule\u003e\", \n            \"  File \\\"/home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages/boto/__init__.py\\\", line 1216, in \u003cmodule\u003e\", \n            \"    boto.plugin.load_plugins(config)\", \n            \"AttributeError: 'module' object has no attribute 'plugin'\"\n        ]\n    }\n    \n    PLAY RECAP *********************************************************************\n    localhost                  : ok=6    changed=4    unreachable=0    failed=1   \n    \n    \nERROR: \nThe command \"molecule --debug create --scenario-name aws-ec2-ubuntu\" exited with 2.\n```\n\nTo fix this, there are two changes needed inside our [.travis.yml](.travis.yml). We need to set `sudo: false` and use the environment variable `BOTO_CONFIG=/dev/null`: \n\n```yaml\nsudo: false\nlanguage: python\n\nenv:\n- EC2_REGION=eu-central-1 BOTO_CONFIG=\"/dev/null\"\n...\n```\n\nAnd we need to add the `BOTO_CONFIG` environment variable to the same line as the already existing variable - otherwise Travis starts multiple builds with only one variable set to each build! See the docs (https://docs.travis-ci.com/user/environment-variables/#defining-multiple-variables-per-item)\n\n\u003e If you need to specify several environment variables for each build, put them all on the same line in the env array\n\nNow head over to Travis and have a look into the log. It should look green and somehow like this: https://travis-ci.org/jonashackt/molecule-ansible-docker-vagrant/builds/477365844\n\n### Use pipenv with TravisCI\n\nAs described in [Replace direct usage of virtualenv and pip with pipenv](https://github.com/jonashackt/pulumi-python-aws-ansible#replace-direct-usage-of-virtualenv-and-pip-with-pipenv), we use the great Python dependency management tool like [pipenv](https://github.com/pypa/pipenv) instead of no or non-pinned (`requirements.txt`) dependencies. So we should also use it with TravisCI!\n\nAnd as TravisCI has the needed `virtualenv` already installed per default in it's `python` machines, we only need to do (and replace the `pip install XYZ` commands):\n\n```yaml\n  # Install pipenv dependency manager\n  - pip install pipenv\n  # Install required (and locked) dependecies from Pipfile.lock. pipenv is smart enough to recognise the existing \n  # virtualenv without a prior pipenv shell command (see https://medium.com/@dirk.avery/quirks-of-pipenv-on-travis-ci-and-appveyor-10d6adb6c55b)\n  - pipenv install\n```\n\n\n### Use pipenv with TravisCI\n\nAs described in [Replace direct usage of virtualenv and pip with pipenv](https://github.com/jonashackt/pulumi-python-aws-ansible#replace-direct-usage-of-virtualenv-and-pip-with-pipenv), we use the great Python dependency management tool like [pipenv](https://github.com/pypa/pipenv) instead of no or non-pinned (`requirements.txt`) dependencies. So we should also use it with TravisCI!\n\nAnd as TravisCI has the needed `virtualenv` already installed per default in it's `python` machines, we only need to do (and replace the `pip install XYZ` commands):\n\n```yaml\n  # Install pipenv dependency manager\n  - pip install pipenv\n  # Install required (and locked) dependecies from Pipfile.lock. pipenv is smart enough to recognise the existing \n  # virtualenv without a prior pipenv shell command (see https://medium.com/@dirk.avery/quirks-of-pipenv-on-travis-ci-and-appveyor-10d6adb6c55b)\n  - pipenv install\n```\n\n\n\n## Use CircleCI to execute Molecule with EC2 infrastructure\n\nAs TravisCI is just one example of a cloud CI provider, let's use another one also - so let's just pick [CircleCI](https://circleci.com), let's go with this CI tool!  \n\n\u003e Using Molecule to develop and test an Ansible role - togehter with the infrastructure provider AWS EC2 - automatically executed by CircleCI after commits or regularly with CircleCI scheduled jobs.\n\n\n\nSo let's do it! First we need to configure CircleCI. Therefore we need to create a [.circleci/config.yml](.circleci/config.yml). As we need the same python package additions as locally, we need to install `boto`, `boto3` and `awscli`:\n\n```yaml\nversion: 2\njobs:\n  build:\n    docker:\n      - image: circleci/python:3.7.5\n\n    environment:\n      EC2_REGION: eu-central-1\n\n    working_directory: ~/molecule-ansible-docker-vagrant\n\n    steps:\n      - checkout\n\n      - run:\n          name: Install Molecule dependencies\n          command: |\n            pip install molecule docker\n\n      - run:\n          name: Run Molecule Testing CircleCI-locally with Docker\n          command: |\n            molecule test\n\n      - run:\n          name: Install Molecule AWS dependencies\n          command: |\n            pip install boto boto3\n            pip install --upgrade awscli\n\n      - run:\n          name: configure AWS CLI\n          command: |\n            aws configure set aws_access_key_id ${AWS_ACCESS_KEY}\n            aws configure set aws_secret_access_key ${AWS_SECRET_KEY}\n            aws configure set default.region ${EC2_REGION}\n            aws configure list\n\n      - run:\n          name: Run Molecule Testing on AWS EC2\n          command: |\n            molecule create --scenario-name aws-ec2-ubuntu\n            molecule converge --scenario-name aws-ec2-ubuntu\n            molecule verify --scenario-name aws-ec2-ubuntu\n            molecule destroy --scenario-name aws-ec2-ubuntu\n\n```\n\nAfter that, we need to configure our AWS CLI to use the correct credentials and AWS region. This can be achieved by usind the `aws configure set` command. Then we need to head over to the settings tab of our CircleCI project (for the current project this can be found at https://circleci.com/gh/jonashackt/molecule-ansible-docker-vagrant/edit#env-vars) and insert the three environment variables `AWS_ACCESS_KEY` \u0026 `AWS_SECRET_KEY`:\n\n![circleci-aws-settings-env-variables](screenshots/circleci-aws-settings-env-variables.png)\n\nThe last part is to add the molecule commands to our `.circleci/config.yml`. \n\n### CircleCI gives Permission denied error at pip install\n\nUsing `circleci/python` Docker image it seems that you can't simply use `pip install xyz`:\n\n```\nPermissionError: [Errno 13] Permission denied: '/usr/local/lib/python3.6/site-packages/ptyprocess'\n```\n\nThis error is also [reported here](https://discuss.circleci.com/t/circleci-python-docker-images-disallow-pip-install-due-to-directory-ownership/12504). To avoid this error, we need to use `sudo` to be able to install the pip packages successfully:\n\n```yaml\nversion: 2\njobs:\n  build:\n    docker:\n      - image: circleci/python:3.7.5\n...\n    steps:\n      - checkout\n\n      - run:\n          name: Install Molecule dependencies\n          command: |\n            sudo pip install molecule docker\n\n      - run:\n          name: Run Molecule Testing CircleCI-locally with Docker\n          command: |\n            molecule test\n```\n\n### \"msg\": \"Error connecting: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))\"\n\nMolecule needs a Docker service available. Since CircleCI is mostly using pure Docker-based images to run it's CI jobs, there's no Docker service inside the Docker containers available. But there's help! [The docs state](https://circleci.com/docs/2.0/building-docker-images/):\n\n\u003e If your job requires docker or docker-compose commands, add the setup_remote_docker step into your .circleci/config.yml\n\nWe need to use `- setup_remote_docker`\n\n```yaml\nversion: 2\njobs:\n  build:\n    docker:\n      - image: circleci/python:3.7.5\n...\n    steps:\n      - checkout  \n      - setup_remote_docker \n```\n\n\u003e Don't use `docker_layer_caching: true`, if you only have the free tier available (see https://github.com/jonashackt/molecule-ansible-docker-vagrant/issues/5)!!\n\nNow head over to CircleCI and have a look into the log. It should look green and somehow like this: https://circleci.com/gh/jonashackt/molecule-ansible-docker-vagrant/16 :\n\n![circleci-aws-full-run-docker-aws](screenshots/circleci-aws-full-run-docker-aws.png)\n\n\n### Use pipenv with CircleCI\n\nAs we already use `pipenv` with our local project setup and with TravisCI, CircleCI builds should also depend on this great dependency-lock supporting Python build tool.\n\nAnd [in the examples](https://circleci.com/docs/2.0/language-python/#install-dependencies) there's already the part we need instead of `pip install XYZ` commands:\n\n```yaml\nversion: 2\njobs:\n  build:\n    docker:\n      - image: circleci/python:3.7.5\n\n    environment:\n      EC2_REGION: eu-central-1\n\n    working_directory: ~/molecule-ansible-docker-vagrant\n\n    steps:\n      - checkout\n      - setup_remote_docker\n\n      - run:\n          name: Install all dependencies with pipenv (incl. python-dev installation for pip packages, that need to be build \u0026 have a Python.h present)\n          command: |\n            sudo apt-get install python-dev\n            sudo pip install pipenv\n            pipenv shell\n            pipenv install\n```\n\nThis gets us (maybe) into the following error:\n\n```\nAn error occurred while installing psutil==5.6.5 ; sys_platform != 'win32' and sys_platform != 'cygwin'\n...\n'    psutil/_psutil_common.c:9:10: fatal error: Python.h: No such file or directory', '     #include \u003cPython.h\u003e', '              ^~~~~~~~~~', '    compilation terminated.', \"    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1\", '    ----------------------------------------', 'ERROR: Command errored out with exit status 1: /home/circleci/.local/share/virtualenvs/molecule-ansible-docker-vagrant-082rgMyr/bin/python3.7 -u -c \\'import sys, setuptools, tokenize; sys.argv[0] = \\'\"\\'\"\\'/tmp/pip-install-fecrj6wj/psutil/setup.py\\'\"\\'\"\\'; __file__=\\'\"\\'\"\\'/tmp/pip-install-fecrj6wj/psutil/setup.py\\'\"\\'\"\\';f=getattr(tokenize, \\'\"\\'\"\\'open\\'\"\\'\"\\', open)(__file__);code=f.read().replace(\\'\"\\'\"\\'\\\\r\\\\n\\'\"\\'\"\\', \\'\"\\'\"\\'\\\\n\\'\"\\'\"\\');f.close();exec(compile(code, __file__, \\'\"\\'\"\\'exec\\'\"\\'\"\\'))\\' install --record /tmp/pip-record-iium4npj/install-record.txt --single-version-externally-managed --compile --install-headers /home/circleci/.local/share/virtualenvs/molecule-ansible-docker-vagrant-082rgMyr/include/site/python3.7/psutil Check the logs for full command output.']\n```\n\nThis seems to be [a knows issue](https://github.com/giampaolo/psutil/issues/1143#issuecomment-334694641), if `python-dev` package isn't available, no PIP packages could be build locally.\n\nSo let's add a `sudo apt-get install python-dev` to our [.circleci/config.yml](.circleci/config.yml).\n\n\nNow using pipenv, we can't simply activate the `pipenv` shell with a `pipenv shell` -\u003e this will stop our CircleCI builds right in the middle ([see this build](https://circleci.com/gh/jonashackt/molecule-ansible-docker-vagrant/84))\n \nThe docs don't mention that one clearly enough - __BUT__ otherwise the commands like `molecule` etc wont be able to find, [see this build](https://circleci.com/gh/jonashackt/molecule-ansible-docker-vagrant/81)!\n\nThe solution is to run every command with a prefixed `pipenv run` - like `pipenv run molecule test`. \n\n\n\n### Schedule regular CircleCI builds with workflow triggers \u0026 cron\n\nAs the docs at https://circleci.com/docs/2.0/triggers/#scheduled-builds state, we need to add the `workflows` section inside our [.circleci/config.yml](.circleci/config.yml) to be able to use cron-scheduled builds. First we introduce a `on-commit:` workflow, which will just be triggered by a normal git commit/push. Nothing new here.\n\nBut the second workflow item `weekly-schedule:` gets us where we wanted to be: using the `cron` trigger we can configure to run our Molecule tests e.g. once on friday 04:45 PM every week:\n\n```yaml\nworkflows:\n  version: 2\n  on-commit:\n    jobs:\n      - build\n  weekly-schedule:\n    triggers:\n      - schedule:\n          # 17:55 every friday (see https://en.wikipedia.org/wiki/Cron)\n          cron: \"55 17 * * 5\"\n          filters:\n            branches:\n              only:\n                - master\n    jobs:\n      - build\n```\n\nNow we should see our builds triggered by this cron regularly:\n\n![circleci-cron-weekly-schedule](screenshots/circleci-cron-weekly-schedule.png)\n\nJust mind the `UTC+2` timezone - for me, configuring `17:55` actually means, that my job will be scheduled to run at `19:55` - so don't think your config is wrong, maybe you're just in another timezone :)\n\n\n## Upgrade to Molecule v3\n\nWith Molecule 3.x our project and especially the `molecule.yml` needs some refinement. [Here's an good overview](https://github.com/ansible-community/molecule/issues/2560).\n\nEspecially the `lint` sections and the `scenario-name` has to go - the latter is derived from the directory name and is thus not doubled anymore. The linting is now configured separately ([see this commit](https://github.com/jonashackt/molecule-ansible-docker-aws/commit/74b0ad7ac011e27bacfcf2e3d5d8ada257d393cb)). \n\nAdditionally now only `Docker`, `Podman` and `Delegated` are supposed to be core-providers. All other providers are now regarded as community-supported providers. For us as Molecule users this means, we need to install separate dependencies - since these providers now also have their own GitHub repo (see https://github.com/ansible-community/molecule-vagrant for example).\n\nTo use Vagrant, we need to add a new dependency to our [Pipfile](Pipfile):\n\n```\nmolecule-vagrant = \"==0.2\"\ntestinfra = \"==4.1.0\"\n```\n\nAs we Testinfra is now also not longer installed by default, we should also add it explicitely. \n\n\n## Use Vagrant on GitHub Actions to execute Molecule\n\nWell that one was on my list for a long time - but as Travis defacto laid off the OpenSource support, I switched to GitHub Actions. \n\nSince GHA is easily able to run Vagrant (see https://github.com/jonashackt/vagrant-github-actions), we can simply use it inside our GHA workflow file [vagrant.yml](.github/workflows/vagrant.yml):\n\n```yaml\nname: vagrant\n\non: [push]\n\njobs:\n  molecule-vagrant-ubuntu:\n    runs-on: macos-10.15\n\n    steps:\n    - uses: actions/checkout@v2\n\n    - name: Cache pipenv virtualenvs incl. all pip packages\n      uses: actions/cache@v2\n      with:\n        path: ~/.local/share/virtualenvs\n        key: ${{ runner.os }}-pipenv-${{ hashFiles('**/Pipfile.lock') }}\n        restore-keys: |\n          ${{ runner.os }}-pipenv-\n\n    - uses: actions/setup-python@v2\n      with:\n        python-version: '3.9'\n\n    - name: Install required dependecies with pipenv\n      run: |\n        pip install pipenv\n        pipenv install\n\n    - name: Molecule testing GHA-locally with Vagrant Ubuntu Bionic\n      run: |\n        pipenv run molecule create --scenario-name vagrant-ubuntu\n        pipenv run molecule converge --scenario-name vagrant-ubuntu\n        pipenv run molecule verify --scenario-name vagrant-ubuntu\n        pipenv run molecule destroy --scenario-name vagrant-ubuntu\n```\n\nCaching the pipenv virtualenv we also get relatively fast cycle times here.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjonashackt%2Fmolecule-ansible-docker-aws","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjonashackt%2Fmolecule-ansible-docker-aws","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjonashackt%2Fmolecule-ansible-docker-aws/lists"}