Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/pwillis-els/terraformsh

A wrapper for Terraform in Bash
https://github.com/pwillis-els/terraformsh

Last synced: about 1 month ago
JSON representation

A wrapper for Terraform in Bash

Awesome Lists containing this project

README 

        
# Requirements

 - Bash (v3+)

 - Terraform

 - AWS CLI (only for aws_bootstrap command)



# About

  Terraformsh is a Bash script that makes it easier to run Terraform by 

  performing common steps for you. It also makes it easy to keep your

  configuration DRY and deploy infrastructure based on a directory

  hierarchy of environments. See [DRY_CODE.md](./DRY_CODE.md) for

  more details.



  Unlike Terragrunt, this script includes no DSL or code generation. All it

  does is make it easier to call Terraform. See [PHILOSOPHY.md](./PHILOSOPHY.md)

  for more details.



  Terraformsh will detect and use Terraform `-var-files` and `-backend-config`

  configuration files across a directory hierarchy. It also has its own 

  configuration file so you don't have to remember any command other than

  `terraformsh` itself (removing the need for a `Makefile`, though you can

  still use one if you want).



  You can override any options with environment variables, command-line options

  and config files. Good conventions like using *.plan* files for changes are the

  default.



## How it works



### Basic operation



  Change to the directory of a Terraform module and run `terraformsh` with any

  Terraform commands and arguments you'd normally use.



       $ cd root-modules/aws/common/

       $ terraformsh plan

       $ terraformsh apply



  Terraformsh will run dependent Terraform commands when necessary. If you run

  `terraformsh plan`, Terraformsh will first run `terraform validate`, but before

  that `terraform get`, but before that `terraform init`. Terraformsh passes

  relevant options to each command as necessary, and you can also override

  those options.



### Automatic *plan* files



  When certain commands are run (`plan`, `apply`, `plan_destroy`, `destroy`)

  Terraformsh will use the appropriate options to create a *plan file*. This way

  you can be sure that an `apply` or `destroy` operation will only happen on

  a plan that has been saved to a file and reviewed. (You can disable this

  automatic behavior by setting *USE_PLANFILE=0* as an environment or configuration

  variable)



  The plan files are, by default, written to the directory where you ran Terraformsh,

  with a naming convention like `tfsh.92h39d9hd9.plan`. You can override this by

  setting environent or configuration variable *TF_PLANFILE* and *TF_DESTROY_PLANFILE*.



### Multiple commands as arguments



  You can pass multiple Terraform commands as options and it'll run them

  in the order you specify.



  Not sure what that looks like? Use the dry-run mode:



        $ ./terraformsh -N plan apply

        ./terraformsh: WARNING: No -b option passed! Potentially using only local state.



        + terraform init -input=false -reconfigure -force-copy

        + terraform get -update=true

        + terraform validate

        + terraform plan -input=false -out=/home/vagrant/git/PUBLIC/terraformsh/tf.104900abc1.plan

        + terraform init -input=false -reconfigure -force-copy

        + terraform apply -input=false /home/vagrant/git/PUBLIC/terraformsh/tf.104900abc1.plan



### Change directory at runtime



  You can tell Terraformsh to change to a module's directory before running commands

  so you don't have to do it yourself (later versions of Terraform have an option

  for this, but earlier ones don't):



        $ ./terraformsh -C ../../../root-modules/aws/common/ plan



### Passing Terraform tfvars files



  You can pass Terraform configuration files using the `-f` or `-b` options.



        $ terraformsh -C ../../../root-modules/aws/common/ \

            -f terraform.tfvars.json \

            -f override.auto.tfvars.json \

            -b backend.tfvars \

            -b backend-key.tfvars \

            plan approve apply



  To make this even simpler, if you pass any argument to Terraformsh after the 

  initial *OPTIONS*, and they match *TFVARS* ('\*.backend.tfvars', '\*.backend.sh.tfvars',

  '\*.tfvars.json', '\*.tfvars', '\*.sh.tfvars.json', '\*.sh.tfvars'), they will

  be automatically loaded with the `-f` and `-b` options.



        # Assuming you already have 'something.tfvars' and 'something.backend.tfvars'

        # in your current working directory, run the following:

        $ terraformsh -C ../../../root-modules/aws/common/ \

            *.tfvars \

            plan approve apply



  Finally, if in any *parent directory* of where you ran Terraformsh, there are

  files named `backend.sh.tfvars`, `terraform.sh.tfvars.json`, or `terraform.sh.tfvars`,

  those will also be loaded automatically (you can disable this with the `-I` option).



        $ mkdir -p some/configs/here

        $ cd some

        $ touch terraform.sh.tfvars

        $ cd configs

        $ touch backend.sh.tfvars

        $ cd here

        $ touch terraform.sh.tfvars

        $ terraformsh -N plan apply

        + terraform init -input=false -reconfigure -force-copy -backend-config /home/vagrant/git/PUBLIC/terraformsh/some/configs/backend.sh.tfvars

        + terraform get -update=true

        + terraform validate -var-file /home/vagrant/git/PUBLIC/terraformsh/some/terraform.sh.tfvars -var-file /home/vagrant/git/PUBLIC/terraformsh/some/configs/here/terraform.sh.tfvars

        + terraform plan -var-file /home/vagrant/git/PUBLIC/terraformsh/some/terraform.sh.tfvars -var-file /home/vagrant/git/PUBLIC/terraformsh/some/configs/here/terraform.sh.tfvars -input=false -out=/home/vagrant/git/PUBLIC/terraformsh/some/configs/here/tf.019c25e289.plan

        + terraform init -input=false -reconfigure -force-copy -backend-config /home/vagrant/git/PUBLIC/terraformsh/some/configs/backend.sh.tfvars

        + terraform apply -input=false /home/vagrant/git/PUBLIC/terraformsh/some/configs/here/tf.019c25e289.plan



### Environment Variables / Configuration



  Don't want to remember what options to pass to terraformsh? You don't have to!

  You can capture anything you want Terraformsh to do in a config file that is

  automatically loaded.



  The config file format is just a bash script. Therefore you can do things like

  'export' arbitrary environment variables for Terraform to load, or even run

  custom code.



  It's *highly recommended* that you **do not** set environment variables like

  Terraform's `TF_VAR_*`, otherwise you will have a mix of variables set in both

  config files and environment variables, and it will make it difficult to track

  down where/how a variable is being set. Stick to static variables in

  `*.tfvars` or `*.tfvars.json` files, and load dynamic variables from Terraform

  with a data source.



  You can set the following variables in a config file (any of:

  `/etc/terraformsh`, `~/.terraformshrc`, `.terraformshrc`, `terraformsh.conf`),

  or set them as environment variables before you call Terraformsh:



    DEBUG=1                     # Enable bash tracing

    TERRAFORM=terraform         # The name of the terraform executable

    TF_PLANFILE=                # Automatically populated by terraformsh

    TF_DESTROY_PLANFILE=        # Automatically populated by terraformsh

    TF_BOOTSTAP_PLANFILE=       # Automatically populated by terraformsh

    PUSH_ERRORED_TFSTATE=0      # Don't push errored.tfstate on failed apply

    USE_PLANFILE=0              # Don't use a plan file for each apply/destroy

    INHERIT_TFFILES=0           # Don't inherit tfvars files in parent directories

    NO_DEP_CMDS=1               # Don't run dependent commands automatically

    NO_CLEANUP_TMP=1            # Don't clean up temporary TF_DATA_DIR

    DRYRUN=1                    # Enable dry-run mode

    CD_DIR=             # The directory to change to before running terraform commands



  The environment variable `TF_DATA_DIR` is automatically overridden by Terraformsh.

  A new temporary directory is created for the data dir, based on *both* the name of

  the directory you ran Terraformsh from, and the Terraform module directory

  you run terraform against (the `-C` option). If you pass your own `TF_DATA_DIR`

  environment variable, Terraformsh will use that instead.



  The following can be set in the Terraformsh config file as Bash arrays, or you

  can set them by passing them to `-E`, such as `-E "PLAN_ARGS=(-no-color -input=false)"`.



    VARFILES=()                     # files to pass to -var-file

    BACKENDVARFILES=()              # files to pass to -backend-config

    CMDS=()                         # the commands for terraformsh to run

    PLAN_ARGS=(-input=false)        # the arguments for 'terraform plan'

    APPLY_ARGS=(-input=false)       # the arguments for 'terraform apply'

    PLANDESTROY_ARGS=(-input=false) # arguments for 'plan -destroy'

    DESTROY_ARGS=(-input=false)     # arguments for 'terraform destroy'

    REFRESH_ARGS=(-input=false)     # arguments for 'terraform refresh'

    INIT_ARGS=(-input=false -reconfigure -force-copy)  # arguments for 'terraform init'

    OH12UPGRADE_ARGS=(-yes)         # arguments for '0.12upgrade'

    OH13UPGRADE_ARGS=(-yes)         # arguments for '0.13upgrade'

    IMPORT_ARGS=(-input=false)      # arguments for 'terraform import'

    GET_ARGS=(-update=true)         # arguments for 'terraform get'

    STATE_ARGS=()                   # arguments for 'terraform state'

    VALIDATE_ARGS=()                # arguments for 'terraform validate'

    WORKSPACE_ARGS=()               # arguments for 'terraform workspace'

    CONSOLE_ARGS=()                 # arguments for 'terraform console

    OUTPUT_ARGS=()                  # arguments for 'terraform output'

    TAINT_ARGS=()                   # arguments for 'terraform taint'

    UNTAINT_ARGS=()                 # arguments for 'terraform untaint'

    FORCEUNLOCK_ARGS=(-force)       # arguments for 'terraform forceunlock'

    SHOW_ARGS=()                    # arguments for 'terraform show'



  To use the 'aws_bootstrap' command, pass the '-b FILE' option and make sure the

  file(s) have the following variables:



    bucket          - The S3 bucket your Terraform state will live in

    dynamodb_table  - The DynamoDB table your Terraform state will be managed in



  An example file: [.terraformshrc-example](.terraformshrc-example)



### Interactive troubleshooting



  Need to troubleshoot some problem by just running 'terraform' yourself? No

  problem, use the `shell` command. It will drop you into a Bash shell after

  first changing to the correct directory and running `terraform init` and

  `terraform get` with all the environment variables set up for you

  (including the automatic `TF_DATA_DIR`).



        $ ./terraformsh -N -C ../../../root-modules/aws/common/ shell

        + cd "../../../root-modules/aws/common/"

        ./terraformsh: WARNING: No -b option passed! Potentially using only local state.



        + terraform init -input=false -reconfigure -force-copy

        + terraform get -update=true

        + bash -i -l



  You can even get Terraformsh to explicitly ask you for confirmation before

  moving to the next command with the `approve` command (since the default is

  to pass `-input=false` to each command for easier use in automation).



  Are you working in a hierarchy of config files, and want to grep all

  the parent directories? Use the built-in `revgrep` command:



        $ terraformsh revgrep -H -e "gcp_project_id"

        terraformsh: Info: Found terraform command 'revgrep'

        terraformsh: Warning: '-H' is not a valid command; passing as an option instead

        terraformsh: Warning: 'project_id' is not a valid command; passing as an option instead

        + cd "/home/vagrant/my-repo/env/product/dev/nonprod/us-west1/tf-state/bootstrap"

        /home/vagrant/git/my-repo/env/product/dev/nonprod/terraform.sh.tfvars:gcloud_project_id = "123456789"



  Want to output one of Terraformsh's plan files as JSON?



        $ terraformsh show -json "$(pwd)/tf.b063520160.plan"



### More Examples



  There are many ways to use Terraformsh, whether you pass all the options

  via environment variables/command-line options, or keep all the commands

  in a configuration file and load everything automatically.



 - Run 'plan' using a `.terraformshrc` file that has all the above options,

   but override terraformsh's internal arguments to 'terraform plan':



        $ terraformsh -E 'PLAN_ARGS=("-compact-warnings" "-no-color" "-input=false")' \

          plan



 - Run 'plan' on a module and pass any configs found in these directories:



        $ terraformsh -C root-modules/my-database/ \

           *.tfvars \

           env/my-database/*.tfvars \

           plan



 - Run 'plan' on a module, implicitly loading configuration files from parent directories:



        $ pwd

        /home/vagrant/git/some-repo/env/non-prod/us-east-2/my-database

        $ echo 'CD_DIR=../../../../modules/my-database/' > terraformsh.conf

        $ echo 'aws_account_id = "0123456789"' > ../../terraform.sh.tfvars

        $ echo 'region = "us-east-2"' > ../terraform.sh.tfvars

        $ echo 'database_name = "some database"' > terraform.sh.tfvars

        $ terraformsh plan



 - You've applied some Terraform using local state, and now you want to migrate it to a remote backend. After you add your new backend tf code, you'd run:



        $ terraformsh -E "INIT_ARGS=()" init -force-copy -migrate-state plan apply



### Having trouble?



 - **Problem: I'm using Terraformsh from two different shell sessions, in the same directory, running the same commands, but one of them is working and the other isn't. What's going on?**

   

   *Solution:* Something's wrong with your environment variables in one of the sessions. If both `TF_DATA_DIR` and `TF_TMPDIR` are set to something starting with `/tmp/tfsh.`, then you probably used `terraformsh shell` and forgot to *exit*.



---



    terraformsh v0.14

    Usage: ./terraformsh [OPTIONS] [TFVARS] COMMAND [..]



# Options



  Pass these OPTIONS before any others (see examples); do not pass them after

  TFVARS or COMMANDs.



    -f FILE         A file passed to Terraform's -var-file option.

                      ( config: VARFILES= )

    -b FILE         A file passed to Terraform's -backend-config option.

                      ( config: BACKENDVARFILES= )

    -C DIR          Change to directory DIR.

                      ( config: CD_DIR= )

    -c file         Specify a '.terraformshrc' configuration file to load.

    -E EXPR         Evaluate an expression in bash ('eval EXPR').

    -I              Disables automatically loading any 'terraform.sh.tfvars',

                    'terraform.sh.tfvars.json', or 'backend.sh.tfvars' files

                    found while recursively searching parent directories.

                      ( config: INHERIT_TFFILES=0 )

    -P              Do not use '.plan' files for plan/apply/destroy commands.

                      ( config: USE_PLANFILE=0 )

    -D              Don't run 'dependency' commands (e.g. don't run "terraform

                    init" before "terraform apply").

                      ( config: NO_DEP_CMDS=1 )

    -N              Dry-run mode (don't execute anything).

                      ( config: DRYRUN=1 )

    -n              Don't remove the temporary TF_DATA_DIR.

                      ( config: NO_CLEANUP_TMP=1 )

    -v              Verbose mode.

                      ( config: DEBUG=1 )

    -h              This help screen.



# Commands



  The following are Terraform commands that terraformsh provides wrappers for

  (there's some Terraformsh-specific logic behind the scenes). Other Terraform

  commands not listed here are passed through to Terraform verbatim.



    plan              Run init, get, validate, `terraform plan @VARFILE_ARG -out $TF_PLANFILE`

    apply             Run init, get, validate, `terraform apply $TF_PLANFILE`

    plan_destroy      Run init, get, validate, `terraform plan -destroy -out=$TF_DESTROY_PLANFILE`

    destroy           Run init, get, validate, `terraform apply $TF_DESTROY_PLANFILE`

    refresh           Run init, `terraform refresh`

    validate          Run init, get, `terraform validate`

    init              Run clean_modules, `terraform init @BACKENDVARFILE_ARG`

    get               Run init, `terraform get [..]`

    show              Run init, `terraform show [..]`

    import            Run init, `terraform import [..]`

    state             Run init, `terraform state [..]`

    taint             Run init, `terraform taint [..]`

    untaint           Run init, `terraform untaint [..]`

    output            Run init, refresh, `terraform output [..]`

    console           Run init, `terraform console [..]`

    workspace         Run init, `terraform workspace [..]`

    force-unlock      Run init, `terraform force-unlock [..]`

    0.12upgrade       Run init, `terraform 0.12upgrade [..]`

    0.13upgrade       Run init, `terraform 0.13upgrade [..]`



  The following commands are specific to terraformsh:



    shell             Run init, get, and `bash -i -l`

    clean             Remove '.terraform/modules/*', terraform.tfstate files, and .plan files

    clean_modules     Run `rm -v -rf .terraform/modules/*`

    approve           Prompts the user to approve the next step, or the program will exit with an error.

    aws_bootstrap     Looks for 'bucket' and 'dynamodb_table' in your '-b' file options.

                      If found, creates the bucket and table and initializes your Terraform state with them.

    revgrep           Run 'grep' on files in all parent directories

    env               Run 'env' command with optional arguments



All arguments after a COMMAND are evaluated for whether they match a Terraform

or Terraformsh command; if they don't, they are assumed to be options and are

passed to the first recognized command that precedes them.