{"id":26522497,"url":"https://github.com/pix4d/terravalet","last_synced_at":"2025-03-21T13:27:21.893Z","repository":{"id":40369587,"uuid":"314482940","full_name":"Pix4D/terravalet","owner":"Pix4D","description":"A tool to help with some Terraform operations","archived":false,"fork":false,"pushed_at":"2024-01-31T13:28:50.000Z","size":162,"stargazers_count":87,"open_issues_count":0,"forks_count":5,"subscribers_count":15,"default_branch":"master","last_synced_at":"2024-11-16T02:17:39.567Z","etag":null,"topics":["go","owner-platform-ci","resurces","terraform"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Pix4D.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-11-20T07:49:27.000Z","updated_at":"2024-10-21T09:02:33.000Z","dependencies_parsed_at":"2024-06-19T00:03:54.170Z","dependency_job_id":"48c01d3c-65c6-44e3-b1e8-a90283c490ca","html_url":"https://github.com/Pix4D/terravalet","commit_stats":null,"previous_names":[],"tags_count":12,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Pix4D%2Fterravalet","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Pix4D%2Fterravalet/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Pix4D%2Fterravalet/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Pix4D%2Fterravalet/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Pix4D","download_url":"https://codeload.github.com/Pix4D/terravalet/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244805254,"owners_count":20513238,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["go","owner-platform-ci","resurces","terraform"],"created_at":"2025-03-21T13:27:21.042Z","updated_at":"2025-03-21T13:27:21.887Z","avatar_url":"https://github.com/Pix4D.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Terravalet\n\nA tool to help with advanced, low-level [Terraform](https://www.terraform.io/) operations:\n\n- Rename resources within the same Terraform state, with optional fuzzy match.\n- Move resources from one Terraform state to another.\n- Import existing resources into Terraform state.\n- Remove existing resources from Terraform state.\n\n**DISCLAIMER Manipulating Terraform state is inherently dangerous. It is your responsibility to be careful and ensure you UNDERSTAND what you are doing**.\n\n## Status\n\nThis is BETA code, although we already use it in production.\n\nThe project follows [semantic versioning](https://semver.org/). In particular, we are currently at major version 0: anything MAY change at any time. The public API SHOULD NOT be considered stable.\n\n## Overall approach and migration scripts\n\nThe overall approach is for Terravalet to generate migration scripts, not to perform any change directly. This for two reasons:\n\n1. Safety. The operator can review the migration scripts for correctness.\n2. Gitops-style. The migration scripts are meant to be stored in git in the same branch (and thus same PR) that performs the Terraform changes and can optionally be hooked to an automatic deployment system.\n\nTerravalet takes as input the output of `terraform plan` for each involved root module and generates one UP and one DOWN migration script.\n\n### Remote and local state\n\nAt least until Terraform 0.14, `terraform state mv` has a bug: if a remote backend for the state is configured (which will always be the case for prod), it will remove entries from the remote state, but it will not add entries to it.\nIt will fail silently and leave an empty backup file, so you will lose your state.\n\nFor this reason Terravalet operates on local state and leaves to the operator the task of performing `terraform state pull` and `terraform state push`.\n\n### Terraform workspaces\n\nBe careful when using Terraform workspaces, since they are invisible and persistent global state :-(. Remember to always explicitly run `terraform workspace select` before anything else.\n\n### Interactions with the \"moved\" block\n\nAfter the creation of Terravalet, Terraform introduced the `moved` block, which can be seen as an alternative to certain usages of Terravalet. See [Terraform: refactoring](https://developer.hashicorp.com/terraform/language/modules/develop/refactoring)) for more information.\n\n## Install\n\n### Install from binary package\n\n1. Download the archive for your platform from the [releases page](https://github.com/Pix4D/terravalet/releases).\n2. Unarchive and copy the `terravalet` executable to a directory in your `$PATH`.\n\n### Install from source\n\n1. Install [Go](https://golang.org/).\n2. Install [Task](https://taskfile.dev/).\n3. Run `task`:\n   ```\n   $ task test build\n   ```\n4. Copy the executable `bin/terravalet` to a directory in your `$PATH`.\n\n## Usage\n\nTerravalet supports multiple operations:\n\n- [Rename resources](#rename-resources-within-the-same-state) within the same Terraform state, with optional fuzzy match.\n- [Move resources](#-move-resources-from-one-state-to-another) from one Terraform state to another.\n- [Import existing resources](#-import-existing-resources) into Terraform state.\n- [Remove existing resources](#removing-existing-resources) from Terraform state.\n\nThey will be explained in the following sections.\n\nYou can also look at the tests and in particular at the files below [testdata/](testdata) for a rough idea.\n\n# Rename resources within the same state\n\nOnly one Terraform root module (and thus only one state) is involved. This actually covers two different use cases:\n\n1. Renaming resources within the same root module.\n2. Moving resources to/from a non-root Terraform module (this will actually _rename_ the resources, since they will get or lose the `module.` prefix).\n\n## Collect information and remote state\n\n```\n$ cd $ROOT_MODULE\n$ terraform workspace select $WS\n$ terraform plan -no-color 2\u003e\u00261 | tee plan.txt\n\n$ terraform state pull \u003e local.tfstate\n$ cp local.tfstate local.tfstate.BACK\n```\n\nThe backup is needed to recover in case of errors. It must be done now.\n\n## Generate migration scripts: exact match, success\n\nTake as input the Terraform plan `plan.txt` (explicit) and the local state `local.tfstate` (implicit) and generate UP and DOWN migration scripts:\n\n```\n$ terravalet rename \\\n    --plan plan.txt --up 001_TITLE.up.sh --down 001_TITLE.down.sh\n```\n\n## Generate migration scripts: exact match, failure\n\nDepending on _how_ the elements have been renamed in the Terraform configuration, it is possible that the exact match will fail:\n\n```\n$ terravalet rename \\\n    --plan plan.txt --up 001_TITLE.up.sh --down 001_TITLE.down.sh\nmatch_exact:\nunmatched create:\n  aws_route53_record.private[\"foo\"]\nunmatched destroy:\n  aws_route53_record.foo_private\n```\n\nIn this case, you can attempt fuzzy matching.\n\n## Generate migration scripts: fuzzy match\n\n**WARNING** Fuzzy match can make mistakes. It is up to you to validate that the migration makes sense.\n\nIf the exact match failed, it is possible to enable [q-gram distance](https://github.com/dexyk/stringosim) fuzzy matching with the `-fuzzy-match` flag:\n\n```\n$ terravalet rename -fuzzy-match \\\n    --plan plan.txt --up 001_TITLE.up.sh --down 001_TITLE.down.sh\nWARNING fuzzy match enabled. Double-check the following matches:\n 9 aws_route53_record.foo_private -\u003e aws_route53_record.private[\"foo\"]\n```\n\n## Run the migration script\n\n1. Review the contents of `001_TITLE.up.sh`.\n2. Run it: `sh ./001_TITLE.up.sh`\n\n## Push the migrated state\n\n1. `terraform state push local.tfstate`. In case of error, DO NOT FORCE the push unless you understand very well what you are doing.\n\n## Recovery in case of error\n\nPush the `local.tfstate.BACK`.\n\n# Move resources from one state to another\n\nTwo Terraform root modules (and thus two states) are involved. The names of the resources stay the same, but we move them from one root module to another.\n\n## Understanding move-after and move-before\n\nConsider root environment (name `1`), represented as list element:\n\n![](docs/list.png)\n\nThere are two ways to split it:\n\n1. By putting some of its contents AFTER itself.\n  ![](docs/list-append.png)\n2. By putting some of its contents BEFORE itself.\n  ![](docs/list-prepend.png)\n\nMove AFTER is more frequent and easier to reason about.\n\nOn the other hand, you will know when you need to move BEFORE `terraform plan` in the BEFORE module (1' in the figure above) will fail with a message similar to this one:\n\n```\nError: Unsupported attribute\n│\n│ on main.tf line 11, in resource \"null_resource\" \"res2\":\n│ 11:     dep = data.terraform_remote_state.prepend_p0.outputs.res1_id\n│ data.terraform_remote_state.prepend_p0.outputs is object with 1 attribute \"pet\"\n│\n│ This object does not have an attribute named \"res1_id\".\n╵\n```\n\nThe error is because we didn't run terraform apply in `p0`, so `p0.outputs` doesn't have yet the attribute `res1_id`.\n\nWhen this happens and you convince yourself that this is expected and not an error on your part, you can still move the state, by using command `move-before`.\n\n## Collect information and remote state\n\nPerform all operations in `topdir`, the directory containing the two root modules.\n\nSomething like:\n\n```\ntopdir/\n    BEFORE/    \u003c== root module\n    AFTER/     \u003c== root module\n```\n\n### Collect information\n\nIf this is a terravalet move-after (the default):\n\n```\n$ terraform -chdir=BEFORE plan -no-color \u003e BEFORE.tfplan\n$ terraform -chdir=AFTER  plan -no-color \u003e AFTER.tfplan\n```\n\nIn this case, you can also perform a basic validation: the number of elements to add compared to the number of elements to destroy must be the same:\n\n   ```\n   $ grep \"Plan:\" BEFORE.tfplan\n   Plan: 229 to add, 0 to change, 229 to destroy.\n   ```\nIf there is a mismatch, then it means that you have missed something. Go back to editing the Terraform files.\n\n\nIf this is a terravalet move-before (special case):\n\n```\n$ terraform -chdir=BEFORE plan -no-color \u003e BEFORE.tfplan\n```\n\n### Collect remote state\n\n```\n$ terraform -chdir=BEFORE state pull \u003e BEFORE.tfstate\n$ terraform -chdir=AFTER  state pull \u003e AFTER.tfstate\n```\n\nBackup the two remote states. This is needed to recover in case of errors and must be done now.\n\n```\n$ cp BEFORE.tfstate BEFORE.tfstate.BACK\n$ cp AFTER.tfstate AFTER.tfstate.BACK\n```\n\n## Generate migration scripts\n\nTake as input the one or two Terraform plans (BEFORE.tfplan and AFTER.tfplan) and the two state files (BEFORE.tfstate and AFTER.tfstate) and generate the 01-migrate-foo_up.sh and 01-migrate-foo_down.sh migration scripts.\n\nIf move-after:\n\n```\n$ terravalet move-after  --script=01-migrate-foo --before=BEFORE --after=AFTER\n```\n\nIf move-before:\n\n```\n$ terravalet move-before --script=01-migrate-foo --before=BEFORE --after=AFTER\n```\n\n## Run the migration script\n\n1. Review the contents of `01-migrate-foo_up.sh`.\n2. Run it: `sh ./01-migrate-foo_up.sh`\n\n## Push the migrated states\n\nIn case of error, DO NOT FORCE the push unless you understand very well what you are doing.\n\n```\n$ terraform -chdir=BEFORE state push - \u003c BEFORE.tfstate\n$ terraform -chdir=AFTER  state push - \u003c AFTER.tfstate\n```\n\n## Recovery in case of error\n\nPush the two backups:\n\n```\n$ terraform -chdir=BEFORE state push - \u003c BEFORE.tfstate.BACK\n$ terraform -chdir=AFTER  state push - \u003c AFTER.tfstate.BACK\n```\n\n# Import existing resources\n\nThe `terraform import` command can import existing resources into Terraform state, but requires to painstakingly write by hand the arguments, one per resource. This is error-prone and tedious.\n\nThus, `terravalet import` creates the import commands for you.\n\nYou must first add to the Terraform configuration the resources that you want to import, and then import them: neither `terraform` nor `terravalet` are able to write Terraform configuration, they only add to the Terraform state.\n\nSince each Terraform provider introduces its own resources, it would be impossible for Terravalet to know all of them. Instead, you write a simple [resource definitions file](#writing-a-resource-definitions-file), so that Terravalet can know how to proceed.\n\nFor concreteness, the examples below refer to the [Terraform GitHub provider](https://registry.terraform.io/providers/integrations/github/latest/docs).\n\n## Generate a plan in JSON format\n\nterraform plan:\n\n```\n$ cd $ROOT_MODULE\n$ terraform plan -no-color 2\u003e\u00261 -out plan.txt\n$ terraform show -json plan.txt | tee plan.json\n```\n\n## Generate import/remove scripts\n\nTake as input the Terraform plan in JSON format `plan.json` and generate UP and DOWN import scripts:\n\n```\n$ terravalet import \\\n    --res-defs  my_definitions.json \\\n    --src-plan  plan.json \\\n    --up import.up.sh --down import.down.sh\n```\n\n## Review the scripts\n\n1. Ensure that the **parent** resources are placed at the top of the `up` script, followed by their **children**.\n2. Ensure that the **child** resources are placed at the top of the `down` script, followed by their **parents**.\n3. Ensure the correctness of parameters.\n\nNOTE: The script modifies the remote state, but it is not dangerous because it only imports new resources if they already exist and it doesn't create or destroy anything.\n\nTerraform will try to import as much as possible, if the corresponding address in state doesn't exist yet, it means it should be created later using `terraform apply`, actually the resource is in `.tf` configuration, but not yet in real world.\n\n## Run the import script\n\n```\nsh ./import.up.sh\n```\n\n### Example\n\nHere is a new plan, scripts have been already generated:\n\n```\n $ terraform plan\n .....\n Plan: 6 to add, 0 to change, 0 to destroy.\n```\nThese are new resources, let's run the import script and run the plan again:\n\n```\n$ sh import.up.sh\nmodule.github.github_repository.repos[\"test-import-gh\"]: Importing from ID \"test-import-gh\"...\nmodule.github.github_repository.repos[\"test-import-gh\"]: Import prepared!\n  Prepared github_repository for import\nmodule.github.github_repository.repos[\"test-import-gh\"]: Refreshing state... [id=test-import-gh]\n\nImport successful!\n.....\n```\n\nDuring the run the following error can happen:\n\n```\nError: Cannot import non-existent remote object\n\nWhile attempting to import an existing object to\ngithub_team_repository.all_teams[\"test-import-gh.integration\"], the provider\ndetected that no object exists with the given id. Only pre-existing objects\ncan be imported; check that the id is correct and that it is associated with\nthe provider's configured region or endpoint, or use \"terraform apply\" to\ncreate a new remote object for this resource.\n```\n\nIn this specific case the out-of-band resource didn't have a setting yet about teams, so it's normal.\n\nNext plan should be different:\n\n```\n$ terraform plan\n.....\nPlan: 3 to add, 2 to change, 0 to destroy.\n```\n\nIn conclusion, the plan now is close to real resources states and terraform is now aware of them.\nIn every case plan doesn't contain any `destroy` sentence.\n\n## Rollback\n\nRun `import.down.sh` script that remove the same resources from terraform state that have been imported with `import.up.sh`.\n\n## Writing a resource definitions file\n\nTerravalet doesn't know anything about resources, it just parses the plan and uses the resources configuration file passed via the flag `res-defs`. An example can be found in [testdata/import/terravalet_imports_definitions.json](testdata/import/terravalet_imports_definitions.json).\n\nThe idea is to tell Terravalet where to search the data to build the up/down scripts. The correct information can be found on the [specific provider documentation](https://registry.terraform.io/browse/providers). Under the hood, Terravalet matches the parsed plan and resources definition file.\n\n1. The JSON resources definition is a map of resources type objects identified by their own name as a key.\n2. The resource type object has an optional `priority`: import statement for that resource must be placed at the top of up.sh and at the bottom of down.sh (resources that must be imported before others).\n3. The resource type object has an optional `separator`: in case of multiple arguments it is mandatory and it will be used to join them. Using the example below, `tag, owner` will be joined into the string `\u003ctag_value\u003e:\u003cowner_value\u003e`.\n4. The resource type object must have `variables`: a list of fields names that are the keys in the plan to retrieve the correct values building the import statement. Using the example below, Terravalet will search for keys `tag` and `owner` in terraform plan for that resource.\n\n```json\n{\n  \"dummy_resource1\": {\n    \"priority\": 1,\n    \"separator\": \":\",\n    \"variables\": [\n      \"tag\",\n      \"owner\"\n    ]\n  }\n}\n```\n\n## Error cases\n\nIgnorable errors:\n\n1. Resource X doesn't exist yet, it resides only in new terraform configuration.\n2. Resource X exists, but depends on resource Y that has not been imported yet (should be fine setting the priority).\n\nNON ignorable errors:\n\n1. Provider specific argument ID is wrong.\n\n# Removing existing resources\n\nAlthough `terraform state rm` allows to remove _individual_ resources, but when a real-world resource is composed of multiple terraform resources, using `terraform state rm` becomes tedious and error-prone. Even worse when multiple high-level resources are removed together.\n\nThus, `terravalet remove` parses a plan file and creates all the `state rm` commands for you.\n\n1. Remove the resources in the Terraform configuration files.\n2. Generate the plan file:\n   ```\n   $ terraform -chdir=\u003cthe tf root\u003e plan -no-color \u003e remove-plan.txt\n   ```\n3. Run `terravalet remove`. As usual, this is a safe operation, since it will only generate a script file:\n   ```\n   $ terravalet remove --up=remove.sh --plan=remove-plan.txt\n   ```\n4. Carefully examine the generated script file `remove.sh`!\n5. Execute the scrip.\n   ```\n   $ sh ./remove.sh\n   ```\n\n# Making a release\n\n## Setup\n\n1. Install [gopass](https://github.com/gopasspw/gopass) or equivalent.\n2. Configure a fine-grained personal access token, scoped to the terravalet repository:\n    * Go to [Fine-grained personal access tokens](https://github.com/settings/tokens?type=beta)\n    * Click on \"Generate new token\"\n    * Give it a name like \"terravalet-releases\"\n    * Select \"Resource owner\" -\u003e Pix4D\n    * Select \"Repository access\" -\u003e \"Only select repositories\" -\u003e Terravalet\n    * Select \"Repository permissions\"\n      * \"Contents\" -\u003e RW\n    * Generate the token \n3. Store the token securely with a tool like `gopass`. The name `GITHUB_TOKEN` is expected by `github-release`\n   ```\n   $ gopass insert gh/terravalet/GITHUB_TOKEN\n   ```\n\n## Each time\n\n1. Update [CHANGELOG](CHANGELOG.md)\n2. Update this README and/or additional documentation.\n3. Make and merge a PR.\n4. Ensure your local master branch is up-to-date:\n   ```\n   $ git checkout master\n   $ git pull\n   ```\n5. Begin the release process with\n   ```\n   $ RELEASE_TAG=v0.1.0 gopass env gh/terravalet task release\n   ```\n6. Finish the release process by following the instructions printed by `task` above.\n7. To recover from a half-baked release, see the hints in the [Taskfile](Taskfile.yml).\n\n# History and credits\n\nThe idea of migrations comes from [tfmigrate](https://github.com/minamijoyo/tfmigrate). Then this blog [post](https://medium.com/@lynnlin827/moving-terraform-resources-states-from-one-remote-state-to-another-c76f8b76a996)  made me realize that `terraform state mv` had a bug and how to workaround it.\n\n# License\n\nThis code is released under the MIT license, see file [LICENSE](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpix4d%2Fterravalet","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpix4d%2Fterravalet","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpix4d%2Fterravalet/lists"}