{"id":20334922,"url":"https://github.com/mdb/tfmigrate-demo","last_synced_at":"2025-07-04T00:34:26.211Z","repository":{"id":185907135,"uuid":"674242605","full_name":"mdb/tfmigrate-demo","owner":"mdb","description":"A reference example and demo illustrating the use of tfmigrate for migrating Terraform resources between distinct Terraform projects and remote states","archived":false,"fork":false,"pushed_at":"2023-08-30T16:56:41.000Z","size":26,"stargazers_count":1,"open_issues_count":2,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-17T16:55:32.228Z","etag":null,"topics":["localstack","terraform","tfenv","tfmigrate"],"latest_commit_sha":null,"homepage":"https://mikeball.info/blog/using-tfmigrate-to-codify-and-automate-terraform-state-operations/","language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mdb.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-08-03T13:18:02.000Z","updated_at":"2023-10-05T11:36:42.000Z","dependencies_parsed_at":null,"dependency_job_id":"630b5e02-8380-43a6-9bb1-39033935f5d3","html_url":"https://github.com/mdb/tfmigrate-demo","commit_stats":null,"previous_names":["mdb/tfmigrate-demo"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mdb/tfmigrate-demo","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mdb%2Ftfmigrate-demo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mdb%2Ftfmigrate-demo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mdb%2Ftfmigrate-demo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mdb%2Ftfmigrate-demo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mdb","download_url":"https://codeload.github.com/mdb/tfmigrate-demo/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mdb%2Ftfmigrate-demo/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263426394,"owners_count":23464795,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["localstack","terraform","tfenv","tfmigrate"],"created_at":"2024-11-14T20:38:35.528Z","updated_at":"2025-07-04T00:34:26.192Z","avatar_url":"https://github.com/mdb.png","language":"HCL","funding_links":[],"categories":[],"sub_categories":[],"readme":"# tfmigrate-demo\n\nA demo showing how [tfmigrate](https://github.com/minamijoyo/tfmigrate) can be\nused to codify and automate the migration of Terraform resources between root module\nproject configurations, each using different S3 remote states.\n\nThe demo also shows how `tfmigrate` enables teams to codify state migrations as HCL,\nsubjecting the migrations to code review and CI/CD, similar to other Terraform\nchanges codified as HCL.\n\nSee [Using tfmigrate to Codify and Automate Terraform State Operations](https://mikeball.info/blog/using-tfmigrate-to-codify-and-automate-terraform-state-operations/)\nas the corresponding blog post.\n\nOverview:\n\n* `bootstrap` is a minimal Terraform project that creates a [localstack](https://localstack.cloud/)\n  `tfmigrate-demo` S3 bucket for use hosting `project-one` and `project-two`'s\n   Terraform remote states\n* `project-one` is a minimal Terraform 0.13.7 project that creates `foo.txt` and\n  `bar.txt` files and uses `s3://tfmigrate-demo/project-one/terraform.tfstate` as\n  its remote state backend.\n* `project-two` is a minimal Terraform 1.4.6 project that creates a `baz.txt` file\n  and uses `s3://tfmigrate-demo/project-two/terraform.tfstate` as its remote state\n  backend.\n* `project-one` and `project-two` each feature a `.terraform-version` file. This\n  ensures [tfenv](https://github.com/tfutils/tfenv) selects the proper Terraform\n  for use in each project.\n* `migration.hcl` is a [tfmigrate](https://github.com/minamijoyo/tfmigrate) migration that orchestrates the migration\n  of `local_file.bar` from management in `project-one` to management in `project-two`.\n  `tfmigrate` enables teams to codify migrations as HCL, subjecting the\n  migrations to code review and CI/CD, alongside terraform HCL configurations.\n* `.tfmigrate.hcl` is a `tfmigrate` configuration file specifying that\n  `tfmigrate`'s migration history be persisted to\n  `s3://tfmigrate-demo/tfmigrate/history.json`.\n\nSee [PR 2](https://github.com/mdb/tfmigrate-demo/pull/2) for an example GitHub\nActions workflow that fails its `tfmigrate plan` step. See [PR 3](https://github.com/mdb/tfmigrate-demo/pull/3) for an example\nGitHub Actions workflow that successfully performs a `tfmigrate apply`.\n\nSee `.github/workflows/pr.yaml` for the GitHub Actions workflow configuration.\n\n## More detailed problem statement\n\nHow can we codify and automate the migration of a Terraform-managed\nresource from one root module project configuration to a different root module\nproject configuration, each using different S3 remote states?\n\nAnd what if each root module project uses a different version of Terraform?\n\nSolution: use [tfmigrate](https://github.com/minamijoyo/tfmigrate) in concert\nwith [tfenv](https://github.com/tfutils/tfenv).\n\n## See the demo in GitHub Actions\n\nThe workflow described in \"Try the demo for yourself\" (below) is automated and demoed in [GitHub Actions](https://github.com/mdb/tfmigrate-demo/actions).\n\n[PR 2](https://github.com/mdb/tfmigrate-demo/pull/2) triggers an example GitHub\nActions workflow that fails its `tfmigrate plan` step: https://github.com/mdb/tfmigrate-demo/actions/runs/5776326609/job/15655320734\n\n[PR 3](https://github.com/mdb/tfmigrate-demo/pull/3) triggers an example GitHub\nActions workflow that successfully performs a `tfmigrate apply` step: https://github.com/mdb/tfmigrate-demo/actions/runs/5776333280/job/15655332928\n\nSee `.github/workflows/pr.yaml` for the GitHub Actions workflow configuration.\n\n## Real World GitOps Workflow\n\nNote this demo is a bit contrived, in large part because it uses ephemeral local\nTerraform projects whose resources and state aren't persistant. A real world\nGitOps-esque workflow against an existing project would look more like...\n\n1. Open a PR containing the desired `tfmigrate` migration HCL and desired Terraform\n   configuration changes.\n2. CI detects the presence of a `tfmigrate` migration HCL and verifies the\n   changes via `tfmigrate plan`.\n3. Following successful CI and code review approval, the PR is merged.\n4. CI/CD detects the presence of the new `tfmigrate` migration HCL  and performs\n   `tfmigrate plan` and `tfmigrate apply` to perform the migration, similar to\n   what's done via `terraform plan` and `terraform apply` for other Terraform\n   changes.\n\n## Try the demo yourself locally\n\n### Install dependencies\n\n```\nbrew install tfmigrate\n```\n\n```\nbrew install tfenv\n```\n\nThe demo also assumes [Docker](https://www.docker.com/) is installed and running.\n\n### Clone `tfmigrate-demo`\n\n```\ngit clone git@github.com:mdb/tfmigrate-demo.git \\\n  \u0026\u0026 cd tfmigrate-demo\n```\n\n### Bootstrap `localstack` environment\n\nRun `localstack` to simulate AWS APIs locally:\n\n```\nmake up\n```\n\nCreate a `localstack` `tfmigrate-demo` S3 bucket. This will be used to host\n`project-one` and `project-two`'s Terraform remote state files.\n\n```\nmake bootstrap\n```\n\n### `terraform apply` `project-one`\n\nInitially, `apply`-ing `project-one` results in the creation of 2 files in the\n`tfmigrate-demo` root using Terraform 0.13.7:\n\n1. `foo.txt` (its Terraform state resource address is `local_file.foo`)\n2. `bar.txt` (its Terraform state resource address is `local_file.bar`)\n\n```\nmake apply-one\n```\n\n### `terraform apply` `project-two`\n\nInitially, `apply`-ing `project-two` results in the creation of 1 file in the\n`tfmigrate-demo` root using Terraform 1.4.6:\n\n1. `baz.txt` (its Terraform state resource address is `local_file.baz`)\n\n```\nmake apply-two\n```\n\n### Use `tfmigrate` to migrate `local_file.bar`\n\nRun `tfmigrate list --status=unapplied` to view any outstanding migrations:\n\n```\ntfmigrate list --status=unapplied\n2023/08/15 19:30:44 [INFO] AWS Auth provider used: \"StaticProvider\"\nmigration.hcl\n```\n\nBased on `tfmigrate list`'s output, the `migration.hcl` file contains an\nunapplied migration that seeks to move `local_file.bar` from `project-one` to `project-two`:\n\n```hcl\nmigration \"multi_state\" \"mv_local_file_bar\" {\n  from_dir = \"project-one\"\n  to_dir   = \"project-two\"\n\n  actions = [\n    \"mv local_file.bar local_file.bar\",\n  ]\n}\n```\n\nNext, use [tfmigrate](https://github.com/minamijoyo/tfmigrate) to `plan` this\nmigration of `local_file.bar` from `project-one`'s Terraform state to\n`project-two`'s Terraform state using the migration instructions codified in\n`migration.hcl`, which move `local_file.bar` from `project-one` to\n`project-two`.\n\nNote that `tfmigrate plan` will \"dry run\" the migration using a local, dummy\ncopy of each project's remote state and will performs a `terraform plan`,\nerroring if the migration results in any unexpected plan changes.\n\nAlso note that `tfmigrate` automatically uses the correct `terraform` CLI version\nrequired by each project, as each project's `.terraform-version` file triggers\n`tfenv` to ensure the correct version is used.\n\nInitially, observe that `tfmigrate plan` fails, as we haven't\nyet moved `local_file.bar`'s recource declaration HCL from `project-one` to\n`project-two.`\n\n```\ntfmigrate plan\n2023/08/15 19:40:23 [INFO] AWS Auth provider used: \"StaticProvider\"\n2023/08/15 19:40:23 [INFO] [runner] unapplied migration files: [migration.hcl]\n2023/08/15 19:40:23 [INFO] [runner] load migration file: migration.hcl\n2023/08/15 19:40:23 [INFO] [migrator] multi start state migrator plan\n2023/08/15 19:40:24 [INFO] [migrator@project-one] terraform version: 0.13.7\n2023/08/15 19:40:24 [INFO] [migrator@project-one] initialize work dir\n2023/08/15 19:40:24 [INFO] [migrator@project-one] get the current remote state\n2023/08/15 19:40:24 [INFO] [migrator@project-one] override backend to local\n2023/08/15 19:40:24 [INFO] [executor@project-one] create an override file\n2023/08/15 19:40:24 [INFO] [migrator@project-one] creating local workspace folder in: project-one/terraform.tfstate.d/default\n2023/08/15 19:40:24 [INFO] [executor@project-one] switch backend to local\n2023/08/15 19:40:25 [INFO] [migrator@project-two] terraform version: 1.4.6\n2023/08/15 19:40:25 [INFO] [migrator@project-two] initialize work dir\n2023/08/15 19:40:26 [INFO] [migrator@project-two] get the current remote state\n2023/08/15 19:40:26 [INFO] [migrator@project-two] override backend to local\n2023/08/15 19:40:26 [INFO] [executor@project-two] create an override file\n2023/08/15 19:40:26 [INFO] [migrator@project-two] creating local workspace folder in: project-two/terraform.tfstate.d/default\n2023/08/15 19:40:26 [INFO] [executor@project-two] switch backend to local\n2023/08/15 19:40:27 [INFO] [migrator] compute new states (project-one =\u003e project-two)\n2023/08/15 19:40:27 [INFO] [migrator@project-one] check diffs\n2023/08/15 19:40:28 [ERROR] [migrator@project-one] unexpected diffs\n2023/08/15 19:40:28 [INFO] [executor@project-two] remove the override file\n2023/08/15 19:40:28 [INFO] [executor@project-two] remove the workspace state folder\n2023/08/15 19:40:28 [INFO] [executor@project-two] switch back to remote\n2023/08/15 19:40:28 [INFO] [executor@project-one] remove the override file\n2023/08/15 19:40:28 [INFO] [executor@project-one] remove the workspace state folder\n2023/08/15 19:40:28 [INFO] [executor@project-one] switch back to remote\nterraform plan command returns unexpected diffs in project-one from_dir: failed to run command (exited 2): terraform plan -state=/var/folders/46/sz1dzp417x7fk46kb3jv8cn00000gp/T/tmp3859825156 -out=/var/folders/46/sz1dzp417x7fk46kb3jv8cn00000gp/T/tfplan265876470 -input=false -no-color -\ndetailed-exitcode\nstdout:\nRefreshing Terraform state in-memory prior to plan...\nThe refreshed state will be used to calculate this plan, but will not be\npersisted to local or remote state storage.\n\nlocal_file.foo: Refreshing state... [id=94dd9e08c129c785f7f256e82fbe0a30e6d1ae40]\n\n------------------------------------------------------------------------\n\nAn execution plan has been generated and is shown below.\nResource actions are indicated with the following symbols:\n  + create\n\nTerraform will perform the following actions:\n\n  # local_file.bar will be created\n  + resource \"local_file\" \"bar\" {\n      + content              = \"Hi\"\n      + content_base64sha256 = (known after apply)\n      + content_base64sha512 = (known after apply)\n      + content_md5          = (known after apply)\n      + content_sha1         = (known after apply)\n      + content_sha256       = (known after apply)\n      + content_sha512       = (known after apply)\n      + directory_permission = \"0777\"\n      + file_permission      = \"0777\"\n      + filename             = \"./../bar.txt\"\n      + id                   = (known after apply)\n    }\n\nPlan: 1 to add, 0 to change, 0 to destroy.\n\n------------------------------------------------------------------------\n\nThis plan was saved to: /var/folders/46/sz1dzp417x7fk46kb3jv8cn00000gp/T/tfplan265876470\n\nTo perform exactly these actions, run the following command to apply:\n    terraform apply \"/var/folders/46/sz1dzp417x7fk46kb3jv8cn00000gp/T/tfplan265876470\"\n\n\nstderr:\n```\n\nNext, remove the `local_file.bar` declaration from `project-one/main.tf` and add\nit to `project-two/main.tf`, such that each project configuration reflects the\ndesired migration end state:\n\n```hcl\nresource \"local_file\" \"bar\" {\n  content  = \"Hi\"\n  filename = \"${path.module}/../bar.txt\"\n}\n```\n\nRe-run `tfmigrate plan`; now it's successful:\n\n```\ntfmigrate plan\n2023/08/15 19:41:16 [info] aws auth provider used: \"staticprovider\"\n2023/08/15 19:41:16 [info] [runner] unapplied migration files: [migration.hcl]\n2023/08/15 19:41:16 [info] [runner] load migration file: migration.hcl\n2023/08/15 19:41:16 [info] [migrator] multi start state migrator plan\n2023/08/15 19:41:17 [info] [migrator@project-one] terraform version: 0.13.7\n2023/08/15 19:41:17 [info] [migrator@project-one] initialize work dir\n2023/08/15 19:41:17 [info] [migrator@project-one] get the current remote state\n2023/08/15 19:41:17 [info] [migrator@project-one] override backend to local\n2023/08/15 19:41:17 [info] [executor@project-one] create an override file\n2023/08/15 19:41:17 [info] [migrator@project-one] creating local workspace folder in: project-one/terraform.tfstate.d/default\n2023/08/15 19:41:17 [info] [executor@project-one] switch backend to local\n2023/08/15 19:41:18 [info] [migrator@project-two] terraform version: 1.4.6\n2023/08/15 19:41:18 [info] [migrator@project-two] initialize work dir\n2023/08/15 19:41:19 [info] [migrator@project-two] get the current remote state\n2023/08/15 19:41:19 [info] [migrator@project-two] override backend to local\n2023/08/15 19:41:19 [info] [executor@project-two] create an override file\n2023/08/15 19:41:19 [info] [migrator@project-two] creating local workspace folder in: project-two/terraform.tfstate.d/default\n2023/08/15 19:41:19 [info] [executor@project-two] switch backend to local\n2023/08/15 19:41:19 [info] [migrator] compute new states (project-one =\u003e project-two)\n2023/08/15 19:41:20 [info] [migrator@project-one] check diffs\n2023/08/15 19:41:21 [info] [migrator@project-two] check diffs\n2023/08/15 19:41:21 [info] [executor@project-two] remove the override file\n2023/08/15 19:41:21 [info] [executor@project-two] remove the workspace state folder\n2023/08/15 19:41:21 [info] [executor@project-two] switch back to remote\n2023/08/15 19:41:22 [info] [executor@project-one] remove the override file\n2023/08/15 19:41:22 [info] [executor@project-one] remove the workspace state folder\n2023/08/15 19:41:22 [info] [executor@project-one] switch back to remote\n2023/08/15 19:41:22 [info] [migrator] multi state migrator plan success!\n```\n\nFinally, to perform the migration, `apply` the `migration.hcl`:\n\n```\ntfmigrate apply\n2023/08/15 19:43:30 [INFO] AWS Auth provider used: \"StaticProvider\"\n2023/08/15 19:43:30 [INFO] [runner] unapplied migration files: [migration.hcl]\n2023/08/15 19:43:30 [INFO] [runner] load migration file: migration.hcl\n2023/08/15 19:43:30 [INFO] [migrator] start multi state migrator plan phase for apply\n2023/08/15 19:43:31 [INFO] [migrator@project-one] terraform version: 0.13.7\n2023/08/15 19:43:31 [INFO] [migrator@project-one] initialize work dir\n2023/08/15 19:43:31 [INFO] [migrator@project-one] get the current remote state\n2023/08/15 19:43:32 [INFO] [migrator@project-one] override backend to local\n2023/08/15 19:43:32 [INFO] [executor@project-one] create an override file\n2023/08/15 19:43:32 [INFO] [migrator@project-one] creating local workspace folder in: project-one/terraform.tfstate.d/default\n2023/08/15 19:43:32 [INFO] [executor@project-one] switch backend to local\n2023/08/15 19:43:32 [INFO] [migrator@project-two] terraform version: 1.4.6\n2023/08/15 19:43:32 [INFO] [migrator@project-two] initialize work dir\n2023/08/15 19:43:33 [INFO] [migrator@project-two] get the current remote state\n2023/08/15 19:43:33 [INFO] [migrator@project-two] override backend to local\n2023/08/15 19:43:33 [INFO] [executor@project-two] create an override file\n2023/08/15 19:43:33 [INFO] [migrator@project-two] creating local workspace folder in: project-two/terraform.tfstate.d/default\n2023/08/15 19:43:33 [INFO] [executor@project-two] switch backend to local\n2023/08/15 19:43:34 [INFO] [migrator] compute new states (project-one =\u003e project-two)\n2023/08/15 19:43:34 [INFO] [migrator@project-one] check diffs\n2023/08/15 19:43:35 [INFO] [migrator@project-two] check diffs\n2023/08/15 19:43:35 [INFO] [executor@project-two] remove the override file\n2023/08/15 19:43:35 [INFO] [executor@project-two] remove the workspace state folder\n2023/08/15 19:43:35 [INFO] [executor@project-two] switch back to remote\n2023/08/15 19:43:36 [INFO] [executor@project-one] remove the override file\n2023/08/15 19:43:36 [INFO] [executor@project-one] remove the workspace state folder\n2023/08/15 19:43:36 [INFO] [executor@project-one] switch back to remote\n2023/08/15 19:43:36 [INFO] [migrator] start multi state migrator apply phase\n2023/08/15 19:43:36 [INFO] [migrator@project-two] push the new state to remote\n2023/08/15 19:43:36 [INFO] [migrator@project-one] push the new state to remote\n2023/08/15 19:43:37 [INFO] [migrator] multi state migrator apply success!\n2023/08/15 19:43:37 [INFO] [runner] add a record to history: migration.hcl\n2023/08/15 19:43:37 [INFO] [runner] save history\n2023/08/15 19:43:37 [INFO] AWS Auth provider used: \"StaticProvider\"\n2023/08/15 19:43:37 [INFO] [runner] history saved\n```\n\n### Verify `project-one` and `project-two` have no outstanding Terraform plan diffs\n\nTerraform plan and apply `project-one`; observe there are `No changes. Infrastructure is up-to-date`:\n\n```\nmake apply-one\n```\n\nTerraform plan and apply `project-two`; observe there are `No changes. Infrastructure is up-to-date`:\n\n```\nmake apply-two\n```\n\n### Check migration history\n\n`.tfmigrate.hcl` configures `tfmigrate`'s [history](https://github.com/minamijoyo/tfmigrate#history-block),\nwhich records migration history and status as a JSON document.\n\nTo view the history JSON:\n\n```\ncurl http://localhost.localstack.cloud:4566/tfmigrate-demo/tfmigrate/history.json\n{\n    \"version\": 1,\n    \"records\": {\n        \"migration.hcl\": {\n            \"type\": \"multi_state\",\n            \"name\": \"mv_local_file_bar\",\n            \"applied_at\": \"2023-08-15T19:43:37.127656-04:00\"\n        }\n    }\n}\n```\n\n`tfmigrate list` can be used to view all migrations:\n\n```\ntfmigrate list\nmigration.hcl\n```\n\nAlternatively, `tfmigrate list --status=unapplied` reports any outstanding, unapplied migrations:\n\n```\ntfmigrate list --status=unapplied\n```\n\n### Tear down `localstack` mock AWS environment\n\n```\nmake down\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmdb%2Ftfmigrate-demo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmdb%2Ftfmigrate-demo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmdb%2Ftfmigrate-demo/lists"}