{"id":26140728,"url":"https://github.com/dpguthrie/dbt-cloud-migration-tool","last_synced_at":"2025-06-26T18:33:27.916Z","repository":{"id":274891175,"uuid":"924404340","full_name":"dpguthrie/dbt-cloud-migration-tool","owner":"dpguthrie","description":"Tool that helps migrate accounts","archived":false,"fork":false,"pushed_at":"2025-01-29T23:59:34.000Z","size":0,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-30T00:26:57.240Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dpguthrie.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-01-29T23:52:25.000Z","updated_at":"2025-01-29T23:59:37.000Z","dependencies_parsed_at":"2025-01-30T00:37:24.159Z","dependency_job_id":null,"html_url":"https://github.com/dpguthrie/dbt-cloud-migration-tool","commit_stats":null,"previous_names":["dpguthrie/dbt-cloud-migration-tool"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpguthrie%2Fdbt-cloud-migration-tool","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpguthrie%2Fdbt-cloud-migration-tool/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpguthrie%2Fdbt-cloud-migration-tool/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpguthrie%2Fdbt-cloud-migration-tool/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dpguthrie","download_url":"https://codeload.github.com/dpguthrie/dbt-cloud-migration-tool/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242961724,"owners_count":20213316,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-11T02:56:24.990Z","updated_at":"2025-03-11T02:56:25.729Z","avatar_url":"https://github.com/dpguthrie.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# dbt Cloud Migration Tool\n\nThis tool facilitates the migration of dbt Cloud resources between different instances/regions. It uses [dbtcloud-terraforming](https://github.com/dbt-labs/dbtcloud-terraforming) to export existing configurations and [terraform-provider-dbtcloud](https://github.com/dbt-labs/terraform-provider-dbtcloud) to apply them to the target instance.\n\n## Prerequisites\n\n- Terraform \u003e= 1.0\n- Access to both source and target dbt Cloud instances with admin privileges\n- Service tokens for both source and target instances\n- [dbtcloud-terraforming](https://github.com/dbt-labs/dbtcloud-terraforming) tool\n- [terraform-provider-dbtcloud](https://github.com/dbt-labs/terraform-provider-dbtcloud)\n\n## Migration Process\n\n### 1. Install dependencies\n\n#### Install Terraform\n\n**Mac / Linux**\n```bash\nbrew tap hashicorp/tap \u0026\u0026 brew install hashicorp/tap/terraform\n```\n**Windows**\n```powershell\nchoco install terraform\n```\n\n#### Install `dbtcloud-terraforming`\n\n**Mac / Linux**\n```bash\nbrew install dbt-labs/dbt-cli/dbtcloud-terraforming\n```\n**Windows**\n\nCheck out the README for [dbtcloud-terraforming](https://github.com/dbt-labs/dbtcloud-terraforming) for instructions on how to install for Windows.\n\n### 2. Export Configuration from Source Instance\n\nFirst, set up environment variables for your source instance:\n```bash\nexport DBT_CLOUD_HOST_URL=\"https://YOUR_SOURCE_REGION.getdbt.com/api\"\nexport DBT_CLOUD_TOKEN=\"your_source_service_token\"\nexport DBT_CLOUD_ACCOUNT_ID=your_source_account_id\n```\n\nReview and modify `resource_types.txt` to specify which resources to export. See the [dbtcloud-terraforming](https://github.com/dbt-labs/dbtcloud-terraforming) README for available resource types.\n\nThen export the configuration:\n```bash\n./migrate.sh\n```\n\nOr if you haven't set environment variables, provide them as command line arguments:\n```bash\n./migrate.sh \\\n  --source-host \"https://cloud.getdbt.com/api\" \\\n  --source-token \"your_source_service_token\" \\\n  --source-account-id \"your_source_account_id\"\n```\n\n### 3. Review and Modify Configuration\n\n1. Check the generated `target/resources.tf` file\n   - Environment references (`deferring_environment_id`) are automatically updated to use Terraform resource references\n   - Unsupported credential types are automatically set to `null`\n   - A backup of the original file is saved as `resources.tf.bak`\n2. Make any necessary adjustments:\n   - Modify or remove `credential_id` values that were not automatically handled\n   - Adjust any environment-specific settings\n\n### 4. Apply Configuration to Target Instance\n\n1. Set up target instance configuration:\n```bash\ncd target\ncp terraform.tfvars.example terraform.tfvars\n# Edit terraform.tfvars with your target instance details\n```\n\n2. Initialize Terraform:\n```bash\nterraform init\n```\n\n3. Preview changes:\n```bash\nterraform plan\n```\n\n4. Apply changes:\n```bash\nterraform apply\n```\n\n5. To remove resources:\n```bash\nterraform destroy\n```\n\nFor more advanced Terraform usage, refer to the [Terraform CLI documentation](https://www.terraform.io/cli/commands).\n\n### 5. Post-Migration Tasks\n\n1. **Verify Repository Connections**:\n   - Reconnect Git repositories\n   - Verify branch configurations\n   - Test repository access\n\n2. **Update Credentials**:\n   - Add repository / warehouse credentials\n   - Configure environment variables\n   - Set up service account tokens\n\n3. **Verify User Access**:\n   - Ensure users have appropriate access\n   - Test SSO if applicable\n   - Verify group memberships\n\n4. **Test Jobs and Environments**:\n   - Run test jobs\n   - Verify environment configurations\n   - Check scheduler settings\n\n## Known Limitations\n\n1. **Credentials and Secrets**:\n   - Repository credentials (SSH keys, tokens) must be manually recreated\n   - Service account tokens must be regenerated\n   - Environment variables containing secrets must be manually recreated\n\n2. **Project Data**:\n   - Actual project data/artifacts are not migrated\n   - Job run history is not preserved\n   - Job artifacts and logs are not transferred\n\n3. **User Management**:\n   - User accounts must exist in the target instance\n   - User permissions must be manually verified\n   - SSO configurations must be set up separately\n\n## Usage Example\n\n```bash\n# 1. Clone this repository\ngit clone https://github.com/dpguthrie/dbt-cloud-migration-tool\ncd dbt-cloud-migration-tool\n\n# 2. Install dependencies\nbrew tap hashicorp/tap \u0026\u0026 brew install hashicorp/tap/terraform\nbrew install dbt-labs/dbt-cli/dbtcloud-terraforming\n\n# 3. Set up environment variables for source instance\nexport DBT_CLOUD_HOST_URL=\"https://cloud.getdbt.com/api\"\nexport DBT_CLOUD_TOKEN=\"my_service_token\"\nexport DBT_CLOUD_ACCOUNT_ID=\"12345\"\n\n# 4. Export source configuration\n./migrate.sh\n\n# 5. Configure target instance\ncd target\ncp terraform.tfvars.example terraform.tfvars\n# Edit terraform.tfvars with your target instance details\n\n# 6. Apply changes using Terraform directly\nterraform init\nterraform plan\nterraform apply\n```\n\n## Security Considerations\n\n1. **API Tokens**:\n   - Use service account tokens with limited scope\n   - Rotate tokens after migration\n   - Never commit tokens to version control\n\n2. **Sensitive Data**:\n   - Manually handle all secrets and credentials\n   - Use environment variables for sensitive values\n   - Verify security settings in target instance\n\n3. **Access Control**:\n   - Review and verify all permission assignments\n   - Implement principle of least privilege\n   - Document all access changes\n\n## Troubleshooting\n\n### Common Issues\n\n1. **Resource Conflicts**:\n   - Ensure unique resource names in target instance\n   - Check for naming conflicts before migration\n   - Use resource mapping file for resolution\n\n2. **Connection Issues**:\n   - Verify API access and tokens\n   - Check network connectivity\n   - Confirm region endpoints\n\n3. **Permission Errors**:\n   - Verify admin access in both instances\n   - Check token permissions\n   - Review resource-specific access requirements\n\n## Contributing\n\nContributions are welcome! Please read our [Contributing Guidelines](CONTRIBUTING.md) for details on how to submit pull requests.\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details. ","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdpguthrie%2Fdbt-cloud-migration-tool","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdpguthrie%2Fdbt-cloud-migration-tool","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdpguthrie%2Fdbt-cloud-migration-tool/lists"}