{"id":15654020,"url":"https://github.com/emicklei/gmig","last_synced_at":"2025-06-28T01:34:57.552Z","repository":{"id":29366867,"uuid":"121479314","full_name":"emicklei/gmig","owner":"emicklei","description":"Google Cloud Platform migrations tool for infrastructure-as-code","archived":false,"fork":false,"pushed_at":"2025-03-26T13:51:40.000Z","size":2792,"stargazers_count":27,"open_issues_count":4,"forks_count":10,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-15T01:07:56.639Z","etag":null,"topics":["gcloud","gcloud-migrations","gcp","golang","google-cloud-platform","iam-policy","infrastructure-as-code","migrations","tool"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/emicklei.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGES.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-02-14T06:38:26.000Z","updated_at":"2025-03-26T13:50:58.000Z","dependencies_parsed_at":"2024-04-05T21:30:02.019Z","dependency_job_id":"a67fdc95-1b21-405d-afa5-36ce8c7c1d5b","html_url":"https://github.com/emicklei/gmig","commit_stats":null,"previous_names":[],"tags_count":66,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/emicklei%2Fgmig","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/emicklei%2Fgmig/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/emicklei%2Fgmig/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/emicklei%2Fgmig/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/emicklei","download_url":"https://codeload.github.com/emicklei/gmig/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248986312,"owners_count":21194025,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gcloud","gcloud-migrations","gcp","golang","google-cloud-platform","iam-policy","infrastructure-as-code","migrations","tool"],"created_at":"2024-10-03T12:48:59.546Z","updated_at":"2025-04-15T01:08:05.726Z","avatar_url":"https://github.com/emicklei.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# gmig - GCP migrations\n\npronounced as `gimmick`.\n\n[![Build Status](https://app.travis-ci.com/emicklei/gmig.svg?branch=master)](https://app.travis-ci.com/github/emicklei/gmig)\n[![Go Report Card](https://goreportcard.com/badge/github.com/emicklei/gmig)](https://goreportcard.com/report/github.com/emicklei/gmig)\n[![GoDoc](https://godoc.org/github.com/emicklei/gmig?status.svg)](https://godoc.org/github.com/emicklei/gmig)\n\nManage Google Cloud Platform (GCP) infrastructure using migrations that describe incremental changes such as additions or deletions of resources.\nThis work is inspired by MyBatis migrations for SQL database setup.\n\n[Introduction blog post](http://ernestmicklei.com/2018/03/introducing-gmig-infrastructure-as-code-for-gcp/)\n\nYour `gmig` infrastructure is basically a folder with incremental change files, each with a timestamp prefix (for sort ordering) and readable name.\n\n    /010_create_some_account.yaml\n    /015_add_permissions_to_some_account.yaml\n    /my-gcp-production-project\n        gmig.yaml\n\nEach change is a single YAML file with one or more shell commands that change infrastructure for a project.\n\n    # add loadrunner service account\n\n    do:\n    - gcloud iam service-accounts create loadrunner --display-name \"LoadRunner\"\n\n    undo:\n    - gcloud iam service-accounts delete loadrunner\n\nA change must have at least a `do` section and optionally an `undo` section.\nThe `do` section typically has a list of gcloud commands that create resources but any available tool can be used.\nAll lines will be executed at once using a single temporary shell script so you can use shell variables to simplify each section.\nThe `undo` section typically has an ordered list of gcloud commands that deletes the same resources (in reverse order if relevant).\nEach command in each section can use the following environment variables: `$PROJECT`,`$REGION`,`$ZONE`,`$GMIG_CONFIG_DIR`, and any additional environment variables populated from the target configuration (see `env` section in the configuration below).\n\n## State\n\nInformation about the last applied migration to a project is stored as a Google Storage Bucket object.\nTherefore, usage of this tool requires you to have create a Bucket and set the permissions (Storage Writer) accordingly.\nTo view the current state of your infrastructure related to each migration, you can add the `view` section to the YAML file, such as:\n\n    # add loadrunner service account\n\n    do:\n    - gcloud iam service-accounts create loadrunner --display-name \"LoadRunner\"\n\n    undo:\n    - gcloud iam service-accounts delete loadrunner\n\n    view:\n    - gcloud iam service-accounts describe loadrunner\n\nand use the `view` subcommand.\n\n## Conditional migration\n\nCommands (do,undo,view) can be made conditional by adding an `if` section.\nYou can only use custom environment variables and configuration parameters (PROJECT,ZONE,REGION) in expressions.\nIf the expression evaluates to true then the `do` (up), `undo` (down) and `view` (view) commands are executed.\n\n    if: PROJECT == \"your-project-id\"\n    do:\n    - gcloud condig list\n\nor with combinations:\n\n    if: (PROJECT == \"your-project-id\") \u0026\u0026 (ZONE == \"my-zone\")\n    do:\n    - gcloud condig list\n\nFor available operators, see [Language-Definition](https://github.com/antonmedv/expr/blob/master/docs/Language-Definition.md)\n\n## Help\n\n    NAME:\n    gmig - Google Cloud Platform infrastructure migration tool\n\n    USAGE:\n    gmig [global options] command [command options] [arguments...]\n\n    COMMANDS:\n        init     Create the initial configuration, if absent.\n        new      Create a new migration file from a template using a generated timestamp and a given title.\n        up       Runs the do section of all pending migrations in order, one after the other.\n                 If a migration file is specified then stop after applying that one.\n        down     Runs the undo section of the last applied migration only.\n        down-all Runs the undo section of all applied migrations.\n        plan     Log commands of the do section of all pending migrations in order, one after the other.\n        status   List all migrations with details compared to the current state.\n        view     Runs the view section of all applied migrations to see the current state reported by your infrastructure.\n        force    state | do | undo\n        util     create-named-port | delete-named-port\n        export   project-iam-policy | storage-iam-policy\n        help, h  Shows a list of commands or help for one command\n\n    GLOBAL OPTIONS:\n    -q                   quiet mode, accept any prompt\n    -v                   verbose logging\n    --help, -h           show help\n    --print-version, -V  print only the version\n\n## Getting started\n\n### Installation\n\nYou need to compile it using the [Go SDK](https://golang.org/dl/).\n\n    go install github.com/emicklei/gmig@latest\n\n### init \\\u003cpath\u003e\n\nPrepares your setup for working with migrations by creating a `gmig.json` file in a target folder.\n\n    gmig init my-gcp-production-project\n\nThen your filesystem will have:\n\n    /my-gcp-production-project/\n        gmig.yaml\n\nYou must change the file `gmig.yaml` to set the Project and Bucket name.\n\n    # gmig configuration file\n    #\n    # Google Cloud Platform migrations tool for infrastructure-as-code. See https://github.com/emicklei/gmig.\n\n    # [project] must be the Google Cloud Project ID where the infrastructure is created.\n    # Its value is available as $PROJECT in your migrations.\n    #\n    # Required by gmig.\n    project: my-project\n\n    # [region] must be a valid GCP region. See https://cloud.google.com/compute/docs/regions-zones/\n    # A region is a specific geographical location where you can run your resources.\n    # Its value is available as $REGION in your migrations.\n    #\n    # Not required by gmig but some gcloud and gsutil commands do require it.\n    # region: europe-west1\n\n    # [zone] must be a valid GCP zone. See https://cloud.google.com/compute/docs/regions-zones/\n    # Each region has one or more zones; most regions have three or more zones.\n    # Its value is available as $ZONE in your migrations.\n    #\n    # Not required by gmig but some gcloud and gsutil commands do require it.\n    # zone: europe-west1-b\n\n    # [bucket] must be a valid GCP Storage bucket.\n    # A Google Storage Bucket is used to store information (object) about the last applied migration.\n    # Bucket can contain multiple objects from multiple applications. Make sure the [state] is different for each app.\n    #\n    # Required by gmig.\n    bucket: my-bucket\n\n    # [state] is the name of the object that hold information about the last applied migration.\n    # Required by gmig.\n    state: myapp-gmig-last-migration\n\n    # [env] are additional environment values that are available to each section of a migration file.\n    # This can be used to create migrations that are independent of the target project.\n    # By convention, use capitalized words for keys.\n    # In the example, \"myapp-cluster\" is available as $K8S_CLUSTER in your migrations.\n    #\n    # Not required by gmig.\n    env:\n      K8S_CLUSTER: myapp-cluster\n\nIf you decide to store state files of different projects in one Bucket then set the state object name to reflect this, eg. `myproject-gmig-state`.\nIf you want to apply the same migrations to different regions/zones then choose a target folder name to reflect this, eg. `my-gcp-production-project-us-east`. Values for `region` and `zone` are required if you want to create Compute Engine resources. The `env` map can be used to parameterize commands in your migrations. In the example, all commands will have access to the value of `$K8S_CLUSTER`.\n\n### new \\\u003ctitle\u003e\n\nCreates a new migration for you to describe a change to the current state of infrastructure.\n\n    gmig new \"add storage view role to cloudbuild account\"\n\nUsing a combination of the options `--do`, `--undo` and `--view`, you can set the commands directly for the new migration.\n\n### status \\\u003cpath\u003e [--migrations folder]\n\nList all migrations with an indicator (applied,pending) whether is has been applied or not.\n\n    gmig status my-gcp-production-project/\n\nRun this command in the directory where all migrations are stored. Use `--migrations` for a different location.\n\n### plan \\\u003cpath\u003e [stop] [--migrations folder]\n\nLog commands of the `do` section of all pending migrations in order, one after the other.\nIf `stop` is given, then stop after that migration file.\n\n### up \\\u003cpath\u003e [stop] [--migrations folder]\n\nExecutes the `do` section of each pending migration compared to the last applied change to the infrastructure.\nIf `stop` is given, then stop after that migration file.\nUpon each completed migration, the `gmig-last-migration` object is updated in the bucket.\n\n    gmig up my-gcp-production-project\n\n### down \\\u003cpath\u003e [--migrations folder]\n\nExecutes one `undo` section of the last applied change to the infrastructure.\nIf completed then update the `gmig-last-migration` object.\n\n    gmig down my-gcp-production-project\n\n### down-all \\\u003cpath\u003e [--migrations folder]\n\nExecutes `undo` section of all applied change to the infrastructure.\nUpdates the `gmig-last-migration` object after each successfull step.\n\n    gmig down-all my-gcp-production-project\n\n### view \\\u003cpath\u003e [migration file] [--migrations folder]\n\nExecutes the `view` section of each applied migration to the infrastructure.\nIf `migration file` is given then run that view only.\n\n    gmig view my-gcp-production-project\n\n### template [-w] source-file\n\nProcesses the source-file as a Go template and write the result to stdout.\nIf the `-w` is given then rewrite the source with the processed content.\nThe following functions are available:\n\n#### env\n\nThis function takes the first argument and does a lookup in the available OS environment values.\nExample of a configuration snippet that needs the environment dependent value for $PROJECT.\n\n    project: {{ env \"PROJECT\" }}\n\nExample:\n\n    gmig template some-config.template.yaml \u003e some-config.yaml\n\n## Export existing infrastructure\n\nExporting migrations from existing infrastructure is useful when you start working with `gmig` but do not want to start from scratch.\nSeveral sub commands are (or will become) available to inspect a project and export migrations to reflect the current state.\nAfter marking the current state in `gmig` (using `force-state`), new migrations can be added that will bring your infrastructure to the next state.\nThe generated migration can ofcourse also be used to just copy commands to your own migration.\n\n### export project-iam-policy \\\u003cpath\u003e\n\nGenerate a new migration by reading all the IAM policy bindings from the current infrastructure of the project.\n\n    gmig -v export project-iam-policy my-project/\n\n### export storage-iam-policy \\\u003cpath\u003e\n\nGenerate a new migration by reading all the IAM policy bindings, per Google Storage Bucket owned by the project.\n\n    gmig -v export storage-iam-policy my-project/\n\n## Working around migrations\n\nSometimes you need to fix things because you made a mistake or want to reorganise your work. Use the `force` and confirm your action.\n\n### force state \\\u003cpath\u003e \\\u003cfilename\u003e\n\nExplicitly set the state for the target to the last applied filename. This command can be useful if you need to work from existing infrastructure. Effectively, this filename is written to the bucket object.\nUse this command with care!.\n\n    gmig force state my-gcp-production-project 010_create_some_account.yaml\n\n### force do \\\u003cpath\u003e \\\u003cfilename\u003e\n\nExplicitly run the commands in the `do` section of a given migration filename.\nThe `gmig-last-migration` object is `not` updated in the bucket.\nUse this command with care!.\n\n    gmig force do my-gcp-production-project 010_create_some_account.yaml\n\n### force undo \\\u003cpath\u003e \\\u003cfilename\u003e\n\nExplicitly run the commands in the `undo` section of a given migration filename.\nThe `gmig-last-migration` object is `not` updated in the bucket.\nUse this command with care!.\n\n    gmig force undo my-gcp-production-project 010_create_some_account.yaml\n\n## export-env \\\u003cpath\u003e\n\nExport all available environment variable from the configuration file and also export $PROJECT, $REGION and $ZONE\nUse this command with care!.\n\n    eval $(gmig export-env my-gcp-production-project)\n\n## GCP utilities\n\n### util create-named-port \\\u003cinstance-group\u003e \\\u003cname:port\u003e\n\nThe Cloud SDK has a command to [set-named-ports](https://cloud.google.com/sdk/gcloud/reference/compute/instance-groups/set-named-ports) but not a command to add or delete a single name:port mapping. To simplify the migration command for creating a name:port mapping, this gmig util command is added.\nFirst it calls `get-named-ports` to retrieve all existing mappings. Then it will call `set-named-ports` with the new mapping unless it already exists.\n\n### util delete-named-port \\\u003cinstance-group\u003e \\\u003cname:port\u003e\n\nThe Cloud SDK has a command to [set-named-ports](https://cloud.google.com/sdk/gcloud/reference/compute/instance-groups/set-named-ports) but not a command to add or delete a single name:port mapping. To simplify the migration command for deleting a name:port mapping, this `gmig` util command is added.\nFirst it calls `get-named-ports` to retrieve all existing mappings. Then it will call `set-named-ports` without the mapping.\n\n### util add-path-rules-to-path-matcher [config folder] -url-map [url-map-name] -service [backend-service-name] -path-matcher [path-matcher-name] -paths \"/v1/path/_, /v1/otherpath/_\"\n\nThe Cloud SDK has a command to [add a patch matcher](https://cloud.google.com/sdk/gcloud/reference/compute/url-maps/add-path-matcher) with a set of paths but not a command update the path rules of an existing path matcher in the url map. To write a migration that changes the set of paths (add,remove), this `gmig` util command is added.\nFirst is [exports](https://cloud.google.com/sdk/gcloud/reference/compute/url-maps/export) an URL map, updates the paths of the rules of a path-matcher, then imports the changed URL map. Because this migration is changing a regional resource which is typically shared by multiple services, the patching of the URL map is executed using a global lock (using the Bucket from the config).\n\n## Examples\n\nThis repository has a number of [examples](https://github.com/emicklei/gmig/tree/master/examples) of migrations.\n\n\u0026copy; 2022, ernestmicklei.com. MIT License. Contributions welcome.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Femicklei%2Fgmig","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Femicklei%2Fgmig","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Femicklei%2Fgmig/lists"}