{"id":44527933,"url":"https://github.com/inspec/magic-modules","last_synced_at":"2026-02-13T18:19:53.788Z","repository":{"id":37900467,"uuid":"502952604","full_name":"inspec/magic-modules","owner":"inspec","description":null,"archived":false,"fork":false,"pushed_at":"2024-11-28T10:12:11.000Z","size":3996,"stargazers_count":0,"open_issues_count":6,"forks_count":3,"subscribers_count":4,"default_branch":"main","last_synced_at":"2024-11-28T11:20:39.540Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/inspec.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":".github/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":".github/SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-06-13T12:44:38.000Z","updated_at":"2024-11-28T10:11:38.000Z","dependencies_parsed_at":"2023-10-16T20:39:29.893Z","dependency_job_id":"894feb70-02fa-4baa-b99d-e1a6f25e5034","html_url":"https://github.com/inspec/magic-modules","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":"sa-progress/magic-modules","purl":"pkg:github/inspec/magic-modules","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/inspec%2Fmagic-modules","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/inspec%2Fmagic-modules/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/inspec%2Fmagic-modules/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/inspec%2Fmagic-modules/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/inspec","download_url":"https://codeload.github.com/inspec/magic-modules/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/inspec%2Fmagic-modules/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29414277,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-13T06:24:03.484Z","status":"ssl_error","status_checked_at":"2026-02-13T06:23:12.830Z","response_time":78,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-02-13T18:19:51.637Z","updated_at":"2026-02-13T18:19:53.775Z","avatar_url":"https://github.com/inspec.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Magic Modules\n\n\u003cimg src=\"mmv1/images/magic-modules.svg\" alt=\"Magic Modules Logo\" width=\"300\" align=\"right\" /\u003e\n\n## Overview\n\nMagic Modules is a tool used to autogenerate support in a variety of open source DevOps\ntools for Google Cloud Platform. [GCP \"resource\"](https://cloud.google.com/docs/overview/#gcp_resources)\ndefinitions are encoded in a shared data file, and that data is used to fill in\ntool-specific templates across each of the tools Magic Modules\ngenerates.\n\nMagic Modules generates GCP support for:\n\n* Terraform\n* Ansible\n\nIn addition, Magic Modules generates support for several companion\nfeatures/tools:\n\n* Terraform Google Inventory Mapper\n* Terraform in Cloud Shell\n\nImportantly, Magic Modules *isn't* full code generation. Every change is made\nmanually; more than a code generator, Magic Modules is a force multiplier for\ndevelopment. While many Magic Modules resources are defined exactly based on the\nGCP API, we use Magic Modules to preemptively solve issues across each tool by\nencoding our field-tested learnings from other tools in those definitions. In\neffect, an issue solved in one tool will be solved for each other tool.\n\n## Getting Started with Magic Modules\n\nYou can try out Magic Modules immediately with Open in Cloud Shell below; if\nyou're getting set up on a local workstation, this guide serves as a reference\nto help you get it set up.\n\n[![Open in Cloud Shell](http://gstatic.com/cloudssh/images/open-btn.svg)](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/GoogleCloudPlatform/magic-modules\u0026tutorial=mmv1/TUTORIAL.md)\n\n### Requirements\n\nTo get started, you'll need:\n\n* Ruby 3.1.0\n  * You can use `rbenv` to manage your Ruby version(s)\n* [`Bundler`](https://github.com/bundler/bundler)\n  * This can be installed with `gem install bundler`\n* If you are getting \"Too many open files\" ulimit needs to be raised.\n  * Mac OSX: `ulimit -n 1000`\n\n### Preparing Magic Modules / One-time setup\n\n**Important:**\nCompiling Magic Modules can be done directly from the `mmv1` directory within this repository.\nIn the future we will add hybrid generation with multiple generators. All the information below\npertains only to the contents of the `mmv1` directory, and commands should be executed from\nthat directory.\n\n\nTo get started right away, use the bootstrap script with:\n\n```bash\n./scripts/bootstrap\n```\n\n---\n\nOtherwise, follow the manual steps below:\n\nIf you're generating the Terraform providers (`google` and `google-beta`),\nyou'll need to check out the repo(s) you're generating in your GOPATH. For\nexample:\n\n```bash\ngit clone https://github.com/hashicorp/terraform-provider-google.git $GOPATH/src/github.com/hashicorp/terraform-provider-google\ngit clone https://github.com/hashicorp/terraform-provider-google-beta.git $GOPATH/src/github.com/hashicorp/terraform-provider-google-beta\n```\n\nMagic Modules won't work with old versions of the Terraform provider repos. If\nyou're encountering issues with vendoring and paths, make sure both MM and the\nTerraform provider are running on up to date copies of `main`.\n\nOnce you've prepared the target folders for the tools, run the following to\nfinish getting Magic Modules set up by installing the Ruby gems it needs to run:\n\n```bash\ncd mmv1\nbundle install\n```\n\nNow, you can verify you're ready with:\n\n```bash\n./scripts/doctor\n```\n\n### Generating the Terraform Providers\n\nBefore making any changes, you can compile the Terraform provider you're working\non by running the following command. If Magic Modules has been installed\ncorrectly, you'll get no errors.\n\nThe following commands should be run from the root directory of the repository.\nOUTPUT_PATH should be set to the location of your provider repository, which is\nrecommended to be inside your GOPATH.\n\n```bash\nmake terraform VERSION=ga OUTPUT_PATH=\"$GOPATH/src/github.com/hashicorp/terraform-provider-google\"\nmake terraform VERSION=beta OUTPUT_PATH=\"$GOPATH/src/github.com/hashicorp/terraform-provider-google-beta\"\n\n# Only generate a specific product (plus all common files)\nmake terraform VERSION=ga OUTPUT_PATH=\"$GOPATH/src/github.com/hashicorp/terraform-provider-google\" PRODUCT=dataproc\n```\n\nIt's worth noting that Magic Modules will only generate new files when run\nlocally. The \"Magician\"- the Magic Modules CI system- handles deletion of old\nfiles when creating PRs.\n\n#### Generating terraform-validator\n\nYou can compile terraform-validator by running the following command.\nIf Magic Modules has been installed correctly, you'll get no errors.\n\n```bash\nmake validator OUTPUT_PATH=\"/path/to/your/terraform-validator\"\n```\n\n### Making changes to resources\n\nOnce again, see the Open in Cloud Shell example above for an interactive example\nof making a Magic Modules change; this section will serve as a reference more\nthan a specific example.\n\nMagic Modules mirrors the GCP REST API; there are [products](https://github.com/GoogleCloudPlatform/magic-modules/blob/main/mmv1/api/product.rb)\nsuch as Compute or Container (GKE) that contains [resources](https://github.com/GoogleCloudPlatform/magic-modules/blob/main/mmv1/api/resource.rb),\n[GCP resources](https://cloud.google.com/docs/overview/#gcp_resources) such as\nCompute VM Instances or GKE Clusters.\n\nProducts are separate folders under [`products/`], and each folder contains a\nfile named `api.yaml` that contains the resources that make up the API\ndefinition.\n\nResources are made up of some metadata like their `\"name\"` in the API such as\nAddress or Instance, some additional metadata (see the fields in [resource.rb](https://github.com/GoogleCloudPlatform/magic-modules/blob/main/mmv1/api/resource.rb)),\nand the meat of a resource, its fields. They're represented by `properties` in\nMagic Modules, an array of [types](https://github.com/GoogleCloudPlatform/magic-modules/blob/main/mmv1/api/type.rb).\n\nAdding a new field to a resource in Magic Modules is often as easy as adding a\n`type` to the `properties` array for the resource. See [this example](https://github.com/GoogleCloudPlatform/magic-modules/pull/1126/files#diff-fb4f76e7d870258668a3beac48bf164c)\nwhere a field was added to all the tools (currently only Terraform) that support\nbeta fields.\n\n#### Tool-specific overrides\n\nWhile most small changes won't require fiddling with overrides, each tool has\n\"overrides\" when it needs to deviate from the definition in `api.yaml`. This is\noften minor differences- the naming of a field, or whether it's required or not.\n\nYou can find them under the folder for a product, with the name `{{tool}}.yaml`.\nFor example, Ansible's overrides for Cloud SQL are present at `products/sql/ansible.yaml`\n\nYou can find a full reference for each tool under `overrides/{{tool}}/resource_override.rb`\nand `overrides/{{tool}}/property_override.rb`, as well as some other tool-specific\nfunctionality.\n\n#### Making changes to handwritten files\n\nThe Google providers for Terraform have a large number of handwritten files,\nwritten before Magic Modules was used with them. While conversion is ongoing,\nmany resources are still managed by hand. You can modify handwritten files\nunder the `third_party/terraform` directory.\n\nFeatures that are only present in certain versions need to be \"guarded\" by\nwrapping those lines of code in version guards;\n\n```erb\n\u003c% unless version == 'ga' -%\u003e\n  // beta-only code\n\u003c% end -%\u003e\n```\n\n### Testing your changes\n\nOnce you've made changes to resource definition, you can run Magic Modules\nto generate changes to your tool; see\n[\"Generating the Terraform Providers\"](#generating-the-terraform-providers)\nabove if you need a refresher. Once it's generated, you should run the\ntool-specific tests as if you were submitting a PR against that tool.\n\nYou can run tests in the `{{output_folder}}` you generated the tool in.\nSee the following tool-specific documentation for more details on testing that\ntool;\n\nTool             | Testing Guide\n-----------------|--------------\nansible          | [instructions](https://docs.ansible.com/ansible/devel/dev_guide/testing.html)\nterraform        | [`google` provider testing guide](https://github.com/hashicorp/terraform-provider-google/blob/main/.github/CONTRIBUTING.md#tests)\nterraform (beta) | [`google-beta` provider testing guide](https://github.com/hashicorp/terraform-provider-google-beta/blob/main/.github/CONTRIBUTING.md#tests)\n\nDon't worry about testing every tool, only the primary tool you're making\nchanges against. The Magic Modules maintainers will ensure your changes work\nagainst each tool.\n\nIf your changes have unintended consequences in another tool, a reviewer will\ninstruct you to mark the field excluded or provide specific feedback on what\nchanges to make to the tool-specific overrides in order for them to work\ncorrectly.\n\n### Submitting a PR\n\nBefore creating a commit, if you've modified any .rb files, make sure you run\n`rake test`! That will run rubocop to ensure that the code you've written will\npass Travis.\n\nTo run rubocop automatically before committing, add a Git pre-commit hook with:\n\n```bash\ncp .github/pre-commit .git/hooks\n```\n\nOnce you've created your commit(s), you can submit the code normally as a PR in\nthe GitHub UI. The PR template includes some instructions to make sure we\ngenerate good PR messages for the tools' repo histories.\n\nOnce your PR is submitted, one of the Magic Modules maintainers will review it.\nThey'll look over the code before running the \"Magician\", the Magic Modules CI\nsystem that generates PRs against each tool. Each review pass, your reviewer\nwill run the Magician again to update the PRs against the tools.\n\nIf there are multiple tools affected, that first reviewer will be the \"primary\"\nreviewer, and for each other affected tool a maintainer for that specific tool\nwill make a pass. The primary reviewer will make it clear which other\nmaintainers need to review, and prompt them to review your code; you will\ncommunicate primarily with the first reviewer.\n\nEven when multiple tools are affected, this will generally be a quick look by\nthat maintainer with no changes needing to be made.\n\nOnce you've gotten approvals from the primary reviewer and the reviewers for\nany affected tools, the primary reviewer will merge your changes.\n\n## Glossary\n\nThe maintainers of the repository will tend to use specific jargon to describe\nconcepts related to Magic Modules; here's a quick reference of what some of\nthose terms are.\n\nTerm          | Definition\n--------------|--------------\ntool          | One of the OSS DevOps projects Magic Modules generates GCP support in\nprovider      | Synonym for tool as referred to inside the codebase\ndownstream(s) | A PR created by the Magician against a tool\nupstream      | A PR created against Magic Modules or the Magic Modules repo\nThe Magician  | The Magic Modules CI system that drives the GitHub robot `modular-magician`\n\n## Compiling MMv1 + tpgtools\n\nWe are currently developing a new generation tool for Terraform called tpgtools.\nThis relies on a [declarative client library](https://github.com/GoogleCloudPlatform/declarative-resource-client-library)\nthat handles the actuation of GCP resources. We plan to gradually move resources\nfrom being generated by the existing Magic Modules Ruby code (mmv1) to using\ntpgtools. While we move resources over there will be a period of time when\nboth generators are in use. To assist with generation we have a series of `make`\ntargets that will run the compilers in tandem to generate the Terraform provider.\n\nSample Usage to compile at beta:\n`make OUTPUT_PATH=/path/to/terraform-provider-google-beta VERSION=beta`\n\nTarget single product:\n`make OUTPUT_PATH=/path/to/terraform-provider-google VERSION=ga PRODUCT=compute`\n\nTarget single resource\n`make OUTPUT_PATH=/path/to/terraform-provider-google VERSION=ga PRODUCT=compute RESOURCE=image`\n\nFor more advanced usage of mmv1 compiler flags, please execute the compiler directly\nfrom within the mmv1 directory.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finspec%2Fmagic-modules","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Finspec%2Fmagic-modules","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finspec%2Fmagic-modules/lists"}