{"id":14069685,"url":"https://github.com/jthegedus/meta-cloud-builders","last_synced_at":"2025-08-22T05:39:27.512Z","repository":{"id":83212244,"uuid":"219237388","full_name":"jthegedus/meta-cloud-builders","owner":"jthegedus","description":"Build custom GCP Cloud Build builders from a config file. Deploy all your GCP Cloud Build Triggers from config files.","archived":false,"fork":false,"pushed_at":"2023-01-29T12:42:40.000Z","size":2867,"stargazers_count":6,"open_issues_count":4,"forks_count":2,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-28T21:14:30.914Z","etag":null,"topics":["build","cloud-builders","docker","google-cloud-build","google-cloud-platform","google-containers"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jthegedus.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null},"funding":{"github":["jthegedus"]}},"created_at":"2019-11-03T01:43:01.000Z","updated_at":"2023-05-10T14:00:40.000Z","dependencies_parsed_at":"2023-03-01T20:31:08.794Z","dependency_job_id":null,"html_url":"https://github.com/jthegedus/meta-cloud-builders","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jthegedus%2Fmeta-cloud-builders","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jthegedus%2Fmeta-cloud-builders/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jthegedus%2Fmeta-cloud-builders/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jthegedus%2Fmeta-cloud-builders/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jthegedus","download_url":"https://codeload.github.com/jthegedus/meta-cloud-builders/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244022650,"owners_count":20385134,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["build","cloud-builders","docker","google-cloud-build","google-cloud-platform","google-containers"],"created_at":"2024-08-13T07:07:08.475Z","updated_at":"2025-03-19T22:30:39.150Z","avatar_url":"https://github.com/jthegedus.png","language":"Shell","funding_links":["https://github.com/sponsors/jthegedus"],"categories":["Shell"],"sub_categories":[],"readme":"# Meta Cloud Builders\n\n\u003e Supercharge your Cloud Building with these custom Builders!\n\n- [`meta-cloud-builder`](#meta-cloud-builder): Build custom Cloud Build images (builders) from a config file.\n- [`meta-triggers`](#meta-triggers): Deploy all your Cloud Build Triggers from config files (`yaml` or `json`).\n- [Contributing](#contributing)\n- [Todo](#todo)\n- [Testing](#testing)\n- [License](#license)\n\n## meta-cloud-builder\n\n\u003e Build custom Cloud Build images (builders) from a config file.\n\nManually build this image into your project once (I suggest Cloud Shell).\n\n```shell\n# clone\ngit clone https://github.com/jthegedus/meta-cloud-builders\n# build\ncd meta-cloud-builders/meta-cloud-builder\ngcloud builds submit .\n# validate\ngcloud container images list --filter meta-cloud-builder\n```\n\nCreate a `.yaml` or `.json` file with repos and the cloud-builder you wish to build into your project:\n\n```yaml\n# .cicd/builders/custom-builders.yaml\n- repo: https://github.com/jthegedus/meta-cloud-builder\n  builders:\n    - meta-cloud-builder\n    - meta-triggers\n- repo: https://github.com/GoogleCloudPlatform/cloud-builders-community\n  builders:\n    - cancelot\n    - cache\n```\n\nthen pass this config file into the `meta-cloud-builder` step as an arg:\n\n```yaml\n# .cicd/builders.cloudbuild.yaml\nsteps:\n  - name: gcr.io/$PROJECT_ID/meta-cloud-builder\n    id: \"build custom cloud build builder(s) from a config file\"\n    waitFor:\n      - \"-\"\n    args:\n      - \".cicd/builders/custom-builders.yaml\"\n      - \"--async\"\ntags:\n  - cloud-builders\n```\n\nThis builder then invokes `gcloud builds submit ...` for cloud-builders defined in the configuration file. You can pass any `gcloud builds submit` flags to the builder except the `--config` and `SOURCE` aspects. [See the flags here](https://cloud.google.com/sdk/gcloud/reference/builds/submit).\n\n\u003cdetails\u003e\n\u003csummary\u003eAutomate building your builders with a Trigger\u003c/summary\u003e\n\n### Triggers\n\nNow with [Cloud Build Triggers being created via `.yaml` config](https://cloud.google.com/blog/products/devops-sre/cloud-build-brings-advanced-cicd-capabilities-to-github) we can run this `builders.cloudbuild.yaml` whenever we make a change to this config file.\n\n```yaml\n# .cicd/triggers/builders.trigger.yaml\nname: cloud-builders\ndescription: Build custom Cloud Build builders into my gcr project on change\ngithub:\n  owner: \u003corg/user_name\u003e\n  name: \u003crepo_name\u003e\n  push:\n    branch: master\nfilename: .cicd/builders.cloudbuild.yaml\nincludedFiles:\n  - .cicd/builders.cloudbuild.yaml      # the Cloud Build job file\n  - .cicd/builders/custom-builders.yaml # the config file\n```\n\nFor security purposes, I would suggest only running this trigger on pushes to `master` so that changes must be approved before they are applied.\n\nImport the Trigger:\n\n```shell\ngcloud beta builds triggers import --source=.cicd/triggers/builders.trigger.yaml\n```\n\n### Schedule\n\nWIP: the message-body might need changing to run a GitHub-based Trigger - see this [Cloud Build Issue](https://issuetracker.google.com/issues/142550612).\n\nWith custom Cloud Builders you are almost always going to want the latest images from the source. Since we cannot trigger off of changes to external repos, we can at least rebuild these containers on a regular basis, say daily or weekly.\n\n```shell\ngcloud scheduler jobs create http \\\n  build-custom-cloud-builders \\\n  --description=\"Build custom cloud-builders on a schedule\" \\\n  --schedule=\"0 0 * * SUN\" \\\n  --time-zone=\"AEST\"\n  --http-method=\"POST\" \\\n  --uri=https://cloudbuild.googleapis.com/v1/projects/[PROJECTID]/triggers/[TRIGGERID]:run \\\n  --message-body={\"branchName\": \"master\"} \\\n  --oauth-service-account-email=[EMAIL_ADDRESS]@appspot.gserviceaccount.com\n```\n\nJust fill in `PROJECTID`, `TRIGGERID` and create a Service Account and fill in the `EMAIL_ADDRESS` accordingly.\n\nSuggested schedule intervals:\n- daily: `0 0 * * *`\n- every sunday: `0 0 * * SUN`\n\n\u003c/details\u003e\n\n## meta-triggers\n\n\u003e Deploy all your Cloud Build Triggers from config files (`yaml` or `json`).\n\nPair this builder with a Trigger and have it run on changes to your Trigger config files.\n\nManually build this image into your project once.\n\n```shell\n# clone\ngit clone https://github.com/jthegedus/meta-cloud-builders\n# build\ncd meta-cloud-builders/meta-triggers\ngcloud builds submit .\n# validate\ngcloud container images list --filter meta-triggers\n```\n\nThe builder runs by recursively checking a directory for files matching a sufix. The defaults are:\n\n- dir: `.` - the repo root\n- suffix: `.*\\.trigger\\.(json|yaml)` - a Grep -E regex\n\nYou can override the params, but they are positionaly args so to override the `suffix` you must override the `dir`.\n\n```yaml\n# .cicd/apply-triggers.cloudbuild.yaml\nsteps:\n  - name: gcr.io/$PROJECT_ID/meta-triggers\n    id: \"Watch for changes to all Cloud Build Triggers in provided dir\"\n    waitFor:\n      - \"-\"\n    args:\n      - \".cicd/\"\n      - \".*\\\\.trigger\\\\.(json|yaml)\"\n```\n\nNow we want to run this Cloud Build Job any time a file in the `--dir` changes. So we setup this trigger once:\n\n```yaml\n# .cicd/triggers/meta-trigger.trigger.yaml\nname: meta-trigger\ndescription: \"Trigger to apply Triggers on change\"\ngithub:\n  owner: \u003corg/user_name\u003e\n  name: \u003crepo_name\u003e\n  push:\n    branch: master\nfilename: .cicd/apply-triggers.cloudbuild.yaml\nincludedFiles:\n  - .cicd/**\n```\n\nFor security purposes, I would suggest only running this trigger on pushes to `master` so that changes must be approved before they are applied.\n\n\u003e NB: this will perform a `gcloud beta builds triggers import --source=\"\"` of Trigger configs that do not change.\n\n🚨 Trigger `includedFiles` uses [Go Regex with additions](https://cloud.google.com/cloud-build/docs/running-builds/create-manage-triggers#build_trigger), whereas the regex used by meta-triggers is Grep -E Perl regex. Take care.\n\n## Contributing\n\nContributions of any kind are welcome! Feel free to tackle something from the todo list below.\n\n### Todo\n\n- [ ] meta-triggers: only update triggers that did change by diffing the existing config against the config in files.\n- [ ] meta-triggers: fix the [`run-meta-triggers.trigger.yaml`](.cicd/triggers/run-meta-triggers.trigger.yaml) includedFiles regex to include any file in the repo that matches the `*.trigger.(json|yaml)` regex.\n- general improvements also welcome, though focus on readability.\n\n### Testing\n\n- meta-cloud-builder: run from the repo root dir to test:\n\n```shell\ngcloud builds submit ./meta-cloud-builder/test/ --config=./meta-cloud-builder/test/cloudbuild.yaml\n```\n\n- meta-triggers: run from the repo root dir to test:\n\n```shell\ngcloud builds submit ./meta-triggers/test/ --config=./meta-triggers/test/cloudbuild.yaml\n```\n\n## License\n\n[MIT License](https://github.com/jthegedus/meta-cloud-builder/blob/master/LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjthegedus%2Fmeta-cloud-builders","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjthegedus%2Fmeta-cloud-builders","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjthegedus%2Fmeta-cloud-builders/lists"}