{"id":22700239,"url":"https://github.com/simp/puppetsync","last_synced_at":"2025-10-18T20:31:34.114Z","repository":{"id":54299316,"uuid":"256260809","full_name":"simp/puppetsync","owner":"simp","description":"Bolt project to standardize assets across many GitHub repos and PR the changes (Should probably be called \"boltsync\")","archived":false,"fork":false,"pushed_at":"2024-02-07T16:35:31.000Z","size":550,"stargazers_count":1,"open_issues_count":1,"forks_count":4,"subscribers_count":8,"default_branch":"main","last_synced_at":"2024-04-19T11:05:03.507Z","etag":null,"topics":["baseline","bolt","git","puppet"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/simp.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-04-16T15:50:31.000Z","updated_at":"2024-05-28T18:53:14.987Z","dependencies_parsed_at":"2023-01-29T05:00:50.664Z","dependency_job_id":"6fd4b6fc-0dc2-463d-8e73-b76d1cd8621c","html_url":"https://github.com/simp/puppetsync","commit_stats":null,"previous_names":[],"tags_count":17,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simp%2Fpuppetsync","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simp%2Fpuppetsync/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simp%2Fpuppetsync/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simp%2Fpuppetsync/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/simp","download_url":"https://codeload.github.com/simp/puppetsync/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":229013251,"owners_count":18006191,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["baseline","bolt","git","puppet"],"created_at":"2024-12-10T06:10:38.610Z","updated_at":"2025-10-18T20:31:34.055Z","avatar_url":"https://github.com/simp.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# puppetsync\n\n\u003c!-- vim-markdown-toc GFM --\u003e\n\n* [Description](#description)\n* [Setup](#setup)\n  * [Requirements](#requirements)\n  * [Quickstart](#quickstart)\n    * [Initial setup](#initial-setup)\n    * [Preparing to run `puppetsync`](#preparing-to-run-puppetsync)\n    * [Running `puppetsync`](#running-puppetsync)\n    * [Running `puppetsync::approve_github_prs`](#running-puppetsyncapprove_github_prs)\n    * [Running `puppetsync::merge_github_prs`](#running-puppetsyncmerge_github_prs)\n* [Usage](#usage)\n  * [Syncing repos](#syncing-repos)\n  * [Inspecting pipeline stages](#inspecting-pipeline-stages)\n* [Reference](#reference)\n  * [Environment variables](#environment-variables)\n  * [Puppetsync `config`](#puppetsync-config)\n  * [Puppetsync `repolist`](#puppetsync-repolist)\n  * [Plans](#plans)\n    * [`puppetsync`](#puppetsync)\n    * [`puppetsync::approve_github_prs`](#puppetsyncapprove_github_prs)\n    * [`puppetsync::merge_github_prs`](#puppetsyncmerge_github_prs)\n  * [Manually installing dependencies](#manually-installing-dependencies)\n* [Troubleshooting](#troubleshooting)\n  * [Error: `Ignoring \u003cx\u003e because its extensions are not built.`](#error-ignoring-x-because-its-extensions-are-not-built)\n* [Limitations](#limitations)\n\n\u003c!-- vim-markdown-toc --\u003e\n\n## Description\n\nRun [Puppet Bolt Plans][bolt] to manage your GitHub repos' code like infrastructure-as-code!\n\n  * Enforce a \"baseline\" across multiple GitHub repos, using [Puppet][puppet] and [Bolt][bolt] tasks\n  * Automate the CM/workflow for each change: Jira, git commit messages, GitHub forks \u0026 PRs\n  * Separate plans to submit, approve, and merge GitHub PRs, so different roles can run them (if required by CM policy)\n\n\n![Puppetsync Plans Overview](assets/puppetsync_plans_overview.png)\n\n\n## Setup\n\n### Requirements\n\n* [Puppet Bolt 3.0+][bolt], installed from an [OS package][bolt-install] (don't\n  use the RubyGem)\n* The `git` command must be available\n  * SSH + ssh-agent must be set up to push changes\n* Some specific [environment variables](#environment-variables) are required\n  to handle API authentication (e.g., GitHub)\n* Runtime dependencies (installed by `./Rakefile install`)\n  * Puppet modules (defined in bolt project's `bolt-project.yaml`):\n  * Ruby Gems (defined in `gem.deps.rb`): octokit, jira-ruby, etc\n\n\n### Quickstart\n\n#### Initial setup\n\n1. Before running any plans, from the top level of this repository:\n\n   ```sh\n   command -v rvm \u0026\u0026 rvm use system    # Make sure you're using packaged `bolt`\n   ./Rakefile install                  # Install Puppet module + Ruby Gem deps;\n   bolt plan show                      # Verify `puppetsync::` plans are visible\n   ```\n\n2. Set the [environment variable](#environment-variables) `GITHUB_API_TOKEN`.\n   * Before using the main `puppetsync::` plan, also set `JIRA_USER`,\n     `JIRA_API_TOKEN`, and `GITLAB_API_TOKEN`\n\n4. At this point, you are ready to run a plan.  The plan to run will depend on your role:\n\n   | Plan | Role | Purpose |\n   | --- | --- | --- |\n   | **`puppetsync`** | Maintainer | Applies baseline to each repo, submits changes as PRs |\n   | **`puppetsync::approve_github_prs`** | Approver | Approves all PRs from a specific `puppetsync` session |\n   | **`puppetsync::merge_github_prs`** | Maintainer | Merges all approved PRs from a specific `puppetsync` |\n\n\n#### Preparing to run `puppetsync`\n\n:warning: You only need to change these files when preparing a new\n`puppetsync`!\n\n1. Find/add a repolist file for the repos you want to affect under\n   `data/sync/repolists/`\n2. Copy and customize a Puppetsync config file under `data/sync/configs/` to\n   fit your workflow\n3. Set [environment variables](#environment-variables) for GITHUB, GITLAB, and\n   JIRA API authentication\n4. (Optional) Develop Puppet code/Hiera data/Tasks to provide new features\n\nNote: If you are just approving or merging PRs, you will reuse the repolist and\nconfig files from the puppetsync session you used.\n\n#### Running `puppetsync`\n\n\n```sh\n# (PROTIP: don't actually expose API tokens on the CLI when running commands)\n\nGITHUB_API_TOKEN=$GITHUB_API_TOKEN \\\n  JIRA_USER=$JIRA_USER \\\n  JIRA_API_TOKEN=$JIRA_API_TOKEN \\\n  GITLAB_API_TOKEN=$JIRA_API_TOKEN \\\n    bolt plan run puppetsync config={CONFIG_NAME} repolist={REPOLIST_NAME}\n```\n\nSee the [`puppetsync`](#puppetsync) reference for details.\n\n#### Running `puppetsync::approve_github_prs`\n\n\n```sh\nGITHUB_API_TOKEN=$GITHUB_API_TOKEN \\\n    bolt plan run puppetsync::approve_github_prs\n```\n\nSee the [`puppetsync::approve_github_prs`](#puppetsyncapprove_github_prs) reference for details.\n\n#### Running `puppetsync::merge_github_prs`\n\n```sh\nGITHUB_API_TOKEN=$GITHUB_API_TOKEN \\\n    bolt plan run puppetsync::merge_github_prs\n```\n\nSee the [`puppetsync::merge_github_prs`](#puppetsyncmerge_github_prs) reference for details.\n\n\n## Usage\n\n### Syncing repos\n\nAfter [setup](#setup), sync all repos by running:\n\n        /opt/puppetlabs/bin/bolt plan run puppetsync \\\n          config=CONFIG_NAME repolist=REPOLIST_NAME\n\nIf the config and repolist's  `latest.yaml` files are symlinked to the target\nconfig and repolist, you don't have to specify `config=` or `repolist=`\n(for brevity, following examples will assume this is\nthe case):\n\n        /opt/puppetlabs/bin/bolt plan run puppetsync\n\nTo see what's going on under the hood (potentially less irritating when\n`apply()` appears to hang for a long time when updating a lot of repos):\n\n        /opt/puppetlabs/bin/bolt plan run puppetsync --log-level info\n\n        # Alternatively (warning: LOTS of info):\n        /opt/puppetlabs/bin/bolt plan run puppetsync --log-level debug\n\n\n### Inspecting pipeline stages\n\nTo list all pipeline stages in a plan (and inspect which stages will be\nskipped), run:\n\n        bolt plan run puppetsync options='{\"list_pipeline_stages\": true}' \\\n          config=CONFIG_NAME repolist=REPOLIST_NAME\n\nThese steps can be specified/commented out in the [Puppetsync `config`] file,\nunder the corresponding plan.\n\n**Example:**\n\n```sh\nbolt plan run puppetsync \\\n  options='{\"list_pipeline_stages\": true}' \\\n  github_token=x jira_username=x  jira_token=x\n\n#   Starting: plan puppetsync\n#   ===== SKIPPING PIPELINE STAGE DUE TO CONFIGURATION: install_gems\n#   - checkout_git_feature_branch_in_each_repo\n#   - ensure_jira_subtask\n#   - apply_puppet_role\n#   - modernize_gitlab_files\n#   - lint_gitlab_ci\n#   - git_commit_changes\n#   - ensure_github_fork\n#   - ensure_git_remote\n#   - git_push_to_remote\n#   - ensure_gitlab_remote\n#   - git_push_to_gitlab\n#   - ensure_github_pr\n#   Finished: plan puppetsync in 0.04 sec\n#   Plan completed successfully with no result\n```\n\nAt the time the command above was run, the corresponding\n[Puppetsync `config`] file contained the following\n`puppetsync.plans.sync.stages`:\n\n```yaml\npuppetsync:\n  plans:\n    sync:\n      stages:\n        # - install_gems\n        - checkout_git_feature_branch_in_each_repo\n        - ensure_jira_subtask\n        - apply_puppet_role\n        - modernize_gitlab_files\n        - lint_gitlab_ci\n        - git_commit_changes\n        - ensure_github_fork\n        - ensure_git_remote\n        - git_push_to_remote\n        - ensure_gitlab_remote\n        - git_push_to_gitlab\n        - ensure_github_pr\n```\n\n\n## Reference\n\n### Environment variables\n\nThese environment variables are necessary to create Jira subtasks:\n\n| Env variable | Purpose   |                           |\n| ------------ | -------   | ------------------------- |\n| `JIRA_USER`  | Jira user | Probably an email address |\n| `JIRA_API_TOKEN` | Jira API token | You MUST generate an API token (basic auth no longer works). To do so, you must have Jira instance access rights.  You can generate a token here: https://id.atlassian.com/manage/api-tokens |\n\nThese environment variables are necessary to fork GitHub repositories and submit Pull Requests:\n\n| Env variable       | Purpose          |     |\n| ------------       | -------          | --- |\n| `GITHUB_API_TOKEN` | GitHub API token |     |\n\nThese environment variables are necessary to use GitLab's CI lint API:\n\n| Env variable       | Purpose                   |                      |\n| ------------       | -------                   | -------------------- |\n| `GITLAB_API_TOKEN` | GitLab Personal API Token | Requires `api` scope |\n\n(Recommended) To stop Bolt from collecting analytics, set this environment variable:\n\n| Env variable                  | Purpose                                                                           |     |\n| ------------                  | -------                                                                           | --- |\n| `BOLT_DISABLE_ANALYTICS=true` | Prevent bolt's analytics from phoning home to tell Puppet about everything you do |     |\n\n### Puppetsync `config`\n\nThe workflow of a specific Puppetsync session (sync -\u003e apply -\u003e merge of\nrelated PRs) is controlled by a single configuration data structure, defined\nin Hiera using the key `puppetsync::plan_config`.\n\nTypically, the `puppetsync::plan_config` data structure is defined in its own\nHiera YAML file, located at `data/sync/configs/{CONFIG_NAME}.yaml`.\nThe Hiera file's name is is the `CONFIG_NAME` in a Puppetsync plan's\n`config=CONFIG_NAME`\n\nExample:\n\n```yaml\n---\npuppetsync::plan_config:\n  permitted_project_types:\n    - pupmod\n    - pupmod_skeleton\n  plans:\n    # clone_git_repos: false     # set to `false` when applying manual updates on a second run\n    # clear_before_clone: false  # set to `false` when applying manual updates on a second run\n    sync:\n      stages:\n        - install_gems\n        - checkout_git_feature_branch_in_each_repo\n        - ensure_jira_subtask\n        - apply_puppet_role\n        - modernize_gitlab_files\n        - lint_gitlab_ci\n        - git_commit_changes\n        - ensure_github_fork\n        - ensure_git_remote\n        - git_push_to_remote\n        - ensure_gitlab_remote\n        - git_push_to_gitlab\n        - ensure_github_pr\n\n    approve_github_pr:\n      clone_git_repos: false\n      stages:\n        - install_gems\n        - approve_github_pr_for_each_repo\n\n    merge_github_pr:\n      clone_git_repos: false\n      stages:\n        - install_gems\n        - merge_github_pr_for_each_repo\n\njira:\n  parent_issue: SIMP-7035\n  project: SIMP\n  jira_site: https://simp-project.atlassian.net\n  subtask_title: 'Update .travis.yml pipeline in %COMPONENT%'\n\n  # optional subtask fields:\n  subtask_story_points: 1\n  subtask_assignee: 'chris.tessmer'\n\ngit:\n  commit_message: |\n    (%JIRA_PARENT_ISSUE%) Update to new Travis CI pipeline\n\n    This patch updates the Travis Pipeline to a static, standardized format\n    that uses project variables for secrets. It includes an optional\n    diagnostic mode to test the project's variables against their respective\n    deployment APIs (GitHub and Puppet Forge).\n\n    [%JIRA_PARENT_ISSUE%] #comment Update to latest pipeline in %COMPONENT%\n    [%JIRA_SUBTASK%] #close\n\ngithub:\n  pr_user: op-ct  # This should be the account that *submitted* the PRs (Used\n                  # by idempotency checks when approving/merging PRs)\n  approval_message: ':+1: lgtm'\n```\n\n### Puppetsync `repolist`\n\nData about each repo/branch to target.\nThe data is defined in a Hiera YAML file, located at\n`data/sync/repolists/{REPOLIST_NAME}.yaml`.\nEach repolist is named after its file.\n\n\n```yaml\npuppetsync::repos_config:\n\n  https://github.com/simp/pupmod-simp-acpid:\n    branch: master\n\n  https://github.com/simp/pupmod-simp-aide:\n    branch: master\n\n  https://github.com/simp/pupmod-simp-at:\n    branch: master\n\n  https://github.com/simp/pupmod-simp-auditd:\n    branch: master\n\n  # ... and so on\n```\n\n### Plans\n\nEach plan:\n\n* Reads its config data from the [Puppetsync `config`] file\n* Reads its repolist data from the [Puppetsync `repolist`] file\n* Has its own specific configuration under the keys (`plans.sync`,\n  `plans.approve_github_pr`, and `plans.merge_github_pr`)\n* Executes its workflow as a series of pipeline stages for each repo in the\n  repolist file (in Hiera at `data/sync/configs/{CONFIG_FILE}.yaml`).\n\n#### `puppetsync`\n\nThe main plan (`puppetsync`) clones and updates each repo in the [Puppetsync `repolist`].\nIt (idempotently) ensures a Jira subtask and GitHub PR exists for each change.\n\nWorkflow:\n\n1. Clone `:git` repositories defined in the [Puppetsync `repolist`]\n   * (disable with `clone_git_repos: false`)\n\nIt will then execute the following pipeline stages for each repo (in parallel):\n\n2. Ensure a Jira subtask exists to track the change\n3. Check out a new git feature branch\n4. Apply Puppet manifests to enforce a common repository asset baseline\n5. Commit changes to git with a templated commit message (`git.commit_message`)\n6. Ensure the user has forked repository on GitHub\n7. Push changes up to the user's forked repository\n8. Submit a Pull Request to merge the changes back the original repository and branch\n\nIf an individual repo encounters failures during a stage, it will be held back\nwhile the other repos proceed with their workflows.\n\nAll failures are summarized after the full plan finishes executing.\n\n#### `puppetsync::approve_github_prs`\n\nIdempotently approves every open PR from user `github.pr_user` on\nbranch `jira.parent_issue` for each repo in the [Puppetsync `repolist`].\n\n#### `puppetsync::merge_github_prs`\n\nIdempotently merges every approved PR from user `github.pr_user` on\nbranch `jira.parent_issue` for each repo in the [Puppetsync `repolist`].\n\n### Manually installing dependencies\n\nUse `bolt` to download the project's dependencies from `bolt-project.yaml` and\n`gems.deps.rb`:\n\n       /opt/puppetlabs/bolt/bin/gem install --user-install -g gem.deps.rb\n       /opt/puppetlabs/bin/bolt module install\n\nThe Rakefile can be used as a shortcut:\n\n      ./Rakefile install\n\nRun `./Rakefile -T` \u0026 `./Rakefile -D` to see other tasks \u0026 descriptions\n\n## Troubleshooting\n\n### Error: `Ignoring \u003cx\u003e because its extensions are not built.`\n\n**Cause:** Running `bolt plan run puppetsync` from a Ruby interpreter other\nthan the `bolt` package.\n\n**Fix:** Make sure you're not using RVM.  If necessary, invoke the packaged\nbolt executable directly:\n\n```sh\ncommand -v rvm \u0026\u0026 rvm use system    # make sure you're using the packaged `bolt`\n./Rakefile install                  # Install Puppet module and Ruby Gem deps\nbolt plan show --filter puppetsync  # Validate bolt is working\n\n## ^^^ If that still didn't work:\n# /opt/puppetlabs/bolt/bin/bolt show --filter puppetsync\n```\n\n**Characteristic error messages:**\n\n```\nIgnoring bcrypt_pbkdf-1.0.1 because its extensions are not built. Try: gem pristine bcrypt_pbkdf --version 1.0.1\nIgnoring byebug-11.1.3 because its extensions are not built. Try: gem pristine byebug --version 11.1.3\nIgnoring byebug-11.1.1 because its extensions are not built. Try: gem pristine byebug --version 11.1.1\nIgnoring byebug-11.0.1 because its extensions are not built. Try: gem pristine byebug --version 11.0.1\nIgnoring ed25519-1.2.4 because its extensions are not built. Try: gem pristine ed25519 --version 1.2.4\nIgnoring executable-hooks-1.6.0 because its extensions are not built. Try: gem pristine executable-hooks --version 1.6.0\nIgnoring ffi-1.12.2 because its extensions are not built. Try: gem pristine ffi --version 1.12.2\n...\n```\n\n## Limitations\n\n* Requires git to be configured with SSH, with keys loaded into a running agent\n* Probably only works from an \\*nix host\n\n[bolt]: https://puppet.com/docs/bolt/latest/bolt.html\n[puppet]: https://puppet.com/docs/puppet/latest/\n[bolt-install]: https://puppet.com/docs/bolt/latest/bolt_installing.html\n[Puppetsync `config`]: #puppetsync-config\n[Puppetsync `repolist`]: #puppetsync-repolist\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsimp%2Fpuppetsync","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsimp%2Fpuppetsync","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsimp%2Fpuppetsync/lists"}