{"id":22700280,"url":"https://github.com/simp/gitlab-beaker-cleanup-driver","last_synced_at":"2025-04-13T06:05:24.437Z","repository":{"id":43831734,"uuid":"250332855","full_name":"simp/gitlab-beaker-cleanup-driver","owner":"simp","description":"GitLab CI custom executor to help Beaker jobs clean up their own VMs","archived":true,"fork":false,"pushed_at":"2022-02-16T21:12:29.000Z","size":239,"stargazers_count":4,"open_issues_count":0,"forks_count":2,"subscribers_count":12,"default_branch":"main","last_synced_at":"2025-04-13T06:04:45.574Z","etag":null,"topics":["beaker","ci","custom-executor","gitlab","gitlab-ci","gitlab-runner","puppet","vagrant","virtualbox"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/simp.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-03-26T17:57:45.000Z","updated_at":"2024-07-24T21:00:25.000Z","dependencies_parsed_at":"2022-09-17T07:50:24.568Z","dependency_job_id":null,"html_url":"https://github.com/simp/gitlab-beaker-cleanup-driver","commit_stats":null,"previous_names":[],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simp%2Fgitlab-beaker-cleanup-driver","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simp%2Fgitlab-beaker-cleanup-driver/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simp%2Fgitlab-beaker-cleanup-driver/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simp%2Fgitlab-beaker-cleanup-driver/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/simp","download_url":"https://codeload.github.com/simp/gitlab-beaker-cleanup-driver/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248670437,"owners_count":21142904,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["beaker","ci","custom-executor","gitlab","gitlab-ci","gitlab-runner","puppet","vagrant","virtualbox"],"created_at":"2024-12-10T06:10:50.358Z","updated_at":"2025-04-13T06:05:24.387Z","avatar_url":"https://github.com/simp.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Beaker Cleanup Driver for GitLab Runners\n\n\u003c!-- vim-markdown-toc GFM --\u003e\n\n* [Description](#description)\n* [Requirements](#requirements)\n* [Setup](#setup)\n  * [1. Installing the executor files](#1-installing-the-executor-files)\n  * [2. Registering new Runner(s) using the configuration template file](#2-registering-new-runners-using-the-configuration-template-file)\n* [Reference](#reference)\n  * [Environment variables](#environment-variables)\n  * [How the executor knows what VMs to clean up](#how-the-executor-knows-what-vms-to-clean-up)\n  * [Troubleshooting](#troubleshooting)\n    * [journald logs on syslog identifier `beaker-cleanup-driver`](#journald-logs-on-syslog-identifier-beaker-cleanup-driver)\n    * [journald logs for `gitlab-runner` service](#journald-logs-for-gitlab-runner-service)\n      * [Limitations](#limitations)\n\n\u003c!-- vim-markdown-toc --\u003e\n\n![](assets/beaker-cleanup-driver.gif)\n\n## Description\n\n**beaker-cleanup-driver** is a GitLab CI runner [custom executor][custom\nexecutor] that properly cleans up each CI job's child processes and VMs―even if\nthe job failed, was cancelled, or timed out (probably).\n\n1. Drop-in alternative to the shell executor\n2. Tracks all processes created by a CI job and stops them after it terminates.\n3. Includes special logic to clean up Virtual Box VMs and Vagrant sessions\n   * (Using their respective tooling in order to properly reclaim their disk resources)\n\nIt is designed to run multi-node acceptance tests using Puppet's\n[Beaker][beaker] while avoiding the shell executor's tendency to\naccumulate hundreds of gigs of disk space and running VM processes, left\nover from jobs that didn't terminate cleanly.\n\n\n## Requirements\n\n* Gitlab CI Runner 11+\n* Linux host (tested on CentOS 7)\n* bash\n\n## Setup\n\n### 1. Installing the executor files\n\nAs root, clone the contents of this repository into a directory on the runner,\nand make sure that the `gitlab-runner` user can read and execute the directory\npath and the `*.sh` scripts:\n\n```sh\n\n# We'll keep using this $CUSTOM_EXECUTOR_DIR in the next example, too\nCUSTOM_EXECUTOR_DIR=/opt/simp/gitlab-runner/beaker-cleanup-driver\nTHIS_REPOSITORY_URL=\"\u003cthe url of this git repository\u003e\"\n\numask 0027\ngit clone \"$THIS_REPOSITORY_URL\" \"$CUSTOM_EXECUTOR_DIR\"\n\n# Ensure that gitlab-runner can read and execute the scripts\nchgrp -R gitlab-runner \"$CUSTOM_EXECUTOR_DIR\"\nchmod -R g=u-w \"$CUSTOM_EXECUTOR_DIR\"\n```\n\n### 2. Registering new Runner(s) using the configuration template file\n\nA Gitlab Runner [configuration template file][configuration template file]\nis provided to simplify registration.\n\n1. Tailor the template file `beaker-cleanup-driver.template.toml`\n\n   * Make sure the `_exec` entries in the `[runners.custom]` section are under the\n     path where you cloned the repository (`$CUSTOM_EXECUTOR_DIR`).\n   * Tailor settings to taste (not everybody needs `log_level = 'debug')\n\n2. Register a new GitLab Runner using the `--template-config` option:\n\n```sh\ngitlab-runner register \\\n  --name \"$RUNNER_NAME\" \\\n  --registration-token \"$REGISTRATION_TOKEN\" \\\n  --url \"${CI_SERVER_URL:-https://gitlab.com/}\" \\\n  --executor custom \\\n  --template-config \"$CUSTOM_EXECUTOR_DIR/beaker-cleanup-driver.template.toml\" \\\n  --tag-list \"${RUNNER_TAG_LIST:-beaker}\" \\\n  --non-interactive \\\n  --paused \\\n  --locked\n```\nFor more information, see GitLab's documentation about [Registering Runners][registering runners]\n\n## Reference\n\n\n### Environment variables\n\nYou can set environment variables (in the `job_env` key of the executor's `config_exec.sh` script\u003csup\u003e[[0]]\u003c/sup\u003e)\n\n\n| Environment Variable | Purpose                                                             | Default                      |\n|----------------------|---------------------------------------------------------------------|------------------------------|\n| `CI_RUNNER_USER`     | non-privileged build user                                           | `gitlab-runner`              |\n| `CI_RUNNER_USER_DIR` | parent path for non-privileged build user's build/cache directories | `/var/lib/${CI_RUNNER_USER}` |\n\n### How the executor knows what VMs to clean up\n\nBeaker (and its VM orchestration) does not lend itself to using PID files, so\ninstead the custom executor \"tags\" every process it starts with a unique\nenvironment variable:\n\n1. The exec script in each stage sets an environment variable `$_CI_JOB_TAG` to\n   a value that is unique to the job\n2. Every runner script for the job executes with `$_CI_JOB_TAG` set to this same\n   value.\n3. When the `cleanup_exec` stage executes, all processes with `$_CI_JOB_TAG` set\n   to that value will be cleaned up.\n   - Running VirtualBox VMs will be shut down and deleted\n   - `vagrant global-status --prune` will run if any VMs were shut down\n   - after a short grace period, all remaining processes\n4. The environment variables are found by grepping `/proc/*/environ` for the\n   job's unique `$_CI_JOB_TAG` value\n5. VMs are idenitified by grepping each `$_CI_JOB_TAG` pid's associated\n   `/proc/$pid/cmdline` for VBoxHeadless\n\nThe technique of tagging processes for later cleanup using environment\nvariables (and identifying them via `/proc/*/environ`) was informed by the\ndiscussion at https://serverfault.com/a/274613 .\n\n\n### Troubleshooting\n\n#### journald logs on syslog identifier `beaker-cleanup-driver`\n\nThe custom executor logs its activity to the Runner console output and to\njournald under the syslog identifier `beaker-cleanup-driver`.\n\nYou can follow these messages via `journalctl`:\n\n        journalctl -t beaker-cleanup-driver -f\n\nWhile troubleshooting, you are likely to only be interested in messages from a\nsingle job.  To see a specific job's progress, grep for the job's unique\n`_CI_JOB_TAG` value (shown in the Runner console output at the beginning of\nevery exec stage and sub-stage):\n\n        journalctl -t beaker-cleanup-driver -n 3000 -e  | egrep _CI_JOB_TAG=runner-1751173-project-17669670-concurrent-0-487152645 -A5 -B6 | less\n\n#### journald logs for `gitlab-runner` service\n\nThe `gitlab-runner` unit's journald log can also be useful for correlating the\nRunner's communication and setup events during troubleshooting.  Depending on\nthe gitlab-runner's `debug_level`, some executor messages (ONLY from the\n`cleanup_exec` stage) will also appear in this log.\n\nYou can follow the unit's log via `journalctl`:\n\n        journalctl -u gitlab-runner -f -n 3000\n\nThe gitlab-runner's log can have a lot of noise in addition to the executor\noutput To help filter the noise, you can grep the logs:\n\n        journalctl -u gitlab-runner  -e -n 3000   | egrep '==|--' | less\n\nTo follow a specific job's progress, grep for the _CI_JOB_TAG value (shown in\nthe Runner console output at the beginning of every exec stage and sub-stage):\n\n        journalctl -u gitlab-runner -e | grep _CI_JOB_TAG=runner-1751173-project-17669670-concurrent-0-485904412\n\n\n##### Limitations\n\n* The `gitlab-runner` unit's journald log only includes output from the\n  `cleanup_exec` script (when `log_level` is `warn` or higher)\n* There can be a signifcant delay before the messages show up at all.\n\n[registering runners]: https://docs.gitlab.com/runner/register/\n[configuration template file]: https://docs.gitlab.com/runner/register/#runners-configuration-template-file\n[custom executor]: https://docs.gitlab.com/runner/executors/custom.html\n[beaker]: https://github.com/puppetlabs/beaker\n[0]: https://docs.gitlab.com/runner/executors/custom.html#config\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsimp%2Fgitlab-beaker-cleanup-driver","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsimp%2Fgitlab-beaker-cleanup-driver","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsimp%2Fgitlab-beaker-cleanup-driver/lists"}