{"id":13542557,"url":"https://github.com/travis-ci/packer-templates","last_synced_at":"2025-04-05T11:09:52.079Z","repository":{"id":32673328,"uuid":"36261810","full_name":"travis-ci/packer-templates","owner":"travis-ci","description":"Templates for Packer!","archived":false,"fork":false,"pushed_at":"2025-03-17T15:52:51.000Z","size":45759,"stargazers_count":106,"open_issues_count":104,"forks_count":54,"subscribers_count":19,"default_branch":"master","last_synced_at":"2025-03-29T10:07:45.072Z","etag":null,"topics":["ascii-art","docker-images","google-cloud","packer","packer-builder","travis-ci"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/travis-ci.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2015-05-26T00:24:25.000Z","updated_at":"2025-03-10T11:24:51.000Z","dependencies_parsed_at":"2023-10-10T15:23:20.385Z","dependency_job_id":"e71df058-002d-4c5d-88f5-696685de64a2","html_url":"https://github.com/travis-ci/packer-templates","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/travis-ci%2Fpacker-templates","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/travis-ci%2Fpacker-templates/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/travis-ci%2Fpacker-templates/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/travis-ci%2Fpacker-templates/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/travis-ci","download_url":"https://codeload.github.com/travis-ci/packer-templates/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247325693,"owners_count":20920714,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ascii-art","docker-images","google-cloud","packer","packer-builder","travis-ci"],"created_at":"2024-08-01T10:01:10.632Z","updated_at":"2025-04-05T11:09:52.046Z","avatar_url":"https://github.com/travis-ci.png","language":"Ruby","funding_links":[],"categories":["google-cloud","Ruby"],"sub_categories":[],"readme":"# packer-templates [![Build Status](https://travis-ci.org/travis-ci/packer-templates.svg?branch=master)](https://travis-ci.org/travis-ci/packer-templates)\n\nCollection of Packer templates used for various infrastructure layers.\n\n## How to build stuff\n\nTo build a given template, one may use the `make` implicit builder, like the\nfollowing for `ci-stevonnie`:\n\n``` bash\nmake ci-stevonnie\n```\n\nor, with a specific builder:\n\n``` bash\nmake ci-stevonnie BUILDER=docker\n```\n\nor forget about the `Makefile` and run with `packer` directly:\n\n``` bash\npacker build -only=docker \u003c(bin/yml2json \u003c ci-stevonnie.yml)\n```\n\n## env config bits\n\nMost of the templates in here require some env vars.  Take a look at\n[`.example.env`](./.example.env) for an example.  Use of\n[autoenv](https://github.com/kennethreitz/autoenv) is encouraged but not\nrequired.\n\n## packer template types\n\nThere are two primary types of templates present at the top level: those\nintended for use as execution environment for jobs flowing through Travis CI,\nand those used for various backend fun in the Travis CI infrastructure.  The\nformer type all have the prefix `ci-`, described in more detail below:\n\n### stacks\n\nThere are two primary types of stacks: those targeting Ubuntu 14.04 (trusty),\nand those targeting Ubuntu 16.04 (xenial) that run on GCE and Docker.\n\nTake a peek at what's what:\n\n``` bash\nmake stacks-trusty\n```\n\n``` bash\nmake stacks-xenial\n```\n\nThere may be some subtle variations, but for the most part each stack is built\nvia the following steps.\n\n#### git metadata file input\n\nThe generated files in `./tmp/git-meta/` are copied onto the provisioned machine\nat `/var/tmp/git-meta/` for later use by the `./packer-scripts/packer-env-dump`\nscript.\n\n#### purge file input\n\nA git-tracked file in `./packer-assets` is copied onto the provisioned machine\nat `/var/tmp/purge.txt` for later use by the `./packer-scripts/purge` script.\n\n#### packages file input\n\nA git-tracked file in `./packer-assets` is copied onto the provisioned machine\nat `/var/tmp/packages.txt` for later use by both the\n`travis_packer_templates::default` recipe and the serverspec suites via\n`./cookbooks/lib/support.rb`.\n\n#### write packer and travis env vars\n\nThe script at `./packer-scripts/packer-env-dump` creates a directory on the\nprovisioned machine at `/.packer-env` which is intended to be in the [envdir\nformat](https://cr.yp.to/daemontools/envdir.html).  Any environment variables\nthat match `^(PACKER|TRAVIS)`, and (if present) the files previously written to\n`/var/tmp/git-meta/` are copied or written into `/.packer-env/`.\n\n#### remove default users\n\nThe script at `./packer-scripts/remove-default-users` will perform a best-effort\nremoval of users defined in `${DEFAULT_USERS}` (default `vagrant ubuntu`).  The\nprimary reasons for this are general tidyness and to try to free up uid 2000.\n\n#### pre-chef bootstrapping\n\nThe script at `./packer-scripts/pre-chef-bootstrap` is responsible for ensuring\nthe provisioned machine has all necessary packages and users for the Chef\nprovisioning process.  The steps executed include:\n\n- remove the \"partner\" APT source list file\n- remove all cached APT list files\n- install APT packages needed by Chef\n- ensure `/var/run/sshd` dir exists\n- ensure `sshd: ALL: ALLOW` exists in `/etc/hosts.allow`\n- ensure there is a `travis` user\n- change the `travis` user password to `travis`\n- ensure `#includedir /etc/sudoers.d` exists in `/etc/sudoers`\n- ensure the `/etc/sudoers.d` dir exists\n- ensure the `/etc/sudoers.d/travis` file exists with specific permissions\n- ensure the `/home/travis/.ssh` dir exists\n- ensure the `/home/travis/.ssh/authorized_keys` file exists\n- add `/var/tmp/*_rsa.pub` to `/home/travis/.ssh/authorized_keys`\n- ensure `/home/travis/.ssh/authorized_keys` perms are `0600`\n- ensure the `/home/travis/bin` dir exists\n\n#### cloning travis-cookbooks\n\nThe script at `./packer-scripts/clone-travis-cookbooks` is responsible for `git\nclone`'ing [travis-cookbooks](https://github.com/travis-ci/travis-cookbooks)\ninto `/tmp/chef-stuff` on the provisioned machine.  Optional env vars supported\nby this script are:\n\n- `TRAVIS_COOKBOOKS_BRANCH` - the branch specified during `git clone`\n- `TRAVIS_COOKBOOKS_EDGE_BRANCH` - the default branch used if\n  `TRAVIS_COOKBOOKS_BRANCH` is not defined\n- `TRAVIS_COOKBOOKS_URL` - the git clone remote (default\n  `https://github.com/travis-ci/travis-cookbooks.git`)\n- `TRAVIS_COOKBOOKS_SHA` - a git tree-ish to which the clone will be checked\n  out if defined (default not set)\n\nOnce the clone is complete, the clone directory is written to\n`/.packer-env/TRAVIS_COOKBOOKS_DIR` and the head sha is written to\n`/.packer-env/TRAVIS_COOKBOOKS_SHA`.\n\n#### chef provisioning\n\nThe `chef-solo` provisioner will typically have *no json data*, but instead will\nleave all attribute and effective run list definition to a single wrapper\ncookbook located in `./cookbooks/`.\n\n##### chef wrapper cookbook layout\n\nEach wrapper cookbook must contain at least a `metadata.rb` and a\n`recipes/default.rb`.  Typically, the `attributes/default.rb` is defined and\ncontains all override attribute settings.  The earliest version of Chef used by\neither trusty or xenial stacks is `12.9`, which means that *all* cookbook\ndependencies must be declared in `metadata.rb`, a requirement that is also\nenforced by the `foodcritic` checks.\n\nFor example, the minimal trusty image \"ci-stevonnie\" has a wrapper cookbook at\n`./cookbooks/travis-ci_stevonnie` that looks like this:\n\n```\ncookbooks/travis_ci_stevonnie\n├── README.md\n├── attributes\n│   └── default.rb\n├── metadata.rb\n├── recipes\n│   └── default.rb\n└── spec\n    ├── ...\n```\n\n#### travis user double check\n\nThe script at `./packer-scripts/ensure-travis-user` is responsible for ensuring\nthe existence of the `travis` user and its home directory permissions,\noptionally setting the password to a random string.  The list of operations is:\n\n- ensure the `travis` user exists\n- set the `travis` user password\n- ensure `/home/travis` exists\n- ensure `/home/travis/.ssh/authorized_keys` and `/home/travis/.ssh/known_hosts`\n  both exist and have permissions of `0600`\n- blank out `/home/travis/.ssh/authorized_keys`\n- ensure `/home/travis` is fully owned by `travis:travis`\n\nOptional env vars supported by this script are:\n\n- `TRAVIS_USER_PASSWORD` - a string (default \"travis\")\n- `TRAVIS_OBFUSCATE_PASSWORD` - if non-empty, causes\n  `TRAVIS_USER_PASSWORD` to be set to a random string\n\n#### purging undesirable packages\n\nThe script at `./packer-scripts/purge` is responsible for purging packages that\nare not desirable for the CI environment, such as the Chef that was installed\nprior for the Chef provisioner.  Additionally, any package names present in\n`/var/tmp/purge.txt` will be purged.  Optional env vars supported by this script\nare:\n\n- `APT_GET_UPGRADE_DURING_CLEANUP` - if non-empty, triggers an `apt-get -y\n  upgrade` prior to package purging.\n- `CLEAN_DEV_PACKAGES` - if non-empty, purges any packages matching `-dev$`\n\n#### disabling apparmor\n\nThe script at `./packer-scripts/disable-apparmor` is responsible for disabling\napparmor if detected.  This is done primarily so that services such as\nPostgreSQL and Docker may be used in the CI environment without first updating\napparmor configuration and restaring said services.\n\n#### running server specs\n\nThe script at `./packer-scripts/run-serverspecs` is responsible for running the\nserverspec suites via the rspec executable that is part of the `chefdk` package.\nThe list of operations is:\n\n- install the `chefdk` package\n- create a `sudo-bash` wrapper for use in some specs\n- ensure all spec files are owned by `travis:travis`\n- run each suite defined in `${SPEC_SUITES}`\n- optionally remove the `chefdk` package\n\nOptional env vars supported by this script are:\n\n- `PACKER_CHEF_PREFIX` - directory in which to find packer chef stuff (default\n  `/tmp`)\n- `SPEC_RUNNER` - string used to wrap execution of rspec (default `sudo -u\n  travis HOME=/home/travis -- bash -lc`)\n- `SPEC_SUITES` - comma-delimited string of spec suites to run (default not set)\n- `SKIP_CHEFDK_REMOVAL` - if non-empty do not remove the `chefdk` package and\n  APT source\n\n#### removing undesirable files\n\nThe script at `./packer-scripts/cleanup` is responsible for removing files and\ndirectories that are unnecessary for the CI environment or otherwise add\nunnecessary mass to the mastered image.  The list of operations is:\n\n- recursively remove a bunch of files and directories\n- conditionally remove `/var/lib/apt/lists/*`\n- conditionally remove `/var/lib/man-db`\n- conditionally remove `/home/travis/linux.iso` and `/home/travis/shutdown.sh`\n- empty all files in `/var/log`\n\nOptional env vars supported by this script are:\n\n- `CLEANUP_APT_LISTS` - if non-empty, trigger removal of `/var/lib/apt/lists/*`\n- `CLEANUP_MAN_DB` - if non-empty, trigger removal of `/var/lib/man-db`\n\n#### minimizing image size\n\nThe script at `./packer-scripts/minimize` is responsible for reducing the size\nof the provisioned image by squeezing out all of the empty space into a\ncontiguous area using the [same method as\nbento](https://github.com/chef/bento/blob/0d78beb7df68b025a0354f8eee58d81102d192f1/scripts/common/minimize.sh).\nThe list of operations is:\n\n- exit `0` if `$PACKER_BUILDER_TYPE` is either `googlecompute` or `amazon-ebs`,\n  as minimizing like this is superfluous on those builders\n- if `$PACKER_BUILDER_TYPE` is not `docker`, turn off swap and zero out the swap\n  partition if available.\n- write zeros to `/EMPTY` until the disk is out of space\n- remove `/EMPTY` and run `sync`\n- if the `vmware-toolbox-cmd` is available, run disk shrink operations for both\n  `/` `/boot` paths.\n\n#### registering the image with job-board\n\nThe script at `./bin/job-board-register` is responsible for \"registering\" the\nmastered image in a post-processing step by making an HTTP request to the\n[job-board](https://github.com/travis-ci/job-board) images API.  The list of\noperations is:\n\n- source any available env vars exported from the provisioned VM\n- dump any env vars with prefixes `^(PACKER|TRAVIS|TAGS|IMAGE_NAME)`\n- define a `TAGS` env var that will be used as the `tags` HTTP request param.\n- define a URI-escaped query string from several env vars\n- perform the HTTP request to job-board with `curl` and pipe the response\n  through `jq`\n\nRequired env vars for this script are:\n\n- `JOB_BOARD_IMAGES_URL` - the URL including `PATH_INFO` prefix to job-board\n- `IMAGE_NAME` - the name of the image, typically the same as that used by the\n  target infrastructure\n\nOptional env vars supported by this script are:\n\n- `PACKER_ENV_DIR` - path to the envdir containing packer-specific env vars,\n  default `/.packer-env`\n- `TAGS` - initial value for tags set during job-board registration\n- `GROUP` - value used in `group` tag, default `edge` if edge conditions match,\n  else `dev`\n- `DIST` - value used in `dist` tag, default either Linux release codename or OS\n  X product version\n- `OS` - value used in `os` tag, default lowercase value of `uname`, mapped to\n  `osx` on Darwin\n\nFor more info on the relationship between a given packer build artifact and\njob-board, see [job-board details](#job-board-details) below.\n\n###  job-board details\n\nThe [job-board](https://github.com/travis-ci/job-board) application is\nresponsible for tracking stack image metadata and presenting a queryable API\nthat is used by the [travis-worker API image\nselector](https://github.com/travis-ci/worker/blob/master/image/api_selector.go).\nAs described above, each stack image is registered with job-board along with a\n`group`, `os`, `dist`, and map of `tags`.  When travis-worker requests a stack\nimage identifier, it performs a series of queries with progressively lower\nspecificity.\n\nOf the values assigned to each stack image, the `tags` map is perhaps most\nmysterious, in part because it is so loosely defined.  This is intentional, as\nthe number of values that could be considered \"tags\" varies enough that\nmaintaining them all as individual columns would result in (opinions!) too much\noverhead in the form of schema management and query complexity.\n\nThe implementation of the [job-board-register\nscript](./lib/job_board_registrar.rb) includes a process that converts the\n`languages` and `features` arrays present in `/.job-board-register.yml`, written\nfrom the values present in chef attributes at\n`travis_packer_templates.job_board.{features,languages}`, into \"sets\"\nrepresented as `{key} =\u003e true`.  For example, if a given wrapper cookbook\ncontains attributes like this:\n\n``` ruby\noverride['travis_packer_templates']['job_board']['languages'] = %w(\n  fribble\n  snurp\n  zzz\n)\n```\n\nthen the tags generated for registration with job-board would be equivalent to:\n\n``` json\n{\n  \"language_fribble\": true,\n  \"language_snurp\": true,\n  \"language_zzz\": true\n}\n```\n\n#### job-board tagsets\n\nA \"tagset\" is the \"set\" (as in the type) of the \"tags\" applied during job-board\nregistration of a particular stack image, including `languages` and `features`.\nAt the time of this writing, both tagsets are used during serverspec runs, and\nonly the `languages` tagset is considered during selection via the job-board\nAPI.\n\n#### tagset relationships\n\nBecause the [travis-worker API image\nselector](https://github.com/travis-ci/worker/blob/master/image/api_selector.go)\nis querying job-board for stack images that match a particular language, it is\nimportant for us to ensure reasonably consistent image selection by way of\nasserting the `languages` values *do not* overlap between certain stacks (an\n\"exclusive\" relationship).  Additionally, it is important that we ensure certain\nstack `features` are subsets of others (an \"inclusive\" relationship).\n\nPart of the CI process for *this* repository makes assertions about such\nexclusive and inclusive relationships by way of the\n[check-job-board-tags](./bin/check-job-board-tags) script.  The exact\nrelationships being enforced may be viewed like so:\n\n``` bash\n./bin/check-job-board-tags --list-only\n```\n\n##### exclusive relationships\n\nAn exclusive tagset relationship is equivalent to asserting that the set\nintersection is the empty set, e.g.:\n\n``` ruby\ntagset_a = %w(a b c)\ntagset_b = %w(d e f)\nassert (tagset_a \u0026 tagset_b).empty?\n```\n\n##### inclusive relationships\n\nAn inclusive tagset relationship is equivalent to asserting that all members\nof one tagset are present in another, or that a tagset's intersection with its\nsuperset is equivalent to itself, e.g.:\n\n``` ruby\ntagset_a = %w(a b c d e f)\ntagset_b = %w(f d b)\nassert (tagset_a \u0026 tagset_b).sort == tagset_b.sort\n```\n\n\n## Testing cookbook changes\n\nWhen submitting changes to this repository, please be aware that\nthe top level-specs are shallow and don't include logic changes in the cookbooks.\n\nAny cookbook specs are ran as part of the actual image building\nprocess, which is triggered when any of the `ci-\u003cimage-name\u003e.yml`\ntemplates are modified.\n\nThe image build is ran as part of the\n[packer-build](https://github.com/travis-infrastructure/packer-build)\nrepo on the branch corresponding to each template and is triggered by\n[travis-packer-build](https://github.com/travis-ci/travis-packer-build).\n\nThis can be installed and invoked locally by running `bundle install`\nand then `bundle exec travis-packer-build [options]`.\n\nExample:\n\n```\nbundle exec travis-packer-build \\\n\t-I ci-sardonyx.yml \\\n\t--target-repo-slug=\"travis-infrastructure/packer-build\" \\\n\t--github-api-token=\"\u003cyour-token-here\u003e\" \\\n\t--body-tmpl=\".packer-build-pull-request-false-tmpl.yml\"\n```\n\nYou can specify the branch using `-B` (if you don't want to build from master).\n\nThe file `.packer-build-pull-request-false-tmpl.yml` here is just an\nexample, but you can also create a different template that specifies\nother travis-cookbooks or packer-template branches.\n\nAdditionaly, if you just want to test a change in\n[travis-cookbooks](https://github.com/travis-ci/travis-cookbooks), you\ncan use the shortcut script in `./bin/packer-build-cookbooks-branch`:\n\n```\n./bin/packer-build-cookbooks-branch \u003ctravis-cookbooks-branch-name\u003e \u003ctemplate-name\u003e\n```\n\n**Note:**  *The above script expects the `GITHUB_API_TOKEN`\nenvironment variable to be set.*\n\nOnce created, the images will be registered in job-board under the\n`group: dev` tag.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftravis-ci%2Fpacker-templates","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftravis-ci%2Fpacker-templates","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftravis-ci%2Fpacker-templates/lists"}