{"id":20018077,"url":"https://github.com/unidata/thredds-test-environment","last_synced_at":"2025-05-04T23:30:39.475Z","repository":{"id":45084083,"uuid":"332935260","full_name":"Unidata/thredds-test-environment","owner":"Unidata","description":"A packer + ansible project to build and deploy an Amazon Machine Image and Docker Container for use by the various THREDDS projects at Unidata for running automated tests.","archived":false,"fork":false,"pushed_at":"2025-05-03T03:08:46.000Z","size":99,"stargazers_count":0,"open_issues_count":0,"forks_count":4,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-05-03T04:19:07.542Z","etag":null,"topics":["ansible","netcdf-java","packer","rosetta","tds","thredds"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Unidata.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2021-01-26T01:28:59.000Z","updated_at":"2025-05-03T03:08:51.000Z","dependencies_parsed_at":"2025-02-25T21:21:01.754Z","dependency_job_id":"34e15f34-55d1-498a-93c4-c1d4b819804c","html_url":"https://github.com/Unidata/thredds-test-environment","commit_stats":{"total_commits":42,"total_committers":2,"mean_commits":21.0,"dds":"0.16666666666666663","last_synced_commit":"489f92b88f28e8695fde8e0901ed66a008575ba7"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Unidata%2Fthredds-test-environment","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Unidata%2Fthredds-test-environment/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Unidata%2Fthredds-test-environment/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Unidata%2Fthredds-test-environment/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Unidata","download_url":"https://codeload.github.com/Unidata/thredds-test-environment/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252414108,"owners_count":21744042,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ansible","netcdf-java","packer","rosetta","tds","thredds"],"created_at":"2024-11-13T08:19:43.820Z","updated_at":"2025-05-04T23:30:39.467Z","avatar_url":"https://github.com/Unidata.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# THREDDS Test Environment\n\nA packer + ansible project to build Docker images for use by the various THREDDS projects at Unidata for running automated tests.\nThe Docker images are used as the basis for Jenkins build nodes as well as a custom GitHub action for testing the various THREDDS projects.\nThe image is based off the latest available image for `ubuntu 24.04`.\nThe docker images produced by packer are called \"thredds-test-environment\" (for use by Jenkins) of `thredds-test-action` (for use by the GitHub action).\n\n## Requirements\n\n* Packer ([download](https://www.packer.io/downloads))\n* Docker, for building the Docker images ([download](https://www.docker.com/products/docker-desktop))\n\nIf this is your first time running the build, you will need to initialize packer using:\n\n~~~\npacker init .\n~~~\n\nWhile this project's packer configuration relies heavily on Ansible, you do not need to have Ansible installed locally.\nWe utilize the `ansible-local` provisioner of Packer, which means Ansible is run on the remote/guest machine, and not on the build machine (the machine running packer).\nThis does mean that part of the Packer build process includes installing Ansible on the remote/guest machine (see `packer/provisioners/scripts/bootstrap-common.sh`), which does add some time to the total build process (approximately two minutes).\nHowever, Since Ansible does not support the use of Windows systems as a control node, the ability create the THREDDS test environment images across all major platforms outweighs the extra cost in time.\n\n## Building the images\n\nTo generate the golden Docker images, start off by validating the packer configuration by first moving into the `packer/` directory and then running:\n\n~~~bash\npacker validate .\n~~~\n\nOnce validated, you may run the build using:\n\n~~~bash\npacker build --only=\u003ctype\u003e thredds-test-env.pkr.hcl\n~~~\n\n`\u003ctype\u003e` will one or more (separated by commas) of the following:\n* `docker.docker-jenkins`: Provision a Docker container and generate and tag a local Docker image (`docker.unidata.ucar.edu/thredds-test-environment:20.04`).\n* `docker.docker-github-action`: Provision a Docker container for use with GitHub Actions and tag a local docker image (`ghcr.io/unidata/thredds-test-action:v3`).\n* `docker.docker-export`: Provision a Docker container and generate a local Docker image as a file (`image.tar`).\n\nTypically, we would run the following to update the Jenkins and Github Action Docker images at the same time:\n\n~~~bash\npacker build --only=docker.docker-jenkins,docker.docker-github-action thredds-test-env.pkr.hcl\n~~~\n\nThe Docker image builds takes about 30 minutes to create on MacOS.\nPacker will run the builders in parallel, so the total time to create the `thredds-test-environment` images is around an hour.\n\nIf using `docker-jenkins`, then once the image is built you can test out the environment by using:\n\n~~~bash\ndocker run -i -t --rm docker.unidata.ucar.edu/thredds-test-environment:24.04\n~~~\n\nThese images are built using the multiplatform feature of Docker, and only `linux/amd64` images are produced.\nIf you are on a `arm64` based mac, you will need to run:\n\n~~~bash\nDOCKER_DEFAULT_PLATFORM=linux/amd64 docker run -i -t --rm docker.unidata.ucar.edu/thredds-test-environment:24.04\n~~~\n\nNote that images are not pushed as part of this build process.\nPushes can be done via the normal docker mechanisms, e.g. `docker image push docker.unidata.ucar.edu/thredds-test-environment:24.04` and `docker image push ghcr.io/unidata/thredds-test-action:v3`.\n\n## Project layout\n\nInside the packer directory is a file called `thredds-test-env.pkr.hcl`.\nThis contains the packer configuration for the builders (docker builder), the provisioners (shell scripts, ansible playbooks), and the post-processors (tagging the docker image).\n\nThe provisioners directory contains the provisioner configurations used by packer.\nThere are three types of provisioners used by the packer configuration: ansible, file, and scripts.\n\nThe `files/` directory contains Docker entrypoint scripts:\n\n* `entrypoint.sh`: drives the GitHub Action.\n* `jenkins-agent.sh`: connects a container to the jenkins control .\n\nThe `scripts/` directory contains the following shell scripts:\n\n* `bootstrap-common.sh`: Install and configure ansible (common to both builders)\n* `cleanup.sh`: Runs after the ansible provisioner and cleans up the `apt` cache, as well as some general build environment things.\n\nThe `ansible/` directory contains the ansible playbooks that are used to configure the testing environment.\nThe `ansible/` directory is laid out as follows:\n\n* `site.yml`: The main ansible playbook that references all other playbooks used by the build\n* `roles/`: Playbooks organized by common tasks\n\nWe use the following roles when provisioning our images:\n\n* `cleanup`: General cleanup related tasks, such as remove the temporary build directory and running `ldconfig`\n* `corretto`: Obtain and install LTS versions of Corretto.\n* `general-packages`: Install general packages needed for the build environment using the OS package manager.\n* `init`: Initialize the build environment by ensuring the temporary ansible build directory exists.\n* `libnetcdf-and-deps`: Configure, build, and install `zlib`, `HDF5`, and `netCDF-C`.\n* `maven`: Obtain and install the Apache Maven software project management and comprehension tool. \n* `security`:\n  * Add the `ubuntu` user.\n  * Add a default `maven-settings.xml` file configured to publish to the Unidata artifacts server.\n  * Configure `ssh` (uses modified version of a task from Jeff Geerling's [ansible-role-security](https://github.com/geerlingguy/ansible-role-security) project - see `packer/provisioners/ansible/roles/security/README.md`).\n  * Configure a system wide bash environment.\n* `temurin`: Obtain and install LTS versions of Temurin.\n* `test-data-mount-prep`: Prepare the environment to mount the `thredds-test-data` datasets when available (currently used on Jenkins worker nodes).\n* `zulu`: Obtain and install LTS versions of Zulu.\n\n## THREDDS Test Environment Highlights\n\n### netCDF-C\n * location: `/usr/thredds-test-environment`\n * version: `4.9.2`\n * dependencies (same location):\n   * zlib version: `1.3.1`\n   * hdf5 version: `1.14.6`\n\n### maven:\n * location: `/usr/thredds-test-environment/mvn`\n * version: `3.6.3`\n\n### Java:\n * Temurin (latest version available from adoptium.net)\n   * 8 (`/usr/thredds-test-environment/temurin8`)\n   * 11 (`/usr/thredds-test-environment/temurin11`)\n   * 17 (`/usr/thredds-test-environment/temurin17`)\n   * 21 (`/usr/thredds-test-environment/temurin21`)\n\n * Zulu (latest version available from azul.com)\n   * 8 (`/usr/thredds-test-environment/zulu8`)\n   * 11 (`/usr/thredds-test-environment/zulu11`)\n   * 17 (`/usr/thredds-test-environment/zulu17`)\n   * 21 (`/usr/thredds-test-environment/zulu21`)\n\n * Corretto (latest version available from docs.aws.amazon.com/corretto/)\n   * 8 (`/usr/thredds-test-environment/corretto8`)\n   * 11 (`/usr/thredds-test-environment/corretto11`)\n   * 17 (`/usr/thredds-test-environment/corretto17`)\n   * 21 (`/usr/thredds-test-environment/corretto21`)\n\n### Bash functions:\n * `select-java \u003cvendor\u003e \u003cversion\u003e` (where version is 8, 11, 17, or 21, and vendor is `temurin`, `zulu`, or `corretto`)\n * `get_pw \u003ckey\u003e`\n\n### Latest version available via the OS Package Manager\n  * sed\n  * dos2unix\n  * git\n  * fonts-dejavu\n  * fontconfig\n  * openssh-server\n\n## Example Timings\n\n### Docker Image\n\n~~~\n    docker.docker-jenkins: Saturday 08 March 2025  01:34:12 +0000 (0:00:02.971)       0:25:25.026 ********\n    docker.docker-jenkins: ===============================================================================\n    docker.docker-jenkins: Wait for the HDF5 async test task to complete. ------------------------ 483.06s\n    docker.docker-jenkins: libnetcdf-and-deps : Make hdf5. --------------------------------------- 327.68s\n    docker.docker-jenkins: libnetcdf-and-deps : Configure netCDF-c. ------------------------------ 174.98s\n    docker.docker-jenkins: libnetcdf-and-deps : Configure hdf5. ---------------------------------- 148.53s\n    docker.docker-jenkins: temurin : Fetch latest Temurin Java builds. ---------------------------- 53.85s\n    docker.docker-jenkins: zulu : Unpack Zulu Java Installations. --------------------------------- 39.85s\n    docker.docker-jenkins: general-packages : Install os managed tools. --------------------------- 39.50s\n    docker.docker-jenkins: general-packages : Install os managed tools. --------------------------- 34.55s\n    docker.docker-jenkins: libnetcdf-and-deps : Install netCDF-c. --------------------------------- 32.28s\n    docker.docker-jenkins: corretto : Unpack Corretto Java Installations. ------------------------- 28.85s\n    docker.docker-jenkins: temurin : Unpack Temurin Java Installations. --------------------------- 28.05s\n    docker.docker-jenkins: zulu : Fetch latest Zulu Java builds. ---------------------------------- 27.61s\n    docker.docker-jenkins: corretto : Fetch latest Corretto Java builds. -------------------------- 16.80s\n    docker.docker-jenkins: general-packages : Install os managed tools. --------------------------- 11.56s\n    docker.docker-jenkins: libnetcdf-and-deps : Download and unpack hdf5. -------------------------- 9.03s\n    docker.docker-jenkins: libnetcdf-and-deps : Install zlib. -------------------------------------- 6.08s\n    docker.docker-jenkins: security : Update SSH configuration to be more secure. ------------------ 5.03s\n    docker.docker-jenkins: libnetcdf-and-deps : Configure zlib. ------------------------------------ 4.44s\n    docker.docker-jenkins: zulu : Read versions of installed Zulu. --------------------------------- 4.24s\n    docker.docker-jenkins: cleanup : Remove packages that are not needed in final environment. ----- 3.46s\n~~~\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Funidata%2Fthredds-test-environment","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Funidata%2Fthredds-test-environment","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Funidata%2Fthredds-test-environment/lists"}