{"id":19705172,"url":"https://github.com/linux-system-roles/test-harness","last_synced_at":"2025-04-29T15:30:28.494Z","repository":{"id":23960211,"uuid":"89464947","full_name":"linux-system-roles/test-harness","owner":"linux-system-roles","description":"Test harness for linux system roles","archived":false,"fork":false,"pushed_at":"2022-11-09T08:39:26.000Z","size":317,"stargazers_count":10,"open_issues_count":17,"forks_count":18,"subscribers_count":9,"default_branch":"master","last_synced_at":"2024-07-22T17:36:30.893Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/linux-system-roles.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-04-26T09:47:36.000Z","updated_at":"2021-12-01T21:24:54.000Z","dependencies_parsed_at":"2023-01-14T00:08:41.556Z","dependency_job_id":null,"html_url":"https://github.com/linux-system-roles/test-harness","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linux-system-roles%2Ftest-harness","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linux-system-roles%2Ftest-harness/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linux-system-roles%2Ftest-harness/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linux-system-roles%2Ftest-harness/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/linux-system-roles","download_url":"https://codeload.github.com/linux-system-roles/test-harness/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224178037,"owners_count":17268786,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-11T21:26:41.096Z","updated_at":"2024-11-11T21:26:41.772Z","avatar_url":"https://github.com/linux-system-roles.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Integration Tests for Linux System Roles\n\n![CI Status](https://github.com/linux-system-roles/test-harness/workflows/tox/badge.svg)\n[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/ambv/black)\n\n## Commands\n\nThe CI system can be controlled with a few commands as comments in pull requests:\n\n* `[citest]` - Trigger a re-test for all machines\n* `[citest bad]` - Trigger a re-test for all machines with an error or failure status\n* `[citest pending]` - Trigger a re-test for all machines with a pending status\n* `[citest commit:\u003csha1\u003e]` - Whitelist a commit to be tested if the submitter is not trusted\n\nIt is also possible to whitelist all commits in a PR with the `needs-ci` tag.\nHowever, this should be only used with trusted submitters since they can still\nchange the commits in the PR.\n\n## Deployment Workflow\n\nTo make changes to the CI environment, the changes should be first committed\nto the `master` branch, and deployed to the `staging` environment.  Once they\nare confirmed to be working, the changes should be merged to the `production`\nbranch and deployed to the `production` environment.  For the centos7 based\ntests, use `linux-system-roles-centos7` for production and use\n`linux-system-roles-centos7-staging` for staging.\n\nSteps:\n\n1. Make changes to `master` branch.\n2. Git commit.\n3. Git push to your github fork.\n4. Deploy changes to the `staging` environment.\n5. If you want to test changes to the run-tests script that have not been\n   merged into the upstream `master` branch:\n```\n$ oc edit bc linux-system-roles-staging\n# look for the source.git.ref - change the uri to point to your fork\n$ oc start-build --follow linux-system-roles-staging\n... build logs ...\n# if the build succeeds, a new staging pod should rollout - the\n# dc has a trigger on new images - if not, do this\n$ oc rollout latest dc/linux-system-roles-staging\n```\n6. Test one or more PRs to ensure that the staging changes are working.  Be\n   sure to check the staging pod logs for problems too.\n7. If there are errors in building or testing, repeat from Step 1.\n8. Submit a PR against `test-harness` in github for the `master` branch.\n9. Once the PR is merged, `git pull` the changes to your local `master`\n   branch.\n10. Deploy the changes to the `staging` environment.  (If you edited the\n    buildconfig to test your unmerged changes, this step will replace them)\n11. Merge the changes from the `master` branch to `production` branch and\n    submit a PR against `test-harness` in github for the `production` branch.\n12. Once the PR is merged, `git pull` the changes to your local `production`\n    branch.\n13. Deploy the changes to the `production` environment.\n\n## Installation\n\nA podman container which runs integration tests for open pull requests on\n[linux system roles](https://linux-system-roles.github.io) repositories. It runs\nall playbooks matching `tests/tests*.yml` (intentionally identical to\nthe pattern used in the Fedora Standard Test Interface) of the\nrepository against various virtual machines.\n\nIf using OpenShift, see below.\n\nTo build a container for local testing, run\n\n```\nbuildah [--log-level debug] bud -f Dockerfile[.ext] -t lsr:{latest,$name} .\n```\nIf `buildah` seems to hang, Ctrl-C, then rerun with the log level.  For example\n```\nbuildah --log-level debug bud -f Dockerfile -t lsr:latest .\n```\nYou can also use `podman` or `docker`.\n```\npodman build -t linuxsystemroles/test-harness:latest .\n```\n\nIf `podman` isn't working, try `docker`:\n\n```\nsudo docker build -t linuxsystemroles/test-harness:latest .\n```\n\nMultiple of these containers can be started in parallel. They try to not step\non each others feet by synchronizing via GitHub commit statuses.\n\nThe test runner needs a `config.json` in the `/config` mount, which specifies\nthe repositories to monitor for open pull requests, the cloud images to use,\na location where to copy (via ssh) the test results to, and a publicly\naccessible URL for these results. For example:\n\n```json\n{\n  \"repositories\": [\n    \"linux-system-roles/network\",\n    \"linux-system-roles/selinux\",\n    \"linux-system-roles/timesync\",\n    \"linux-system-roles/tuned\",\n    \"linux-system-roles/kdump\",\n    \"linux-system-roles/firewall\",\n    \"linux-system-roles/postfix\"\n  ],\n  \"images\": [\n    {\n      \"name\": \"fedora-27\",\n      \"source\": \"https://download.fedoraproject.org/pub/fedora/linux/releases/27/CloudImages/x86_64/images/Fedora-Cloud-Base-27-1.6.x86_64.qcow2\",\n      \"upload_results\": true,\n      \"setup\": \"dnf install -yv python2 python2-dnf libselinux-python\"\n    },\n    {\n      \"name\": \"fedora-28\",\n      \"source\": \"https://download.fedoraproject.org/pub/fedora/linux/releases/28/Cloud/x86_64/images/Fedora-Cloud-Base-28-1.1.x86_64.qcow2\",\n      \"upload_results\": true,\n      \"min_ansible_version\": \"2.7\",\n      \"setup\": [\n        {\n          \"name\": \"Setup\",\n          \"hosts\": \"all\",\n          \"become\": true,\n          \"gather_facts\": false,\n          \"tasks\": [\n            {\"raw\": \"dnf install -yv python2 python2-dnf libselinux-python\"}\n          ]\n        }\n      ]\n    },\n    {\n      \"name\": \"centos-6\",\n      \"source\": \"https://cloud.centos.org/centos/6/images/CentOS-6-x86_64-GenericCloud-1804_02.qcow2c\",\n      \"upload_results\": true\n    },\n    {\n      \"name\": \"centos-7\",\n      \"source\": \"https://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud-1804_02.qcow2c\",\n      \"upload_results\": true\n    }\n  ],\n  \"results\": {\n    \"destination\": \"user@example.org:public_html/logs\",\n    \"public_url\": \"https://example.com/logs\"\n  },\n  \"logging\": {\n    \"level\": \"WARNING\",\n    \"filename\": \"/var/log/test-harness_HOSTNAME.log\",\n    \"format\": \"%(asctime)s: %(levelname)s: %(message)s\",\n    \"datefmt\": \"%Y-%m-%dT%H:%M:%S%z\",\n    \"style\": \"%\",\n    \"file_level\": \"NOTSET\",\n    \"stderr_level\": \"WARNING\"\n  }\n}\n```\n\nThe `setup` key contains either a list of Ansible plays which will be saved as\na playbook and executed before the test run, or a single shell command which\nwill be executed using the Ansible `raw` module (so the two examples above are\nexactly equivalent).\n\nUse the `min_ansible_version` key to indicate the minimum version of ansible\nthat can be used.  For example, if you need to use ansible version 2.8 or\nlater to manage Fedora 32 hosts, use something like this:\n```json\n    {\n      \"name\": \"fedora-32\",\n      \"source\": \"http://download.fedoraproject.org/pub/fedora/linux/releases/32/Cloud/x86_64/images/Fedora-Cloud-Base-32-1.6.x86_64.qcow2\",\n      \"upload_results\": true,\n      \"min_ansible_version\": \"2.8\"\n    },\n```\nThis means the CI environment will run tests using only ansible 2.8 or later\nto manage Fedora 32 hosts.\n\nThe container needs a `/secrets` mount, which must contain these files:\n\n* `github-token`: A GitHub access token with `public_repo`, `repo:status` and\n  `read:org` scopes. The `read:org` permission is required to identify users as\n  members to organiziations in case they set their organization membership to\n  private. Without it, the test harness can still update CI status but might\n  ignore pull requests or command comments unexpectedly.\n* `id_rsa` and `id_rsa.pub`: A ssh key with to access the server in\n  `results.destination`.\n* `known_hosts`: A ssh `known_hosts` file which contains the public key of that\n  server.\n\nThe test runner writes images into the `/cache` mount and reuses existing ones.\n\nThe container must have access to `/dev/kvm` for fast virtualization.\n\n## Running on OpenShift\n\nIn this section we first describe how to setup an OpenShift environment and run\nthe CI system on it. Then we give several tips concerning maintenance the\nentire system and backing up important data.\n\n### Preparing OpenShift Environment\n\nIn our use case, we create OpenShift project with two deployments. The first\none will be used for testing and will be based on\n[stable version](https://github.com/linux-system-roles/test-harness/tree/master)\nof the CI. The second one, based on\n[staging version](https://github.com/linux-system-roles/test-harness/tree/staging)\nof the CI, will be used to test new features of the CI before they will become\npart of the stable version (i.e. before merging them into the\n[master branch](https://github.com/linux-system-roles/test-harness/tree/master)).\n\nBefore start, get the `config.json` and `config-staging.json`. Then create an\nempty directory and put the secrets inside it (files `github-token`, `id_rsa`,\n`id_rsa.pub` and `known_hosts` discussed before). The requirements put on\n`config{,-staging}.json` and secrets were discussed above.\n\nIn what follows, suppose you have set following environment variables:\n\n* `PROJECT_NAME`: A name of OpenShift project.\n* `CONFIG_PATH`: A path to directory with `config.json` and `config-staging.json`\n* `SECRETS_PATH`: A path to the directory with secrets.\n* `SCC_NAME`: The name of the SecurityContextConstraints that you will add your\n              ServiceAccount to for privileged operations (e.g. `privileged`)\n\nFirst, you need to be logged on (replace `[URL]` with a real URL of your\nOpenShift server):\n\n```\n$ oc login [URL]\n```\n\n### Installing CI System in OpenShift using Ansible\n\nNOTE: Ansible will be running all of the commands on `localhost`, instead of\nexecuting the tasks remotely in the cluster.  That is, the Ansible playbook\nwill be using the `oc` OpenShift command line tool, and the python OpenShift\nAPI library, to execute API calls in the OpenShift cluster to provision and\nconfigure the resources in the OpenShift cluster.  So make sure you are logged\ninto the OpenShift cluster first:\n\n```\n$ oc login [URL]\n```\n\nThen execute the playbook:\n\n```\n$ ansible-playbook [-e test_harness_secrets_dir=$SECRETS_PATH] \\\n    [-e test_harness_config_dir=$CONFIG_PATH] \\\n    -e test_harness_scc=$SCC_NAME \\\n    ansible/openshift-playbook.yml\n```\n\nParameters:\n* `test_harness_secrets_dir` - Default\n  `$HOME/rhel-system-roles/test-harness-config/secrets` - See `SECRETS_PATH`\n* `test_harness_config_dir` - Default\n  `$HOME/rhel-system-roles/test-harness-config/config` - See `CONFIG_PATH`\n* `test_harness_scc` - Required - See `SCC_NAME`\n* `test_harness_namespace` - Default `lsr-test-harness`\n* `test_harness_sa` - Default `system:serviceaccount:{{ test_harness_namespace }}:tester`\n* `test_harness_need_node_selector` - Default `true`\n* `test_harness_run_as_root` - Default `true`\n* `test_harness_node_selector` - Default `{\"system-roles-ci\": \"true\"}`\n* `test_harness_use_staging` - Default `true` if on the staging branch, `false` otherwise\n* `test_harness_use_production` - Default `false` - Must explicitly set and be\n  on the production/master branch\n\nThe environment, staging or production, that will be deployed/updated depends\non which git branch you have checked out.  If you are on the staging branch\n(currently `master`), then the playbook will deploy/update the staging\nenvironment.  If you are on the production branch (currently `production`),\nand have explicitly set `test_harness_use_production` to `true`, then the\nplaybook will deploy/update the production environment.  You will get an error\nif not on one of these branches.  You will get an error if you have\nuncommitted git changes (see `git status -uno --porcelain`).  It will also\nchange the DeploymentConfig to use the nodeSelector above, and will configure\nthe DeploymentConfig to run as root.\n\n### Installing CI System manually using OpenShift commands\n\nMake sure you are logged into the OpenShift cluster first:\n\n```\n$ oc login [URL]\n```\n\nCreate a fresh OpenShift project:\n\n```\n$ oc new-project ${PROJECT_NAME} --display-name=${PROJECT_NAME}\n```\n\nThe `--display-name` parameter is optional and it is what is displayed in\nOpenShift web console.\n\nIf you have already OpenShift project created and you want to use it, you can\nselect it by typing:\n\n```\n$ oc project ${PROJECT_NAME}\n```\n\nNow its time to create objects. Type:\n\n```\n$ oc create -f ./openshift-objects.yml\n```\n\nThis will create `ServiceAccount` object named `tester` and `ImageStream` and\n`DeploymentConfig` objects both named `linux-system-roles` for us, just as they\nare specified in `./openshift-objects.yml`. Be careful: if some of the object\ndoes exist so far, the `create` command ends with an error. In this case, you\ncan use `replace` command to just replace the existing object (suppose that\n`./openshift-object.yml` contains a specification of object to be replaced):\n\n```\n$ oc replace -f ./openshift-object.yml\n```\n\nBecause we need also staging deployment, lets create `DeploymentConfig` object\nnamed `linux-system-roles-staging`:\n\n```\n$ oc create -f ./openshift-objects-staging.yml\n```\n\n**Note:** If `ImageStream` was created from an older version of\n`./openshift-objects.yml`, it is possible that `staging` tag is missing. To\ncheck this, please type (`is` refers to `ImageStreams` resource,\n`linux-system-roles` is the name of `ImageStream` we want to inspect):\n\n```\n$ oc get is linux-system-roles -o jsonpath='{range .spec.tags[*]}{.name}{\"\\n\"}{end}'\nlatest\nstaging\n$\n```\n\nIf you are not seeing `staging` in the above command output, then you must add\n`staging` tag by yourself. To do this, type:\n\n```\n$ ./add-staging-tag linux-system-roles\n```\n\nand now you should have `staging` tag in your `linux-system-roles` image\nstream.\n\nThe CI system launches virtual machines (VMs for short), so the container\nneeds to have `/dev/kvm` device (in other words, the container must be\nprivileged). Containers are launched using service account named `tester`.\nSuch account must be present in a list of privileged account. If you are\na cluster-admin, you can simply type\n\n```\n$ oc edit scc $SCC_NAME\n```\n\nand then add under the `users` list a user with the name `tester`. Please keep\nthe following format for items in the `users` list:\n\n```\nsystem:serviceaccount:${PROJECT_NAME}:tester\n```\n\n**Tip:** `oc edit` will open file to be edited in `vi`, which is a default\neditor. To change this behavior, set `OC_EDITOR` environment variable to point\nto your favorite editor.\n\nIf you are done with editing, the snippet from your editor should looks like\nthis (if you named your project `lsr-test-harness`):\n\n```yaml\nusers:\n  - system:serviceaccount:lsr-test-harness:tester\n```\n\nAfter you save and quit the editor, `tester` should become the new privileged\nuser. You can check this with\n\n```\n$ oc get scc -o jsonpath='{range .items[?(.allowPrivilegedContainer==true)].users[*]}{@}{\"\\n\"}{end}' | grep -Ee ':tester$'\nsystem:serviceaccount:lsr-test-harness:tester\n$\n```\n\nIf you are not a cluster-admin, ask someone more powerful to make `tester`\nprivileged user for you.\n\nSome OpenShift servers selects nodes in which deployments should run according\nto node selectors. For example, if you need to run your deployments on bare\nmetal node and OpenShift server selects such a node for deployments that have\nnode selector set to `system-roles-ci = \"true\"`, run the following command for\nboth `linux-system-roles` and `linux-system-roles-staging` deployment configs:\n\n```\n$ ./apply-node-selectors linux-system-roles\n$ ./apply-node-selectors linux-system-roles-staging\n```\n\nIf you need to run containers as a root, run:\n\n```\n$ ./grant-root-privileges linux-system-roles\n$ ./grant-root-privileges linux-system-roles-staging\n```\n\nNow its time to create `ConfigMap` and `Secret` objects. This will create two\n`ConfigMap` objects named `config` and `config-staging` and one `Secret` object\nnamed `secrets`:\n\n```\n$ oc create configmap config --from-file=config.json=${CONFIG_PATH}/config.json\n$ oc create configmap config-staging --from-file=config.json=${CONFIG_PATH}/config-staging.json\n$ oc create secret generic secrets --from-file=${SECRETS_PATH}\n```\n\nOnce all required objects are created, import the images to kick up our\ndeployments:\n\n```\n$ oc import-image linux-system-roles\n$ oc import-image linux-system-roles:staging\n```\n\nCongratulations! Now you can check via OpenShift web console that the things\nare properly set up. You can browse the statuses, scale a number of pods in\neach deployment up or down and do many other operations.\n\n## Logging\n\nIf you want to turn on debug logging for a particular deployment, just set\n`TEST_HARNESS_DEBUG=true` in the deployment config e.g.\n`oc set env dc/linux-system-roles TEST_HARNESS_DEBUG=true`\n\nIf you need to do more fine-grained log level setting, use `oc edit configmap\nconfig` or `oc edit configmap config-staging` and edit the `\"logging\"` section -\nchange both the `\"level\"` and the `\"stderr_level\"` or `\"file_level\"`.  You will\nneed to rollout the `dc` for the change to go into effect (there is no trigger\nfor configmap changes):\n\n```\n$ oc edit configmap config\n...\n$ oc rollout latest dc/linux-system-roles\n$ oc get pods -w\n```\n\nFor staging:\n\n```\n$ oc edit configmap config-staging\n...\n$ oc rollout latest dc/linux-system-roles-staging\n$ oc get pods -w\n```\n\nWhen using file logging e.g.\n\n```\n  \"logging\": {\n    \"filename\": \"/var/log/test-harness_HOSTNAME.log\",\n    \"file_level\": \"DEBUG\",\n```\n\nThe file is written inside the container.  If you want to see the file, you will need\nto use `oc exec` or `oc rsh`:\n\n```\n$ oc get pods\nNAME                                  READY     STATUS       RESTARTS   AGE\nlinux-system-roles-12-ddtrs           1/1       Running      57         55d\n...\n$ oc exec linux-system-roles-12-ddtrs -- cat /var/log/test-harness_linux-system-roles-12-ddtrs.log\n```\n\n## run-tests Command Line Args and Environment Variables\n\nThe `run-tests` script can be controlled via command line arguments,\nenvironment variables, and the config.json config file.  When running in a pod\nin a Kubernetes/OpenShift environment, the easiest way to customize the pod is\nvia environment variables.  The environment variables correspond to the\ncommand line arguments to `run-tests`. In general, the name of the environment\nvariable is the name of the command line argument, in all upper case, with `_`\ninstead of `-`, and with the prefix `TEST_HARNESS_`.  For example, the command\nline option `--use-images` can be set via the env. var.\n`TEST_HARNESS_USE_IMAGES`.  See `run-tests` usage for the complete list.\n\nThe precedence is:\n* command line args are highest precedence\n* environment variables are next highest\n* settings in the config.json file are next highest\n* defaults are lowest precedence\n\nFor example, in a running OpenShift cluster, if you wanted to use a custom\nvariant in the string used for the PR status:\n```\noc set env dc/linux-system-roles-staging TEST_HARNESS_VARIANT=my_custom_name\n```\nThis will cause the staging pods to be redeployed and use `my_custom_name` for\nthe PR status reporting string e.g. `centos-7/ansible-29 (my_custom_name)`.\nTo reset:\n```\noc set env dc/linux-system-roles-staging TEST_HARNESS_VARIANT-\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flinux-system-roles%2Ftest-harness","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flinux-system-roles%2Ftest-harness","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flinux-system-roles%2Ftest-harness/lists"}