{"id":28395127,"url":"https://github.com/databiosphere/terra-test-runner","last_synced_at":"2026-02-08T11:34:10.946Z","repository":{"id":39151064,"uuid":"306382704","full_name":"DataBiosphere/terra-test-runner","owner":"DataBiosphere","description":null,"archived":false,"fork":false,"pushed_at":"2025-06-02T01:59:30.000Z","size":335,"stargazers_count":1,"open_issues_count":3,"forks_count":1,"subscribers_count":20,"default_branch":"main","last_synced_at":"2025-06-02T11:48:49.211Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DataBiosphere.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2020-10-22T15:35:17.000Z","updated_at":"2024-12-12T14:40:11.000Z","dependencies_parsed_at":"2024-12-11T17:22:46.065Z","dependency_job_id":"cd98c7d7-141b-4b9c-b656-9c745a0ea273","html_url":"https://github.com/DataBiosphere/terra-test-runner","commit_stats":{"total_commits":100,"total_committers":11,"mean_commits":9.090909090909092,"dds":0.5700000000000001,"last_synced_commit":"ae2938b76a7732e6bf0d517a3bace59c67b1ff9f"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/DataBiosphere/terra-test-runner","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fterra-test-runner","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fterra-test-runner/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fterra-test-runner/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fterra-test-runner/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DataBiosphere","download_url":"https://codeload.github.com/DataBiosphere/terra-test-runner/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fterra-test-runner/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262172565,"owners_count":23270044,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-31T19:39:12.525Z","updated_at":"2026-02-08T11:34:10.937Z","avatar_url":"https://github.com/DataBiosphere.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# terra-test-runner\n\nThis library is used for running tests using Terra service client libraries.\n\nInitial proposal document:\n\n* [Performance Testing Infrastructure Proposal](https://docs.google.com/document/d/11PZIXZwOyd394BFOlBsDjOGjZdC-jwTr_n92nTJvFxw)\n\nJump to sections below:\n\n\u003c!-- TOC --\u003e\n* [terra-test-runner](#terra-test-runner)\n  * [Terminology](#terminology)\n      * [Test Script](#test-script)\n      * [Test Configuration](#test-configuration)\n      * [Test Run](#test-run)\n      * [Test Runner](#test-runner)\n      * [Test Suite](#test-suite)\n  * [Execute a test run](#execute-a-test-run)\n      * [Results are written to files](#results-are-written-to-files)\n      * [Override the server from the command line](#override-the-server-from-the-command-line)\n      * [Lock server during test run](#lock-server-during-test-run)\n      * [Run against a local server](#run-against-a-local-server)\n      * [Use a local Data Repo client JAR file](#use-a-local-data-repo-client-jar-file)\n      * [Set the directory where SA key files live](#set-the-directory-where-sa-key-files-live)\n      * [Running Resiliency Tests](#running-resiliency-tests)\n  * [Collect measurements generated by the server](#collect-measurements-generated-by-the-server)\n      * [Generated during a time interval](#generated-during-a-time-interval)\n      * [Generated during a test run](#generated-during-a-test-run)\n  * [Example workflows](#example-workflows)\n      * [Execute test run and collect measurements](#execute-test-run-and-collect-measurements)\n      * [Collect measurements for a time interval](#collect-measurements-for-a-time-interval)\n      * [Execute a test suite](#execute-a-test-suite)\n  * [Write a new test](#write-a-new-test)\n      * [Add a new test configuration](#add-a-new-test-configuration)\n      * [Add a new test script](#add-a-new-test-script)\n      * [Add a new server specification](#add-a-new-server-specification)\n      * [Add a new deployment script](#add-a-new-deployment-script)\n      * [Add a new test user specification](#add-a-new-test-user-specification)\n      * [Add a new service account specification](#add-a-new-service-account-specification)\n  * [Add a new test suite](#add-a-new-test-suite)\n  * [Collect a new measurement type](#collect-a-new-measurement-type)\n      * [Add a new measurement collection script](#add-a-new-measurement-collection-script)\n      * [Add a new measurement list](#add-a-new-measurement-list)\n  * [Development](#development)\n      * [Requirements](#requirements)\n      * [Package structure](#package-structure)\n      * [BigQuery Schema](#bigquery-schema)\n      * [Debug a test configuration or script](#debug-a-test-configuration-or-script)\n      * [Dependencies](#dependencies)\n      * [Linters](#linters)\n      * [Testing plugin changes locally](#testing-plugin-changes-locally)\n      * [Publish to Artifactory](#publish-to-artifactory)\n  * [Troubleshooting](#troubleshooting)\n\u003c!-- TOC --\u003e\n\n## Terminology\n\nThis testing infrastructure aims to separate the test from the system\nconfiguration that it runs in. This is so that it's easier to profile variations\nof API calls and environment without rewriting a lot of test code.\n\n#### Test Script\n\nA test script contains several API calls to perform some user action(s) or user\njourney. They can be broken down into three parts:\n\n1. Setup (e.g. Create a dataset)\n2. User Journey (e.g. Make one bulk file load API call)\n3. Cleanup (e.g. Delete dataset)\n\nThe User Journey part contains the API call(s) that we want to profile and it\nmay be scaled to run multiple journeys in parallel. By contrast, the Setup and\nCleanup parts contain the API call(s) that we do not want to profile and will\nnot be scaled to run multiple in parallel. For the example above, we could run\nmultiple bulk file loads in parallel to test the performance, but the dataset\ncreation and deletion would be done only once.\n\n#### Test Configuration\n\nA test configuration describes how to set up the test environment, which test\nscript(s) to run, how to scale the test script(s), and how to add stress into\nthe system.\n\nThe environment specification includes settings for:\n\n* Deployment (e.g. developer’s dev namespace, performance testing Kubernetes\n  cluster, how to deploy there)\n* Kubernetes (e.g. initial size of replica set)\n* Application (e.g. Stairway thread pool size, maximum number of bulk file\n  loads)\n\nThe scripts to run specification includes the:\n\n* Set of different User Journeys to run (e.g. bulk file load \u0026 DRS lookup, just\n  DRS lookups) and any parameters they require (e.g. source/destination bucket\n  region, number of files per bulk load) specified as key-value pairs.\n* Number of each User Journey to run (e.g. 500 bulk file loads \u0026 300 DRS\n  lookups, 1000 DRS lookups)\n* Timeout of client threads (e.g. each bulk file load should take \u003c 15 minutes,\n  each DRS lookup should take \u003c 30 seconds)\n\nThe scaling specification includes:\n\n* Number of different users making the User Journey calls.\n\nRetry settings (0.1.6-SNAPSHOT):\n\n* Added `maxRetries` and `timeToWait` properties.\n  * `maxRetries`: maximum number of retries per user journey thread (default: 3)\n  * `timeToWait`: wait time (milliseconds) till next retry (default: 2000 ms)\n\nThe application specification includes (this section applies to resiliency test\ntype)\n\n* Default values of component label key-value pair for locating a Terra\n  application.\n    * The default component label key-value pair is set to\n        * { \"app.kubernetes.io/component\": \"api\" }\n\n* The default setting may not apply to your application, In that case, you need\n  to configure the following fields in the application specification within the\n  test configuration\n    * componentLabel\n    * apiComponentLabel\n\n**This section will be updated as more pieces of the test configuration are\nimplemented.** See the\n[Performance Testing Infrastructure Proposal](https://docs.google.com/document/d/11PZIXZwOyd394BFOlBsDjOGjZdC-jwTr_n92nTJvFxw)\nfor more details on the desired end goal.\n\n#### Test Run\n\nA test run is a single execution of a test configuration.\n\n#### Test Runner\n\nThe test runner executes test configurations. The steps involved in each test\nrun are:\n\n* Re-deploy the API (i.e. Helm delete then upgrade) with the application\n  properties specified by the configuration.\n* Modify the Kubernetes environment, as specified by the configuration.\n* Run the Setup for each test script.\n* Create a client thread pool for each test script specification.\n* Kick off some number of threads, each running one User Journey, as specified\n  by the configuration.\n* Wait until all threads either finish or time out.\n* Run the Cleanup for each test script.\n* Teardown the API deployment, if applicable.\n\nThe implementation of the test runner is where the bulk of the testing\ninfrastructure code lives.\n\n#### Test Suite\n\nA test suite is a collection of test configurations that have some similar\npurpose. For example, a smoke test suite to detect major performance problems\nquickly or a very long running suite to detect possible memory leaks. The test\nconfigurations are run serially.\n\n## Execute a test run\n\nFind a test configuration or suite to execute. Each configuration is a JSON file\nin the resources/configs directory.\n\nCall the Gradle runTest task and pass it the name of the test configuration or\nsuite to execute, along with the output directory to write the results to.\n\n```\n./gradlew runTest --args=\"configOrSuiteFileName outputDirectoryName\"\n  configOrSuiteFileName = file name of the test configuration or suite JSON file\n  outputDirectoryName = name of the directory where the results will be written\n```\n\n#### Results are written to files\n\nFor test configurations, the results are written to the output directory\nspecified in the runTest Gradle command.\n\nFor test suites, the results are written to sub-directories of the output\ndirectory specified in the runTest Gradle command.\n\n#### Override the server from the command line\n\nThe environment variable TEST_RUNNER_SERVER_SPECIFICATION_FILE optionally\noverrides the server configuration specified by either the test suite or\nconfiguration. The server specification is determined by the following, in\norder:\n\n1. environment variable\n2. test suite server property (if running a test suite)\n3. test configuration server property\n\n```\nexport TEST_RUNNER_SERVER_SPECIFICATION_FILE=\"mmdev.json\"\n./gradlew runTest --args=\"configOrSuiteFileName outputDirectoryName\"\n```\n\n#### Lock server during test run\n\nThe Test Runner has multiple functionalities that directly manipulate the\nspecified server: it can deploy specific settings and manipulate kubernetes.\nWhen you are testing, we want to assert that your test one is the only one\nrunning on the specified server. We accomplish this by creating kubernetes\nsecrets specific to the namespace of the designated server. Switching to this \"\nlocking\" mode is as simple as switching the gradle command from \"runTest\" to \"\nlockAndRunTest,\"\nas demonstrated below. Note: We currently require the\nTEST_RUNNER_SERVER_SPECIFICATION_FILE environment variable to be set, and this\noverrides any server specification in test and suite configs.\n\n```\nexport TEST_RUNNER_SERVER_SPECIFICATION_FILE=\"mmdev.json\"\n./gradlew lockAndRunTest --args=\"configOrSuiteFileName outputDirectoryName\"\n  configOrSuiteFileName = file name of the test configuration or suite JSON file\n  outputDirectoryName = name of the directory where the results will be written\n```\n\nIt is possible that the locks can get out of sync, so if you think you've\nincorrectly hit an \"unable to lock\" error, you can you the following commands to\ndirectly lock or unlock the namespace. Make sure the\nTEST_RUNNER_SERVER_SPECIFICATION_FILE environment variable is set.\n\n```\nexport TEST_RUNNER_SERVER_SPECIFICATION_FILE=\"mmdev.json\"\n./gradlew lockNamespace\n./gradlew unlockNamespace\n```\n\n#### Run against a local server\n\nThere is a localhost.json server specification file in the resources/server\ndirectory. This file contains a filepath to the top-level directory of the\njade-data-repo Git repository. Executing a test against this configuration, will\nstart a local Data Repo server by executing the Gradle bootRun task from that\ndirectory. This is useful for debugging or testing local server code changes.\n\nYou need to modify the path for your own machine. See\ndeploymentScript.parameters (version 0.1.2-SNAPSHOT or below) or deploymentScript.parametersMap (version 0.1.3-SNAPSHOT or above).\n\nFor version up to 0.1.2-SNAPSHOT\n```json\n{\n  \"name\": \"localhost\",\n  \"description\": \"Server running locally. Supports launching the server in a separate process. Does not support modifying Kubernetes post-deployment.\",\n  \"datarepoUri\": \"http://localhost:8080/\",\n  \"samUri\": \"https://sam.dsde-dev.broadinstitute.org\",\n  \"samResourceIdForDatarepo\": \"broad-jade-dev\",\n  \"deploymentScript\": {\n    \"name\": \"LaunchLocalProcess\",\n    \"parameters\": {\n      \"tdr-file-path\": \"file:///Users/marikomedlock/Workspaces/jade-data-repo/\"\n    }\n  },\n  \"skipDeployment\": false,\n  \"skipKubernetes\": true\n}\n```\nFor version up to 0.1.3-SNAPSHOT or above\n```json\n{\n  \"name\": \"localhost\",\n  \"description\": \"Server running locally. Supports launching the server in a separate process. Does not support modifying Kubernetes post-deployment.\",\n  \"datarepoUri\": \"http://localhost:8080/\",\n  \"samUri\": \"https://sam.dsde-dev.broadinstitute.org\",\n  \"samResourceIdForDatarepo\": \"broad-jade-dev\",\n  \"deploymentScript\": {\n    \"name\": \"LaunchLocalProcess\",\n    \"parametersMap\": {\n      \"tdr-file-path\": \"file:///Users/marikomedlock/Workspaces/jade-data-repo/\"\n    }\n  },\n  \"skipDeployment\": false,\n  \"skipKubernetes\": true\n}\n```\n\n#### Use a local Data Repo client JAR file\n\nThe version of the Data Repo client JAR file is specified in the build.gradle\nfile in this sub-project. This JAR file is fetched from the Broad Institute\nMaven repository. You can override this to use a local version of the Data Repo\nclient JAR file by specifying a Gradle project property, either with a command\nline argument\n\n`./gradlew -Pdatarepoclientjar=/Users/marikomedlock/Workspaces/jade-data-repo/datarepo-client/build/libs/datarepo-client-1.0.39-SNAPSHOT.jar run --args=\"configs/BasicUnauthenticated.json`\n\nor an environment variable.\n\n```\nexport ORG_GRADLE_PROJECT_datarepoclientjar=../datarepo-client/build/libs/datarepo-client-1.0.39-SNAPSHOT.jar\n./gradlew runTest --args=\"configs/BasicUnauthenticated.json /tmp/TestRunnerResults\"\n```\n\nThis is useful for debugging or testing local server code changes that affect\nthe generated client library (e.g. new API endpoint). You can generate the Data\nRepo client library with the Gradle assemble task of the datarepo-client\nsub-project.\n\n```\ncd /Users/marikomedlock/Workspaces/jade-data-repo/datarepo-client\n../gradlew clean assemble\nls -la ./build/libs/*jar\n```\n\n#### Set the directory where SA key files live\n\nThe Test Runner looks for service account key files in the directory specified\nin the service account JSON file\n(e.g. jade-k8-sa.json). You can override this directory with an environment\nvariable.\n\n```\nexport TEST_RUNNER_SA_KEY_DIRECTORY_PATH=\"/github/workspace\"\n./gradlew runTest --args=\"configs/BasicUnauthenticated.json /tmp/TestRunnerResults\"\n```\n\n#### Running Resiliency Tests\n\n***Background***\n\nTest Runner Framework supports resiliency tests in addition to Integration,\nPerformance, and Connected tests. As a premier genomic platform for biomedical\nresearch, the [*Terra.Bio*](https://terra.bio) migration to *PaaS* or Cloud\ninfrastructures (GCP, AWS, Azure) to create the new *MCTerra* platform, is a key\nstep to advance the field of genomic science. Without the migration and data\ncrunching capability that comes with the migration, it would be difficult for\nresearchers to develop, experiment with and evaluate next generation *SOTA*\nalgorithms for genomic computing at scale. Our goal is to ensure that the *\nMCTerra* platform continues to function and scale properly as demand continues\nto increase.\n\nGiven the above context, *Containerization* has become the *de facto* go to\nparadigm for managing shared cloud resources the *MCTerra* platform requires\nduring its lifecycles. Although autoscaling and dynamic load balancing are some\nof the key techniques for managing shared cloud resources, policies on how these\ntechniques should apply are often set by *development*, *devOps*, *QA* managers\niteratively over the course of the software lifecycle. Without the proper tool\nto analyze a containerized application, stakeholders will be forced to set these\npolicies in *ad hoc* manners, risking either insufficient resources to meet\ndemand or too many idling resources. Resiliency testing fills the gap by\ngenerating metrics that stakeholders can use to understand the performance of\ncontainerized applications and make comparisons across cloud vendors. Resiliency\ntests can trigger additional logging in targeted *MCTerra* service components\nfor further analysis.\n\n***What is a resiliency test?***\n\nTest Runner provides the framework to target containerized applications with\nspecific loads while at the same time controlling the cloud resources given to\nthe containerized applications. During a resiliency test flight, the framework\nspawns concurrent threads to scale cloud resources up and down while delivering\nload to the target *MCTerra* service components at scale according to user\nspecifications. Resiliency tests can be integrated with a CI/CD pipeline such as\nGitHub Action Workflows, or they can be run locally for debugging purposes.\n\nTest Runner Framework supports resiliency tests on a ***namespaced*** test\nenvironment. The following discussion assumes a valid namespace already exists\nin a Kubernetes cluster. Please refer\nto [Creating the namespace](https://github.com/DataBiosphere/terra/blob/main/docs/dev-guides/personal-environments.md#step-3-creating-the-namespace)\nfor more details about namespace creation in *MCTerra*.\n\n***Requirements on running resiliency test***\n\nAt a high level, running resiliency tests within namespaces require a set of\npermissions to manipulate cluster resources with Kubernetes API. These\npermissions are namespace scoped, no resiliency tests will have cluster-wide\naccess.\n\nThe required namespace permissions are specified in the 3 manifest templates\nwhich comes with the Test Runner Library distribution.\nThe `setup-k8s-testrunner.sh` script templates the formation of the actual\nmanifests for deploying to a namespace. The `setup-k8s-testrunner.sh` script\nalso carries out the following functions:\n\n* Provision the Kubernetes Service Account, RBAC Role and RoleBinding for Test\n  Runner.\n* Export credentials of the Test Runner Kubernetes Service Account to Vault.\n\n***Setting up existing namespaces for resiliency tests***\n\nTo set up a namespace for Test Runner resiliency tests, simply run the command\nas provided in the following example (`terra-zloery` namespace for example).\n\nThe first argument is the `kubectl context` mentioned elsewhere in this\ndocument.\n\nThe second argument is the Terra namespace (without the `terra-` prefix).Without\n\nThe third argument is just some text to describe the application itself.\n\n```shell script\n$ ./setup-k8s-testrunner.sh gke_terra-kernel-k8s_us-central1-a_terra-integration zloery workspacemanager\n```\n\nIn summary, the script automatically templates in the\nvariables `__KUBECONTEXT__, __NAMESPACE__, __APP__` based on the 3 arguments\npresented above and creates the necessary namespace objects in Kubernetes that\nenables Test Runner to control the namespace. Please follow\nthis [link](https://github.com/DataBiosphere/terra-test-runner/blob/main/setup-k8s-testrunner.sh)\nto find out more details about this script.\n\nThe above script consumes the following template manifest files representing\nobjects in Kubernetes namespace. There is no need to apply these template files\nmanually.\n\n\u003cdetails\u003e\n\n\u003csummary\u003etestrunner-k8s-serviceaccount.yml.template\u003c/summary\u003e\n\n```yaml\n# Do not modify this template file.\n\n# This template file is used for setting a Test Runner K8S Service Account\n# for running resiliency tests in a namespace.\n#\n# This template file is to be used in conjunction with the other template files\n#\n#   testrunner-k8s-role.yml.template\n#   testrunner-k8s-rolebinding.yml.template\n#\n# within an automation pipeline and is not meant to be run separately or manually.\n\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n  labels:\n    app.kubernetes.io/component: __APP__\n  name: testrunner-k8s-sa\n  namespace: terra-__NAMESPACE__\n```\n\n\u003csummary\u003etestrunner-k8s-role.yml.template\u003c/summary\u003e\n\n```yaml\n# Do not modify this template file.\n\n# This template file is used for setting a Test Runner privileged RBAC role\n# for running resiliency tests in a namespace.\n#\n# This template file is to be used in conjunction with the other template files\n#\n#   testrunner-k8s-sa.yml.template\n#   testrunner-k8s-rolebinding.yml.template\n#\n# within an automation pipeline and is not meant to be run separately or manually.\n\napiVersion: rbac.authorization.k8s.io/v1\nkind: Role\nmetadata:\n  name: testrunner-k8s-role\n  # A k8s namespace: e.g. terra-wsmtest, terra-ichang.\n  # Avoid using default or system namespaces such as kube-system.\n  namespace: terra-__NAMESPACE__\n  labels:\n    app.kubernetes.io/component: __APP__\nrules:\n- apiGroups: [ \"\" ]\n  resources: [ \"pods\", \"pods/exec\" ]\n  verbs: [ \"get\", \"list\", \"watch\", \"delete\", \"patch\", \"create\", \"update\" ]\n- apiGroups: [ \"extensions\", \"apps\" ]\n  resources: [ \"deployments\", \"deployments/scale\" ]\n  verbs: [ \"get\", \"list\", \"watch\", \"delete\", \"patch\", \"create\", \"update\" ]\n```\n\n\u003csummary\u003etestrunner-k8s-rolebinding.yml.template\u003c/summary\u003e\n\n```yaml\n# Do not modify this template file.\n\n# This template file is used for binding a Test Runner K8S Service Account\n# to a privileged RBAC role for running resiliency tests in a namespace.\n#\n# This template file is to be used in conjunction with the other template files\n#\n#   testrunner-k8s-sa.yml.template\n#   testrunner-k8s-role.yml.template\n#\n# within an automation pipeline and is not meant to be run separately or manually.\n\napiVersion: rbac.authorization.k8s.io/v1\nkind: RoleBinding\nmetadata:\n  name: testrunner-k8s-sa-rolebinding\n  # A k8s namespace: e.g. terra-wsmtest, terra-ichang.\n  # Avoid using default or system namespaces such as kube-system.\n  namespace: terra-__NAMESPACE__\n  labels:\n    app.kubernetes.io/component: __APP__\nsubjects:\n# Authorize In-Cluster Service Account\n- kind: ServiceAccount\n  name: testrunner-k8s-sa\n  namespace: terra-__NAMESPACE__\nroleRef:\n  kind: Role\n  name: testrunner-k8s-role\n  apiGroup: rbac.authorization.k8s.io\n```\n\n\u003c/details\u003e\n\n***Rendering credentials for resiliency tests***\n\nThe Kubernetes credentials stored in Vault needs to be rendered by means of\nthe `./render-k8s-config.sh` script in the application repository before kicking\noff resiliency tests.\n\nUsing namespace `terra-zloery` as example:\n\n```shell script\n$ ./render-k8s-config.sh zloery\n```\n\nNow the namespace (`terra-zloery` in above example) is ready for resiliency\ntesting through Test Runner Framework.\n\n## Collect measurements generated by the server\n\nFind the list of measurements to collect. Each list is a JSON file in the\nresources/measurementlists directory. There are 2 options for how to call the\ncollectMeasurements task.\n\n#### Generated during a time interval\n\nCall the Gradle collectMeasurements task and pass it the list of measurements to\ncollect, the output directory to write the results to, the server to query, and\nthe start and end timestamps for the time interval.\n\n```\n./gradlew collectMeasurements --args=\"measurementListFileName outputDirectoryName serverFileName startTimestamp endTimestamp\"\n  measurementListFileName = file name of the measurement list JSON file\n  outputDirectoryName = name of the directory where the results will be written\n  serverFileName = name of the server JSON file\n  startTimestamp = start of the interval; format must be yyyy-mm-dd hh:mm:ss[.fffffffff] (UTC timezone)\n  endTimestamp = end of the interval; format must be yyyy-mm-dd hh:mm:ss[.fffffffff] (UTC timezone)\n```\n\n#### Generated during a test run\n\nCall the Gradle collectMeasurements task and pass it the list of measurements to\ncollect, and the output directory to write the results to. If the output\ndirectory is the same one where a test run results are written, then it will\npull the start and end timestamps from the summary file for the test run.\n\n```\n./gradlew collectMeasurements --args=\"measurementListFileName outputDirectoryName\"\n  measurementListFileName = file name of the measurement list JSON file\n  outputDirectoryName = name of the same directory that contains the Test Runner results\n```\n\n## Example workflows\n\n#### Execute test run and collect measurements\n\nExecute a test configuration and collect the measurements generated by the\nserver during the run.\n\n```\n  ./gradlew runTest --args=\"configs/basicexamples/BasicUnauthenticated.json /tmp/TestRunnerResults\"\n  ./gradlew collectMeasurements --args=\"BasicKubernetes.json /tmp/TestRunnerResults\"\n```\n\n#### Collect measurements for a time interval\n\nCollect the measurements generated by the server for a particular time interval.\n\n```\n  ./gradlew collectMeasurements --args=\"AllMeasurements.json /tmp/TestRunnerResults mmdev.json '2020-08-20 13:18:34' '2020-08-20 13:18:35.615628881'\"\n```\n\n#### Execute a test suite\n\nExecute a set of test configurations.\n\n```\n  ./gradlew runTest --args=\"suites/BasicSmoke.json /tmp/TestRunnerResults\"\n```\n\n## Write a new test\n\n#### Add a new test configuration\n\nA test configuration is an instance of the TestConfiguration POJO class,\nserialized into JSON and saved in the resources/configs directory. Below are the\navailable fields:\n\n* name: Name of the configuration\n* description: (optional) Description of the configuration\n* serverSpecificationFile: Name of a file in the resources/servers directory\n  that specifies the server to test against\n* billingAccount: Google billing account to use\n* kubernetes: Kubernetes-related settings that will be set after deploying the\n  application and before executing any tests\n    * numberOfInitialPods: (optional) Initial number of pods, defaults to 1\n* application: Application-related settings that will be set before deploying\n  the application and executing any tests\n    * maxStairwayThreads: (optional) defaults to 20\n    * maxBulkFileLoad: (optional) defaults to 1000000\n    * loadConcurrentFiles: (optional) defaults to 80\n    * loadConcurrentIngests: (optional) defaults to 2\n    * inKubernetes: (optional) defaults to false\n    * loadHistoryCopyChunkSize: (optional) defaults to 1000\n    * loadHistoryWaitSeconds: (optional) defaults to 2\n    * loadDriverWaitSeconds: (optional) defaults to 1\n* testScripts: List of test script specifications (i.e. instance of the\n  TestScriptSpecification POJO class, serialized into JSON). Each specification\n  should include the below required fields:\n    * name: Name of the test script class to run\n    * numberOfUserJourneyThreadsToRun: Integer number of user journeys to run\n    * userJourneyThreadPoolSize: Integer number of user journeys to run in\n      parallel (i.e. size of the thread pool)\n    * expectedTimeForEach: Integer number of time units indicating the maximum\n      amount of time a user journey thread will be allowed to execute.\n    * expectedTimeForEachUnit: String representation of the Java TimeUnit\n      class (e.g. MILLISECONDS, SECONDS, MINUTES)\n* testUserFiles: List of names of files in the resources/testusers directory\n  that specify the users whose crendentials will be used to run the test scripts\n\n#### Add a new test script\n\nA test script is a sub-class of the TestScript base class. It specifies the\nsetup and cleanup methods that are run once at the beginning and end of a test\nrun, respectively. It also specifies the userJourney method, which will be\nlaunched in multiple threads in parallel, as specified by the test\nconfiguration.\n\n#### Add a new server specification\n\nA server specification is an instance of the ServerSpecification POJO class,\nserialized into JSON and saved in the resources/servers directory. Below are the\navailable fields:\n\n* name: Name of the server\n* description: (optional) Description of the server\n* uri: URI of the Data Repo instance\n* clusterName: Name of the Kubernetes cluster where the Data Repo instance is\n  deployed\n* clusterShortName: Name of the cluster, stripped of the region and project\n  qualifiers\n* region: Region where the cluster is running\n* zone: Zone where the cluster is running\n* project: Google project under which the cluster is running\n* componentLabel: A Kubernetes metadata label key used to identify the service\n  deployment\n* apiComponentLabel: The value associated with the `componentLabel`, defaults\n  to `api` if unspecified.\n* namespace: (optional) Name of the Kubernetes namespace where the Data Repo\n  instance is deployed\n* deploymentScript: Name of the deployment script class to run. Only required if\n  skipDeployment is false\n* skipDeployment: (optional) true to skip the deployment script, default is\n  false\n* skipKubernetes: (optional) true to skip the post-deployment Kubernetes\n  modifications, default is false\n\n#### Add a new deployment script\n\nA deployment script is a sub-class of the DeploymentScript base class. It\nspecifies the deploy, waitForDeployToFinish, and optional teardown methods that\nare run once at the beginning and end of a test run, respectively.\n\n#### Add a new test user specification\n\nA test user specification is an instance of the TestUserSpecification POJO\nclass, serialized into JSON and saved in the resources/testusers directory.\nBelow are the required fields:\n\n* name: Name of the test user\n* userEmail: Email of the test user\n* delegatorServiceAccountFile: Name of a file in the resources/serviceaccounts\n  directory that specifies the service account with permission to fetch\n  domain-wide delegated credentials for the test user\n\nAll test users must be registered in Terra and there must be a service account\nthat can fetch domain-wide delegated credentials for the user. Jade has already\nsetup test users (see src/main/resources/application-integrationtest.properties)\nand the jade-k8-sa service account to delegate for them. It's probably easiest\nto reuse one of these test users when adding new tests.\n\n#### Add a new service account specification\n\nA service account specification is an instance of the\nServiceAccountSpecification POJO class, serialized into JSON and saved in the\nresources/serviceaccounts directory. Below are the required fields:\n\n* name: Name of the service account\n* jsonKeyFilePath: JSON-formatted file that includes the client ID and private\n  key\n\nThe service account email is stored in the JSON-formatted file.\n\nThe JSON key file for the jade-k8-sa service account match the paths used by the\nrender-configs script in the main datarepo project. Jade stores these files in\nVault and uses the script to fetch them locally for each test run.\n\n## Add a new test suite\n\nA test suite is an instance of the TestSuite POJO class, serialized into JSON\nand saved in the resources/suites directory. Below are the available fields:\n\n* name: Name of the test suite\n* description: (optional) Description of the test suite\n* serverSpecificationFile: Name of a file in the resources/servers directory\n  that specifies the server to test against\n* testConfigurationFiles: List of names of files in the resources/configs\n  directory that specify the test configurations to include in this suite\n\nThe server specification file for the test suite overrides the server\nspecification file for all test configurations contained in the suite.\n\n## Collect a new measurement type\n\n#### Add a new measurement collection script\n\nA measurement collection script is a sub-class of the\nMeasurementCollectionScript base class. It specifies the \"query\"\nto run on the server-generated data to pull down the relevant measurement data,\ncalculate summary statistics on it, and write it out to a file.\n\nThere are two base classes that query Google logs and metrics. If the new\nmeasurement type relies on either of these data stores, it can extend these base\nclasses to reuse the Google client library class and serialization logic when\nwriting out the results to a file\n\n#### Add a new measurement list\n\nA measurement list is an instance of the MeasurementList POJO class, serialized\ninto JSON and saved in the resources/measurementlists directory. Below are the\navailable fields:\n\n* name: Name of the measurement list\n* description: (optional) Description of the measurement list\n* measurementCollectionScripts: List of measurement collection script\n  specifications (i.e. instance of the MeasurementCollectionScriptSpecification\n  POJO class, serialized into JSON). Each specification should include the below\n  fields:\n    * name: Name of the measurement collection script class to run\n    * description: Description of the parametrized metric\n    * parameters: (optional) parameters to pass to the metric collection script (up to version 0.1.2-SNAPSHOT, replaced with parametersMap since version 0.1.3-SNAPSHOT)\n    * parametersMap: (optional) parametersMap to pass to the metric collection script (version 0.1.3-SNAPSHOT or above)\n\n## Development\n\n#### Requirements\n\nJava 17\n\n#### Package structure\n\nAll the Java code is in the src/main/java directory.\n\n* The runner package contains code for executing test configurations, including\n  the POJO classes used to specify a test configuration and suite.\n* The collector package contains code for collecting measurements generated by\n  the server, including the POJO class used to specify a list of measurements.\n* The scripts package contains the \"user-specified\" code:\n    * **testscripts**: user journeys\n    * **deploymentscripts**: methods of application deployment\n    * **disruptivescripts**: types of stress/disruption to apply to the\n      application\n    * **measurementcollectionscripts**: metric definitions\n    * **utils**: utility methods used only **within** this package\n* The common package contains code that may be useful for the test runner and\n  measurement collector code, in addition to the user-specified code in the\n  scripts package.\n\n#### BigQuery Schema\n\nThe `bq` folder consists of a couple `BigQuery` JSON table schema used for\nstoring test results. The `create-testrunner-bq-tables.sh` is used for setting\nup the `BigQuery` tables to store the test results, it requires a Google Project\nID as input. Please refer\nto `Design Document: Test Runner Cloud Function Reporting Infrastructure`[https://docs.google.com/document/d/1sm5J85K9Ihph4fffICePJAkOiRxvYtzt7Aol7U3M5AE/edit#] for\narchitecture related to streaming test results from `GCS` bucket to `BigQuery`.\n\n#### Debug a test configuration or script\n\n* The Gradle run task just calls the main method of the RunTest or\n  MeasurementCollector class. To debug, add a Run/Debug Configuration in\n  IntelliJ that calls this method with the same arguments.\n\n![Intellij run configuration](https://user-images.githubusercontent.com/10929390/158693901-1292131b-7fce-426d-b791-0b1420848a25.png)\n\n* To debug a test script without the test runner, for example to make sure the\n  API calls are coded correctly, add a main method that executes the\n  setup/userjourney/cleanup steps. You can copy the main method in the\n  TestScript base class and paste it into the test script class you want to\n  debug. Then change the method to call the constructor of the test script class\n  you want to debug. Run the test script main method in debug mode.\n\n#### Dependencies\n\nWe use Gradle's [dependency locking](https://docs.gradle.org/current/userguide/dependency_locking.html)\nto ensure that builds use the same transitive dependencies, so they're reproducible. \nThis means that adding or updating a dependency requires telling Gradle to save the change. \nIf you're getting errors that mention \"dependency lock state\" after changing a dep, you need to do this step.\n\n```\n./gradlew dependencies --write-locks\n```\n\n#### Linters\n\nRun the linters before putting up a PR. The output of the spotbugs plugin is in\nthe build/reports/spotbugs/main.txt\n\n```\n./gradlew spotlessApply spotbugsMain\ncat build/reports/spotbugs/main.txt\n```\n\n#### Testing plugin changes locally\n\nIf you're making changes to the plugin, you can test them locally by publishing\nto your local Maven repository. Update the version in `build.gradle`, replacing\n`0.1.10` with the new version you're testing:\n\n```\nversion '0.1.10-SNAPSHOT'\n```\n\nPublish to your local Maven repository:\n\n```shell\n./gradlew publishToMavenLocal\n```\n\nIn the `build.gradle` of the project you're testing with,  update the version for \nthe test-runner plugin, and add `mavenlocal()` to your plugin repositories:\n\n```groovy\nrepositories {\n    mavenLocal()\n}\ndependencies {\n    implementation 'bio.terra:terra-test-runner:0.1.10-SNAPSHOT'\n}\n```\n\nNow you can build your project with the new version of the plugin.\n\n\n#### Publish to Artifactory\n\nPublish the Test Runner library after making changes, once your PR is merged up.\n\n```\n./tools/publish.sh\n```\n\nThe library is published to the URL:\nhttps://console.cloud.google.com/artifacts/maven/dsp-artifact-registry/us-central1/libs-snapshot-standard/bio.terra:terra-test-runner?project=dsp-artifact-registry\n\n## Troubleshooting\n\n* Check that the server specification file property of the test configuration\n  points to the correct URL you want to test against.\n\n* Check that your IP address is included on the IP whitelist for the cluster\n  you're testing against. The easiest way to do this is to connect to the\n  Non-Split Broad VPN, because the VPN IP addresses are already included on the\n  IP whitelist for the Jade dev cluster.\n\n* Check that you are calling the correct version of Gradle (6.1.1 or higher).\n  Use the Gradle wrapper in the sub-project\n  (`.gradlew`), not the one in the parent directory (`../.gradlew`).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabiosphere%2Fterra-test-runner","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatabiosphere%2Fterra-test-runner","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabiosphere%2Fterra-test-runner/lists"}