{"id":20747512,"url":"https://github.com/linuxboot/contest","last_synced_at":"2025-08-08T16:09:30.611Z","repository":{"id":37457084,"uuid":"397307178","full_name":"linuxboot/contest","owner":"linuxboot","description":"Run continuous and on-demand system testing for real and virtual hardware","archived":false,"fork":false,"pushed_at":"2024-05-09T21:45:37.000Z","size":3351,"stargazers_count":19,"open_issues_count":38,"forks_count":18,"subscribers_count":7,"default_branch":"develop","last_synced_at":"2025-08-02T06:07:04.908Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/linuxboot.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS","dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-08-17T15:33:10.000Z","updated_at":"2025-05-27T13:21:36.000Z","dependencies_parsed_at":"2024-05-09T21:44:43.119Z","dependency_job_id":"0fff96ae-35f9-4aea-9cd2-5d23e22dd77d","html_url":"https://github.com/linuxboot/contest","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/linuxboot/contest","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linuxboot%2Fcontest","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linuxboot%2Fcontest/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linuxboot%2Fcontest/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linuxboot%2Fcontest/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/linuxboot","download_url":"https://codeload.github.com/linuxboot/contest/tar.gz/refs/heads/develop","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linuxboot%2Fcontest/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":269449163,"owners_count":24419029,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-08T02:00:09.200Z","response_time":72,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-17T08:13:26.974Z","updated_at":"2025-08-08T16:09:30.573Z","avatar_url":"https://github.com/linuxboot.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ConTest\n[![Tests](https://github.com/linuxboot/contest/workflows/Tests/badge.svg?branch=master)](https://github.com/facebookincubator/contest/actions?query=workflow%3ATests+branch%3Amaster)\n[![codecov](https://codecov.io/gh/linuxboot/contest/branch/master/graph/badge.svg)](https://codecov.io/gh/facebookincubator/contest)\n[![Go Report Card](https://goreportcard.com/badge/github.com/linuxboot/contest)](https://goreportcard.com/report/github.com/facebookincubator/contest)\n\n\u003cimg src=\"https://raw.githubusercontent.com/linuxboot/contest/main/assets/SVG/ConTest_logo_color_transparent.svg\" width=\"150px\"\u003e\n\nConTest is a framework for system testing that aims at covering multiple use cases:\n* system firmware testing\n* provisioning testing\n* system regressions\n* microbenchmarks\n\nand more. It provides a modular and pluggable set of interfaces to define your own methods to:\n* fetch the information about which systems to run your test on\n* describe and validate the initial state for the test systems, for a test to be able to run\n* describe the type of tests to run (test definitions)\n* implement system-specific actions (e.g. how to reboot it) and measurements (e.g. did the host boot into the OS?)\n* maintain test state for each system on a customizable persistent storage\n* describe and validate the final state for the test machines, for a test to be considered successful\n* log the test progress and the final results, and allowing multiple backends\n\n\n## Examples\n\nYou can look at the tools implemented under [cmds/*](cmds/) for usage examples.\n\n## Requirements\n\nConTest is developed and heavily tested in production on Linux with MySQL.\nThere is some minimal testing with MacOS and MariaDB, but no production experience.\nYou will also need a recent version of Go (we recommend at least Go 1.15 at the\nmoment).\n\n\n## Quick Start\n\nWe offer a few Docker files to help setup ConTest. The fastest way to\nhave a ConTest instance up and running is to bring up the server and MySQL\ncontainers via the default docker-compose configuration: just run\n`docker-compose up --build` from the root of the source tree.\nTo use MariaDB instead, run `docker-compose -f docker-compose.mariadb.yml up --build`\nNote: you might need to set DOCKER_BUILDKIT=1 in env for Docker to interpret\nARGs like $TARGETARCH inside the scripts.\n\nContainers can be also orchestrated separately:\n\n* [docker/mysql](docker/mysql) will configure and bring up a MySQL\n  instance with a pre-populated ConTest database that you can use with a local\n  instance. Just run `docker-compose mysql up` from the root of the source tree.\n* [docker/mariadb](docker/mariadb) is an alternative to MySQL. Don't start both!\n* [docker/contest](docker/contest) supports running ConTest as standalone instance\nas explained at the beginning of this section and also supports integration tests.\nFor a full test run please see `run_tests.sh`.\n\n\n## Building ConTest\n\nThe recommended procedure is to download the ConTest framework via `go get`:\n\n```\ngo get -u github.com/linuxboot/contest\n```\nAfter successfully running this command, the ConTest framework will be available in your `GOPATH`, and you can start building your custom system testing infrastructure on top of it.\n\nYou can also have a look at the sample `contest` program:\n```\ngo get -u github.com/linuxboot/contest/cmds/contest\n```\n\nThis will download and build the `contest` sample program, and will add it to your Go binary path.\n\nTo build the sample server from a local checkout of the Git repo, e.g. for\ndevelopment purposes:\n\n```\ncd cmds/contest\ngo build .\n```\n\n## Running the sample server\n\nThere is a sample server under [cmds/contest](cmds/contest). All it does is to\nregister various plugins and user functions, and running ConTest's job manager\nwith a specific API listener.\nYou may want to adapt it to your needs by removing unnecessary plugins and\nadding your custom ones, if necessary.\nAdditionally, the sample server uses the HTTP API listener. You may want to use\na different one or build your own.\n\nAfter building the sample server as explained in the [Building\nConTest](#building-contest) section, run it with no arguments:\n```\n$ cd cmds/contest\n$ go build .\n$ ./contest\n[2023-05-10T13:59:26.361+02:00 I pluginregistry.go:63] Registering target manager csvfiletargetmanager\n[2023-05-10T13:59:26.361+02:00 I pluginregistry.go:63] Registering target manager targetlist\n[2023-05-10T13:59:26.361+02:00 I pluginregistry.go:76] Registering test fetcher literal\n[2023-05-10T13:59:26.361+02:00 I pluginregistry.go:76] Registering test fetcher uri\n[2023-05-10T13:59:26.361+02:00 I pluginregistry.go:89] Registering test step cmd\n[2023-05-10T13:59:26.361+02:00 I pluginregistry.go:89] Registering test step cpucmd\n[2023-05-10T13:59:26.361+02:00 I pluginregistry.go:89] Registering test step echo\n[2023-05-10T13:59:26.362+02:00 I pluginregistry.go:89] Registering test step exec\n[2023-05-10T13:59:26.362+02:00 I pluginregistry.go:89] Registering test step gathercmd\n[2023-05-10T13:59:26.362+02:00 I pluginregistry.go:89] Registering test step randecho\n[2023-05-10T13:59:26.362+02:00 I pluginregistry.go:89] Registering test step sleep\n[2023-05-10T13:59:26.362+02:00 I pluginregistry.go:89] Registering test step sshcmd\n[2023-05-10T13:59:26.362+02:00 I pluginregistry.go:113] Registering reporter noop\n[2023-05-10T13:59:26.362+02:00 I pluginregistry.go:113] Registering reporter targetsuccess\n[2023-05-10T13:59:26.362+02:00 I server.go:207] Using database URI for primary storage: contest:contest@tcp(localhost:3306)/contest?parseTime=true\n[2023-05-10T13:59:26.370+02:00 I server.go:221] Storage version: 7\n[2023-05-10T13:59:26.370+02:00 I server.go:227] Using database URI for replica storage: contest:contest@tcp(localhost:3306)/contest?parseTime=true\n[2023-05-10T13:59:26.372+02:00 I server.go:241] Storage version: 7\n[2023-05-10T13:59:26.372+02:00 I server.go:269] Locker engine set to auto, using DBLocker\n[2023-05-10T13:59:26.375+02:00 D storage.go:52] consistency model check: \u003cnil\u003e\n[2023-05-10T13:59:26.375+02:00 I jobmanager.go:228] Found 0 zombie jobs for /yourservername\n[2023-05-10T13:59:26.375+02:00 D httplistener.go:253] Serving a listener\n[2023-05-10T13:59:26.376+02:00 I httplistener.go:230] Started HTTP API listener on :8080\n```\n\nThe server is informing us that it started the HTTP API listener on port 8080, after registering various types of plugins: target managers, test fetchers, test steps, and reporters.\n\n## Database\n\nConTest requires a database to store its state, events and other data.\n\n* DB name and default user are defined in [docker/mysql/initdb.sql](docker/mysql/initdb.sql)\n* DB schema is defined in [db/rdbms/schema/v0/create_contest_db.sql](db/rdbms/schema/v0/create_contest_db.sql)\n* DB migrations are defined in the [db/rdbms/migration directory](db/rdbms/migration)\n\n### Using docker\n\nWe provide a docker image to bring up a database, so you can use it with the\nsample server. Just run\n```\ndocker-compose up mysql\n```\nfrom the top directory of ConTest's source code.\n\n### Using a local mysql instance\n\nIf you don't want to run Docker, or if you have already a mysql instance you\nwant to use, you can:\n\n1. Either create a \"contest\" local database and a \"contest\" user as described\nin the `initdb.sql` script:\n\n```\nmysql -u root -p \u003c docker/mysql/initdb.sql\n```\n\n2. Load the database schema\n\n```\nmysql -u contest -p \u003c db/rdbms/schema/v0/create_contest_db.sql\n```\n\n3. Run all migrations\n\n```\ncd tools/migration/rdbms\ngo run . -dir ../../../db/rdbms/migration up\n```\n\nOnce the database is up, it will possible to submit test jobs through the client,\nas shown in the next section.\n\n\n## Submitting jobs to the sample server\n\nConTest has no official CLI, because every user is different. However we provide\na sample CLI based on cleartext HTTP that will communicate to the HTTP API. All\nyou need to do is building the sample ConTest server as described above, and run\nthe CLI tool under [cmds/clients/contestcli](cmds/clients/contestcli).\nThe tool is very simple:\n```\n$ cd cmds/clients/contestcli\n$ go build .\n$ ./contestcli -h\nUsage of contestcli:\n\n  contestcli [args] command\n\ncommand: start, stop, status, retry, version\n  start\n        start a new job using the job description passed via stdin\n  stop int\n        stop a job by job ID\n  status int\n        get the status of a job by job ID\n  retry int\n        retry a job by job ID\n  version\n        request the API version to the server\n\nargs:\n  -addr string\n      ConTest server [scheme://]host:port[/basepath] to connect to (default \"http://localhost:8080\")\n  -r string\n      Identifier of the requestor of the API call (default \"contestcli\")\nexit status 2\n```\n\nTo run a job, just feed a [job descriptor](#job-descriptors) to the client's `start` command, e.g. using\n[start-literal.json](cmds/clients/contestcli/descriptors/start-literal.json):\n\n```\n$ ./contestcli start \u003c descriptors/start-literal.json\n\n[..output showing the job descriptor...]\n\nThe server responded with status 200 OK\n{\n  \"ServerID\": \"barberio-ubuntu-9QCS5S2\",\n  \"Type\": \"ResponseTypeStart\",\n  \"Data\": {\n    \"JobID\": 12\n  },\n  \"Error\": null\n}\n```\n\nThen we can get the status of the job using the `status` command and the job ID returned by the `start` request:\n```\n$ go run . status 12 | jq\nRequesting URL http://localhost:8080/status with requestor ID 'contestcli-http'\n  with params:\n    jobID: [32]\n    requestor: [contestcli-http]\n\nThe server responded with status 200 OK{\n  \"ServerID\": \"barberio-ubuntu-9QCS5S2\",\n  \"Type\": \"ResponseTypeStatus\",\n  \"Data\": {\n    \"Status\": {\n      \"EndTime\": \"0001-01-01T00:00:00Z\",\n      \"JobReport\": {\n        \"FinalReports\": [\n          {\n            \"Data\": \"I did nothing at the end, all good\",\n            \"ReportTime\": \"2020-02-10T20:39:54Z\",\n            \"Success\": true\n          }\n        ],\n        \"JobID\": 32,\n        \"RunReports\": [\n          [\n            {\n              \"Data\": {\n                \"AchievedSuccess\": \"100.00%\",\n                \"DesiredSuccess\": \"\u003e80.00%\",\n                \"Message\": \"All tests pass success criteria: 100.00% \u003e 80.00%\"\n              },\n              \"ReportTime\": \"2020-02-10T20:39:48Z\",\n              \"Success\": true\n            },\n            {\n              \"Data\": \"I did nothing on run #1, all good\",\n              \"ReportTime\": \"2020-02-10T20:39:48Z\",\n              \"Success\": true\n            }\n          ],\n          [\n            {\n              \"Data\": {\n                \"AchievedSuccess\": \"100.00%\",\n                \"DesiredSuccess\": \"\u003e80.00%\",\n                \"Message\": \"All tests pass success criteria: 100.00% \u003e 80.00%\"\n              },\n              \"ReportTime\": \"2020-02-10T20:39:51Z\",\n              \"Success\": true\n            },\n            {\n              \"Data\": \"I did nothing on run #2, all good\",\n              \"ReportTime\": \"2020-02-10T20:39:51Z\",\n              \"Success\": true\n            }\n          ],\n          [\n            {\n              \"Data\": {\n                \"AchievedSuccess\": \"100.00%\",\n                \"DesiredSuccess\": \"\u003e80.00%\",\n                \"Message\": \"All tests pass success criteria: 100.00% \u003e 80.00%\"\n              },\n              \"ReportTime\": \"2020-02-10T20:39:54Z\",\n              \"Success\": true\n            },\n            {\n              \"Data\": \"I did nothing on run #3, all good\",\n              \"ReportTime\": \"2020-02-10T20:39:54Z\",\n              \"Success\": true\n            }\n          ]\n        ]\n      },\n      \"Name\": \"test job\",\n      \"StartTime\": \"2020-02-10T20:39:48Z\",\n      \"State\": \"\",\n      \"TestStatus\": [\n        {\n          \"TestName\": \"Literal test\",\n          \"TestStepStatus\": [\n            {\n              \"Events\": [],\n              \"TargetStatus\": [\n                {\n                  \"Error\": null,\n                  \"InTime\": \"2020-02-10T20:39:54Z\",\n                  \"OutTime\": \"2020-02-10T20:39:54Z\",\n                  \"Target\": {\n                    \"FQDN\": \"\",\n                    \"ID\": \"1234\",\n                    \"Name\": \"example.org\"\n                  }\n                }\n              ],\n              \"TestStepLabel\": \"cmd\",\n              \"TestStepName\": \"Cmd\"\n            }\n          ]\n        }\n      ]\n    }\n  },\n  \"Error\": null\n}\n```\n\n## How does ConTest work\n\nConTest is a framework, not a program. You can use the framework to create your own system testing infrastructure on top of it.\nWe also provide a few sample programs under the [cmds/*](cmds/) directory that you can look at.\n\n### Interfaces and plugins\n\nConTest is heavily based on interfaces and plugins. *Interfaces* are contracts\nbetween the framework and its components, while *plugins* are implementations of\nsuch interfaces.\n\nFrom a high level perspective, the framework is modelled as a sequence of\nstages, starting from the API listener until the reporting (see the\n[design document](#design-document) section).\n\nEach stage is defined by an interface, and can be implemented with a plugin. For\nexample, the API listener defines [an interface](/pkg/api/listener.go) that\nrequires a `Serve` method. The [http listener plugin](/plugins/listeners/http) implements such\nmethod, and exposes various API operations like start or retry, and provides an HTTP endpoint to\ncommunicate with the internal API. Of course you can implement your own plugin\nto use a different protocol. It just has to respect the Listener interface and\ndo something useful.\n\nSimilarly, test steps are defined by the test step interface (see\n[pkg/test/step.go](/pkg/test/step.go)). A plugin simply needs to implement such\ninterface and respect a few basic rules as defined in the developer documentation\n(TODO). See for example the [sshcmd](/plugins/teststeps/sshcmd) plugin.\n\nConTest offers various plugins out of the box, which should be sufficient\nfor many use cases, but if you need more feel free to contribute with a pull\nrequest, or to open an issue for a feature request. We are open to contributions\nthat are useful to the community!\n\n[PLUGINS.md](ConTest’s Step Plugin Author’s Guide) explains the step plugin API.\n\n### Job descriptors\n\nOne of the main concepts in ConTest is the job descriptor. It describes how a test job will behave on each device under test.\nA job descriptor has to declare:\n* how to get the [targets](#targets) to run the test jobs on\n* how to get the [test descriptors](#test-descriptors) that will be executed for each target\n* how to report success or failure\n\nAn example of a job descriptor that runs a command over SSH to a target host\nis available at [cmds/clients/contestcli/descriptors/start.json](cmds/clients/contestcli/descriptors/start.json), reported below and\ncommented for clarity:\n```\n{\n    // The job name. This can be used to search and correlate job events.\n    \"JobName\": \"test job\",\n    // number of times a job should be run. 0 or a negative number means\n    // \"run forever\". Also see TestDescriptors below.\n    \"Runs\": 5,\n    // How much time to wait between runs. Any string suitable for\n    // [time.ParseDuration](https://golang.org/pkg/time/#ParseDuration) will work.\n    // Also see TestDescriptors below.\n    \"RunInterval\": \"5s\",\n    // Tags can be used for search and aggregation. Currently not used.\n    \"Tags\": [\"test\", \"csv\"],\n    // A list of test descriptors that contain all the information to run a\n    // job. At least one test descriptor is required (like in the example below),\n    // but there is virtually no limit to how many descriptors a user can specify.\n    // Each test associated to a descriptor is run sequentially. This is\n    // intentional, and parallelism can be achieved in other ways.\n    //\n    // Note: the Runs parameter above is the number of times that all the test\n    // descriptors are run in total, sequentially.\n    // Note: the RunInterval parameter above is the time the framework will wait\n    // between multiple runs of a list of test descriptors.\n    \"TestDescriptors\": [\n        {\n            // The name of the target manager that will be called to get a list\n            // of targets to run tests against. The plugins must be registered\n            // (see cmds/contest/main.go). Various\n            // target manager plugins are already available in\n            // [plugins/targetmanagers](plugins/targetmanagers).\n            \"TargetManagerName\": \"CSVFileTargetManager\",\n            // parameters to be passed to the target manager. This is dependent\n            // on the actual plugin.\n            \"TargetManagerAcquireParameters\": {\n                // the URI for the CSV file that contains a list of targets in\n                the format \"id,fqdn,ipv4,ipv6\". This is intentionally very simple.\n                \"FileURI\": \"hosts.csv\",\n                // The minimum number of targets needed for the test. If we\n                // don't get at least this number of devices, the job will fail.\n                \"MinNumberDevices\": 10,\n                // The maximum number of targets that we need. The plugin should\n                // try to always return this number of targets, unless fewer are\n                // available.\n                \"MaxNumberDevices\": 20,\n                // the host name prefixes used to filter the devices from the\n                // CSV file. For example, compute001.facebook.com will match in\n                // the example below, but web001.facebook.com will not.\n                \"HostPrefixes\": [\n                    \"compute\",\n                    \"storage\"\n                ]\n            },\n            // parameters that are passed to the target manager when it is asked\n            // to release the targets at the end of a test. This is also depending\n            // on the plugin. In this case there is no parameter for releasing\n            // the targets.\n            \"TargetManagerReleaseParameters\": {\n            },\n            // The name of the plugin used to fetch the test definitions. The\n            // test fetcher plugins must be registered in main.go just like we\n            // do for target managers (see above).\n            // Tests can be stored somewhere else, so the concept of fetching\n            // tests.\n            // Here we are using the URI plugin, which allows specifying file\n            // paths that are local to the ConTest server (note: not local to the\n            // client), but also http:// and https:// in case you prefer to\n            // store files on a remote HTTP endpoint. New plugins can be\n            // developed to implement more complex fetching logic.\n            //\n            // Note: if you prefer to specify the test steps inline in the test\n            // descriptor you can do that using the \"literal\" test fetcher plugin.\n            // See cmds/clients/http/start-literal.json for an example.\n            \"TestFetcherName\": \"URI\",\n            // Parameters to be passed to the test fetcher. This is dependent on\n            // the plugin.\n            \"TestFetcherFetchParameters\": {\n                // Test name used for searching and correlation.\n                \"TestName\": \"My Test Name\",\n                // URI of the JSON containing the actual test step definitions.\n                // This is a file that is local to the ConTest server. You can\n                // also specify an absolute path, or use any of file://, http://\n                // and https:// schemes.\n                //\n                // Note: in the section below we will show a commented example\n                // of test step definitions.\n                \"URI\": \"test_samples/randecho.json\"\n            }\n        }\n    ],\n    // The reporting section is divided in two parts: run reporters and final\n    // reporters. At least one reporter must be specified, of any type, or the\n    // job will be rejected.\n    \"Reporting\": {\n        // Run reporters are plugins that are executed at the end of each test\n        // run.\n        // They are meant to report results of individual runs, as opposed to\n        // the full job inclusive of all of its runs. The number of runs is\n        // specified above in the job descriptor.\n        //\n        // A reporter plugin can implement run reporting, or final reporting, or\n        // both. The two implementations would normally be different and\n        // independent.\n        \"RunReporters\": [\n            {\n                // TargetSuccess is the name of the reporter plugin. Every\n                // plugin has its own specific arguments, so there is no common\n                // format. The plugin will know how to parse the JSON\n                // subdocument.\n                // In this example, TargetSuccess will determine the success of\n                // the job based on how many devices could make it until the end\n                // of the test. Both percentage and target numbers can be\n                // specified.\n                \"Name\": \"TargetSuccess\",\n                \"Parameters\": {\n                    \"SuccessExpression\": \"\u003e80%\"\n                }\n            },\n            {\n                // This is another run reporter. It will be executed after the\n                // above, and it will have its own concept of \"success\". Every\n                // reporter carries its own success indicator, so there is no\n                // overall \"this job was successful\", rather \"this reporter was\n                // successful\". For example, one reporter may succeed if enough\n                // targets made it until the end of the tests, whike another\n                // reporter may measure performance regressions, and fail. The\n                // two results would be independent from each other.\n                \"Name\": \"Noop\"\n            }\n        ],\n        // Final reporters instead will be executed at the very end, and will\n        // normally be used to report the results of the whole set of runs.\n        //\n        // If the job is running in continuous mode (i.e. Runs is 0), the final\n        // reporters will only be run if the job is cancelled.\n        //\n        // In the example below there is only one reporter, but it's possible to\n        // specify more, just like for run reporters.\n        \"FinalReporters\": [\n            {\n                \"Name\": \"noop\"\n            }\n        ]\n    }\n\n    // The name of the reporter plugin. This is used to decide and report\n    whether a job was successful or not.\n    \"ReporterName\": \"TargetSuccess\",\n    // The parameters to a reporter plugin. This is is plugin-dependant too.\n    \"ReporterParameters\": {\n        // The TargetSuccess plugin allows for simple expressions like \"\u003e=50%\"\n        // or \"\u003e10\", meaning respectively \"at least 50% of the targets\", and\n        // \"more than 10 targets\". This is a common reporting scenario, but we\n        // expect users to implement their own custom reporting logic.\n        \"SuccessExpression\": \"\u003e50%\"\n   }\n}\n```\n\n### Test fetchers\n\nTest fetchers are responsible for retrieving the test steps that we want to run\non each target.\n\nTest steps are encoded in JSON format. For example:\n```\n{\n    \"steps\": [\n        {\n            \"name\": \"cmd\",\n            \"label\": \"some label\",\n            \"parameters\": {\n                \"executable\": [\"echo\"],\n                \"args\": [\"Hello, world!\"]\n            }\n        }\n    ]\n}\n```\n\nThe above test steps describe a test run with a single step. Such step will\nuse the `cmd` plugin to execute the command `echo \"Hello, world!\"` on the\nConTest server.\nIf you want to add more test steps, just add more items to the `steps` list.\n\nIn the [job descriptors](#job-descriptors) paragraph we have shown an example of\nusing the `URI` test fetcher. The `URI` plugin lets you get your test steps\nusing an URI, e.g. \"https://example.org/test/my-test-steps.json\". This is\nconvenient when storing and reusing test steps in a remote repository or in a\nlocal file, but one size does not fit all. You may want, for example, specify\nall of your test steps inline. For the purpose we wrote the `literal` test\nfetcher plugin, which allows you to specify the test steps inline in the job\ndescriptor. Just insert the content of your JSON file containing the test steps\ninto the job descriptor. For an example, see\n[start-literal.json](cmds/clients/contestcli-http/start-literal.json).\n\nTODO where are they stored\nTODO square brackets\n\n### Targets\n\nTargets are a generic representation of the entity where test jobs are run on.\n\nEvery job is associated to a list of targets, and what actions (and how) are\nexecuted on each target depends on the plugin.\n\nTargets are currently defined with three properties:\n* **ID**: primary identifier for the target. Must be unique within the scope of this ConTest instance.\n  Common choices are IDs from inventory management systems, DNS short or full names,\n  or textual representations of ipv4/ipv6 addresses. Storage plugins might enforce uniqueness.\n* **FQDN**: DNS-resolvable name of the target, ideally a FQDN. Plugins can use\n  this field to contact the test target.\n* **PrimaryIPv6**/**PrimaryIPv4**: Raw IP address used by plugins to contact the test target.\n\nOnly **ID** is required, but it is recommended to set as many fields as possible\nfor maximum plugin compatibility. Note that no validation is done on FQDNs or IP addresses.\n\nThe `Target` structure is defined in [pkg/target](pkg/target/target.go). Plugin\nconfigurations can access the specific fields via Go templates, as explained in\nmore detail in the [Templates in test step arguments](#templates-in-plugin-configurations)\nsection. For example, to print a target's ID and FQDN with the `cmd` plugin:\n\n```\n{\n    \"steps\": [\n        {\n            \"name\": \"cmd\",\n            \"label\": \"some label\",\n            \"parameters\": {\n                \"executable\": [\"echo\"],\n                \"args\": [\"ID is {{ .ID }} and FQDN is {{ .FQDN }}\"]\n            }\n        }\n    ]\n}\n```\n\nThis will be expanded and executed for every target in the test job.\n\n### Templates in plugin configurations\n\nMany plugins support Go templating in the test step definitions using\n[text/template](https://golang.org/pkg/text/template/). This means that\nit is possible to use functions and substitutions to configure a plugin's action\nmore dynamically. This is useful for example for steps that need target-specific\ninformation, or when the plugin configuration needs some run-time or dynamic\nmanipulation.\n\nThe standard Go templating functions are available, and it is also possible to\nregister user functions.\nThe `Target` object is passed to the template as the root object.\n\nFor example, to instruct the `cmd` plugin to print the name of a target, the\nfollowing snippet can be used in the test step configuration. Note: the `cmd`\nplugin executes a command on the ConTest server, not on the targets.\n\n```\n...\n    {\n        \"name\": \"cmd\",\n        \"label\": \"some label\",\n        \"parameters: {\n            \"executable\": [\"echo\"],\n            \"args\": [\"My ID is {{ .ID }}\"]\n        }\"\n    }\n...\n```\n\nLet's dissect the above.\n\n* the \"name\" directive tells ConTest to run the \"cmd\" plugin. This plugin will\n  be run on every target in the test job. It's the plugin's responsibility to\n  know what to do with the targets: the framework simply routes them to each\n  step's plugin.\n* the \"parameters\" map contains the arguments to pass to the plugin.\n* the \"executable\" parameter specifies what to execute\n* the \"args\" parameter is the list of arguments to pass to the program execution\n\nNote that \"args\" contains only one argument, and this argument uses the Go\ntemplating syntax. `.ID` expands to the value contained in `Target.ID`,\nsince the target is the root object passed to the template. This means that you\ncan also use `.FQDN` or `.PrimaryIPv6` if you want to access other members of the target\nstructure.\nAfter the name expansion is done, the resulting string will be unique per\ntarget, and ConTest will execute the \"echo\" command with this customized output\nfor each target.\n\nSubstitution is not the only thing one can do with templates. Functions are also\navailable, for example to further manipulate our configuration data. All of the\nbuilt-in functions from `text/template` are available. To slice a string one can\nwrite:\n\n```\n...\n    {\n        \"name\": \"cmd\",\n        \"label\": \"some label\",\n        \"parameters: {\n            \"executable\": [\"echo\"],\n            \"args\": [\"The first four letters of my FQDN are {{ slice .FQDN 0 4 }}\"]\n        }\"\n    }\n...\n```\n\nConTest defines additional built-in functions in\n[pkg/test](pkg/test/functions.go). For example, to print the target FQDN after\ncapitalzing its letters, one can write:\n\n```\n...\n    {\n        \"name\": \"cmd\",\n        \"label\": \"some label\",\n        \"parameters: {\n            \"executable\": [\"echo\"],\n            \"args\": [\"My capitalized name is {{ ToUpper .FQDN }}\"]\n        }\"\n    }\n...\n```\n\nConTest also allows the user to register their own functions with\n`test.RegisterFunction` from [pkg/test/functions.go](pkg/test/functions.go).\nSee [cmds/contest/main.go](cmds/contest/main.go) for an example of how to use\nit.\n\nCustom functions are useful to enable more complex use cases. For example, if we\nwant to execute a command on a jump host that depends on the target name, we can\nwrite a custom template function to get this information. There is no limitation\non how to implement it: it can be simple string substitution, or it can be\nretrieved from a backend service requiring authentication.\nFor example, imagine that we wrote a custom template function called `jumphost`,\nthat receives the target fqdn as input and returns the jump host (or an error),\nwe can use it as follows. Note that the \"sshcmd\" plugin runs a command on a\nremote host.\n\n```\n...\n    {\n        \"name\": \"sshcmd\",\n        \"label\": \"some label...\",\n        \"parameters: {\n            \"user\": \"contest\",\n            \"host\": \"{{ jumphost .FQDN }}\",\n            \"private_key_file\": \"/path/to/id_dsa\",\n            \"executable\": [\"echo\"],\n            \"args\": [\"ConTest was here\"]\n        }\"\n    }\n...\n```\n\n\nGo templates allow for more powerful actions, like loops and conditionals, so we\nrecommend reading the [text/template](https://golang.org/pkg/text/template/)\ndocumentation.\n\nLimitations:\n* currently every plugin must explicitly call template expansion. We plan to\n  make this free and automatic for every plugin in the future.\n\n\n## Join the ConTest community\n\n* Website: https://github.com/linuxboot/contest . Please use issues and\n  pull requests!\n* IM: we are on the #ConTest channel on the OSF (Open Source Firmware) Slack team. Get your invite on http://u-root.org if you're not there already!\n* See the CONTRIBUTING file for how to help out.\n\n## License\n\nConTest is MIT licensed, as found in the LICENSE file.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flinuxboot%2Fcontest","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flinuxboot%2Fcontest","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flinuxboot%2Fcontest/lists"}