{"id":13500410,"url":"https://github.com/ovh/venom","last_synced_at":"2025-05-14T00:04:45.413Z","repository":{"id":17585190,"uuid":"82285879","full_name":"ovh/venom","owner":"ovh","description":"🐍 Manage and run your integration tests with efficiency - Venom run executors (script, HTTP Request, web, imap, etc... ) and assertions","archived":false,"fork":false,"pushed_at":"2025-05-09T12:24:55.000Z","size":11507,"stargazers_count":1112,"open_issues_count":29,"forks_count":155,"subscribers_count":33,"default_branch":"master","last_synced_at":"2025-05-09T13:31:56.714Z","etag":null,"topics":["cli","cli-app","command","command-line","golang-application","hacktoberfest","integration-testing","test","testing","xunit"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ovh.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2017-02-17T10:27:52.000Z","updated_at":"2025-05-09T12:24:59.000Z","dependencies_parsed_at":"2023-07-14T03:05:57.572Z","dependency_job_id":"a595066e-058d-4bd7-872a-49ce13535c8f","html_url":"https://github.com/ovh/venom","commit_stats":{"total_commits":428,"total_committers":80,"mean_commits":5.35,"dds":0.691588785046729,"last_synced_commit":"386ac69f1ff92cd7095e7b668516aebf482f7dfc"},"previous_names":["runabove/venom"],"tags_count":62,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ovh%2Fvenom","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ovh%2Fvenom/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ovh%2Fvenom/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ovh%2Fvenom/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ovh","download_url":"https://codeload.github.com/ovh/venom/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254043292,"owners_count":22004915,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","cli-app","command","command-line","golang-application","hacktoberfest","integration-testing","test","testing","xunit"],"created_at":"2024-07-31T22:00:59.478Z","updated_at":"2025-05-14T00:04:45.372Z","avatar_url":"https://github.com/ovh.png","language":"Go","readme":"# 🐍 Venom\n\nVenom is a CLI (Command Line Interface) that aims to create, manage and run your integration tests with efficiency.\n\n\u003ca href=\"https://github.com/ovh/venom/releases/latest\"\u003e\u003cimg alt=\"GitHub release\" src=\"https://img.shields.io/github/v/release/ovh/venom.svg?logo=github\u0026style=flat-square\"\u003e\u003c/a\u003e\n[![GoDoc](https://godoc.org/github.com/ovh/venom?status.svg)](https://godoc.org/github.com/ovh/venom)\n[![Go Report Card](https://goreportcard.com/badge/github.com/ovh/venom)](https://goreportcard.com/report/github.com/ovh/venom)\n[![Discussions](https://img.shields.io/badge/Discussions-OVHcloud-brightgreen)](https://github.com/ovh/venom/discussions)\n\u003ca href=\"https://gitpod.io/#https://github.com/ovh/venom\"\u003e\u003cimg src=\"https://img.shields.io/badge/Contribute%20with-Gitpod-908a85?logo=gitpod\" alt=\"Contribute with Gitpod\"/\u003e\u003c/a\u003e\n \n# Table of content\n\n* [Overview](#overview)\n* [Installing](#installing)\n* [Updating](#updating)\n* [Docker image](#docker-image)\n* [CLI usage](#cli-usage)\n  * [Globstar support](#globstar-support)\n  * [Variables](#variables)\n    * [Variable Definitions Files](#variable-definitions-files)\n    * [Environment Variables](#environment-variables)\n  * [Arguments](#arguments)\n    * [Define arguments with environment variables](#define-arguments-with-environment-variables)\n    * [Use a configuration file](#use-a-configuration-file)\n* [Concepts](#concepts)\n  * [TestSuites](#testsuites)\n  * [Executors](#executors)\n    * [User defined executors](#user-defined-executors)\n  * [Variables](#variables)\n    * [Testsuite variables](#testsuite-variables)\n    * [Variable helpers](#variable-helpers)\n  * [Use outputs from a test step as input of another test step](#use-outputs-from-a-test-step-as-input-of-another-test-step)\n  * [Builtin venom variables](#builtin-venom-variables)\n  * [Assertions](#assertions)\n    * [Keywords](#keywords)\n      * [`Must` Keywords](#must-keywords)\n    * [Using logical operators](#using-logical-operators)\n* [Write and run your first test suite](#write-and-run-your-first-test-suite)\n* [Export tests report](#export-tests-report)\n* [Advanced usage](#advanced-usage)\n  * [Debug your testsuites](#debug-your-testsuites)\n  * [Skip testcase and teststeps](#skip-testcase-and-teststeps)\n  * [Iterating over data](#iterating-over-data)\n* [FAQ](#faq)\n  * [Common errors with quotes](#common-errors-with-quotes)\n* [Use venom in CI/CD pipelines](#use-venom-in-cicd-pipelines)\n* [Hacking](#hacking)\n* [Contributing](#contributing)\n* [License](#license)\n\n# Overview\n\nVenom allows you to handle integration tests the same way you code your application.\nWith Venom, testcases will be managed as code: the readability of the tests means that the tests are part of the code reviews. Thanks to that, write and execute testsuites become easier for developers and teams.\n\nConcretely, you have to write testsuite in a YAML file.\nVenom run executors (scripts, HTTP Request, web, IMAP, etc.) and apply assertions. \nIt can also generate xUnit result files.\n\n\u003cimg src=\"./venom.gif\" alt=\"Venom Demonstration\"\u003e\n\n# Installing\n\n## Install from binaries\n\nYou can find latest binary release from: https://github.com/ovh/venom/releases/latest/.\n\nExample for Linux:\n\n```bash\n$ curl https://github.com/ovh/venom/releases/download/v1.2.0/venom.linux-amd64 -L -o /usr/local/bin/venom \u0026\u0026 chmod +x /usr/local/bin/venom\n$ venom -h\n```\n\n# Updating\n\nYou can update to the latest version with `venom update` command:\n\n```bash\n$ venom update\n```\n\nThe `venom update` command will download the latest version and replace the current binary:\n\n```bash\nUrl to update venom: https://github.com/ovh/venom/releases/download/v1.2.0/venom.darwin-amd64\nGetting latest release from: https://github.com/ovh/venom/releases/download/v1.2.0/venom.darwin-amd64 ...\nUpdate done.\n```\n\nCheck the new version with `venom version` command:\n\n```bash\n$ venom version\nVersion venom: v1.2.0 \n```\n\n# Docker image\n\nInstead of installing (and updating) Venom locally, Venom can be started as a Docker image with following commands. \n\nConsidering your testsuites are in `./tests` directory in your current directory and your test library is under `./tests/lib`, the results will be available under the `results` directory.\n\n```bash\n$ mkdir -p results\n$ docker run --mount type=bind,source=$(pwd)/tests,target=/workdir/tests --mount type=bind,source=$(pwd)/results,target=/workdir/results ovhcom/venom:latest \n```\n\nPlease refer to https://hub.docker.com/r/ovhcom/venom/tags to get the available image tags.\n\n# CLI Usage\n\n`venom` CLI is composed of several commands:\n\n```bash\n$ venom -h\nVenom - RUN Integration Tests\n\nUsage:\n  venom [command]\n\nAvailable Commands:\n  help        Help about any command\n  run         Run Tests\n  update      Update venom to the latest release version: venom update\n  version     Display Version of venom: venom version\n\nFlags:\n  -h, --help   help for venom\n\nUse \"venom [command] --help\" for more information about a command.\n```\n\nYou can see the help of a command with `venom [command] -h`:\n\n```bash\n$ venom run -h\n\nrun integration tests\n\nUsage:\n  venom run [flags]\n\nExamples:\n  Run all testsuites containing in files ending with *.yml or *.yaml: venom run\n  Run a single testsuite: venom run mytestfile.yml\n  Run a single testsuite and export the result in JSON format in test/ folder: venom run mytestfile.yml --format=json --output-dir=test\n  Run a single testsuite and export the result in XML and HTML formats in test/ folder: venom run mytestfile.yml --format=xml --output-dir=test --html-report\n  Run a single testsuite and specify a variable: venom run mytestfile.yml --var=\"foo=bar\"\n  Run a single testsuite and load all variables from a file: venom run mytestfile.yml --var-from-file variables.yaml\n  Run all testsuites containing in files ending with *.yml or *.yaml with verbosity: VENOM_VERBOSE=2 venom run\n\n  Notice that variables initialized with -var-from-file argument can be overrided with -var argument\n\n  More info: https://github.com/ovh/venom\n\nFlags:\n      --format string           --format:json, tap, xml, yaml (default \"xml\")\n  -h, --help                    help for run\n      --html-report             Generate HTML Report\n      --lib-dir string          Lib Directory: can contain user executors. example:/etc/venom/lib:$HOME/venom.d/lib\n      --output-dir string       Output Directory: create tests results file inside this directory\n      --stop-on-failure         Stop running Test Suite on first Test Case failure\n      --var stringArray         --var cds='cds -f config.json' --var cds2='cds -f config.json'\n      --var-from-file strings   --var-from-file filename.yaml --var-from-file filename2.yaml: yaml, must contains a dictionary\n  -v, --verbose count           verbose. -v (INFO level in venom.log file), -vv to very verbose (DEBUG level) and -vvv to very verbose with CPU Profiling\n```\n\n## Run test suites in a specific order\n\n- `venom run 01_foo.yml 02_foo.yml` will run 01 before 02. \n- `venom run 02_foo.yml 01_foo.yml` will run 02 before 01.\n\nIf you want to sort many testsuite files, you can use standard commands, example:\n\n```bash\nvenom run `find . -type f -name \"*.yml\"|sort`\n```\n\n## Globstar support\n\nThe `venom` CLI supports globstar:\n\n```\n$ venom run ./foo/b*/**/z*.yml\n```\n\n## Variables\n\nTo specify individual variables on the command line, use the `--var` option when running the `venom run` commands:\n\n```bash\n$ venom run --var=\"foo=bar\"\n$ venom run --var='foo_list=[\"biz\",\"buz\"]'\n$ venom run --var='foo={\"biz\":\"bar\",\"biz\":\"barr\"}'\n```\n\nThe `--var` option can be used many times in a single command.\n\n### Variable Definitions Files\n\nTo set a lot of variables, it is more convenient to specify their values in a variable definitions file. This file is a YAML dictionary. You have to specify that file on the command line with `--var-from-file`:\n\n```bash\nvenom run --var-from-file variables.yaml\n```\n\n### Environment Variables\n\nAs a fallback for the other ways of defining variables, `venom` tool searches the environment of its own process for environment variables named `VENOM_VAR_` followed by the name of a declared variable.\n\n```bash\n$ export VENOM_VAR_foo=bar\n$ venom run *.yml\n```\n\nYou can also define the environment variable and run your testsuite in one line:\n\n```bash\n$ VENOM_VAR_foo=bar venom run *.yml\n```\n\n## Arguments\n\nYou can define arguments on the command line using the flag name.\n\nFlags are listed in the result of help command.\n\nList of available flags for `venom run` command:\n\n```\nFlags:\n      --format string           --format:json, tap, xml, yaml (default \"xml\")\n  -h, --help                    help for run\n      --html-report             Generate HTML Report\n      --lib-dir string          Lib Directory: can contain user executors. example:/etc/venom/lib:$HOME/venom.d/lib\n      --output-dir string       Output Directory: create tests results file inside this directory\n      --stop-on-failure         Stop running Test Suite on first Test Case failure\n      --var stringArray         --var cds='cds -f config.json' --var cds2='cds -f config.json'\n      --var-from-file strings   --var-from-file filename.yaml --var-from-file filename2.yaml: yaml, must contains a dictionary\n  -v, --verbose count           verbose. -vv to very verbose and -vvv to very verbose with CPU Profiling\n```\n\n### Define arguments with environment variables\n\nYou can also define the arguments with environment variables:\n\n```bash\n# is the same as\nVENOM_FORMAT=json venom run my-test-suite.yml\n\n# is equivalent to\nvenom run my-test-suite.yml --format=json\n```\n\nFlags and their equivalent with environment variables usage:\n\n- `--format=\"json\"` flag is equivalent to `VENOM_FORMAT=\"json\"` environment variable\n- `--lib-dir=\"/etc/venom/lib:$HOME/venom.d/lib\"` flag is equivalent to `VENOM_LIB_DIR=\"/etc/venom/lib\"` environment variable\n- `--output-dir=\"test-results\"` flag is equivalent to `VENOM_OUTPUT_DIR=\"test-results\"` environment variable\n- `--stop-on-failure` flag is equivalent to `VENOM_STOP_ON_FAILURE=true` environment variable\n- `--var foo=bar` flag is equivalent to `VENOM_VAR_foo='bar'` environment variable\n- `--var-from-file fileA.yml fileB.yml` flag is equivalent to `VENOM_VAR_FROM_FILE=\"fileA.yml fileB.yml\"` environment variable\n- `-v` flag is equivalent to `VENOM_VERBOSE=1` environment variable\n- `-vv` flag is equivalent to `VENOM_VERBOSE=2` environment variable\n\nIt is possible to set `NO_COLOR=1` environment variable to disable colors from output.\n\n## Use a configuration file\n\nYou can define the Venom settings using a configuration file `.venomrc`. This configuration file should be placed in the current directory or in the `home` directory.\n\n```yml\nvariables: \n  - foo=bar\nvariables_files:\n  - my_var_file.yaml\nstop_on_failure: true\nformat: xml\noutput_dir: output\nlib_dir: lib\nverbosity: 3\n```\n\nPlease note that the command line flags overrides the configuration file. The configuration file overrides the environment variables.\n\n\n# Concepts\n\n## TestSuites\n\nA test suite is a collection of test cases that are intended to be used to test a software program to show that it has a specified set of behaviors.\nA test case is a specification of the inputs, execution conditions, testing procedure, and expected results that define a single test to be executed to achieve a particular software testing objective, such as to exercise a particular program path or to verify compliance with a specific requirement.\n\nIn `venom` the testcases are executed sequentially within a testsuite. Each testcase is an ordered set of steps. Each step is based on an `executor` that enable some specific kind of behavior.\n\nIn `venom` a testsuite is written in one `YAML` file respecting the following structure:\n\n```yaml\n\nname: Title of TestSuite\ndescription: A detailed description of the TestSuite, in markdown.\ntestcases:\n- name: TestCase with default value, exec cmd. Check if exit code != 1\n  steps:\n  - script: echo 'foo'\n    type: exec\n\n- name: Title of First TestCase\n  steps:\n  - script: echo 'foo'\n    assertions:\n    - result.code ShouldEqual 0\n  - script: echo 'bar'\n    assertions:\n    - result.systemout ShouldNotContainSubstring foo\n    - result.timeseconds ShouldBeLessThan 1\n\n- name: GET http testcase, with 5 seconds timeout\n  steps:\n  - type: http\n    method: GET\n    url: https://eu.api.ovh.com/1.0/\n    timeout: 5\n    assertions:\n    - result.body ShouldContainSubstring /dedicated/server\n    - result.body ShouldContainSubstring /ipLoadbalancing\n    - result.statuscode ShouldEqual 200\n    - result.timeseconds ShouldBeLessThan 1\n\n- name: Test with retries and delay in seconds between each try\n  steps:\n  - type: http\n    method: GET\n    url: https://eu.api.ovh.com/1.0/\n    retry: 3\n    retry_if: # (optional, lets you early break unrecoverable errors)\n    - result.statuscode ShouldNotEqual 403\n    delay: 2\n    assertions:\n    - result.statuscode ShouldEqual 200\n\n```\n\n## Executors\n\n* **amqp**: https://github.com/ovh/venom/tree/master/executors/amqp\n* **couchbase**: https://github.com/ovh/venom/tree/master/executors/couchbase\n* **dbfixtures**: https://github.com/ovh/venom/tree/master/executors/dbfixtures\n* **exec**: https://github.com/ovh/venom/tree/master/executors/exec `exec` is the default type for a step\n* **grpc**: https://github.com/ovh/venom/tree/master/executors/grpc\n* **http**: https://github.com/ovh/venom/tree/master/executors/http\n* **imap**: https://github.com/ovh/venom/tree/master/executors/imap\n* **kafka** https://github.com/ovh/venom/tree/master/executors/kafka\n* **mqtt** https://github.com/ovh/venom/tree/master/executors/mqtt\n* **odbc**: https://github.com/ovh/venom/tree/master/executors/plugins/odbc\n* **ovhapi**: https://github.com/ovh/venom/tree/master/executors/ovhapi\n* **rabbitmq**: https://github.com/ovh/venom/tree/master/executors/rabbitmq\n* **readfile**: https://github.com/ovh/venom/tree/master/executors/readfile\n* **redis**: https://github.com/ovh/venom/tree/master/executors/redis\n* **smtp**: https://github.com/ovh/venom/tree/master/executors/smtp\n* **sql**: https://github.com/ovh/venom/tree/master/executors/sql\n* **ssh**: https://github.com/ovh/venom/tree/master/executors/ssh\n* **web**: https://github.com/ovh/venom/tree/master/executors/web\n\n### User defined executors\n\nYou can define an executor with a single YAML file. This is a good way to abstract technical or functional behaviors and reuse them in complex testsuites.\n\nExample:\n\nfile `lib/customA.yml`:\n\n```yml\nexecutor: hello\ninput:\n  myarg: {}\nsteps:\n- script: echo \"{\\\"hello\\\":\\\"{{.input.myarg}}\\\"}\"\n  assertions:\n  - result.code ShouldEqual 0\n  vars:\n    hello:\n      from: result.systemoutjson.hello\n    all:\n      from: result.systemoutjson\noutput:\n  display:\n    hello: \"{{.hello}}\"\n  all: \"{{.all}}\"\n```\n\nfile `testsuite.yml`:\n\n```yml\nname: testsuite with a user executor\ntestcases:\n- name: testA\n  steps:\n  - type: hello\n    myarg: World\n    assertions:\n    - result.display.hello ShouldContainSubstring World\n    - result.alljson.hello ShouldContainSubstring World\n```\n\nNotice the variable `alljson`. All variables declared in output are automatically converted in a json format with the suffix `json`. In the example above, two implicit variables are available: `displayjson.hello` and `alljson`.\n\nVenom will load user's executors from the directory `lib/` relative to the testsuite path. You add executors source path using the flag `--lib-dir`. \nNote that all folders listed with `--lib-dir` will be scanned recursively to find `.yml` files as user executors.\n\nThe user defined executors work with templating, you can check the templating result in `venom.log`. In this file, if you see an error as `error converting YAML to JSON: yaml: line 14: found unexpected end of stream`, you probably need to adjust indentation with the templating function `indent`. \n\nExample:\n\n```yml\nname: testsuite with a user executor multilines\ntestcases:\n- name: test\n  steps:\n  - type: multilines\n    script: |\n      # test multilines\n      echo \"5\"\n    assertions:\n    - result.alljson ShouldEqual 5\n```\n\nusing this executor:\n\n```yml\nexecutor: multilines\ninput:\n  script: \"echo 'foo'\"\nsteps:\n- type: exec\n  script: {{ .input.script | nindent 4 }}\n  assertions:\n  - result.code ShouldEqual 0\n  vars:\n    all:\n      from: result.systemoutjson\noutput:\n  all: '{{.all}}'\n```\n\n\n```bash\n# lib/*.yml files will be loaded as executors.\n$ venom run testsuite.yml \n\n# executors will be loaded from /etc/venom/lib, $HOME/venom.d/lib and lib/ directory relative to testsuite.yml file.\n$ venom run --lib-dir=/etc/venom/lib:$HOME/venom.d/lib testsuite.yml \n```\n\n## Variables\n\n### Testsuite variables\n\nYou can define variables at the `testsuite` level.\n\n```yaml\nname: myTestSuite\nvars:\n  foo: foo\n  biz:\n    bar: bar\n  aString: '{\"foo\": \"bar\"}'\n\ntestcases:\n- name: first-test-case\n  steps:\n  - type: exec\n    script: echo '{{.foo}} {{.biz.bar}}'\n    assertions:\n    - result.code ShouldEqual 0\n    - result.systemout ShouldEqual \"foo bar\"\n\n- name: foobar\n  steps:\n  - script: echo '{{.aString}}'\n    info: value of aString is {{.aString}}\n    assertions:\n    - result.systemoutjson.foo ShouldEqual bar\n...\n```\n\nEach user variable used in testsuite must be declared in this section. You can override its value at runtime in a number of ways:\n- Individually, with the `--var` command line option.\n- In variable definitions files, either specified on the command line `--var-from-file`.\n- As environment variables.\n\n\n\n### Variable helpers\n\nAvailable helpers and some examples:\n\n- `abbrev`\n- `abbrevboth`\n- `trunc`\n- `trim`\n- `upper`: {{.myvar | upper}}\n- `lower`: {{.myvar | lower}}\n- `title`\n- `untitle`\n- `substr`\n- `repeat`\n- `trimall`\n- `trimAll`\n- `trimSuffix`\n- `trimPrefix`\n- `nospace`\n- `initials`\n- `randAlphaNum`\n- `randAlpha`\n- `randASCII`\n- `randNumeric`\n- `swapcase`\n- `shuffle`\n- `snakecase`\n- `camelcase`\n- `quote`\n- `squote`\n- `indent`\n- `nindent`\n- `replace`: {{.myvar | replace \"_\" \".\"}}\n- `plural`\n- `default`: {{.myvar | default \"\"}}\n- `empty`\n- `coalesce`\n- `toJSON`\n- `toPrettyJSON`\n- `b64enc`\n- `b64dec` {{.result.bodyjson | b64enc}}\n- `escape`: replace ‘_‘, ‘/’, ‘.’ by ‘-’\n\nMore examples are available [here](https://github.com/ovh/venom/tree/master/variable_helpers.md)\n\n## Use outputs from a test step as input of another test step\n\nTo be able to reuse a property from a teststep in a following testcase or step, you have to extract the variable, as the following example. \n\nAfter the first step execution, `venom` extracts a value using a regular expression `foo with a ([a-z]+) here` from the content of the `result.systemout` property returned by the `executor`.\nThen this variable can be reused in another test, with the name `testA.myvariable` with `testA` corresponding to the name of the testcase. A default value could also be supplied if the variable can't be extracted from the output, which can commonly happen when parsing json output.\n\n```yaml\nname: MyTestSuite\ntestcases:\n- name: testA\n  steps:\n  - type: exec\n    script: echo 'foo with a bar here'\n    vars:\n      myvariable:\n        from: result.systemout\n        regex: foo with a ([a-z]+) here\n        default: \"somevalue\"\n\n- name: testB\n  steps:\n  - type: exec\n    script: echo {{.testA.myvariable}}\n    assertions:\n    - result.code ShouldEqual 0\n    - result.systemout ShouldContainSubstring bar\n```\n\n## Builtin venom variables\n\n```yaml\nname: MyTestSuite\ntestcases:\n- name: testA\n  steps:\n  - type: exec\n    script: echo '{{.venom.testsuite}} {{.venom.testsuite.filename}} {{.venom.testcase}} {{.venom.teststep.number}} {{.venom.datetime}} {{.venom.timestamp}} {{.venom.testcase.totalSteps}} {{.venom.testsuite.totalSteps}}'\n    # will display something as: MyTestSuite MyTestSuiteWithVenomBuiltinVar.yml testA 0 2018-08-05T21:38:24+02:00 1533497904 3 5\n\n```\n\nBuiltin variables:\n\n* {{.venom.datetime}}\n* {{.venom.executable}}\n* {{.venom.libdir}}\n* {{.venom.outputdir}}\n* {{.venom.testcase}}\n* {{.venom.testcase.totalSteps}}\n* {{.venom.teststep.number}}\n* {{.venom.testsuite.name}}\n* {{.venom.testsuite.filename}}\n* {{.venom.testsuite.filepath}}\n* {{.venom.testsuite.shortName}}\n* {{.venom.testsuite.workdir}}\n* {{.venom.testsuite}}\n* {{.venom.testsuite.totalSteps}}\n* {{.venom.timestamp}}\n\n\n### Secrets variables\n\nExample:\n\n```yml\nname: Your Testsuite\nvars:\n  foo : this-value-is-secret\nsecrets:\n  - foo\ntestcases:\n- name: myvar_first\n  steps:\n  - type: exec\n    script: \"echo myvar {{.foo}}\"\n```\n\nThe value `this-value-is-secret` will not be printed in your console, `venom.log` and `...dump.json` files.\n\n## Assertions\n\n### Keywords\n\n* ShouldEqual - [example](https://github.com/ovh/venom/tree/master/tests/assertions/ShouldEqual.yml)\n* ShouldNotEqual - [example](https://github.com/ovh/venom/tree/master/tests/assertions/ShouldNotEqual.yml)\n* ShouldAlmostEqual - [example](https://github.com/ovh/venom/tree/master/tests/assertions/ShouldAlmostEqual.yml)\n* ShouldNotAlmostEqual - [example](https://github.com/ovh/venom/tree/master/tests/assertions/ShouldNotAlmostEqual.yml)\n* ShouldBeNil - [example](https://github.com/ovh/venom/tree/master/tests/assertions/ShouldBeNil.yml)\n* ShouldNotBeNil - [example](https://github.com/ovh/venom/tree/master/tests/assertions/ShouldNotBeNil.yml)\n* ShouldBeTrue - [example](https://github.com/ovh/venom/tree/master/tests/assertions/ShouldBeTrue.yml)\n* ShouldBeFalse - [example](https://github.com/ovh/venom/tree/master/tests/assertions/ShouldBeFalse.yml)\n* ShouldBeZeroValue - [example](https://github.com/ovh/venom/tree/master/tests/assertions/ShouldBeZeroValue.yml)\n* ShouldBeGreaterThan - [example](https://github.com/ovh/venom/tree/master/tests/assertions/ShouldBeGreaterThan.yml)\n* ShouldBeGreaterThanOrEqualTo - [example](https://github.com/ovh/venom/tree/master/tests/assertions/ShouldBeGreaterThanOrEqualTo.yml)\n* ShouldBeLessThan - [example](https://github.com/ovh/venom/tree/master/tests/assertions/ShouldBeLessThan.yml)\n* ShouldBeLessThanOrEqualTo - [example](https://github.com/ovh/venom/tree/master/tests/assertions/ShouldBeLessThanOrEqualTo.yml)\n* ShouldBeBetween - [example](https://github.com/ovh/venom/tree/master/tests/assertions/ShouldBeBetween.yml)\n* ShouldNotBeBetween - [example](https://github.com/ovh/venom/tree/master/tests/assertions/ShouldNotBeBetween.yml)\n* ShouldBeBetweenOrEqual - [example](https://github.com/ovh/venom/tree/master/tests/assertions/ShouldBeBetweenOrEqual.yml)\n* ShouldNotBeBetweenOrEqual - [example](https://github.com/ovh/venom/tree/master/tests/assertions/ShouldNotBeBetweenOrEqual.yml)\n* ShouldContain - [example](https://github.com/ovh/venom/tree/master/tests/assertions/ShouldContain.yml)\n* ShouldNotContain - [example](https://github.com/ovh/venom/tree/master/tests/assertions/ShouldNotContain.yml)\n* ShouldJSONContain - [example](https://github.com/ovh/venom/tree/master/tests/assertions/ShouldJSONContain.yml)\n* ShouldNotJSONContain - [example](https://github.com/ovh/venom/tree/master/tests/assertions/ShouldNotJSONContain.yml)\n* ShouldJSONContainWithKey - [example](https://github.com/ovh/venom/tree/master/tests/assertions/ShouldJSONContainWithKey.yml)\n* ShouldJSONContainAllWithKey - [example](https://github.com/ovh/venom/tree/master/tests/assertions/ShouldJSONContainAllWithKey.yml)\n* ShouldNotJSONContainWithKey - [example](https://github.com/ovh/venom/tree/master/tests/assertions/ShouldNotJSONContainWithKey.yml)\n* ShouldContainKey - [example](https://github.com/ovh/venom/tree/master/tests/assertions/ShouldContainKey.yml)\n* ShouldNotContainKey - [example](https://github.com/ovh/venom/tree/master/tests/assertions/ShouldNotContainKey.yml)\n* ShouldBeIn - [example](https://github.com/ovh/venom/tree/master/tests/assertions/ShouldBeIn.yml)\n* ShouldNotBeIn - [example](https://github.com/ovh/venom/tree/master/tests/assertions/ShouldNotBeIn.yml)\n* ShouldBeEmpty - [example](https://github.com/ovh/venom/tree/master/tests/assertions/ShouldBeEmpty.yml)\n* ShouldNotBeEmpty - [example](https://github.com/ovh/venom/tree/master/tests/assertions/ShouldNotBeEmpty.yml)\n* ShouldHaveLength - [example](https://github.com/ovh/venom/tree/master/tests/assertions/ShouldHaveLength.yml)\n* ShouldStartWith - [example](https://github.com/ovh/venom/tree/master/tests/assertions/ShouldStartWith.yml)\n* ShouldNotStartWith - [example](https://github.com/ovh/venom/tree/master/tests/assertions/ShouldNotStartWith.yml)\n* ShouldEndWith - [example](https://github.com/ovh/venom/tree/master/tests/assertions/ShouldEndWith.yml)\n* ShouldNotEndWith - [example](https://github.com/ovh/venom/tree/master/tests/assertions/ShouldNotEndWith.yml)\n* ShouldBeBlank - [example](https://github.com/ovh/venom/tree/master/tests/assertions/ShouldBeBlank.yml)\n* ShouldNotBeBlank - [example](https://github.com/ovh/venom/tree/master/tests/assertions/ShouldNotBeBlank.yml)\n* ShouldContainSubstring - [example](https://github.com/ovh/venom/tree/master/tests/assertions/ShouldContainSubstring.yml)\n* ShouldNotContainSubstring - [example](https://github.com/ovh/venom/tree/master/tests/assertions/ShouldNotContainSubstring.yml)\n* ShouldEqualTrimSpace - [example](https://github.com/ovh/venom/tree/master/tests/assertions/ShouldEqualTrimSpace.yml)\n* ShouldNotExist - [example](https://github.com/ovh/venom/tree/master/tests/assertions/ShouldNotExist.yml)\n* ShouldHappenBefore - [example](https://github.com/ovh/venom/tree/master/tests/assertions/ShouldHappenBefore.yml)\n* ShouldHappenOnOrBefore - [example](https://github.com/ovh/venom/tree/master/tests/assertions/ShouldHappenOnOrBefore.yml)\n* ShouldHappenAfter - [example](https://github.com/ovh/venom/tree/master/tests/assertions/ShouldHappenAfter.yml)\n* ShouldHappenOnOrAfter - [example](https://github.com/ovh/venom/tree/master/tests/assertions/ShouldHappenOnOrAfter.yml)\n* ShouldHappenBetween - [example](https://github.com/ovh/venom/tree/master/tests/assertions/ShouldHappenBetween.yml)\n* ShouldTimeEqual - [example](https://github.com/ovh/venom/tree/master/tests/assertions/ShouldTimeEqual.yml)\n* ShouldMatchRegex - [example](https://github.com/ovh/venom/tree/master/tests/assertions/ShouldMatchRegex.yml)\n* ShouldJSONEqual - [example](https://github.com/ovh/venom/tree/master/tests/assertions/ShouldJSONEqual.yml)\n\n#### `Must` keywords\n\nAll the above assertions keywords also have a `Must` counterpart which can be used to create a required passing assertion and prevent test cases (and custom executors) to run remaining steps.\n\nExample:\n```yml\n- steps:\n  - type: exec\n    script: exit 1\n    assertions:\n      - result.code MustEqual 0\n  # Remaining steps in this context will not be executed\n```\n\n### Using logical operators\n\nWhile assertions use `and` operator implicitly, it is possible to use other logical operators to perform complex assertions.\n\nSupported operators are `and`, `or` and `xor`.\n\n```yml\n- name: Assertions operators\n  steps:\n  - script: echo 1\n    assertions:\n      - or:\n        - result.systemoutjson ShouldEqual 1 \n        - result.systemoutjson ShouldEqual 2\n      # Nested operators\n      - or:\n        - result.systemoutjson ShouldBeGreaterThanOrEqualTo 1\n        - result.systemoutjson ShouldBeLessThanOrEqualTo 1\n        - or:\n          - result.systemoutjson ShouldEqual 1\n```\n\nMore examples are available in [`tests/assertions_operators.yml`](/tests/assertions_operators.yml).\n\n# Write and run your first test suite \n\nTo understand how Venom is working, let's create and run a first testsuite together.\n\nThe first assertions that we will do, in this testsuite, are to check whether the site we want to test (a public REST API, OVHcloud API for example):\n- is accessible (respond with a 200 status code)\n- responds in less than 5 seconds\n- returns a valid response (which is in JSON format)\n\nFirst, create your testsuite in a file called `testsuite.yml` for example.\nOpen it in your favorite editor or IDE and fill it with this content:\n\n```yaml\nname: APIIntegrationTest\n\nvars:\n  url: https://eu.api.ovh.com\n\ntestcases:\n- name: GET http testcase, with 5 seconds timeout\n  steps:\n  - type: http\n    method: GET\n    url: {{.url}}/1.0/\n    timeout: 5\n    assertions:\n    - result.statuscode ShouldEqual 200\n    - result.timeseconds ShouldBeLessThan 1\n    - result.bodyjson ShouldContainKey apis\n    - result.body ShouldContainSubstring /dedicated/server\n    - result.body ShouldContainSubstring /ipLoadbalancing\n```\n\nThen, run your testsuite with the following command:\n\n```bash\n$ venom run \n\n • APIIntegrationTest (testsuite.yml)\n \t• GET-http-testcase-with-5-seconds-timeout SUCCESS\n```\n\nYou wrote and executed your first testsuite with the HTTP executor! :)\n\n# Export tests report\n\nYou can export your testsuite results as a report in several available formats: xUnit (XML), JSON, YAML, TAP.\n\nYou can specify the output directory with the `--output-dir` flag and the format with the `--format` flag (XML by default):\n\n```bash\n$ venom run --format=xml --output-dir=\".\"\n\n# html export\n$ venom run --output-dir=\".\" --html-report\n```\n\nReports exported in XML can be visualized with a xUnit/jUnit Viewer, directly in your favorite CI/CD stack for example in order to see results run after run.\n\n# Advanced usage\n\n## Debug your testsuites\n\nA *venom.log* file is generated for each `venom run` command.\n\nThere are two ways to debug a testsuite:\n - use `-v` flag on venom binary.\n   - `$ venom run -v test.yml` will output details for each step\n   - `$ venom run -vv test.yml` will generate *dump.json* files for each teststep.\n   - `$ venom run -vvv test.yml` will generate *pprof* files for CPU profiling.\n - use `info` keyword in your teststep:\n`test.yml` file:\n```yml\nname: Exec testsuite\ntestcases:\n- name: testA\n  steps:\n  - type: exec\n    script: echo 'foo with a bar here'\n    info:\n      - this a first info\n      - and a second...\n- name: cat json\n  steps:\n  - script: cat exec/testa.json\n    info: \"the value of result.systemoutjson is {{.result.systemoutjson}}\"\n    assertions:\n    - result.systemoutjson.foo ShouldContainSubstrin bar\n```\n\n```bash\n$ venom run test.yml\n\n# output:\n\n • Exec testsuite (exec.yml)\n  • testA SUCCESS\n    [info] this a first info (exec.yml:8)\n    [info] and a second... (exec.yml:9)\n  • testB SUCCESS\n  • sleep 1 SUCCESS\n  • cat json SUCCESS\n    [info] the value of result.systemoutjson is map[foo:bar] (exec.yml:34)\n```\n\n## Skip testcase and teststeps\n\nIt is possible to skip `testcase` according to some `assertions`. For instance, the following example will skip the last testcase.\n\n```yaml\nname: \"Skip testsuite\"\nvars:\n  foo: bar\n\ntestcases:\n- name: init\n  steps:\n  - type: exec\n    script: echo {{.foo}}\n    assertions:\n    - result.code ShouldEqual 0\n    - result.systemout ShouldContainSubstring bar\n\n- name: do-not-skip-this\n  skip: \n  - foo ShouldNotBeEmpty\n  steps:\n  - type: exec\n    script: exit 0\n\n- name: skip-this\n  skip: \n    - foo ShouldBeEmpty\n  steps:\n  - type: exec\n    script: command_not_found\n    assertions:\n    - result.code ShouldEqual 0\n\n```\n\nA `skip` statement may also be placed at steps level to partially execute a testcase. If one condition from the skip block is not true, it's skipped.\n\nIf all steps from a testcase are skipped, the testcase itself will also be treated as \"skipped\" rather than \"passed\"/\"failed\".\n\n```yaml\nname: \"Skip testsuite\"\nvars:\n  foo: bar\n\ntestcases:\n- name: skip-one-of-these\n  steps:\n  - name: do-not-skip-this\n    type: exec\n    script: exit 0\n    assertions:\n    - result.code ShouldEqual 0\n    skip:\n    - foo ShouldNotBeEmpty\n  - name: skip-this\n    type: exec\n    script: exit 1\n    assertions:\n    - result.code ShouldEqual 0\n    skip:\n    - foo ShouldBeEmpty\n\n```\n\n## Iterating over data\n\nIt is possible to iterate over data using `range` attribute.\n\nThe following data types are supported, each exposing contexted variables `.index`, `.key` and `.value`:\n\n- An array where each value will be iterated over (`[]interface{}`)\n  - `.index`/`.key`: current iteration index\n  - `.value`: current iteration item value\n- A map where each key will be iterated over (`map[string]interface{}`)\n  - `.index`: current iteration index\n  - `.key`: current iteration item key\n  - `.value`: current iteration item value\n- An integer to perform target step `n` times (`int`)\n  - `.index`/`.key`/`.value`: current iteration index\n- A templated string which results in one of the above typing (`string`)\n  - It can be either inherited from vars file, or interpolated from a previous step result\n\nFor instance, the following example will iterate over an array of two items containing maps:\n```yaml\n- name: range with hardcoded array\n  steps:\n  - type: exec\n    range:\n      - actual: hello\n        expected: hello\n      - actual: world\n        expected: world\n    script: echo \"{{.value.actual}}\"\n    assertions:\n    - result.code ShouldEqual 0\n    - result.systemout ShouldEqual \"{{.value.expected}}\"\n```\n\nMore examples are available in [`tests/ranged.yml`](/tests/ranged.yml).\n\n# FAQ\n\n## Common errors with quotes\n\nIf you have this kind of error:\n\n```\nerr:unable to parse file \"foo.yaml\": error converting YAML to JSON: yaml: line 8: did not find expected key\n```\n\nthis is probably because you try to use a json value instead of a string. You should have more details in `venom.log` file.\n\nWrong:\n\n```yml\n...\nvars:\n  body: \u003e-\n      {\n        \"the-attribute\": \"the-value\"\n      }\n...\nsteps:\n- type: http\n  body: \"{{.body}}\"\n...\n```\n\nOK:\n\n\n```yml\n...\nvars:\n  body: \u003e-\n      {\n        \"the-attribute\": \"the-value\"\n      }\n...\nsteps:\n- type: http\n  body: '{{.body}}'\n...\n```\n\nNote the simple quote on the value of `body`.\n\n# Use venom in CI/CD pipelines\n\nVenom can be used in dev environment or on your CI server.\nTo display properly the venom output, you probably will have to export the environment variable `IS_TTY=true` before running venom.\n\n# Hacking\n\n[How to write your own executor?](https://github.com/ovh/venom/tree/master/executors#venom-executor)\n\n## How to compile?\n```bash\n$ make build\n```\n\n## How to test?\n\n### Unit tests:\n\n```bash\nmake test\n```\n\n### Integration tests:\n\nPrepare the stack:\n\n```bash\nmake build OS=linux ARCH=amd64\ncp dist/venom.linux-amd64 tests/venom\ncd tests\nmake start-test-stack  # (wait a few seconds)\nmake build-test-binary-docker\n```\n\nRun integration tests:\n\n```bash\nmake run-test\n```\n\nCleanup:\n\n```bash\nmake clean\nmake stop-test-stack\n```\n\n# Contributing\n\n\u003ca href=\"https://gitpod.io/#https://github.com/ovh/venom\"\u003e\u003cimg src=\"https://img.shields.io/badge/Contribute%20with-Gitpod-908a85?logo=gitpod\" alt=\"Contribute with Gitpod\"/\u003e\u003c/a\u003e\n\nPlease read the [contributing guide](./CONTRIBUTING.md) to learn about how you can contribute to Venom ;-).\nThere is no small contribution, don't hesitate!\n\nOur awesome contributors:\n\n\u003ca href=\"https://github.com/ovh/venom/graphs/contributors\"\u003e\n  \u003cimg src=\"https://contrib.rocks/image?repo=ovh/venom\" /\u003e\n\u003c/a\u003e\n\n# License\n\nCopyright 2022 OVH SAS\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n    http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n","funding_links":[],"categories":["Go","Automated API Testing \u0026 Load Testing Tools","cli","\u003ca name=\"Go\"\u003e\u003c/a\u003eGo","Automated Testing"],"sub_categories":["44. [Venom](https://github.com/ovh/venom)"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fovh%2Fvenom","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fovh%2Fvenom","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fovh%2Fvenom/lists"}