{"id":21068939,"url":"https://github.com/kamchatka-volcano/lunchtoast","last_synced_at":"2025-05-16T03:33:41.718Z","repository":{"id":53142487,"uuid":"324651112","full_name":"kamchatka-volcano/lunchtoast","owner":"kamchatka-volcano","description":"A command-line tool for functional testing of console applications","archived":false,"fork":false,"pushed_at":"2024-06-12T15:07:06.000Z","size":731,"stargazers_count":18,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-10-30T02:03:28.533Z","etag":null,"topics":["cli","cli-tool","cpp20","functional-testing","test-automation"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"ms-pl","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kamchatka-volcano.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-12-26T23:17:24.000Z","updated_at":"2024-09-08T18:29:39.000Z","dependencies_parsed_at":"2024-06-11T02:04:07.521Z","dependency_job_id":null,"html_url":"https://github.com/kamchatka-volcano/lunchtoast","commit_stats":null,"previous_names":[],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kamchatka-volcano%2Flunchtoast","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kamchatka-volcano%2Flunchtoast/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kamchatka-volcano%2Flunchtoast/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kamchatka-volcano%2Flunchtoast/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kamchatka-volcano","download_url":"https://codeload.github.com/kamchatka-volcano/lunchtoast/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225405249,"owners_count":17469314,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","cli-tool","cpp20","functional-testing","test-automation"],"created_at":"2024-11-19T18:29:31.627Z","updated_at":"2024-11-19T18:29:32.382Z","avatar_url":"https://github.com/kamchatka-volcano.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🍞lunchtoast\n\n[![build \u0026 test (clang, gcc, MSVC)](https://github.com/kamchatka-volcano/lunchtoast/actions/workflows/build_and_test.yml/badge.svg?branch=master)](https://github.com/kamchatka-volcano/lunchtoast/actions/workflows/build_and_test.yml)\n\n**lunchtoast** - is a command-line functional testing tool that provides a clean and customizable way to launch\nprocesses or shell commands and check their exit codes, console output, and contents of result files.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg width=\"374\" height=\"204\" src=\"doc/lunchtoast.svg\"/\u003e\n\u003c/p\u003e\n\nTest cases are created with simple configuration files, and you can use a set of built-in actions to check the expected\nresult of tested programs:\n\n```\n-Name: test\n-Launch: my_process --out test.res\n-Assert files equal: test.res test.ref\n```\n\nor define your own actions to invoke any shell command you like. For example:\n\n```\n-Check boiler #42 temperature: +54C\n```\n\ncan be configured to run\n\n```\n[ $(curl http://localhost/boiler/42/temperature) == \"+54C\" ]\n```\n\n---\n\n### Table of Contents\n\n* [Test case format](#test-case-format)\n  * [File structure](#file-structure)\n  * [Parameter sections](#parameter-sections)\n  * [Action sections](#action-sections)\n* [Configuration](#configuration)\n  * [Variables](#variables)\n  * [User defined actions](#user-defined-actions)\n* [Command line options](#command-line-options)\n* [Showcase](#showcase)\n* [Build instructions](#build-instructions)\n* [Running unit tests](#running-unit-tests)\n* [Running functional tests](#running-functional-tests)\n* [License](#license)\n\n### Test case format\n\nA test case that can be launched by `lunchtoast` is a directory containing a test file `test.toast`:\n\n```\nmy_test/\n  test.toast\n```\n\nThe test can be launched by simply passing its path to the `lunchtoast` executable:\n\n```shell\nkamchatka-volcano@home:~$ lunchtoast my_test/\n```\n\nMultiple tests can be launched by passing its parent directory:\n\n```\ntests_collection/\n  my_test_1/\n    test.toast\n  my_test_2/\n    test.toast   \n```\n\n```shell\nkamchatka-volcano@home:~$ lunchtoast tests_collection/\n```\n\n#### File structure\n\nTest cases in lunchtoast are written in a simple configuration file format with the `.toast` extension. A test case file\nconsists of multiple sections that define test parameters and actions.  \nSections are created by starting the line with `-` followed by the section's name and parameters, and delimiting the\nsection's value with `:`.   \nThe section's value is everything between `:` and the end of the line. By default, sections consist of only one line.\n\n```\n-SectionName: SectionValue\n```\n\n`lunchtoast` also supports multiline sections, which have a value that spans over multiple lines. To create a multiline\nsection, start a new line with `-` followed by the section's name and parameters, place a `:` delimiter, then start the\nsection's value on the next line. The section's value must be closed with `---` sequence or the end of the file.\n\n```\n-MultilineSectionName: \nSectionValueLine1\nSectionValueLine2\n---\n```\n\nEmpty section values can be created using empty multiline sections:\n\n```\n-EmptySection:\n---\n```\n\nor by omitting the `:` delimiter in single-line sections:\n\n```\n-EmptySection\n```\n\n#### Parameter sections\n\nThe following sections are available to set up test parameters:\n\n- **Name**  \n  Sets the name of the current test. The default value is the name of the test case directory.\n  ```\n  -Name: Example Test\n  ```\n\n- **Description**   \n  Provides information about the test that will be displayed in the report.\n  ```\n  -Description: \n    Detailed info about this test\n  ---\n  ```\n\n- **Suite**  \n  Sets the suite of the test. This currently only affects grouping in the report.\n  ```\n  -Suite: smoke tests\n  ```\n\n- **Enabled**  \n  Enables or disables the test. The default value is \"true\". To disable the test, set this to any other value.\n  ```\n  -Enabled: false\n  ```\n\n- **Contents**  \n  Specifies the files in the test directory that should be preserved. Files not specified in this parameter are deleted\n  at the start of the test and on successful completion. By default, the `test.toast` and `lunchtoast.cfg` files are\n  included and do not need to be specified. Files should be specified relative to the test directory and separated by\n  whitespace. Regular expressions can be used and should be surrounded with braces.\n  ```\n  -Contents: config.ini {.*\\.txt} \n  ```\n\n  The `Contents` section can be automatically generated based on the current state of the test directory by using\n  the `saveContents` command:\n  ```shell\n  kamchatka-volcano@home:~$ lunchtoast saveContents my_test/\n  ```\n\n- **Tags**  \n  Sets the list of tags separated by whitespace. Tags can be used to select or exclude a subset of tests by using\n  the `select` and `skip` command line parameters:\n  ```\n  -Tags: windows network\n  ```\n\n  ```shell\n  kamchatka-volcano@home:~$ lunchtoast test_collection/ -skip=windows,network\n  kamchatka-volcano@home:~$ lunchtoast test_collection/ -select=network\n  ```\n\n#### Action sections\n\nThe following sections are available to set up test actions:\n\n- **Launch**  \n  The `Launch` section launches a shell command, waits for it to complete, and checks its exit code. If the exit code is\n  different from 0, the action fails, and the test execution stops.\n  ```\n  -Launch: readme_proc.sh \u003e test.res\n  ``` \n\n  The default shell command is `bash -ceo pipefail`, but you can change it through the command line:\n  ```shell\n  kamchatka-volcano@home:~$ lunchtoast my_test/ -shell=\"sh -c -e\"\n  ```\n\n  To launch the process directly without invoking it through the system shell, use the `Launch process` format:\n  ```\n  -Launch process: my_proc --write-ouput test.res\n  ```\n\n  To launch the process in the background without waiting for its completion, use the `Launch detached` format:\n  ```\n  -Launch detached: my_proc \n  ```\n  Detached processes cannot be checked for their result, and they are automatically terminated at the end of the test.\n\n  There's also a way to launch processes and wait for their completion while ignoring the output. To use it, use\n  the `Launch ignoring output` format:\n\n  ``` \n  -Launch ignoring output: my_proc\n  ``` \n  Generally, you won't need to use it, but it can be helpful when encountering processes that hang, to confirm that\n  their execution is actually stuck waiting for the output.  \n  \u003cbr/\u003e\n  The previously mentioned launch formats are defined by the keywords \"process\", \"detached\", \"ignoring output\" or \"\n  ignore output\". These keywords can be combined and other words can be used in between them:\n  ```\n  -Launch detached process: my_proc\n  -Launch process and ignore output: my_proc\n  ``` \n\n- **Assert/Expect files equal**  \n  The `Assert files equal` section checks that two files specified in the section's value are equal. If they differ, the\n  action fails, and the test execution stops. To continue execution on assertion failure, use the `Expect files equal`\n  action:\n  ```\n  -Expect files equal: lhs.txt rhs.txt\n  -Assert files equal: 1.res 1.ref \n  ```\n  By default, files are compared in text mode, which normalizes line separators (e.g., `\\n` and `\\r\\n` are considered\n  the same).\n  To compare files in binary mode, use the `Assert data files equal` format:\n  ```\n  -Assert data files equal: lhs.txt rhs.txt\n  ```\n\n- **Assert/Expect `\u003cfilename\u003e`**  \n  The `Assert \u003cfilename\u003e` section compares the content of the file to the section's value. If they differ, the action\n  fails, and the test execution stops. To continue execution on assertion failure, use the `Expect \u003cfilename\u003e` action:\n  ```\n  -Expect 1.res: Hello world\n  -Assert ouput.txt:\n  Hello world\n  Hello moon\n  ---\n  ```\n\n- **Assert/Expect exit code**  \n  The `Assert exit code` section compares the exit code of the `Launch` action to the section's value. If they differ,\n  the `Launch` action fails, and the test execution stops. To continue execution on assertion failure, use\n  the `Expect exit code` action:\n  ```\n  -Launch: my_proc --write-ouput test.res\n  -Assert exit code: 1\n  ```\n  If the exit code of the `Launch` action is not important, you can use `any` or `*` section value to allow any exit\n  code:\n  ```\n  -Launch: my_proc --write-ouput test.res\n  -Expect exit code: any\n  ```\n\n- **Assert/Expect output**  \n  The `Assert output` section compares the output of the `Launch` action to stdout to the section's value. If they\n  differ, the `Launch` action fails, and the test execution stops. To continue execution on assertion failure, use\n  the `Expect output` action:\n  ```\n  -Launch: echo \"Hello world\"\n  -Assert output: Hello world  \n  ```\n\n- **Assert/Expect error output**  \n  The `Assert error output` section compares the output of the `Launch` action to stderr to the section's value. If they\n  differ, the `Launch` action fails, and the test execution stops. To continue execution on assertion failure, use\n  the `Expect error output` action:\n  ```\n  -Launch: echo \"Hello world\" \u003e\u00262\n  -Assert error output: Hello world  \n  ```\n\n  *Note that all three assertions can be used after the `Launch` action simultaneously. However, it is not possible to\n  mix the `Expect` and `Assert` assertions:*\n  ```\n  -Launch: my_proc --write-ouput test.res\n  -Assert exit code: 1\n  -Assert error output: Can't open file test.res for writing     \n  ```\n\n- **Write `\u003cfilename\u003e`**  \n  The `Write \u003cfilename\u003e` action is used to write the section value to the file specified in the first section's\n  parameter:\n  ```\n  -Write input.txt: Hello world\n  ```\n\n- **Wait**  \n  The `Wait` action is used to wait for a specified amount of time by putting the thread to sleep:\n  ```\n  -Wait: 500 ms\n  -Wait: 2 sec\n  ```\n\n### Configuration\n\nAn optional configuration file for `lunchtoast` uses the [`shoal`](https://shoal.eelnet.org) config format and can be\nspecified on the command line using the `config` parameter:\n\n```shell\nkamchatka-volcano@home:~$ lunchtoast my_test/ -config=default.cfg\n```\n\nIf a config file named `lunchtoast.cfg` is placed in any directory that is part of the path passed to `lunchtoast`, it\nis read automatically. For example:\n\n```\n  tests_collection/\n    lunchtoast.cfg\n    my_test_1/\n      test.toast      \n    my_test_2/\n      lunchtoast.cfg\n      test.toast  \n```\n\n```shell\nkamchatka-volcano@home:~$ lunchtoast tests_collection/\n```\n\nIn this example, `my_test_1` is launched with the configuration from tests_collection/lunchtoast.cfg, while `my_test_2`\nuses a merged configuration from both `tests_collection/lunchtoast.cfg` and `tests_collection/my_test_2/lunchtoast.cfg`.\n\n#### Variables\n\nYou can define text variables in config files and use them in any section value of the test case. To use the variable\nvalue, use the format `${{ var }}`:\n\n```\n  -Launch: myprocess ${{ args }}  \n```\n\nIn the configuration file, variables are placed in the `vars` node:\n\n```\n#vars:\n  args = -o result.txt\n---  \n```\n\nTagged tests can use presets of variables defined for their tags. To do this, use a node list `tagVars`, whose elements\ncontain a parameter `tag` and a subnode `vars`:\n\n```\n#tagVars:\n###\n  tag = windows\n  #vars:\n    args = '-shell=\"msys2 -c\"'    \n###    \n  tag = linux\n  #vars:\n    args = '-shell=sh -ce'\n---\n```\n\nWith this configuration, a test tagged with `windows` tag will get a `msys` shell argument when using `${{ args }}`\nvariable.\n\n#### User defined actions\n\n`lunchtoast` provides a way to register shell commands as user actions. This allows you to make verbose commands\nclearer, add domain-specific information to your tests, or implement custom comparison logic for result files.  \nTo register the `Check boiler #\u003cboiler number\u003e temperature` action shown at the beginning of this document, use a node\nlist `actions`, whose elements contain parameters `format` and `command`:\n\n```\n#actions:\n###\n  format = Check boiler #%1 temperature\n  command = `[ $(curl http://localhost/boiler/%1/temperature) == \"%input\" ]`\n```\n\nThe `format` parameter is a template of the user-defined action and can contain string variables matching a sequence of\nnon-whitespace characters. Variables are identified by their indices starting with 1: `%1`, `%2`, etc.\n\nThe `command` parameter specifies the shell command invoked by this action. Variables captured in the `format` command\ncan be used here with the same syntax: `%1`, `%2`, etc. The `%input` variable contains the section's value of the\naction.\n\nBy default, the result of the command is expected to be `0`. You can control the expected result\nwith `checkExitCode`, `checkOutput`, and `checkErrorOutput` parameters. With them, it's possible to rewrite the same\naction in the configuration file like this:\n\n```\n#actions:\n###\n  format = Check boiler #%1 temperature\n  command = curl http://localhost/boiler/%1/temperature\n  checkOutput = \"%input\"\n```\n\nTo simplify action names with multiple variables, it is possible to register them in the `%input` variable by splitting\nthe section value into multiple subsections. Let's add a JSON payload to the `Check boiler #\u003cboiler number\u003e temperature`\ncommand to demonstrate this:\n\n```\n-Check boiler #%1 temperature\":\n#payload:\n{\n  \"access_token\": \"ae92f9f1-825a-453a-8924-556663c5d4b9\" \n}\n#expected_temperature:\n+42C\n---\n```\n\nTo implement this in the configuration file, we use the `%input.payload` and `%input.expected_temperature` variables:\n\n```\n#actions:\n###\n  format = Check boiler #%1 temperature\n  command = curl -d '%input.payload' http://localhost/boiler/42/temperature\n  checkOutput = \"%input.expected_temperature\"\n---\n```\n\nSubsections are created by starting the line with `#` followed by the subsection's name and parameters, placing a `:`\ndelimiter, and starting the section's value on the next line. The subsection's value is closed with `---`, the start of\nthe next subsection or the end of the file.\n\n### Command line options\n\n|                              |                                                                                     |\n|------------------------------|-------------------------------------------------------------------------------------|\n| **Arguments:**               |                                                                                     |\n| `\u003ctestPath\u003e`                 | directory containing tests                                                          |\n| **Parameters:**              |                                                                                     |\n| `-config=\u003cpath\u003e`             | config file for setting variables and actions (optional)                            |\n| `-shell=\u003cstring\u003e`            | shell command (optional, default: bash -ceo pipefail)                               |\n| `-listFailedTests=\u003cpath\u003e`    | write a list of failed tests to the specified file (optional)                       |\n| `-collectFailedTests=\u003cpath\u003e` | copy directories containing failed tests to the specified path (optional)           |\n| `-reportWidth=\u003cint\u003e`         | set the test report's width as the number of characters (optional, default: 48)     |\n| `-reportFile=\u003cpath\u003e`         | write the test report to the specified file (optional)                              |\n| `-searchDepth=\u003cint\u003e`         | the number of descents into child directories levels for tests searching (optional) |\n| `-select=\u003cstring\u003e`           | select tests by tag names (multi-value, optional)                                   | \n| `-skip=\u003cstring\u003e`             | skip tests by tag names (multi-value, optional)                                     |\n| **Flags:**                   |                                                                                     | \n| `--withoutCleanup`           | disable cleanup of test files                                                       |\n| `--help`                     | show usage info and exit                                                            |\n| **Commands:**                |                                                                                     |\n| `saveContents [options]`     | save the current contents of the test directory                                     |\n\n### Showcase\n- [`asyncgi/functional_tests`](https://github.com/kamchatka-volcano/asyncgi/tree/master/functional_tests)\n- [`figcone/functional_tests`](https://github.com/kamchatka-volcano/figcone/tree/master/functional_tests)\n- [`lunchtoast/functional_tests`](https://github.com/kamchatka-volcano/lunchtoast/tree/master/functional_tests)\n\n### Build instructions\n\nTo build `lunchtoast`, you will need a C++20-compliant compiler, CMake, and the Boost.Process library installed on your\nsystem.\n\n```\ngit clone https://github.com/kamchatka-volcano/lunchtoast.git\ncd lunchtoast\ncmake -S . -B build\ncmake --build build\n```\n\nBoost dependencies can be resolved using [`vcpkg`](https://vcpkg.io/en/getting-started.html) by running the build with\nthis command:\n\n```\ncmake -S . -B build -DCMAKE_TOOLCHAIN_FILE=\u003cvcpkg path\u003e/scripts/buildsystems/vcpkg.cmake\n```\n\n### Running unit tests\n\n```\ncd lunchtoast\ncmake -S . -B build -DENABLE_TESTS=ON\ncmake --build build \ncd build/tests \u0026\u0026 ctest\n```\n\n### Running functional tests\n\n`lunchtoast` is tested for regression using the `lunchtoast` itself. Once the development branch is built, functional\ntests are executed using the latest release of `lunchtoast` in the following manner:\n\n* Linux command:\n\n```\n\u003clunchtoast_release\u003e/lunchtoast \u003cdev_branch_dir\u003e/functional_tests -skip=windows -config=linux_vars.shoal -searchDepth=1\n```\n\n* Windows command:\n\n```\n\u003clunchtoast_release\u003e/lunchtoast.exe \u003cdev_branch_dir\u003e/functional_tests -skip=linux -config=windows_vars.shoal -searchDepth=1\n```\n\nTo run functional tests on Windows, it's recommended to use the bash shell from\nthe  [`msys2`](https://www.msys2.org/#installation) project. After installing it, add the following script `msys2.cmd`\nto your system `PATH`:\n\n```bat\n@echo off\nsetlocal\nIF NOT DEFINED MSYS2_PATH_TYPE set MSYS2_PATH_TYPE=inherit\nset CHERE_INVOKING=1\nC:\\\\msys64\\\\usr\\\\bin\\\\bash.exe -leo pipefail %*\n```\n\n### License\n\n**lunchtoast** is licensed under the [MS-PL license](/LICENSE.md)  ","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkamchatka-volcano%2Flunchtoast","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkamchatka-volcano%2Flunchtoast","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkamchatka-volcano%2Flunchtoast/lists"}