{"id":15042456,"url":"https://github.com/styczynski/bash-universal-tester","last_synced_at":"2025-04-14T20:33:44.612Z","repository":{"id":49350347,"uuid":"97726103","full_name":"styczynski/bash-universal-tester","owner":"styczynski","description":":white_check_mark: Universal testing script for bash","archived":false,"fork":false,"pushed_at":"2022-07-24T21:51:16.000Z","size":85,"stargazers_count":17,"open_issues_count":6,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-01T20:31:46.232Z","etag":null,"topics":["bash","bash-script","bash-universal-tester","cli","cli-utilities","script","shell-script","test","test-automation","test-data","test-runner","testing","testing-tools","unit-testing"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/styczynski.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-07-19T14:29:16.000Z","updated_at":"2024-12-08T07:33:05.000Z","dependencies_parsed_at":"2022-09-23T21:21:44.909Z","dependency_job_id":null,"html_url":"https://github.com/styczynski/bash-universal-tester","commit_stats":null,"previous_names":[],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/styczynski%2Fbash-universal-tester","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/styczynski%2Fbash-universal-tester/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/styczynski%2Fbash-universal-tester/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/styczynski%2Fbash-universal-tester/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/styczynski","download_url":"https://codeload.github.com/styczynski/bash-universal-tester/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248593733,"owners_count":21130312,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bash","bash-script","bash-universal-tester","cli","cli-utilities","script","shell-script","test","test-automation","test-data","test-runner","testing","testing-tools","unit-testing"],"created_at":"2024-09-24T20:47:20.482Z","updated_at":"2025-04-14T20:33:44.588Z","avatar_url":"https://github.com/styczynski.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Made by Styczynsky Digital Systems][badge sts]][link styczynski]\n[![Travis build status][badge travis]][link travis] \n\n[![NPM](https://nodei.co/npm/bash-universal-tester.png?mini=true)](https://www.npmjs.com/package/bash-universal-tester) \n\n**Superquick installation via** `npm install -g bash-universal-tester`\n\n# :white_check_mark: bash-universal-tester \u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp; [![Download][badge download]][link download latest]\nUniversal testing script for bash\n\nGeneral purpose awesome **testing-script**\n\n![Screenshot 1][screenshot 1]\n\n## What?\n\nCurrently this script supports:\n\n* Testing native executables with text input/out files (.in/.out) \n\n* Testing native executables with .in/.out files packed to a zip file \n\n* Using custom testing scripts \n\n* Some nice types of output formatting \n\n## Requirements\n\nThis is stand-alone script.\n\n* Works on Linux with bash shell (tested on 4.4.12 and 4.3.39)\n\n* Works on Windows **(BUT YOU NEED BASH EMULATION LIKE [CYGWIN][link cygwin])**\n\n## Installation\n\n* Install via `npm`:\n\n```bash\n\nnpm install -g bash-universal-tester\n\n```\n\n* Download through provided link \u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp; [![Download][badge download]][link download latest]\n\n* Alternatively on bash type: \n\n```bash\n\nwget https://raw.githubusercontent.com/styczynski/bash-universal-tester/master/utest.sh \u0026\u0026 chmod u+x ./utest.sh \u0026\u0026 mv ./utest.sh utest \u0026\u0026 PATH=$PATH:$PWD\n\n```\n\n## Basic usage\n\nBasic usage:\n\t`utest \u003cprog\u003e`\n\t\nThe script tries to autodetect folder with input/output test files.\nAnd sometimes gives suggestions what program you may want to test.\n\n### Basic in + out + non empty err\n\nTo test input folder (only .in and .out): \n\n\tutest \u003cprog\u003e \u003cfolder\u003e\n\t\n### Basic in + out + ignore err (err is not checked anytime)\n\nTo test input folder (only .in and .out): \n\n\tutest --tierr \u003cprog\u003e \u003cfolder\u003e\n\n### Basic in + out + err (missing .err file cause error)\n\nTo test input folder (.in, .out, and .err files): \n\n\tutest --tnerr \u003cprog\u003e \u003cfolder\u003e\n\n### Basic in + out + err (missing .err files are ignored)\n\nTo test input folder (.in, .out, and .err files): \n\n\tutest --tgerr \u003cfolder\u003e \u003cprog\u003e \u003cfolder\u003e\n\n### Globbing input files\n\nLet's suppose we have the following file structure: \n```\ntests\n|- test1 \n|  |- input.in\n|  \\- output.out\n|\n|- test2\n   |- input.in\n   \\- output.out\n```\n\nWe want to feed utest with input files nested in subdirectories.\nFor that purpose just use:\n\n\tutest \u003cprog\u003e \"./tests/**/*.in\"\n\t\n**Note that globbing must be provided in quotes otherwise it will be parsed by shell and won't work!**\n\t\n### Custom input layout\n\nLet's suppose we have the following file structure (even more unfriendly!): \n\n```\ntests\n|- test1 \n|  |- input.txt\n|  \\- out\n|\n|- test2\n   |- input.txt\n   \\- out\n```\n\nWe want to feed utest with input files nested in subdirectories.\nAnd the input files have custom extensions.\nWe must tell utest where to find output files.\nWe use `--tgout` flag that utilizes dynamic variable to generate output path.\nYou can read more about dynamic variables in *variables* section.\n\n\tutest \u003cprog\u003e --tgout \"%input_file_folder/out\" \"./tests/**/input.txt\"\n\t\n**Note that globbing must be provided in quotes otherwise it will be parsed by shell and won't work!**\n\t\n\t\n## Advanced usage\n\n`utest [test_flags] \u003cprog\u003e \u003cdir\u003e [prog_flags]`\n\n* `\u003cprog\u003e` is path to the executable, you want to test\n\n* `\u003cdir\u003e` is the path to folder containing .in/.out files\n\n* `[prog_flags]` are optional conmmand line argument passed to program `\u003cprog\u003e`\n\n* `[test_flags]` are optional flags for test script\n\n\u003cbr\u003e\u003cbr\u003e\n\t  \n|          Switch                   | Parameters | Description  |\n|-----------------------------------|------------|--------------|\n| **--tdebug**                      |            | Disables removing any files (so the utest probably won't do any harm) and enables --tlog flag. |\n| **--tlog**                        |            | Enable logging into utest.log file in the current directory. |\n| **--ttools**                      | *[tools]*  | Sets additional debug tools.\u003cbr\u003e`[tools]` is the coma-separated array of tools names.\u003cbr\u003eTools names can be as the following:\u003cbr\u003e\u003cul\u003e\u003cli\u003e\u003cb\u003esize\u003c/b\u003e - prints the size of input file in bytes.\u003c/li\u003e\u003cli\u003e\u003cb\u003etime\u003c/b\u003e - prints time statistic using Unix time command.\u003c/li\u003e\u003cli\u003e\u003cb\u003estime\u003c/b\u003e - measures time using bash date command (not as precise as time tool).\u003c/li\u003e\u003cli\u003e\u003cb\u003evmemcheck\u003c/b\u003e - uses valgrind memcheck tools to search for application leaks.\u003c/li\u003e\u003cli\u003e\u003cb\u003evmassif\u003c/b\u003e - uses valgrind massif and prints peak memory usage.\u003c/li\u003e\u003c/ul\u003e |\n| **--tscript**                     | *[script]* | Sets output testing command as `[script]`\u003cbr\u003eScript path is path to the executed script/program.\u003cbr\u003eThere exists few built-in testing scripts:\u003cbr\u003e\u003cul\u003e\u003cli\u003eUse \u003cb\u003e--tscript ignore\u003c/b\u003e to always assume output is OK.\u003c/li\u003e |\n| **--tscript-err**                 | *[script]* | Sets stderr output testing command as `[script]`\u003cbr\u003eScript path is path to the executed script/program.\u003cbr\u003eThere exists few built-in testing scripts:\u003cbr\u003e\u003cul\u003e\u003cli\u003eUse \u003cb\u003e--tscript-err ignore\u003c/b\u003e to always assume stderr is OK.\u003c/li\u003e |\n| **--tflags**                      |            | Enables \u003cb\u003e--t(...)\u003c/b\u003e flags interpreting at any place among command line arguments\u003cbr\u003e\u003ci\u003e(by default flags after dir are expected to be program flags)\u003c/i\u003e |\n| **--tsty-format**                 |            | Make tester use \u003ci\u003e!error!\u003c/i\u003e, \u003ci\u003e!info!\u003c/i\u003e etc. output format |\n| **--tterm-format**                |            | Make tester use (default) \u003ci\u003eterm\u003c/i\u003e color formatting |\n| **--tc**\u003cbr\u003e**--tnone-format**    |            | Make tester use \u003ci\u003eclean\u003c/i\u003e (only-text) formatting |\n| **--tpipe-in**                    |*[command]* | Use preprocessing of input. See \u003cb\u003ePiping\u003c/b\u003e section |\n| **--tpipe-out**                   |*[command]* | Use postprocessing of output. See \u003cb\u003ePiping\u003c/b\u003e section |\n| **--tpipe-out-err**               |*[command]* | Use postprocessing of output error stream. See \u003cb\u003ePiping\u003c/b\u003e section |\n| **--ts**                          |            | Skips always oks |\n| **--tierr**                       |            | Always ignore stderr output |\n| **--tgout**                       |  *[dir]*   | Sets \u003ci\u003e(good)\u003c/i\u003e .out input directory\u003cbr\u003e(default is the same as dir/inputs will be still found in dir location/use when .out and .in are in separate locations) |\n| **--tgerr**                       |  *[dir]*   | Same as \u003cb\u003e--tgout\u003c/b\u003e but says where to find good .err files\u003cbr\u003e(by default nonempty .err file means error) |\n| **--terr**                        |  *[dir]*   | Sets .err output directory (default is /out) |\n| **--tout**                        |  *[dir]*   | Set output .out file directory (default is /out) |\n| **--tf**                          |            | Proceeds even if directories do not exists etc. |\n| **--tneed-err**\u003cbr\u003e**--tnerr**    |            | Always need .err files (by default missing good .err files are ignored)\u003cbr\u003eIf \u003cb\u003e--tnerr\u003c/b\u003e flag is used and \u003cb\u003e--tgerr\u003c/b\u003e not specified the good .err files are beeing searched in `[dir]` folder. |\n| **--te**\u003cbr\u003e**--tdefault-no-err** |            | If the .err file not exists (ignored by default) require stderr to be empty |\n| **--tt**                          |            | Automatically create missing .out files using program output |\n| **--tn**                          |            | Skips after-testing summary |\n| **--ta**                          |            | Aborts after +5 errors |\n| **-help**\u003cbr\u003e**--help**           |            | Displays help info |\n| **--tm**                          |            | Use minimalistic mode (less output) |\n| **--tmm**                         |            | Use very minimalistic mode (even less output) |\n| **--tmmm**                        |            | Use the most minimialistic mode (only file names are shown) |\n\n\n\nWherever **-help**, **--help** flags are placed the script always displays its help info.\n\n\u003cbr\u003e\u003cbr\u003e\n\nAbout minimalistic modes: \n\n* In **--tm** mode OKs are not printed / erors are atill full with diff \n\n* In **--tmm** mode errors are only generally descripted / OK at output on success \n\n* In **--tmmm** only names of error files are printed / Nothing at output on success\n\n## Variables\n\nIn `\u003cprog\u003e`, `--tgerr \u003cdir\u003e`, `--tgout \u003cdir\u003e` and config files you can use special dynamic variables.\nThese are the following: \n\n| name                     | description                                                      |\n|--------------------------|------------------------------------------------------------------|\n| **%input_file**          | Current input file name along with extension                     |\n| **%input_file_name**     | Current input file without .in or .out extension                 |\n| **%input_file_folder**   | Directory of current input file                                  |\n| **%input_file_path**     | Full input path                                                  |\n| **%input_file_list**     | List of all input files (separated by space) that will be loaded |\n| **%file_count**          | Number of all input files that will be loaded                    |\n| **%file_index**          | Number of the current input file starting from 1                 |\n| **%ok_index**            | Current number of test that succeeded                            |\n| **%warn_index**          | Current number of test that generated warnnings                  |\n| **%not_exists_index**    | Current number of test that had problems with non existing files |\n| **%param_prog**          | Currently tested command                                         |\n| **%input_prog_flag_acc** | Currently tested command's arguments                             |\n\nExample usage: \n```\nutest \"echo %input_file\" \u003cfolder\u003e\n```\n\n\nMoreover you can use formatting variables (that are set via formatting switches). \nPlease use them instead of hard-coded values, because it's easy and improves \ncustomizability of your output.\n\n| formatting variable name | description                                    |\n|--------------------------|------------------------------------------------|\n|  **%bdebug**             | Begins \u003cb\u003eDEBUG\u003c/b\u003e text section               |\n|  **%edebug**             | Ends \u003cb\u003eDEBUG\u003c/b\u003e text section                 |\n|  **%berr**               | Begins \u003cb\u003eERROR\u003c/b\u003e text section               |\n|  **%eerr**               | Ends \u003cb\u003eERROR\u003c/b\u003e text section                 |\n|  **%binfo**              | Begins \u003cb\u003eINFORMATION\u003c/b\u003e text section         |\n|  **%einfo**              | Ends \u003cb\u003eINFORMATION\u003c/b\u003e text section           |\n|  **%bwarn**              | Begins \u003cb\u003eWARNNING\u003c/b\u003e text section            |\n|  **%ewarn**              | Ends \u003cb\u003eWARNNING\u003c/b\u003e text section              |\n|  **%bbold**              | Begins \u003cb\u003eNOTICE\u003c/b\u003e text section              |\n|  **%ebold**              | Ends \u003cb\u003eNOTICE\u003c/b\u003e text section                |\n|  **%bok**                | Begins \u003cb\u003eOK STATUS\u003c/b\u003e text section           |\n|  **%eok**                | Ends \u003cb\u003eOK STATUS\u003c/b\u003e text section             |\n\nExample usage: \n```yaml\n\ninput: ./test\nexecutions:\n    - prog\nhooks:\n    test_case_start:\n        - @echo %{bwarn}Hello%{ewarn} %input_file %{bok} %ok_index %{eok}\nprog:\n    command: echo Something\n    \n```\n\n\n## Piping\n\nUtest provides easy way to preprocess your input file or postprocess program outputs. \n\nAll you have to do is to use `--tpipe-in \u003ccommand\u003e`, `--tpipe-out \u003ccommand\u003e` or `--tpipe-out-err \u003ccommand\u003e`. \n\nPipes are provided with additional variables: \n\n| name                   | description                                  |\n|------------------------|----------------------------------------------|\n| **%input**             | Pipe program input file path                 |\n| **%output**            | Pipe program output file path                |\n\nFor example let's sort program output alphabetically: \n```\nutest.sh --tpipe-out \"cat %input | sort \u003e %output\" \u003cprog\u003e \u003cfolder\u003e\n```\n\nAdvantage of pipes are that you do not modify in/out files directly. \nAnd you can test programs that may potentailly give not exactly the same answers but which are still correct.\n\n## Configuration file\n\n### Global configuration\n\nInstead of passing all parameters by command line we offer the ability to put everything into single *YAML* file!\n\nUtest seek for `utest.yaml` file in current directory. It can contain all configuration available via\ncommand line switches and flags!\n\nAll config options are listed there: \n\n```yaml\n\ninput: test/*.in\nsilent: false\ngood_output: test/%input_file_name.out\ngood_err: test/%input_file_name.err\nneed_error_files: false\ntesting_script_out: ignore\ntesting_script_err: ignore\nexecutions:\n    - prog1\n    - prog2\nprog1:\n    cwd: ./somefolder\n    command: ./test/totest.sh\n    args: %input_file_name\n    pipes_out:\n        - echo 123 \u003e %output\n    pipes_in:\n        - echo 123 \u003e %output\n    pipes_out_err:\n        - echo 123 \u003e %output\n        - cat %input | echo 123 \u003e %output\nprog2:\n    command: echo 99\n\n```\n\n### Single test configuration\n\nYou can also configure environment for **single test case**!\nJust put `xyz.config.yaml` file next to your input file `xyz.in`.\n\nAll options of config file for single test are listed below: \n\n```yaml\n\nprog2:\n    cwd: ./somefolder\n    args: %input_file_name some additional args\n    input: override_input_file.in\n\n```\n\nYou must identify program by the command it calls. \n\n### Hooks\n\nYou can provide hooks commands for any of testing lifecycle state.\n\nAll available hooks are: \n\n* \u003cb\u003einit\u003c/b\u003e - called when testing begins\n* \u003cb\u003edeinit\u003c/b\u003e - called when testing ends\n* \u003cb\u003etest_case_start\u003c/b\u003e - called when new test file is tested\n* \u003cb\u003etest_case_finish\u003c/b\u003e - called when the test file was tested\n* \u003cb\u003etest_case_fail\u003c/b\u003e - called when test fails\n* \u003cb\u003etest_case_fail_out\u003c/b\u003e - called when test fails on std output (launched after \u003cb\u003etest_case_fail\u003c/b\u003e)\n* \u003cb\u003etest_case_fail_err\u003c/b\u003e - called when test fails on error output (launched after \u003cb\u003etest_case_fail\u003c/b\u003e)\n* \u003cb\u003etest_case_fail_success\u003c/b\u003e - called when test succeeded\n\n**Please note that:**\nYou can add mutliple commands that are executed in order from up to the bottom. \nIf the command begins with `@` character then it's output is directly displayed. \nIf not then utest can change it to be more readable to the user! \n\n```yaml\n\ninput: test/*.in\ngood_output: test/%input_file_name.out\nneed_error_files: false\nexecutions:\n    - prog1\n    - prog2\nhooks:\n    init:\n        - @echo Testing\n        - @echo Prepare input...\n    deinit:\n        - @echo Goodbye\n    test_case_fail:\n        - @echo [%{input_file}]  Test case failed for %{param_prog}\n    test_case_success:\n        - @echo [%{input_file}]  Test case successed for %{param_prog}\n    test_case_fail_out:\n        - @echo [%{input_file}]  FAILED ON OUTPUT\n    test_case_fail_err:\n        - @echo [%{input_file}]  FAILED ON ERR\n        - @echo Whats a shame\n    test_case_start:\n        - @echo New test case jsut started %{input_file}\n    test_case_finish:\n        - @echo The test case was finished\nprog1:\n    command: ./test/totest.sh\n    args: %input_file_name\n    pipes_out:\n        - echo 15 \u003e %output\nprog2:\n    command: echo 159\n\n```\n\n### Custom output format\n\nUsing `--tsilent` flags allows only hooks to write output.\nSo if you use `@` sign along with hooks (see **Hooks** section)\nyou can make utest output any format of the output you want!\n\nExample of outputing `ERR \u003cfile\u003e` only on errors.\n\n```yaml\n\nhooks:\n    test_case_fail:\n        - @echo ERR %{input_file}\n```\n\nSimple enough, right?\n\n[badge travis]: https://travis-ci.org/styczynski/bash-universal-tester.svg?branch=master\n\n[link travis]: https://travis-ci.org/styczynski/bash-universal-tester\n\n[badge sts]: https://img.shields.io/badge/-styczynsky_digital_systems-blue.svg?style=flat-square\u0026logoWidth=20\u0026logo=data%3Aimage%2Fpng%3Bbase64%2CiVBORw0KGgoAAAANSUhEUgAAABYAAAAXCAYAAAAP6L%2BeAAAABmJLR0QA%2FwD%2FAP%2BgvaeTAAAACXBIWXMAAA7DAAAOwwHHb6hkAAAAB3RJTUUH4AgSEh0nVTTLngAAAB1pVFh0Q29tbWVudAAAAAAAQ3JlYXRlZCB3aXRoIEdJTVBkLmUHAAAAm0lEQVQ4y2Pc%2Bkz2PwMNAAs2wVMzk4jSbJY%2BD6ccEwONACMsKIh1JSEgbXKeQdr4PO1cPPQMZiGkoC7bkCQD7%2Fx7znDn35AOClK9PEJSBbNYAJz999UGrOLocsM0KHB5EZ%2FXPxiVMDAwMDD8SP3DwJA6kFka5hJCQOBcDwMDAwPDm3%2FbGBj%2BbR8tNrFUTbiAB8tknHI7%2FuTilAMA9aAwA8miDpgAAAAASUVORK5CYII%3D\n\n[badge download]: https://img.shields.io/badge/-download_me!-green.svg?style=flat-square\u0026logoWidth=10\u0026logo=data%3Aimage%2Fpng%3Bbase64%2CiVBORw0KGgoAAAANSUhEUgAAABkAAAArCAYAAACNWyPFAAAABmJLR0QA%2FwD%2FAP%2BgvaeTAAAACXBIWXMAAA7DAAAOwwHHb6hkAAAAB3RJTUUH4AgTDjEFFOXcpQAAAM1JREFUWMPt2EsOgzAMBFDPJHD%2F80Jid1G1KpR8SqKu7C2QJzwWsoCZSWedb0Tvg5Q%2FlCOOOOKII4444ogjjvxW8bTjYtK57zNTSoCdNm5VBcmRhdua7SJpKaXhN2hmEmO0fd%2BnANXgl2WxbduGAVUFVbUY9rquPVARyDmDpJCktKBK66pACOE5Ia%2FhUlUhaTPm9xM4ZEJScs6YDXwFH0IYgq6Ay%2Bm6C5WAQyYXo9edUQ2oIr1Q5TPUh4iImJkAsMI1AO3O4u4fiV5AROQBGVB7Fu2akxMAAAAASUVORK5CYII%3D\n\n[link styczynski]: http://styczynski.in\n\n[link cygwin]: https://cygwin.com\n\n[screenshot 1]: https://raw.githubusercontent.com/styczynski/bash-universal-tester/master/static/screenshots/screenshot1.png\n\n[link download latest]: https://github.com/styczynski/bash-universal-tester/archive/1.9.5.zip\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstyczynski%2Fbash-universal-tester","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstyczynski%2Fbash-universal-tester","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstyczynski%2Fbash-universal-tester/lists"}