{"id":13908415,"url":"https://github.com/gpac/testsuite","last_synced_at":"2025-04-22T13:24:24.164Z","repository":{"id":38432750,"uuid":"213679007","full_name":"gpac/testsuite","owner":"gpac","description":"GPAC test suite","archived":false,"fork":false,"pushed_at":"2025-04-14T14:02:46.000Z","size":7882,"stargazers_count":12,"open_issues_count":3,"forks_count":13,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-04-14T14:54:13.082Z","etag":null,"topics":["gpac","multimedia","testing"],"latest_commit_sha":null,"homepage":"https://tests.gpac.io","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gpac.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2019-10-08T15:16:09.000Z","updated_at":"2025-04-14T14:02:50.000Z","dependencies_parsed_at":"2024-04-15T13:53:23.680Z","dependency_job_id":null,"html_url":"https://github.com/gpac/testsuite","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gpac%2Ftestsuite","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gpac%2Ftestsuite/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gpac%2Ftestsuite/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gpac%2Ftestsuite/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gpac","download_url":"https://codeload.github.com/gpac/testsuite/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250246985,"owners_count":21398984,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gpac","multimedia","testing"],"created_at":"2024-08-06T23:02:42.884Z","updated_at":"2025-04-22T13:24:24.155Z","avatar_url":"https://github.com/gpac.png","language":"Shell","funding_links":[],"categories":["HarmonyOS"],"sub_categories":["Windows Manager"],"readme":"This is the documentation for creating a new test in GPAC test suite.\n\n# Running the test suite\n\nBefore running the test suite, you should first synchronize external data not hosted in this repository:\n\n```./make_tests.sh -sync-media -clean```\n\nThe external data is currently less than 300 MB.\n\n- to run the entire test suite, do `make_tests.sh` or use the Makefile task `make test_suite`.\n- to perform a quick run on the test suite, do `make_tests.sh -quick` : this will run only a subset of the tests.\n- to run a dedicated test (for example HLS generation test), do `make_tests.sh scripts/hls-gen.sh\n\nRunning the test suite or a given test generates:\n- result/all_logs.txt: logs of all tests that were executed \n- result/all_results.xml: GNU time/gtime statistics and execution status of all tests that were executed\n\nThe different options of make_tests.sh are given by the command:\n```\n./make_tests.sh -h\n```\n\nIt is important to notice that tests are cached, so that only failed tests are re-launched when running the test suite. You can clean the status of the entire test suite by using `./make_tests.sh -clean`, or the status of a given script (for example HLS generation test) by using `./make_tests.sh scripts/hls-gen.sh -clean`.\n\n\n\n# Writing a test\nGPAC test suite is composed of scripts written in the Bash language. Tests are placed in testsuite/scripts/. Each `.sh` file in that folder will be executed when running the entire test suite or may be run individually.\n\nMedia files used in a test can be anywhere (local file, http URL).\n- the environmnent variable `$MEDIA_DIR` can be used to access to data located in testsuite/media (located in this repository). \n- the environmnent variable `$EXTERNAL_MEDIA_DIR` can be used to access to data located in testsuite/external_media (not hosted in this repository). \n\nScripts should use the `$TEMP_DIR` environment variable to get a directory where to place the file they generate.\n\nThis directory is cleaned after each script, unless `-tmp` option is set. This allows preparing variables or files for a series of tests.\n\n__WARNING__\nSince the temp directory is __ONLY__ cleaned after each script, be careful to remove any state file (for example dasher context) produced if you run several tests in the same script, or use different names for each test.\n\n \nA simple GPAC test can be:\n```\n#!/bin/sh\nMP4Box -add $MEDIA_DIR\\foo.bar $TEMP_DIR\\foo.mp4\n```\nor \n```\n#!/bin/sh\ngpac -play $MEDIA_DIR\\foo.mp4\n```\n\nIn order to provide test caching and produce the unified test report for all the tests, with common timing, logging, etc, `make_tests.sh` defines several functions that should be used in a test.\n\nEach test has its own log file `$LOGS` in which you can write (a lot is already in there, such as test name/data and all stderr).\n\nTests may be customized depending on the platform they run on.\nThe environment variable `$GPAC_OSTYPE` can be used to test the binary version used; the currently defined values are `lin32`, `lin64`, `win32`, `win64`, `osx32`, `osx64` and `unknown`.\nThe environment variable `$GPAC_CPU`can be used to test the cpu type; the currently defined values are `x86` and `arm`.\n\n## Simple Testing\nA simple test whose result will be integrated in the test suite report is run using the single_test function.\n\n### single_test \nTwo arguments:\n- `CMD_LINE`: Executes the given `CMD_LINE`. The command line must be passed between single or double quotes. \n- `TESTNAME`: specifies the test name\n\nEach call generates:\n- a log file called `TESTNAME-single-logs.txt` containg test name, command line and all stderr of the test. You should not use file logging of GPAC ( `-lf` or `-logs` options) in your command line.\n- a file called `TESTNAME-single-passed.xml` containg the statistics of the test, as retrieved by GNU time/gtime.\n`TESTNAME` shall be unique in the test suite. In case of doubts, run the test suite script with `-check-name`.\n\nA simple test using `single_test` looks like:\n```\n#!/bin/sh\nsingle_test \"MP4Box -add $MEDIA_DIR\\foo.bar $TEMP_DIR\\foo.mp4\" test01\n```\nor \n```\n#!/bin/sh\nsingle_test \"gpac -play $MEDIA_DIR\\foo.mp4\" test02\n```\n\n### do_play_test \nTwo or three arguments: \n- `TESTNAME`: specifies the test name for logging, as used in `single_test`\n- `VIDEO_ARGS`: specifies the video arguments to be loaded. If not empty, a filter chain will be loaded using these arguments as source (i.e.,  `gpac -i VIDEO_ARGS`)\n- `AUDIO_ARGS`: specifies the audio arguments to be loaded. If not empty, a filter chain will be loaded using these arguments as source (i.e.,  `gpac -i AUDIO_ARGS`). If `-`, argument is ignored but an audio filter subchain ( `aout` or `enc:aac`) will be added to the filter chain.\nPerforms a playback test using `gpac` with the arguments given in `VIDEO_ARGS` and `AUDIO_ARGS`.\nIf `-play` option is specified when running the tests, playbacks the sources.\nIf `-video`  option is specified when running the tests, encodes the result using H264 and AAC in `results/video/$TESTNAME.mp4`.\n\n\n## Aggregating tests\nWhen the results of several tests need to be aggregated, instead of using multiple `single_test` calls, sub-tests can be run within a test using the `do_test` function. \nThis is useful when several tests require the generation of the same input file, for example.\n\nA typical test with two subtests used in a batch will look like:\n```\nmytest()\n{\ntest_begin $1\ndo_test CMD_LINE1 \"Name1\"\ndo_test CMD_LINE1 \"Name2\"\ntest_end\n}\n\nfor i in * ; do\n  mytest $i\ndone\n```\n\n### test_begin \nArgument:\n- `TESTNAME`: specifies the test name for logging, as used in `single_test`\n\nThis function defines overridable variables:\n- `$dump_size`: by default \"192x192\" but can be overriden by your test\n- `$dump_dur`: by default \"5\" seconds but can be overriden by your test\n\n### test_end\nNo argument.\n\nThis function triggers the end of the test and writes all logs and statistics.\n\nThe function evaluates the variable `$result`. If not empty, the test is considered failed, otherwise (default) the test has passed. \nAll results of subtests are automatically appended to the `$result` variable in the `end_test` function.\n\n__WARNING__\nThis variable can not be set in a subshell (eg in `some_function SOME_PARAM \u0026` called during a test), it must be set in the shell calling `test_begin`.\n\n### do_test\nTwo arguments:\n- `CMD_LINE`: the command line to execute\n- `SUBTEST_NAME`: the subtest name as it appears in the logs and in the stats\n\nPerforms a subtest in the current test. If needed, the return value is available in `$ret`. \nFunction does nothing when the test is skipped (see below) or a previous error occured in the parent test `TESTNAME`.\n\n\n## Test failure/success\nA test is considered successful if the execution returned 0.\nSome tests may return a non-zero value and still be considered successful (e.g. negative tests). In that case, errors to be ignored are placed in a file in the gpac/rules directory. If any of the line in this file is found, the test is considered successful. \nThe error file is named:\n- `TESTNAME-stderr.txt` for tests run with the function `single_test`\n- `TESTNAME-SUBTEST_NAME-stderr.txt` for subtests run with `do_test`\n\n## Customized test \nTo customize a test, a file named `$TESTNAME.sh` can be placed in the gpac/rules directory. It is called before the actual test is run and allows custom variables to be set or overriding a given variable in a batch of tests, while still using a single generic test function. Directories variables shall not be modified. Other variables of the test suite documented here are reset at the beginning of each test.  \n\n## Testing and caching results\nIn order to avoid running all tests again whenever one test fails, the GPAC testing environment caches previous test results. If an XML file corresponding to the test or subtest exists in the results folder, the test or subtest is not run. If you want to ignore previous results, use `make_tests.sh -clean` before running the test suite. If you only want to invalidate tests of a given script, use  `make_tests.sh -clean script.sh` \nIn case a test requires generating files before testing, it is recommended to check the variable `test_skip` to check if the test is being skipped because cached. \n\nA typical test with several subtests checking for cache looks like:\n```\ntest_begin TESTNAME\nif [ $test_skip = 1 ] ; then\n return\nfi\n \n#create some file or costly operation\nMP4Box -add 4GB.src -new TESTFILE\ndo_test CMD_LINE1_USING_TESTFILE \"Name1\"\ndo_test CMD_LINE2 \"Name2\"\n\ntest_end\n```\nIn this example, failure to checktest `$test_skip` will make the script import 4GB.src even though all `do_test` calls will be skipped.\n\n## Parallel tests with subscripts\n\nSubtests may be run in subscripts, for example:\n```\ntest_begin TESTNAME\n do_test CMD_LINE1 \"Name1\" \u0026\n do_test CMD_LINE1 \"Name2\" \u0026\ntest_end\n```\n\nTests may also be run in subscript, for example:\n```\nfunction my_test {\n  test_begin $2\n  do_test $1 $2\n  test_end\n}\n\nmy_test CMD_LINE1 \"Name1\" \u0026\nmy_test CMD_LINE2 \"Name2\" \u0026\n```\n\n\n## Testing regressions with hashes\n\nTests may perform all the required checks to detect a regression, such as checking generated file info, counting importing samples, etc ... \nTo simplify test writing, the GPAC testing environment provide simple hashing (SHA-1) functions that you can perform on generated files such as MP4 or TS files, logs, XML dumps, ... Using hashes avoid storing such files while allowing generic regression detection. Because of this functionality, the test suite has a specific mode for generating the hashes, `./make_tests -hash`. Hashes are typically generated only once, upon initial test creation.\nThe test suite script will mark a test as failed if it uses a missing reference hash.\n- To clean hashes for a single test, use `./make_tests -clean-hash mytest.sh`\n- To generate hashes for a single test, use `./make_tests -hash mytest.sh`\n\nThe following functions are available for hash testing.\n\n### do_hash_test \nTwo arguments\n- `FILE`\n- `HASHNAME`\nWhen the test suite is run in hash generation mode, this generates the reference `hash_refs/$TESTNAME-$HASHNAME.hash`. Otherwise, creates a SHA-1 of FILE and compares it with the reference `$TESTNAME-$HASHNAME.hash`. \nFunction does nothing when the test is skipped or a previous error occured in the parent test `TESTNAME`.\n\nWARNING: `HASHNAME` shall be unique for a given test, otherwise hashes will be overwritten.\n\nA subtest having its result reverted from fail to pass through a rule file will prevent the next hash test to be exectuted by setting the variable `skip_next_hash_test` to 1.\nYou can set this variable to 0 before calling  do_hash_test  if you need to force a hash test after a test failure.\n\n### do_hash_test_bin\nSame as `do_hash_test` except skips text mode probing of source file. Should only be used with non-text files.\n \n### do_compare_file_hashes \nTwo arguments\n- `FILE1`\n- `FILE2`\nCompares hashes of `FILE1` and `FILE2`. returns 0 if OK or error code otherwise.\nFunction does nothing when the test is skipped (see below) or a previous error occured in the parent test `TESTNAME`.\n\nA typical test with several subtests and hash testing looks like:\n```\ntest_begin TESTNAME\nif [ $test_skip = 1 ] ; then\n return\nfi\n \n#run some test creating a file FILE1\ndo_test CMD_LINE1_CREATING_FILE1 \"Name1\"\n#perform hash on FILE1 - the hash name may be the same as the subtest names\ndo_hash_test FILE1 \"Name1\"\n\n#run some test inspecting file FILE1, generating report in FILE2\ndo_test CMD_LINE1_INSPECTING_FILE1 \"Name2\"\ndo_hash_test FILE2 \"Name1-inspect\"\n\ntest_end\n```\n\n## Testing events and interactivity\n\nThe test suite has a UI trace generation mode used to record user events (mouse and keyboards) for SVG and BIFS interactivity testing for the media [compositor filter](https://wiki.gpac.io/Filters/compositor/).\n\nIn this modes, all subtests are skipped (i.e.  `$test_skip` is  1), and the variable `$test_ui` is set to 1. A test may use the `do_ui_test` function to perform UI recording. As of GPAC 0.9.0, trace files have to be manually loaded by the test using them, as shown in [BT tests](https://github.com/gpac/testsuite/blob/filters/scripts/bt.sh).\n\n### do_ui_test\nTwo arguments:\n- `FILE`: the command line to execute\n- `SUBTEST_NAME`: the subtest name as it appears in the logs and in the stats\n\nPerforms UI event trace generation on `$FILE` for the subtest using gpac, for a running duration of `$dump_dur` and an output size `$dump_size` (see `begin_test`). \nUI file is generated as `$RULES_DIR/${basename $FILE.*}-$SUBTEST-ui.xml`.\n\nA typical usage is:\n\n```\ntest_begin TESTNAME\ndo_ui_test $MEDIA_DIR/somefile \"play\"\n\nif [ $test_skip = 1 ] ; then\n return\nfi\n\n#create some file or costly operation\nuirec=$RULES_DIR/${basename $FILE.*}-$SUBTEST-ui.xml\ndo_test \"$GPAC -i $MEDIA_DIR/somefile --cfg=Validator:Mode=Play -cfg=Validator:Trace=$uirec -o dump.rgb\"\n\ntest_end\n```\n\nNote that the file has to be generated\n- before testing cache status since all tests are considered as cached in UI modes\n- manually, since subtests are deactivated when the test is cached\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgpac%2Ftestsuite","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgpac%2Ftestsuite","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgpac%2Ftestsuite/lists"}