{"id":28395194,"url":"https://github.com/databiosphere/wdl-conformance-tests","last_synced_at":"2025-06-27T01:31:34.247Z","repository":{"id":117653489,"uuid":"289108345","full_name":"DataBiosphere/wdl-conformance-tests","owner":"DataBiosphere","description":null,"archived":false,"fork":false,"pushed_at":"2024-10-11T14:39:07.000Z","size":1074,"stargazers_count":2,"open_issues_count":2,"forks_count":0,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-06-01T06:52:27.708Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DataBiosphere.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-08-20T20:52:31.000Z","updated_at":"2024-10-11T14:39:11.000Z","dependencies_parsed_at":"2023-10-03T01:38:11.696Z","dependency_job_id":"2a028bc2-a850-4a43-b578-88835f82f1a6","html_url":"https://github.com/DataBiosphere/wdl-conformance-tests","commit_stats":null,"previous_names":["databiosphere/wdl-conformance-tests"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/DataBiosphere/wdl-conformance-tests","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fwdl-conformance-tests","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fwdl-conformance-tests/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fwdl-conformance-tests/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fwdl-conformance-tests/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DataBiosphere","download_url":"https://codeload.github.com/DataBiosphere/wdl-conformance-tests/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fwdl-conformance-tests/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262172451,"owners_count":23270009,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-31T19:39:27.253Z","updated_at":"2025-06-27T01:31:34.237Z","avatar_url":"https://github.com/DataBiosphere.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# WDL (Workflow Description Language) Spec Conformance Tests\n\nNow run WDL Spec Conformance Tests from the comfort of your own environment!  With the\ncomfort of your own runner!\n\nDesigned to be run by any WDL-supporting runner, these tests aim to cover\nthe entire WDL spec eventually.  Running these tests should not only test a runner's\nlevel of compliance with any given version of the WDL spec, running them regularly \nshould also help to ensure that that level of compliance is maintained.\n\nSupported WDL spec versions:\n\n - [WDL Version \"draft-2\"](https://github.com/openwdl/wdl/tree/main/versions/draft-2)\n - [WDL Version 1.0](https://github.com/openwdl/wdl/tree/main/versions/1.0)\n - [WDL Version 1.1](https://github.com/openwdl/wdl/tree/main/versions/1.1)\n\nRunners this has been evaluated against:\n\n - [Toil](https://github.com/DataBiosphere/toil)\n - [miniwdl](https://github.com/chanzuckerberg/miniwdl)\n - [Cromwell](https://github.com/broadinstitute/cromwell)\n\n## WDL Specification Unit Tests\nFor information about running specification tests from the WDL spec, calling a separate script is required. [See `README_UNIT.md`](README_UNIT.md)\n\n## Note\n\nThe tests in this directory are incomplete and are in the process of being added to.  If you'd like \nto help make these tests more complete, we gladly welcome PRs!\n\n## Usage\n\n```\n(venv) quokka@qcore ~/$ python3 run.py --runner toil-wdl-runner --versions 1.1\nTesting runner toil-wdl-runner on WDL versions: 1.1\n\n0: SKIPPED: Standard Lib: Basic test for stdout()\n1: SKIPPED: Standard Lib: Basic test for stderr()\n2: SUCCEEDED: Standard Lib: Basic test for quote()\n3: SUCCEEDED: Standard Lib: Basic test for squote()\n4: SUCCEEDED: Standard Lib: Basic test for sep()\n5: SUCCEEDED: Standard Lib: Basic test for prefix()\n\n...\n\n...\n\n17 tests run, 15 succeeded, 2 failed, 8 skipped\n```\n\n\n## Options\n\nThe `run.py` script, and most other included scripts, support the followign options:\n```\n  -h, --help            show this help message and exit\n  --verbose             Print more information about a test\n  --versions {draft-2,1.0,1.1}, -v {draft-2,1.0,1.1}\n                        Select the WDL versions you wish to test against. Ex: -v=draft-2,1.0\n  --tags TAGS, -t TAGS  Select the tags to run specific tests\n  --numbers NUMBERS, -n NUMBERS\n                        Select the WDL test numbers you wish to run. Can be a comma separated list or hyphen separated inclusive ranges. Ex: -n=1-4,6,8-10\n  --runner {cromwell,toil-wdl-runner,miniwdl}, -r {cromwell,toil-wdl-runner,miniwdl}\n                        Select the WDL runner to use.\n  --threads THREADS     Number of tests to run in parallel. The maximum should be the number of CPU cores (not threads due to wall clock timing).\n  --time                Time the conformance test run.\n  --quiet\n  --exclude-numbers EXCLUDE_NUMBERS\n                        Exclude certain test numbers.\n  --conformance-file CONFORMANCE_FILE\n                       Run the given test suite. \n  --toil-args TOIL_ARGS\n                        Arguments to pass into toil-wdl-runner. Ex: --toil-args=\"caching=False\"\n  --miniwdl-args MINIWDL_ARGS\n                        Arguments to pass into miniwdl. Ex: --miniwdl-args=\"--no-outside-imports\"\n  --cromwell-args CROMWELL_ARGS\n                        Arguments to pass into cromwell. Ex: --cromwell-args=\"--options=[OPTIONS]\"\n  --cromwell-pre-args CROMWELL_PRE_ARGS\n                        Arguments to set java system properties before calling cromwell. This allows things such as setting cromwell config files with --cromwell-pre-\n                        args=\"-Dconfig.file=build/overrides.conf\".\n  --id ID               Specify WDL tests by ID.\n  --repeat REPEAT       Specify how many times to run each test.\n  --jobstore-path JOBSTORE_PATH, -j JOBSTORE_PATH\n                        Specify the PARENT directory for the jobstores to be created in.\n  --progress            Print the progress of the test suite as it runs.\n```\nInvoking `run.py` with no options will result in the entire `conformance.yaml` test suite being run, which can take a long time.\nIncluding `--progress` is recommended when running over a long period of time.\n\nThe tests to run can be specified with `--id`, `--tags`, and `--numbers`. Tests matching any of the selectors will be run:\n```commandline\npython run.py --number 1 --id md5 --tags stderr --version 1.0 --runner toil-wdl-runner\nTesting runner toil-wdl-runner on WDL versions: 1.0\n\n\n=== REPORT ===\n\n1: SUCCEEDED: Standard Lib: Basic test for stderr()\nIteration: 1\n40: SUCCEEDED: Legacy test for stderr_as_output\nIteration: 1\n68: SUCCEEDED: MD5 test\nIteration: 1\n3 tests run, 3 succeeded, 0 failed, 0 skipped\n```\n`--repeat` specifies how many times to run each test. `--threads` allows multiple tests to run simultaneously;\nThis should be set to no more than the number of physical, highest-performance cores in the system (not counting any efficiency cores or the two logical cores per physical core provided by hyperthreading), in order to ensure consistent timings, if running with `--time` or with the performance testing script.\nHowever, there is no thread reservation system; if a test is not guaranteed to run singlethreaded and all threads are in use, the timings may be influenced.\n\n\nBy default, runner logs are only printed for failed tests. `--verbose` forces logs to always print and `--quiet` forces logs to never print.\n\nIf running tests on a cluster, [extra arguments may be necessary](SLURM_README.md).\n## Adding Tests\nTests can be added by editing the test suite file: `conformance.yaml` for conformance tests, or `integration.yaml` for longer integration tests.\n\nFor example, a new test can be added as follows:\n```yaml\n- description: |\n    Example new test description\n  versions: [\"draft-2\", \"1.0\", \"1.1\"] # specify versions to run with\n  id: example_test_id # unique ID of test, no 2 tests should have the same id\n  tags: [\"examples\", \"behavior\"] # specify tags, these don't need to be unique\n  inputs:\n    dir: tests/example_files # path to directory where test is, can be an absolute or relative path (from the test suite file)\n    wdl: example.wdl # wdl file name\n    json: example.json # json file describing test workflow inputs\n  outputs:\n    exampleWf.outputVar: # output name, should be workflowName.outputVariable\n      type: Boolean # expected output type\n      value: True # expected output value\n```\nThe test files then should be under the `tests/example_files` directory and be named `example.wdl` and `example.json`.\nEach expected output should be specified as `wfName.varName`. For example, `exampleWf.outputVar` is the specifier if the WDL is this:\n```wdl\nworkflow exampleWf {\n    output {\n        Boolean outputVar = true\n    }\n}\n```\nOnly one WDL file is used for all WDL versions that a test runs on; the file [will be rewritten and must obey certain formatting conventions](#test-formatting).\n## Test formatting\nThe test runner uses a single WDL file for each test, written for a single WDL version, to run the test across all applicable WDL versions. This is accomplished by rewriting that one WDL file at runtime to produce generated WDL files targeting the other WDL versions.\n\nTest can use either [automatic version conversion](#automatic-version-conversion) or [manual version conversion](#manual-version-conversion) to be runnable across multiple WDL versions from a single file. If a test uses neither, it must be restricted to a single WDL version.\n### Automatic version conversion\nDifferent WDL versions (`draft-2`, `1.0`, `1.1`) all have slightly different syntax. To deal with this, the runner will \nautomatically convert from one version to another before running a WDL test. As long as the format of a WDL file \nfollows these format conventions, the conversion will work properly:\n\nAny input section in WDL code must follow the format:\n```wdl\ninput {\n    ...\n}\n```\nAny command section must follow the format:\n```wdl\ncommand {\n    ...\n} # line with isolated closing brace indicates end of command section\n```\nAny command section can also use triple angle braces:\n```wdl\ncommand \u003c\u003c\u003c\n    ...\n\u003e\u003e\u003e\n```\n\nThese sections should not have other WDL code on the same line, or have unnecessary newlines in between the section declaration and its block.\nFor example, these will not work:\n```wdl\ninput {\n    ...\n} var = 1 # WDL code in input declaration\n```\n```wdl\ninput\n{ # newline between input keyword and open brace\n    ...\n}\n```\n### Manual version conversion\nWhile the format for automatic version conversion should be applicable for most WDL code, there may be WDL syntax that is incompatible \nbetween WDL versions. If a test needs specific syntax differences between versions (beyond changes to section declarations), patch files can be used instead to describe the differences from a base file.\n\nAs long as a test directory has patch files with the name `version_[wdl_version].patch`, the runner will apply\nthe patch files automatically before each test.\n\nFor example, to convert a test file from WDL version `1.1` to `1.0` and `draft-2`, the file structure will look like this:\n```commandline\ntests/basic_stdout\n├── ...\n├── basic_stdout.wdl\n├── version_1.0.patch\n└── version_draft-2.patch\n```\n\nTo help create these patch files, the `patch.py` program is given:\n\n```commandline\nusage: patch.py [-h] --version {1.0,1.1,draft-2} --directory DIRECTORY [--remove REMOVE] [--rename RENAME]\n\nCreate patch files in the right format\n\noptions:\n  -h, --help            show this help message and exit\n  --version {1.0,1.1,draft-2}, -v {1.0,1.1,draft-2}\n                        The base WDL file's version\n  --directory DIRECTORY, -d DIRECTORY\n                        Directory where all the WDL files are\n  --remove REMOVE       Remove WDL files that are not the base\n  --rename RENAME       Rename the base WDL file to the directory (--remove must be set to True too)\n```\n\nTo use this script, first, create the 2-3 different versioned WDL files in a directory. \n\nWhen calling `patch.py`, the base WDL version should be passed into `--version`.\nPatch files will be created converting from that base WDL version.\n\nThe directory containing all these tests should be passed into `--directory`. All the WDL files must follow the naming convention `[testname]_version_[wdl_version]`. For example, `example_version_1.0`, `example_version_draft-2`.\n\n`--remove` can be provided to delete all other WDL files that aren't the base WDL file, and `--rename` can be used to rename the base WDL file to the same name as the directory. (These options are not necessary.)\n\nFor example, if making a new test called `basic_stdout` for all 3 versions:\nFirst create the directory `tests/basic_stdout`. Create the 3 different WDL files, one for each version, and name them appropriately\n\n```commandline\ntests/basic_stdout\n├── basic_stdout.json\n├── basic_stdout_version_1.0.wdl\n├── basic_stdout_version_1.1.wdl\n└── basic_stdout_version_draft2.wdl\n```\n\nThen run the program:\n```commandline\npython patch.py --version 1.1 --directory tests/basic_stdout\n```\n\n`patch.py` will use `basic_stdout_version_1.1.wdl` as the base file for patches, and generate 2 patch files to convert from WDL 1.1 to 1.0 and draft-2.\n```commandline\ntests/basic_stdout\n├── ...\n├── version_1.0.patch # new file\n└── version_draft-2.patch # new file\n```\n\nThen add the tests to the test suite file.\n```yaml\n- description: |\n    Example stdout test\n  versions: [\"draft-2\", \"1.0\", \"1.1\"]\n  id: ...\n  tags: ...\n  inputs:\n    dir: tests/basic_stdout\n    wdl: basic_stdout_version_1.1.wdl\n    json: basic_stdout.json\n  outputs:\n    ...\n```\nThe runner will find and apply the patch files at runtime. (This will take priority over the automatic WDL version conversion.)\n\nTo reverse this process, another program `unpatch.py` will do the opposite of `patch.py`. This is useful if new patch files need to be created, but the original WDL files were deleted:\n```commandline\nusage: unpatch.py [-h] --directory DIRECTORY [--version {1.0,1.1,draft-2}] [--file FILE]\n\nReverse patch.py to get back the original WDL files. This can be useful if the original WDL files no longer exist and a modification that invalidates the existing patches is necessary.\n\noptions:\n  -h, --help            show this help message and exit\n  --directory DIRECTORY, -d DIRECTORY\n                        Directory where all at least the base WDL file and patch files are\n  --version {1.0,1.1,draft-2}, -v {1.0,1.1,draft-2}\n                        The base WDL file's version. If specified, it will find the first WDL file with this version to create patched files from.\n  --file FILE, -f FILE  File name of base WDL file. If specified, it will use it to create patched files from.Takes priority over --version\n```\nGiven a directory `--directory` and the base WDL file version `--version` (or file path `--file`),\nthe script will get back the original WDL files that were used for `patch.py`.\n\nIf the original files used for `patch.py` already exist, then `unpatch.py` will not overwrite them.\n## Running Performance Tests\nThe default runner `run.py` only runs tests and reports on their status in the terminal.\nTo store the runtime of tests (and eventually graph them), the script `run_performance.py` should be used:\n```commandline\npython run_performance.py --runners toil-wdl-runner,miniwdl --output performance_output.csv --progress\n```\nThe output will always be in CSV format. `--runners` can specify a comma separated list of runners to test, and `--all-runners` is a shortcut to specify all of them.\n\n### Performance Testing Options\n[All options](#options) from the normal `run.py` script are also available in `run_performance.py`. For example, if `--id stdout` is provided, then only the `stdout` test will be measured.\n\nThe performance test specific options are:\n```commandline\nArguments for running WDL performance tests:\n  --output OUTPUT, -o OUTPUT\n                        Specify the output CSV file.\n  --all-runners, -a     Specify whether to run with all runners. This will override the --runners argument.\n  --runners RUNNERS     Specify multiple runners in a comma separated list.\n```\n\n## Graphing Performance Tests\nThe script `create_graph.py` can be used to graph the output of the performance tests:\n```commandline\npython create_graph.py -f performance_output.csv\n```\nIf no input CSV file is given, then the program will run the entire performance test suite first.\n### Graph Options\nSimilar to the performance testing script, [all options](#options) from the base runner are carried through. However, there are graphing specific options:\n```commandline\nArguments for graphing runtimes of WDL tests:\n  --from-file FILE, -f FILE\n                        Specify a csv file to read from.\n  --display-num DISPLAY_NUM, -d DISPLAY_NUM\n                        Specify the number of tests to display per graph.\n  --display-all, -a     Display all tests on a single graph. Overrides --display-num.\n  --ignore-runner IGNORE_RUNNER\n                        Specify a runner(s) to ignore in the graph output.\n  --precision PRECISION\n                        Specify the precision when outputting float values. Ex: Default=0 will result in 1 for float value 1.4...\n  --no-labels           Specify to not display extra labels on the graph.\n  --graph-type GRAPH_TYPE\n```\nBy default, separate graphs will be created for every 30 tests.\n\nThis can be controlled with `--display-num`, which overrides the number of tests to display per graph. `--display-all` forces all tests onto a single graph.\n\n`--graph-type` specifies the type of graph to create. The supported options are `box` and `bar`.\n`box` will create a traditional box plot and `bar` will create a bar graph with error bars representing standard deviation. \n### Writing Graphs to a File\nThe graphs can be written to disk by specifying a (base) filename.\n\nFor example, to output a graph with all tests in a PNG with a size of 40 inches by 10 inches (width,height):\n```commandline\npython create_graph.py -f performance_output.csv --output performance_graph.png --dimensions 40,10 -a\n```\nIf `--display-all` is not specified and there are more tests to graph than `--display-num`, then multiple graph files will be created with the format `[output_file][iteration].[fileext]`.\n\nFor example, calling `python create_graph.py -f performance_output.csv --output performance_graph.png` with 90 total tests to graph will result in:\n```commandline\n.\n├── performance_graph1.png\n├── performance_graph2.png\n└── performance_graph3.png\n```\nIf trying to put all the tests into a single graph, it is likely that dimensions should be provided to make all tests fit.\n\nAll output formats implemented by Matplotlib (such as `pdf` and `svg`) are supported.\n### Graph Output Options\nOther than the previous [general](#options) and [graph specific](#graph-options) options, the options for outputting a graph are:\n```commandline\nArguments for specifying how to write the graph to a file.:\n  --output OUTPUT, -o OUTPUT\n                        Instead of displaying the graphs, output it into an image. If --display-num is set to a number less than the total amount of tests in the CSV file, then multiple images will be created with the naming\n                        scheme: [filename][iteration].[fileextension]. For example, if wdl_graph.png is passed, then the first file created will be wdl_graph1.png.\n  --dimensions [DIMENSIONS]\n                        If custom dimensions are needed, this can be called with input format x_size,y_size in inches. Calling this with no value will size the graph accordingly.\n```\n### Example Graphs\nThese are 2 graphs generated from running performance testing on a Slurm cluster.\n![Example box graph](examples/example_box_graph.png)\n![Example bar graph](examples/example_bar_graph.png)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabiosphere%2Fwdl-conformance-tests","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatabiosphere%2Fwdl-conformance-tests","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabiosphere%2Fwdl-conformance-tests/lists"}