{"id":32495324,"url":"https://github.com/tlaplus/validationtestsuite","last_synced_at":"2026-02-24T05:53:45.445Z","repository":{"id":316103336,"uuid":"1061866724","full_name":"tlaplus/ValidationTestSuite","owner":"tlaplus","description":"Tool qualification tests and reports for the TLA+ model checker","archived":false,"fork":false,"pushed_at":"2025-10-02T16:10:42.000Z","size":2321,"stargazers_count":4,"open_issues_count":4,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-10-02T18:09:46.417Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://dl.tlapl.us/validation-test-suite/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tlaplus.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-22T13:45:29.000Z","updated_at":"2025-10-02T16:04:21.000Z","dependencies_parsed_at":"2025-09-23T20:23:55.379Z","dependency_job_id":null,"html_url":"https://github.com/tlaplus/ValidationTestSuite","commit_stats":null,"previous_names":["tlaplus/tool-qualification-suite","tlaplus/validationtestsuite"],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/tlaplus/ValidationTestSuite","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tlaplus%2FValidationTestSuite","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tlaplus%2FValidationTestSuite/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tlaplus%2FValidationTestSuite/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tlaplus%2FValidationTestSuite/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tlaplus","download_url":"https://codeload.github.com/tlaplus/ValidationTestSuite/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tlaplus%2FValidationTestSuite/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":281279073,"owners_count":26473858,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-27T02:00:05.855Z","response_time":61,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-10-27T13:51:35.578Z","updated_at":"2025-10-27T13:51:37.204Z","avatar_url":"https://github.com/tlaplus.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003c!--\nSPDX-FileCopyrightText: Copyright (c) 2022 NVIDIA CORPORATION \u0026 AFFILIATES. All rights reserved.\nSPDX-License-Identifier: Apache-2.0\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\nhttp://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n--\u003e\n\n# Validation Test Suite for TLA+\n\nThe **Validation Test Suite for TLA+** is a comprehensive testing framework that enables the use of TLC in safety-critical software development contexts, including ISO 26262 certification. This Python-based tool automatically generates and executes extensive test cases for [TLC](https://github.com/tlaplus/tlaplus), the TLA+ model checker, to help verify that it satisfies the requirements defined by Automotive Functional Safety Standard ISO 26262 for safety-critical software development.\n\nThe suite runs both TLC and [Apalache](https://github.com/apalache-mc/apalache) model checkers, comparing results to ensure consistency and validates results through cross-checking. It produces test suites that support qualifying TLC for use in safety-critical applications including automotive systems, aerospace applications, medical devices, and industrial control systems.\n\nThis suite was created by Dmitry Kulagin (GitHub: craft095) at NVIDIA and initially contributed by NVIDIA, which successfully used this generator to qualify TLC for their safety-critical applications. Contributions are welcome! Please check the [issues](https://github.com/tlaplus/ValidationTestSuite/issues) for ways to contribute.\n\n## Key Features and Technical Approach\n\nThe suite employs a systematic strategy with the following key capabilities:\n\n- **Systematic Testing**: Defines a constrained TLA+ subset suitable for safety-critical use cases, then automatically creates test suites using pairwise feature combinations of TLA+ language constructs\n- **Cross-Validation**: Runs both TLC and Apalache model checkers, comparing results to ensure consistency and validates results through cross-checking\n- **Comprehensive Coverage**: Tests over 100 TLA+ language features including operators, constants, variables, and other constructs, categorized by type and kind\n- **Multi-Core Support**: Tests TLC with different worker configurations (-workers 1, 2, auto)\n- **Detailed Reporting**: Generates HTML reports with test specifications, execution results, and coverage information\n- **Anomalous Condition Testing**: Includes testing under resource constraints and edge cases\n- **Deviation Documentation**: Documents and explains all divergences between tools to provide complete transparency\n\n## Why This Matters for Safety-Critical Applications\n\nThe qualification suite addresses a critical need in the TLA+ ecosystem: providing the rigorous testing and documentation required to use TLC in safety-critical contexts such as:\n- Automotive systems (ISO 26262)\n- Aerospace applications\n- Medical devices\n- Industrial control systems\n\nBy systematically exercising TLC's behavior across thousands of test cases and documenting known deviations, this suite provides the evidence base needed for tool qualification in regulated industries.\n\n## Prerequisites\n\n* Before cloning the repository ensure that `git-lfs` is installed\n* Python 3.9+; it doesn't work with earlier or newer versions\n* Python libraries:\n  - PyYAML\n  - jinja2\n* OpenJDK Runtime Environment 17\n* bubblewrap 0.4.0 (Linux only)\n* unzip\n\n```bash\nsudo apt install git-lfs\ngit clone https://github.com/tlaplus/ValidationTestSuite.git \u0026\u0026 (cd ValidationTestSuite \u0026\u0026 git lfs install \u0026\u0026 git lfs fetch --all \u0026\u0026 git lfs checkout)\n```\n\n## Usage\n\n### Basic Usage\n\n```bash\npython -m testgen -o \u003cPATH_TO_REPORTS\u003e -s -e --html -x explanation.yaml -w \u003cNUM_OF_WORKERS\u003e $*\n```\n\n* Always run the generator from its top level source directory\n* `\u003cPATH_TO_REPORTS\u003e` is where all the artifacts are put; if `\u003cPATH_TO_REPORTS\u003e` exists, ensure it is empty\n* For better performance, `\u003cNUM_OF_WORKERS\u003e` should correspond to number of physical cores\n\nFor a complete and up-to-date list of all available command-line options, run:\n\n```bash\npython -m testgen --help\n```\n\n### Advanced Usage\n\nTest generator has four independent steps:\n- `--specification` - generate test cases and test case specification report\n- `--execute` - execute tests and create execution report\n- `--coverage` - collect coverage information with JaCoCo\n- `--html` - create HTML report\n\nThese steps can be used together (as for basic usage) or separately.\n\nIt is often convenient to limit scope of test suite. To do so specification step can be supplied with following flags:\n- `--no-symmetry` - do not generate SYMMETRY cases\n- `--filter-any-side VAL0 [VAL1 ...]` - only include test cases with features, whose name include any of `VALn`\n- `--filter-any-side-exact VAL0 [VAL1 ...]` - only include test cases with features, whose name is exactly any of `VALn`\n- `--filter-double-side VAL0 VAL1` - only include test cases, where case feature name include `VAL0` and plug feature name include `VAL1`\n\nThe longest step is test execution. By default, a model is only checked if:\n- There is no previous run results\n- It has changed since previous run\n- Its latest checking result is \"crash\"\n\nSuch behavior saves a lot of time. If it is not desireable, there is an option `--force` to check the model unconditionally.\n\nFinally, there is `--debug` option to enable verbose mode.\n\n## Validation Reports and Releases\n\nThe most recent validation report is available at https://dl.tlapl.us/validation-test-suite. Both recent and older reports can be found on the [GitHub releases page](https://github.com/tlaplus/ValidationTestSuite/releases). Each release's SHA identifies the Git commit from which it was created. The TLC tool is stored in the Git repository using Git LFS.\n\n## Further Reading\n\n- [Test Strategy](test-strategy.md) - Comprehensive overview of the systematic testing approach, including feature identification, pairwise combinations, and test case generation methodology\n- [Symmetry Testing](symmetry.md) - Specialized testing strategy for TLC's `SYMMETRY` optimization setting and its interactions with other configuration options\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftlaplus%2Fvalidationtestsuite","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftlaplus%2Fvalidationtestsuite","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftlaplus%2Fvalidationtestsuite/lists"}