{"id":16836481,"url":"https://github.com/jcfr/cmake-add_test-pass_regular_expression","last_synced_at":"2026-01-04T13:04:37.564Z","repository":{"id":153687585,"uuid":"630307607","full_name":"jcfr/cmake-add_test-PASS_REGULAR_EXPRESSION","owner":"jcfr","description":"Describe behavior of CTest associated with the PASS_REGULAR_EXPRESSION test property","archived":false,"fork":false,"pushed_at":"2023-04-20T06:37:04.000Z","size":9,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-01-24T10:24:45.212Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"CMake","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jcfr.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-04-20T05:28:37.000Z","updated_at":"2023-04-20T05:54:17.000Z","dependencies_parsed_at":null,"dependency_job_id":"b466f4c9-1568-4c83-a01c-72294fcae21c","html_url":"https://github.com/jcfr/cmake-add_test-PASS_REGULAR_EXPRESSION","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jcfr%2Fcmake-add_test-PASS_REGULAR_EXPRESSION","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jcfr%2Fcmake-add_test-PASS_REGULAR_EXPRESSION/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jcfr%2Fcmake-add_test-PASS_REGULAR_EXPRESSION/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jcfr%2Fcmake-add_test-PASS_REGULAR_EXPRESSION/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jcfr","download_url":"https://codeload.github.com/jcfr/cmake-add_test-PASS_REGULAR_EXPRESSION/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244147343,"owners_count":20405942,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-13T12:13:36.210Z","updated_at":"2026-01-04T13:04:37.497Z","avatar_url":"https://github.com/jcfr.png","language":"CMake","readme":"# CTest: PASS_REGULAR_EXPRESSION test property\n\nThis project aims to provide a basis for discussing the behavior of CTest associated with the `PASS_REGULAR_EXPRESSION` test property.\n\n## Configuring the test case\n\nDownload, configure and run tests:\n\n\n```\ngit clone https://github.com/jcfr/cmake-add_test-PASS_REGULAR_EXPRESSION\n\ncmake -S cmake-add_test-PASS_REGULAR_EXPRESSION -B cmake-add_test-PASS_REGULAR_EXPRESSION-build\n\ncmake --build cmake-add_test-PASS_REGULAR_EXPRESSION-build\n\nctest --test-dir cmake-add_test-PASS_REGULAR_EXPRESSION-build\n```\n\nOutput:\n\n```\nctest --test-dir cmake-add_test-PASS_REGULAR_EXPRESSION-build/\nInternal ctest changing into directory: /tmp/cmake-add_test-PASS_REGULAR_EXPRESSION-build\nTest project /tmp/cmake-add_test-PASS_REGULAR_EXPRESSION-build\n    Start 1: ExpectedPass\n1/4 Test #1: ExpectedPass .....................   Passed    0.02 sec\n    Start 2: ExpectedFail\n2/4 Test #2: ExpectedFail .....................***Failed    0.01 sec\n    Start 3: RegexCheckExpectedPass\n3/4 Test #3: RegexCheckExpectedPass ...........   Passed    0.01 sec\n    Start 4: RegexCheckExpectedFail\n4/4 Test #4: RegexCheckExpectedFail ...........   Passed    0.01 sec\n\n75% tests passed, 1 tests failed out of 4\n\nTotal Test time (real) =   0.06 sec\n\nThe following tests FAILED:\n    2 - ExpectedFail (Failed)\nErrors while running CTest\nOutput from these tests are in: /tmp/cmake-add_test-PASS_REGULAR_EXPRESSION-build/Testing/Temporary/LastTest.log\nUse \"--rerun-failed --output-on-failure\" to re-run the failed cases verbosely.\n```\n\n## Expected behavior\n\nTest `RegexCheckExpectedFail` should be failing since a exception is raised and the exit code is different from zero.\n\n## Observations\n\nWhile writing this test case and reviewing the CMake documentation, the following differences in the description associated with the `PASS_REGULAR_EXPRESSION` property were observed:\n\n| CMake version | `PASS_REGULAR_EXPRESSION` description |\n|--|--|\n| \u003c= 3.20.6 | The output must match this regular expression for the test to pass. |\n| \u003e= 3.21 | The output must match this regular expression for the test to pass. The process exit code is ignored. |\n\nThe documentation updates was introduced in [cmake@91b8676f8](https://github.com/Kitware/CMake/commit/91b8676f8):\n\n```\nHelp: Clarify {PASS,FAIL}_REGULAR_EXPRESSION semantics w.r.t. exit code\n\nAlso cross-reference them with each other and `SKIP_REGULAR_EXPRESSION`.\n```\n\n## Attempted workaround\n\n### Set FAIL_REGULAR_EXPRESSION in addition of PASS_REGULAR_EXPRESSION\n\n:heavy_check_mark: Also specifying `FAIL_REGULAR_EXPRESSION` to check for error message representative of Python eception allows to ensure the test fail independently of th return code.\n\n:warning: This is not a satisfactory solution as any non python error would not be detected.\n\nRe-configure and run tests:\n\n```\ncmake -S cmake-add_test-PASS_REGULAR_EXPRESSION -B cmake-add_test-PASS_REGULAR_EXPRESSION-build \\\n  -DWITH_FAIL_REGULAR_EXPRESSION_WORKAROUND:BOOL=1\n\ncmake --build cmake-add_test-PASS_REGULAR_EXPRESSION-build\n\nctest --test-dir cmake-add_test-PASS_REGULAR_EXPRESSION-build\n```\n\nOutput:\n\n```\nInternal ctest changing into directory: /tmp/cmake-add_test-PASS_REGULAR_EXPRESSION-build\nTest project /tmp/cmake-add_test-PASS_REGULAR_EXPRESSION-build\n    Start 1: ExpectedPass\n1/7 Test #1: ExpectedPass ....................................   Passed    0.04 sec\n    Start 2: ExpectedFail\n2/7 Test #2: ExpectedFail ....................................***Failed    0.01 sec\n    Start 3: RegexCheckExpectedPass\n3/7 Test #3: RegexCheckExpectedPass ..........................   Passed    0.01 sec\n    Start 4: RegexCheckExpectedFail\n4/7 Test #4: RegexCheckExpectedFail ..........................   Passed    0.01 sec\n    Start 5: WorkaroundPassAndFail_RegexCheckExpectedPass\n5/7 Test #5: WorkaroundPassAndFail_RegexCheckExpectedPass ....   Passed    0.01 sec\n    Start 6: WorkaroundPassAndFail_RegexCheckExpectedFail0\n6/7 Test #6: WorkaroundPassAndFail_RegexCheckExpectedFail0 ...***Failed  Required regular expression not found. Regex=[hello\n]  0.01 sec\n    Start 7: WorkaroundPassAndFail_RegexCheckExpectedFail1\n7/7 Test #7: WorkaroundPassAndFail_RegexCheckExpectedFail1 ...***Failed  Error regular expression found in output. Regex=[Traceback]  0.01 sec\n\n57% tests passed, 3 tests failed out of 7\n\nTotal Test time (real) =   0.12 sec\n\nThe following tests FAILED:\n\t  2 - ExpectedFail (Failed)\n\t  6 - WorkaroundPassAndFail_RegexCheckExpectedFail0 (Failed)\n\t  7 - WorkaroundPassAndFail_RegexCheckExpectedFail1 (Failed)\nErrors while running CTest\nOutput from these tests are in: /tmp/cmake-add_test-PASS_REGULAR_EXPRESSION-build/Testing/Temporary/LastTest.log\nUse \"--rerun-failed --output-on-failure\" to re-run the failed cases verbosely.\n\n```\n\n## Proposed solutions\n\n* Introduce an opt-in policy to ensure test fail if return code is non-zero\n\nor\n\n* Introduce new test properties like `PASS_RETURN_CODE \u003cvalue\u003e` and `FAIL_RETURN_CODE \u003cvalue\u003e`\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjcfr%2Fcmake-add_test-pass_regular_expression","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjcfr%2Fcmake-add_test-pass_regular_expression","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjcfr%2Fcmake-add_test-pass_regular_expression/lists"}