{"id":49627452,"url":"https://github.com/specmatic/labs-tests","last_synced_at":"2026-05-05T08:43:54.005Z","repository":{"id":349998513,"uuid":"1204519434","full_name":"specmatic/labs-tests","owner":"specmatic","description":null,"archived":false,"fork":false,"pushed_at":"2026-04-28T07:24:47.000Z","size":456,"stargazers_count":0,"open_issues_count":2,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-28T08:13:36.547Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/specmatic.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-08T04:46:46.000Z","updated_at":"2026-04-23T05:50:44.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/specmatic/labs-tests","commit_stats":null,"previous_names":["specmatic/labs-tests"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/specmatic/labs-tests","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/specmatic%2Flabs-tests","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/specmatic%2Flabs-tests/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/specmatic%2Flabs-tests/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/specmatic%2Flabs-tests/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/specmatic","download_url":"https://codeload.github.com/specmatic/labs-tests/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/specmatic%2Flabs-tests/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32642293,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-04T10:08:07.713Z","status":"online","status_checked_at":"2026-05-05T02:00:06.033Z","response_time":54,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-05-05T08:43:50.107Z","updated_at":"2026-05-05T08:43:53.998Z","avatar_url":"https://github.com/specmatic.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# labs-tests\n\nAutomation harnesses for Specmatic labs live in lab-named folders in this repo.\n\nCurrent automation scope validates CLI/runtime behavior and generated artifacts.\nIt does not automate Specmatic Studio flows, but the comparison report can indicate whether a lab README documents a Studio component.\n\nPrerequisites:\n\n- Python `3.14.x`\n- Docker with the daemon running\n- sibling upstream checkout at `../labs`\n\nCurrent labs:\n\n- [`api-coverage`](api-coverage/)\n  README: [`api-coverage/README.md`](api-coverage/README.md)\n- [`api-resiliency-testing`](api-resiliency-testing/)\n  README: [`api-resiliency-testing/README.md`](api-resiliency-testing/README.md)\n- [`api-security-schemes`](api-security-schemes/)\n  README: [`api-security-schemes/README.md`](api-security-schemes/README.md)\n- [`async-event-flow`](async-event-flow/)\n  README: [`async-event-flow/README.md`](async-event-flow/README.md)\n- [`backward-compatibility-testing`](backward-compatibility-testing/)\n  README: [`backward-compatibility-testing/README.md`](backward-compatibility-testing/README.md)\n- [`continuous-integration`](continuous-integration/)\n  README: [`continuous-integration/README.md`](continuous-integration/README.md)\n- [`data-adapters`](data-adapters/)\n  README: [`data-adapters/README.md`](data-adapters/README.md)\n- [`dictionary`](dictionary/)\n  README: [`dictionary/README.md`](dictionary/README.md)\n- [`external-examples`](external-examples/)\n  README: [`external-examples/README.md`](external-examples/README.md)\n- [`filters`](filters/)\n  README: [`filters/README.md`](filters/README.md)\n- [`kafka-avro`](kafka-avro/)\n  README: [`kafka-avro/README.md`](kafka-avro/README.md)\n- [`kafka-sqs-retry-dlq`](kafka-sqs-retry-dlq/)\n  README: [`kafka-sqs-retry-dlq/README.md`](kafka-sqs-retry-dlq/README.md)\n- [`mcp-auto-test`](mcp-auto-test/)\n  README: [`mcp-auto-test/README.md`](mcp-auto-test/README.md)\n- [`overlays`](overlays/)\n  README: [`overlays/README.md`](overlays/README.md)\n- [`partial-examples`](partial-examples/)\n  README: [`partial-examples/README.md`](partial-examples/README.md)\n- [`workflow-in-same-spec`](workflow-in-same-spec/)\n  README: [`workflow-in-same-spec/README.md`](workflow-in-same-spec/README.md)\n- [`quick-start-api-testing`](quick-start-api-testing/)\n  README: [`quick-start-api-testing/README.md`](quick-start-api-testing/README.md)\n- [`quick-start-async-contract-testing`](quick-start-async-contract-testing/)\n  README: [`quick-start-async-contract-testing/README.md`](quick-start-async-contract-testing/README.md)\n- [`quick-start-contract-testing`](quick-start-contract-testing/)\n  README: [`quick-start-contract-testing/README.md`](quick-start-contract-testing/README.md)\n- [`quick-start-mock`](quick-start-mock/)\n  README: [`quick-start-mock/README.md`](quick-start-mock/README.md)\n- [`schema-design`](schema-design/)\n  README: [`schema-design/README.md`](schema-design/README.md)\n- [`schema-resiliency-testing`](schema-resiliency-testing/)\n  README: [`schema-resiliency-testing/README.md`](schema-resiliency-testing/README.md)\n- [`response-templating`](response-templating/)\n  README: [`response-templating/README.md`](response-templating/README.md)\n\nSetup the sibling upstream labs checkout and Docker images from the repo root with:\n\n```bash\npython3 setup.py\n```\n\nTo force `../labs` back to the latest `main` before refreshing Docker images:\n\n```bash\npython3 setup.py --refresh-labs --force\n```\n\nRun every available lab harness from the repo root and build the consolidated and comparison reports with:\n\n```bash\npython3 run_all.py\n```\n\nRebuild the consolidated and comparison reports from the existing lab snapshots without rerunning labs:\n\n```bash\npython3 rebuild_reports.py\n```\n\nRefresh an individual lab report from previously captured artifacts without rerunning the lab:\n\n```bash\npython3 api-coverage/run.py --refresh-report\n```\n\n```bash\npython3 api-resiliency-testing/run.py --refresh-report\n```\n\n```bash\npython3 api-security-schemes/run.py --refresh-report\n```\n\n```bash\npython3 async-event-flow/run.py --refresh-report\n```\n\n```bash\npython3 backward-compatibility-testing/run.py --refresh-report\n```\n\n```bash\npython3 continuous-integration/run.py --refresh-report\n```\n\n```bash\npython3 data-adapters/run.py --refresh-report\n```\n\n```bash\npython3 dictionary/run.py --refresh-report\n```\n\n```bash\npython3 external-examples/run.py --refresh-report\n```\n\n```bash\npython3 filters/run.py --refresh-report\n```\n\n```bash\npython3 kafka-avro/run.py --refresh-report\n```\n\n```bash\npython3 kafka-sqs-retry-dlq/run.py --refresh-report\n```\n\n```bash\npython3 mcp-auto-test/run.py --refresh-report\n```\n\n```bash\npython3 overlays/run.py --refresh-report\n```\n\n```bash\npython3 workflow-in-same-spec/run.py --refresh-report\n```\n\n```bash\npython3 partial-examples/run.py --refresh-report\n```\n\n```bash\npython3 quick-start-api-testing/run.py --refresh-report\n```\n\n```bash\npython3 quick-start-async-contract-testing/run.py --refresh-report\n```\n\n```bash\npython3 quick-start-contract-testing/run.py --refresh-report\n```\n\n```bash\npython3 quick-start-mock/run.py --refresh-report\n```\n\n```bash\npython3 schema-resiliency-testing/run.py --refresh-report\n```\n\n```bash\npython3 schema-design/run.py --refresh-report\n```\n\n```bash\npython3 response-templating/run.py --refresh-report\n```\n\nOutputs are written to:\n\n- `output/consolidated-report/consolidated-report.json`\n- `output/consolidated-report/consolidated-report.html`\n- `output/consolidated-report/labs-comparison.json`\n- `output/consolidated-report/labs-comparison.html`\n- `output/consolidated-report/setup-output.json`\n- `output/labs/\u003clab-name\u003e-output/` for each lab run\n\nEach lab’s `output/` directory is copied into `output/labs/\u003clab-name\u003e-output/` after the run completes. The consolidated report uses those copied folders so the links remain stable even after the live lab output is cleaned up or refreshed.\n\n`run_all.py` starts by clearing the generated `output/labs/` and `output/consolidated-report/` folders before regenerating reports, so stale files from earlier runs do not leak into a new report set. `rebuild_reports.py` does not clean the output tree; it only refreshes the consolidated and comparison reports from the existing lab snapshots.\n\nEach individual lab run also clears its own `\u003clab\u003e/output/` directory before a normal run starts. Refresh-only runs skip that cleanup so they can rebuild from the saved artifacts already on disk.\n\nFor Docker-based labs, a normal run also performs a best-effort runtime cleanup before the first phase starts and again after the lab finishes. This keeps stale containers, networks, or volumes from an earlier lab attempt from leaking into later results without adding heavy cleanup work between every command.\n\nFailure messages should be explicit and actionable.\n\nValidation focus:\n\n- the upstream lab `README.md` is the source of truth\n- the console output from the automated lab run should match the README\n- the generated CTRF JSON and sibling Specmatic HTML report should match the README and console output\n- when a README documents commands, it should provide command sections for Windows, macOS, and Linux\n- OS-specific command sections should use appropriate fenced block languages such as `shell`/`bash` for macOS and Linux, and `powershell`/`cmd` for Windows\n- every documented command section should be followed by a console output snippet\n- OS-specific command sections should have matching OS-specific console output snippets\n- all README console output snippets should use `terminaloutput` fenced blocks\n- when README console output includes timestamps, comparison logic should ignore the datetime stamp and focus on the meaningful output content\n- Studio-only phases that are not automated yet should be reported as known limitations or skipped validations, not as failures\n- intentional differences that are part of the lab design should be recorded as expected differences, not counted in the failure index\n- copied source snapshots such as `specmatic.yaml`, example JSON files, or service source files may still be archived for inspection, but they should not drive pass/fail assertions by themselves\n- the shared README template is configured in [`lablib/readme_expectations.py`](/Users/anand.bagmar/projects/specmatic/labs-tests/lablib/readme_expectations.py)\n  - `README_TEMPLATE` defines the shared H1/H2/H3 schema and section-level command/output expectations\n  - `LAB_README_OVERRIDES` defines lab-specific exceptions or manual Studio allowances\n  - `EXPECTED_README_H2_SEQUENCE` is derived from that template for compatibility with existing comparison logic\n\nREADME command/output conventions:\n\n- use executable fenced blocks for commands:\n  - `shell`, `bash`, `sh`, or `zsh` for macOS and Linux\n  - `powershell`, `ps1`, `cmd`, or `bat` for Windows\n- place a `terminaloutput` fenced block immediately after each documented command block\n- when commands differ by OS, include a matching `terminaloutput` block for each OS-specific command\n\nWhen a command or validation fails, the message should always say:\n\n- what failed\n- what the impact is\n- what action is needed to fix it\n\nPrefer concrete paths, commands, and missing artifacts over vague summaries or raw log excerpts.\n\nNon-failing validation states:\n\n- use `assert_skipped(...)` for validations that are intentionally not implemented yet, such as documented Studio-only steps that labs-tests does not automate yet\n- use `assert_expected(...)` for intentional differences that should stay visible in the report but should not count as failures\n- skipped and expected validations should remain visible in the HTML report, but they should not appear in the failure index or contribute to the failure count\n\nHow to mark an intentional difference as expected in a lab runner:\n\nUse `assert_expected(...)` inside a phase's `extra_assertions`.\n\nExample:\n\n```python\nfrom lablib.scaffold import assert_expected, detail\n\ndef baseline_assertions(context):\n    return [\n        assert_expected(\n            \"This baseline mismatch is intentional and should stay visible without failing the lab.\",\n            category=\"readme\",\n            code=\"readme.intentional-baseline-difference\",\n            details=[\n                detail(\"Reason\", \"The README documents this mismatch as the before state.\"),\n                detail(\"Action\", \"Do not fix this in labs-tests; fix only if the upstream README changes.\"),\n            ],\n        )\n    ]\n```\n\nHow to ignore a shared README validation without showing anything in the rendered README:\n\nAdd an HTML comment to the upstream README. GitHub and browsers do not render it, but labs-tests will read it.\n\nExample:\n\n```md\n\u003c!-- labs-tests: ignore readme.os_commands.coverage --\u003e\n\u003c!-- labs-tests: ignore readme.command_output.followup readme.output.terminaloutput_fence --\u003e\n```\n\nSupported shared README ignore codes currently include:\n\n- `readme.commands.minimum_count`\n- `readme.commands.executable_fences`\n- `readme.os_commands.coverage`\n- `readme.os_commands.fence_languages`\n- `readme.os_output.path_coverage`\n- `readme.command_output.followup`\n- `readme.output.terminaloutput_fence`\n- `readme.os_output.command_coverage`\n- `readme.tests_run_summary.matches_console`\n- `readme.structure.single_h1`\n- `readme.structure.required_h2_sections`\n- `readme.structure.required_h2_order`\n- `readme.structure.unexpected_h2_sections`\n\nWhen an ignore annotation is present:\n\n- the validation is shown as `skipped`\n- it remains visible in the report for traceability\n- it does not count as a failure\n\nHow to mark test-count comparison as not applicable for a lab:\n\n- add `test_counts: false` to the lab README front matter\n\nExample:\n\n```yaml\n---\ntest_counts: false\n---\n```\n\nThe comparison report will then show those phases as `Expected`, and the README / Console / CTRF / HTML cells will render as `Not Applicable`.\n\nGitHub Actions workflow:\n\n- `.github/workflows/labs-tests.yml`\n- runs `python3 run_all.py --refresh-labs --force --labs-branch dynamic-labs` by default\n- accepts an optional space-separated `labs` workflow input to run only selected labs\n- accepts an optional `labs_branch` workflow input; until the `dynamic-labs` work is merged, the default branch is `dynamic-labs`\n- accepts an optional `manage_license` workflow input; by default the workflow creates or replaces `../labs/license.txt` before the run and restores or removes it afterward\n- emits a 60-second heartbeat while the suite is still running, so quiet phases remain visibly active in Actions\n- uses a 40-minute timeout for the workflow job and the main lab execution step\n- publishes a GitHub job summary based on `output/consolidated-report/consolidated-report.json`\n- includes the consolidated report path and comparison report path in the GitHub job summary so workflow runs can be checked quickly\n- uploads `output/` plus every lab-local `*/output/` folder as the `specmatic-labs-reports` artifact\n\nLicense lifecycle:\n\n- `python3 run_all.py` manages `../labs/license.txt` by default\n- local runs read `status.license` from `~/.specmatic/license.json` and write it to `../labs/license.txt`\n- GitHub Actions runs read `SPECMATIC_LICENSE_KEY` and write it to `../labs/license.txt`\n- after the run completes, the original `../labs/license.txt` content is restored, or the file is removed if it did not exist before the run\n- use `python3 run_all.py --no-manage-license ...` to opt out locally\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspecmatic%2Flabs-tests","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fspecmatic%2Flabs-tests","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspecmatic%2Flabs-tests/lists"}