{"id":14969253,"url":"https://github.com/cucumber/compatibility-kit","last_synced_at":"2026-02-24T02:13:08.312Z","repository":{"id":37856926,"uuid":"501185680","full_name":"cucumber/compatibility-kit","owner":"cucumber","description":"Platform-agnostic set of acceptance tests for validating cucumber implementations","archived":false,"fork":false,"pushed_at":"2025-09-30T16:08:50.000Z","size":4530,"stargazers_count":6,"open_issues_count":8,"forks_count":6,"subscribers_count":76,"default_branch":"main","last_synced_at":"2025-09-30T18:09:02.445Z","etag":null,"topics":["cucumber","javascript","polyglot-release","ruby","tidelift"],"latest_commit_sha":null,"homepage":"https://cucumber.io/","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cucumber.png","metadata":{"funding":{"open_collective":"cucumber","github":"cucumber"},"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2022-06-08T09:35:27.000Z","updated_at":"2025-09-30T16:08:08.000Z","dependencies_parsed_at":"2023-11-16T18:03:22.526Z","dependency_job_id":"ea2b4bd6-5e38-4293-adb3-21c52bd334ed","html_url":"https://github.com/cucumber/compatibility-kit","commit_stats":{"total_commits":403,"total_committers":19,"mean_commits":"21.210526315789473","dds":0.8089330024813896,"last_synced_commit":"4b54a551d7716bfdd550cf89d091fca5fcaaa591"},"previous_names":[],"tags_count":51,"template":false,"template_full_name":null,"purl":"pkg:github/cucumber/compatibility-kit","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cucumber%2Fcompatibility-kit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cucumber%2Fcompatibility-kit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cucumber%2Fcompatibility-kit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cucumber%2Fcompatibility-kit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cucumber","download_url":"https://codeload.github.com/cucumber/compatibility-kit/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cucumber%2Fcompatibility-kit/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278323964,"owners_count":25968237,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-04T02:00:05.491Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cucumber","javascript","polyglot-release","ruby","tidelift"],"created_at":"2024-09-24T13:41:26.565Z","updated_at":"2025-10-04T14:29:20.643Z","avatar_url":"https://github.com/cucumber.png","language":"TypeScript","funding_links":["https://opencollective.com/cucumber","https://github.com/sponsors/cucumber"],"categories":[],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003e\n  \u003cimg alt=\"\" width=\"75\" src=\"https://github.com/cucumber.png\"/\u003e\n  \u003cbr\u003e\n  compatibility kit\n\u003c/h1\u003e\n\u003cp align=\"center\"\u003e\n  \u003cb\u003eTools for validating \u003ca href=\"https://github.com/cucumber/messages\"\u003eCucumber Messages\u003c/a\u003e conformance\u003c/b\u003e\n\u003c/p\u003e\n\nThe Cucumber Compatibility Kit (CCK) aims to validate a Cucumber implementation's support for the\n[Cucumber Messages protocol](https://github.com/cucumber/messages).\n\nFor example, the first-party Cucumber implementations for JavaScript, JVM and Ruby all use the CCK\nto ensure they are emitting well-formed and correct messages. The CCK is also used as a source of useful test data when developing and testing formatters and other downstream tools.\n\n## Overview\n\nThe kit is composed of features, step definitions, and messages:\n\n- **features**, once executed, emit an exhaustive set of Messages as specified by the protocol\n- **step definitions** that are used in conjunction with the features to generate the Messages,\nand also serve as reference step definitions for implementations adopting the kit\n- **Messages**, serialized as `.ndjson` files, are the reference: a given feature\n  from the kit, executed using its dedicated step definitions, **must** emit the\n  corresponding Messages\n\nThe Messages in the kit are generated using fake-cucumber, a barebones reference implementation\nthat's part of the [devkit](./devkit/) module in this repo.\n\n## Adoption\n\nTo adopt the compatibility kit when writing a Cucumber implementation, an implementer must have some prerequisites, do\nsome setup in the implementation project and then write a dynamic test that is fed by the kit.\n\n### Prerequisites\n\nFirst, make sure your implementation is sufficiently testable - you should be able to execute a test run\nprogramatically for a given set of feature files and step definitions, and obtain the Messages\noutput (via a formatter or some other means) to make assertions on.\n\n### Setup\n\nPull in the latest version of the CCK package for your platform. If there isn't one yet, consider just copying over the\nfiles you need from the [devkit](./devkit/) for now - you can figure out the publishing later once it's all working.\n\nNext, for each feature in the CCK suite, write step definitions that work with your implementation - these should\nbe _equivalent_ to the reference step definitions from the devkit, but with the appropriate language, syntax etc.\n\nNB: For some flavours of Cucumber, how you write code to do certain actions can vary **heavily!**\nPlease ensure that the steps are equivalent!\n\n### Testing\n\nNow you can build your CCK test. For each feature in the CCK suite, it should:\n\n1. Execute a test run, scoped to:\n   - Only the feature file(s) in question\n   - Only the step definition file(s) that you wrote for this feature\n2. Capture the Messages output of that test run\n3. Load the reference messages from the `.ndjson` file and assert that the output you captured matches it\n\n### Notes and exceptions\n\nWhen we say \"matches\" above, that's heavily qualified - there are many individual fields that will vary from\nrun to run, like generated identifiers, timestamps, durations and file URLs. You can freely exclude these\nfields from your assertion - the key thing to capture is the number, type and order\u003csup\u003e*\u003c/sup\u003e of the messages, and the\ncontent that would be consistent across runs. It's also not unheard of for implementations to fix up the order of\nmessages, so it matches the CCK, although it's best to try to match it naturally if you can.\n\nBear in mind that if you, for example, omit or mis-type a step definition, the failure you get might be\nnon-obvious e.g. a discrepancy between actual and expected messages. Many of the features in the CCK represent a test\nrun that fails, so this is not something that should cause your test to fail, although you might find it useful to\ncapture the stderr (or equivalent) output for debugging purposes.\n\nA small minority of features also have an `.arguments.txt` that specify additional arguments/options that are\npertinent to the feature - see `retry` for an example. You can adapt these how you see fit - they may or may not\nline up directly with your CLI options schema.\n\nThere may be some features in the CCK suite that cover functionality that your implementation doesn't have.\nFor example, there's one for `retry` which only a subset of Cucumber implementations support. You can filter these\nout of your test until/unless you're ready to support them.\n\n#### _*_ Ordering\n\nWhile the samples provide a strictly ordered set of messages. Messages themselves are only\n[partially ordered](https://en.wikipedia.org/wiki/Partially_ordered_set). But this is rather hard to capture with approval\ntesting.\n\nFor example when executing scenarios in parallel, the messages from different scenarios are allowed to interleave while\nstill retaining their order for each scenario. Likewise, feature files may be processed in parallel. So while for a\nsingle feature file, the derived `Source`, `GherkinDocument`, `Pickles`, `TestCase` and `TestStep`, ect, ect may come in\norder, these may interleave with the same messages from another feature file.\n\nFinally, some orderings are not specified at all. While `TestRunStarted` is typically the first message, it could also\nbe `StepDefinition`, `Source` or another message without dependencies.\n\nSo at best we have the following guidelines:\n* `{TestRun,TestCase,TestStep}Started` messages are emitted before their `{TestRun,TestCase,TestStep}Finished`\n  counterparts.\n* `TestCase{Started,Finished}` are bookended by `TestRun{Started,Finished}`.\n* `TestStep{Started,Finished}` are bookended by `TestCase{Started,Finished}`.\n* `Attachment` messages are bookended by `TestStep{Started,Finished}`.\n* In general, prior to emitting a message all messages it references or depends on should have been emitted.\n\nWhen in doubt, keep in mind that that applications should be able to process the events in a streaming manner.\n\n### Existing implementations\n\n- [cucumber-js](https://github.com/cucumber/cucumber-js/tree/main/compatibility)\n- [cucumber-node](https://github.com/cucumber/cucumber-node/tree/main/test/cck)\n- [cucumber-jvm](https://github.com/cucumber/cucumber-jvm/tree/main/compatibility)\n- [cucumber-ruby](https://github.com/cucumber/cucumber-ruby/tree/main/compatibility)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcucumber%2Fcompatibility-kit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcucumber%2Fcompatibility-kit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcucumber%2Fcompatibility-kit/lists"}