{"id":13828013,"url":"https://github.com/zaptst/zap","last_synced_at":"2025-07-09T05:31:14.937Z","repository":{"id":66095142,"uuid":"142369328","full_name":"zaptst/zap","owner":"zaptst","description":"A streamable structured interface for real time reporting of developer tools.","archived":false,"fork":false,"pushed_at":"2018-09-04T18:25:29.000Z","size":37,"stargazers_count":120,"open_issues_count":4,"forks_count":1,"subscribers_count":12,"default_branch":"master","last_synced_at":"2024-08-05T09:17:08.918Z","etag":null,"topics":["reporting","tap","test","testing"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zaptst.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2018-07-26T00:54:56.000Z","updated_at":"2024-02-25T21:00:50.000Z","dependencies_parsed_at":"2023-04-28T15:00:38.966Z","dependency_job_id":null,"html_url":"https://github.com/zaptst/zap","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zaptst%2Fzap","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zaptst%2Fzap/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zaptst%2Fzap/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zaptst%2Fzap/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zaptst","download_url":"https://codeload.github.com/zaptst/zap/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225486500,"owners_count":17481912,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["reporting","tap","test","testing"],"created_at":"2024-08-04T09:02:26.650Z","updated_at":"2024-11-20T07:31:24.646Z","avatar_url":"https://github.com/zaptst.png","language":null,"funding_links":[],"categories":["Others"],"sub_categories":[],"readme":"# ZAP (\"Ze'report Anything Protocol\")\n\n\u003e A streamable structured interface for real time reporting of developer tools.\n\n**Important!** This specification is a work in progress and is seeking feedback\nfrom potential users and any developer tooling authors. It may change\ndramatically based on feedback.\n\n## Motivation\n\nThere are thousands of different developer tools across every programming\nlanguage. Each of them has their own way of reporting the results of tests\nwhile they are running and after they are completed.\n\nBut we need some standard formats for reporting their results so that tools can\nwork together without having to support hundreds of different formats for every\nlanguage and tool.\n\nToday there exists two prominent formats for reporting test results:\n\n**JUnit XML**\n\n```xml\n\u003c?xml version=\"1.0\" encoding=\"UTF-8\" ?\u003e\n\u003ctestsuites id=\"20140612_170519\" name=\"New_configuration (14/06/12 17:05:19)\" tests=\"225\" failures=\"1262\" time=\"0.001\"\u003e\n  \u003ctestsuite id=\"codereview.cobol.analysisProvider\" name=\"COBOL Code Review\" tests=\"45\" failures=\"17\" time=\"0.001\"\u003e\n    \u003ctestcase id=\"codereview.cobol.rules.ProgramIdRule\" name=\"Use a program name that matches the source file name\" time=\"0.001\"\u003e\n      \u003cfailure message=\"PROGRAM.cbl:2 Use a program name that matches the source file name\" type=\"WARNING\"\u003e\nWARNING: Use a program name that matches the source file name\nCategory: COBOL Code Review – Naming Conventions\nFile: /project/PROGRAM.cbl\nLine: 2\n      \u003c/failure\u003e\n    \u003c/testcase\u003e\n  \u003c/testsuite\u003e\n\u003c/testsuites\u003e\n```\n\n**Test Anything Protocol (aka \"TAP\")**\n\n```tap\n1..4\nok 1 - Input file opened\nnot ok 2 - First line of the input valid\nok 3 - Read the rest of the file\nnot ok 4 - Summarized correctly # TODO Not written yet\n```\n\nEach of these has different strengths and weaknesses.\n\n**JUnit** is easy to parse and lots of structured information about results, but\ncannot be streamed as tests run. You have to wait until your tests are finished\nbefore generating the XML.\n\n**TAP** is great for streaming test results as they run and can be used to\nreport progress to developers. But it is hard to parse and contains very little\nstructured information.\n\n**ZAP** is a new reporting format that tries to get the best of both systems, it\ncontains well structured information about results and can be streamed and\nparsed very easily.\n\n#### Goals\n\n- **Streaming:** Allow developer tools to report test information to consumers\n  in real time as tests execute.\n- **Concurrency:** Allow tools to be run concurrently across threads or\n  processes and allow interweaving of events in output.\n- **Structured:** Use a well-defined structure that is easy to parse and\n  understand with lots of information.\n- **Definitive:** At any given point, it should be clear what state things are\n  in: \"Passing\" or \"Failing\". When a completion state is given, it should not\n  change.\n- **Extensible:** Have a clear way of adding additional information to output\n  that is still meets the above goals.\n\n#### Non-goals\n\n- **Readability:** It should be easy to build reporting tools on top of ZAP\n  that is human readable, but ZAP itself should be designed for machines.\n\n#### Comparison\n\n|                 | XML | TAP | ZAP |\n| ---------------:|:---:|:---:|:---:|\n| **Streaming**   |  ❌  |  ✅  |  ✅  |\n| **Concurrency** | N/A |  👌  |  ✅  |\n| **Structured**  |  ✅  |  ❌  |  ✅  |\n| **Extensible**  |  ✅  |  ❌  |  ✅  |\n| **Readable**    |  ❌  |  ✅  |  ❌  |\n\n#### Considerations\n\nThere are many different types of development tools that would like a standard\nway to report results via a format like ZAP.\n\nThey generally fall into one of two categories:\n\n1. **Producers**\n  - Testing Frameworks\n  - Linters/Code Quality\n  - Type Checkers\n  - Compilers/Build Tools\n2. **Consumers**\n  - CI/CD Platforms/Services\n  - Task Runners/Build Frameworks\n  - Reporters (CLI or GUI)\n\nMany of these tools already use JUnit or TAP, however many are unhappy with\nthese formats and would like something better. ZAP should consider all of these\ntools and provide guidance on implementations.\n\n## Spec\n\n### Format\n\nThe output is stream of events written in newline-delimited json:\n\n```js\n{\"kind\":\"group\",\"event\":\"started\",\"id\":\"0\",\"time\":304.09246500208974,\"content\":[{\"message\":\"DatabaseConnection\",\"source\":[{\"file\":\"/path/to/test.js\",\"start\":{\"line\":3,\"column\":6},\"end\":{\"line\":3,\"column\":26}}]}]}\n{\"kind\":\"item\",\"event\":\"started\",\"id\":\"0.0\",\"time\":318.4295700006187,\"content\":[{\"message\":\"db.connect()\",\"source\":[{\"file\":\"/path/to/test.js\",\"start\":{\"line\":4,\"column\":8},\"end\":{\"line\":4,\"column\":20}}]}]}\n{\"kind\":\"check\",\"event\":\"failed\",\"id\":\"0.0.0\",\"time\":379.8125419989228,\"content\":[{\"message\":\"Expected:\\n { port: 5432 }\\nActual:\\n  { port: 8000 }\",\"source\":[{\"file\":\"/path/to/test.js\",\"start\":{\"line\":42,\"column\":6}}]}]}\n{\"kind\":\"item\",\"event\":\"failed\",\"id\":\"0.0\",\"time\":390.98526199907064,\"content\":[{\"message\":\"db.connect()\",\"source\":[{\"file\":\"/path/to/test.js\",\"start\":{\"line\":4,\"column\":8},\"end\":{\"line\":4,\"column\":20}}]}]}\n{\"kind\":\"group\",\"event\":\"failed\",\"id\":\"0\",\"time\":464.20705600082874,\"content\":[{\"message\":\"DatabaseConnection\",\"source\":[{\"file\":\"/path/to/test.js\",\"start\":{\"line\":3,\"column\":6},\"end\":{\"line\":3,\"column\":26}}]}]}\n```\n\nIndividual events have this shape:\n\n```ts\ninterface Event {\n  kind: string,\n  event: string,\n  id: string,\n  time: number,\n  status?: string,\n  content: Array\u003c{\n    message: string,\n    source?: Array\u003c{\n      file: string,\n      start?: { line: number, column?: number },\n      end?: { line: number, column?: number }\n    }\u003e\n  }\u003e\n}\n```\n\n### Fields\n\n### `kind`\n\nThe type of entity the event is for.\n\n- `\"group\"` Contains `\"group\"`'s, `\"item\"`'s, or `\"check\"`'s, passes or fails\n  based on children.\n- `\"item\"` Contains `\"check\"`'s, passes or fails based on children.\n- `\"check\"` Represents an individual assertion that either passes or fails.\n\nYou can have any of these (including `\"check\"`'s) with top-level `\"id\"`'s, you\ndon't have to wrap everything in a `\"group\"` or `\"item\"`.\n\nNote that if a `\"group\"` or `\"item\"` fails, they should always have at least\none child kind that failed. i.e. You should always have at least one reported\n`\"check\"` that failed.\n\n#### Kind Examples\n\nThe structure of reported event kinds may differ a lot based on the tool,\n\n##### Testing Framework\n\n```yaml\n- item: Test\n  - check: Assertion\n- group: Suite\n  - item: Test\n    - check: Assertion\n    - check: Assertion\n  - item: Test\n    - check: Assertion\n  - group: Nested Suite\n    - item: Test\n      - check: Assertion\n      - check: Assertion\n      - check: Assertion\n```\n\n#### Linter\n\n```yaml\n- group: File 1\n  - check: Lint Error\n  - check: Lint Error\n- group: File 2\n  - check: Lint Error\n  - check: Lint Error\n```\n\n#### Type Checker\n\n```yaml\n- check: Type Error\n- check: Type Error\n- check: Type Error\n- check: Type Error\n- check: Type Error\n- check: Type Error\n```\n\n### `event`\n\nThe name of the event.\n\n- `\"started\"` Execution has started.\n- `\"info\"` Still executing, but has some information.\n- `\"completed\"` Execution has completed.\n\nYou don't need to send a `\"started\"` event in order to send a `\"completed\"`\nevent. You only really need to send it when there is a gap in time between when\nsomething began running and when it completed. For example, a simple check does\nnot need to report when it started running.\n\n### `status`\n\n- `\"running\"`\n- `\"passed\"`\n- `\"failed\"`\n- `\"errored\"`\n- `\"skipped\"`\n\nAll statuses except for `\"running\"` should be considered final. They should\nonly be updated by restarting an entity.\n\n```yaml\n# Bad\n- 0: started (running)\n- 0: info (failed)\n- 0: completed (passed)\n\n# Good\n- 0: started (running)\n- 0: completed (failed)\n- 0: started (running) \"Retry\"\n- 0: completed (passed)\n\n# Bad\n- 0: started (running)\n  - 0.0: started (running)\n  - 0.0: completed (passed)\n  - 0.1: started (running)\n  - 0.1: completed (failed)\n- 0: completed (failed)\n  - 0.1: started (running)\n  - 0.1: completed (passed) \"Retry\"\n- 0: completed (passed)\n\n# Good\n- 0: started (running)\n  - 0.0: started (running)\n  - 0.0: completed (passed)\n  - 0.1: started (running)\n  - 0.1: completed (failed)\n- 0: completed (failed)\n- 0: started (running) \"Retry\"\n  - 0.1: started (running)\n  - 0.1: completed (passed) \"Retry\"\n- 0: completed (passed)\n```\n\nA parent's status should be `\"passed\"` when all children `\"passed\"`:\n\n```yaml\n- 0: passed\n  - 0.0: passed\n  - 0.1: passed\n```\n\nA parent may have a status of `\"passed\"` or `\"failed\"` when a child was\n`\"skipped\"`. Skips may have multiple different reasons. They are not all\nfailures depending on the situation.\n\n```yaml\n- 0: passed\n  - 0.0: passed\n  - 0.1: skipped\n- 1: failed\n  - 1.0: passed\n  - 1.1: skipped\n```\n\nBut when one or more children fail or error, you *must* fail the parent.\n\n```yaml\n- 0: failed\n  - 0.0: failed\n  - 0.1: passed\n- 2: failed\n  - 2.0: errored\n  - 2.1: passed\n- 1: failed\n  - 1.0: failed\n  - 1.1: errored\n```\n\nInversely, if you may not fail a parent if all the children passed, express\nyour failure with another `\"check\"`. Although it can error.\n\n```yaml\n# Bad\n- 0: failed\n  - 0.0: passed\n  - 0.1: passed\n\n# Good\n- 0: failed\n  - 0.0: passed\n  - 0.1: passed\n  - 0.2: errored \"An error occurred when cleaning up the database\"\n\n# Good\n- 0: errored \"An error occurred when cleaning up the database\"\n  - 0.0: passed\n  - 0.1: passed\n```\n\nA parent can also choose to update their status to `\"failed\"` early when a\nchild has failed and there are more children still running.\n\n```yaml\n- 0: failed\n  - 0.0: failed\n  - 0.1: running\n  - 0.2: running\n```\n\n### `id`\n\nThe identifier for the entity.\n\nA string which a series of numbers separated by periods `.` (i.e. `2.3.1.12`)\n\n```js\n{ \"kind\": \"group\", ... \"id\": \"0\", ... }\n{ \"kind\": \"item\", ... \"id\": \"0.0\", ... }\n{ \"kind\": \"check\", ... \"id\": \"0.0.0\", ... }\n{ \"kind\": \"check\", ... \"id\": \"0.0.1\", ... }\n{ \"kind\": \"item\", ... \"id\": \"0.0\", ... }\n{ \"kind\": \"item\", ... \"id\": \"0.1\", ... }\n{ \"kind\": \"check\", ... \"id\": \"0.1.0\", ... }\n{ \"kind\": \"item\", ... \"id\": \"0.1\", ... }\n{ \"kind\": \"item\", ... \"id\": \"0.2\", ... }\n{ \"kind\": \"check\", ... \"id\": \"0.2.0\", ... }\n{ \"kind\": \"check\", ... \"id\": \"0.2.1\", ... }\n{ \"kind\": \"item\", ... \"id\": \"0.2\", ... }\n...\n```\n\nThis is a flat way to describe a tree of groups, items, and checks.\n\n```yaml\n- group: \"0\"\n  - item: \"0.0\"\n    - check: \"0.0.0\"\n    - check: \"0.0.1\"\n  - item: \"0.1\"\n    - check: \"0.1.0\"\n  - item: \"0.2\"\n    - check: \"0.2.0\"\n    - check: \"0.2.1\"\n- group: \"1\"\n  - item: \"1.0\"\n    - check: \"1.0.0\"\n- group: \"2\"\n  - item: \"2.0\"\n    - check: \"2.0.0\"\n    - check: \"2.0.1\"\n```\n\nYou can determine the \"parent\" of the entity by stripping the last `.number`\n(i.e. `s/\\.\\d+$//`)\n\n### `time`\n\nThe offset time the event occured.\n\nA number in milliseconds with arbitrary precision up to nanoseconds.\n\n```js\n{ ... \"time\": 257.0332779996097  ... }\n{ ... \"time\": 304.6154760001227  ... }\n{ ... \"time\": 397.152665999718   ... }\n{ ... \"time\": 425.59379499964416 ... }\n{ ... \"time\": 496.1821520002559  ... }\n```\n\nTo calculate the duration of an entity you can compare the entity's `\"start\"`\nand `\"completed\"` events.\n\n```js\n{ \"id\": \"0.0\", \"event\": \"started\", \"time\": 304.6154760001227 ... }\n{ \"id\": \"0.0\", \"event\": \"completed\", \"time\": 425.59379499964416 ... }\n```\n\n```\n425.5937949996ms - 304.6154760001227ms = 120.9783189995ms = 0.12s\n```\n\n**Important!** You should not assume that event times from entities with\ndifferent `\"id\"`'s have anything to do with one another. They could have been\nrun in separate parallel processes, or even across different machines.\n\n```js\n{ \"id\": \"1.8\", \"event\": \"started\", \"time\": 304.6154760001227 ... } // from process 1\n{ \"id\": \"4.3\", \"event\": \"started\", \"time\": 4235.59379499964416 ... } // from process 2\n{ \"id\": \"2.2\", \"event\": \"started\", \"time\": 204.152665999718 ... } // from process 3\n// none of these times have anything to do with one another\n```\n\n### `content`\n\nThe content of the event.\n\nAn array of messages and (optionally) source locations.\n\n```js\n{ ...\n  \"content\": [\n    { \"message\": \"number\", \"source\": [{ \"file\": \"/path/to/source/file.js\", \"start\": { \"line\": 15, \"column\": 12 }, \"end\": { \"line\": 15, \"column\": 19 } }] },\n    { \"message\": \"is not compatible with\" },\n    { \"message\": \"string\", \"source\": [{ \"file\": \"/path/to/source/file.js\", \"start\": { \"line\": 19, \"column\": 8 }, \"end\": { \"line\": 19, \"column\": 14 } }] }\n  ]\n  ...\n}\n```\n\nEvery element in the `\"content\"` array should be an object with the following\nshape:\n\n```ts\ninterface ContentPart {\n  message: string,\n  source?: Array\u003c{\n    file: string,\n    start?: { line: number, column?: number },\n    end?: { line: number, column?: number }\n  }\u003e\n}\n```\n\n#### Content Targets\n\nWhen interpreting elements of `\"content\"` you should not attempt to \"fill in\"\nthe missing elements.\n\n- If there is no `\"source\"`, do not make one up or try associating\n  it with the other elements in the `\"content\"` array.\n- If there is a `\"file\"` but no `\"start\"` or `\"end\"` positions, assume it means\n  the file itself, not the range of the file's content.\n- If there is a `\"line\"` but no `\"column\"`, assume it means the entire line,\n  not range of the line's content.\n- If there is a `\"start\"` but no `\"end\"`, assume it means that exact position,\n  not a range of a single characters, or to the rest of the file.\n\n#### Content Positions\n\nWhen interpreting a position:\n\n- `\"line\"` is 1-index based\n- `\"column\"` is 0-index based\n\nThis means that when you are interpreting a `\"column\"` it is the index *before*\na character. It targets this \"in between\" position and not the character itself.\n\nFor example if we have the following line:\n\n```\n012345\n```\n\nAnd we are selecting columns 1 through 4, we would get the following selection:\n\n```\n0[123]45\n  ^^^\n```\n\n#### Content Examples\n\n##### A message with no location:\n\n```js\n{ ...\n  \"content\": [{\n    \"message\": \"No tests found\"\n  }]\n}\n```\n\n```\nNo tests found\n```\n\n##### An entire file:\n\n```js\n{ ...\n  \"content\": [{\n    \"message\": \"File named incorectly\",\n    \"source\": [{\n      \"file\": \"/path/to/file.js\"\n    }]\n  }]\n}\n```\n\n```\n/path/to/file.js: File named incorrectly\n```\n\n##### A specific line in a file:\n\n```js\n{ ...\n  \"content\": [{\n    \"message\": \"File too long\",\n    \"source\": [{\n      \"file\": \"/path/to/file.js\",\n      \"start\": { \"line\": 10000 }\n    }]\n  }]\n}\n```\n\n```\n/path/to/file.js\n   9998 |   \"lorem\",\n   9999 |   \"ipsum\",\n\u003e 10000 |   \"dolor\",\n  10001 |   \"sit\"\n  10002 |   \"amet\",\nFile too long\n```\n\n##### A specific position in a file:\n\n```js\n{ ...\n  \"content\": [{\n    \"message\": \"Missing closing brace\",\n    \"source\": [{\n      \"file\": \"/path/to/file.js\",\n      \"start\": { \"line\": 24, \"column\": 4 }\n    }]\n  }]\n}\n```\n\n```\n/path/to/file.js\n  22 |     for (let item of items) {\n  23 |       total += item.price;\n\u003e 24 |\n     |     ^ Missing closing brace\n  25 |     return toPriceString(total);\n  26 |   }\n```\n\n##### A range of lines in a file:\n\n```js\n{ ...\n  \"content\": [{\n    \"message\": \"...\",\n    \"source\": [{\n      \"file\": \"/path/to/file.js\",\n      \"start\": { \"line\": 12 },\n      \"end\": { \"line\": 14 }\n    }]\n  }]\n}\n```\n\n```\n/path/to/file.js\n  11 |   ...\n\u003e 12 |     ...\n\u003e 13 |     ...\n\u003e 14 |     ...\n  15 |   ...\n...\n```\n\n##### A range in a file:\n\n```js\n{ ...\n  \"content\": [{\n    \"message\": \"Variable name `public` is reserved word\",\n    \"source\": [{\n      \"file\": \"/path/to/file.js\",\n      \"start\": { \"line\": 9, \"column\": 6 },\n      \"end\": { \"line\": 9, \"column\": 13 }\n    }]\n  }]\n}\n```\n\n```\n/path/to/file.js\n   7 |\n   8 |   function isPublic(item) {\n\u003e  9 |     let public = item.public;\n     |         ^^^^^^ Variable name `public` is reserved word\n  10 |     let childrenPublic = item.children.every(child =\u003e {\n  11 |       return isPublic(child);\n```\n\n##### Multiple ranges in a file:\n\n```js\n{ ...\n  \"content\": [{\n    \"message\": \"Binding `active` declared multiple times\",\n    \"source\": [{\n      \"file\": \"/path/to/file.js\",\n      \"start\": { \"line\": 9, \"column\": 6 },\n      \"end\": { \"line\": 9, \"column\": 13 }\n    }, {\n      \"file\": \"/path/to/file.js\",\n      \"start\": { \"line\": 27, \"column\": 6 },\n      \"end\": { \"line\": 27, \"column\": 13 }\n    }]\n  }]\n}\n```\n\n```\n/path/to/file.js\n   8 |   function isActive(item) {\n\u003e  9 |     let active = item.active;\n     |         ^^^^^^\n  10 |     let childrenActive = item.children.every(child =\u003e {\n    ...\n  26 |\n  27 |     let active = item.children.filter(child =\u003e {\n     |         ^^^^^^\n  28 |       return isActive(child);\nBinding `active` declared multiple times\n```\n\n##### Multi-part message\n\n```js\n{ ...\n  \"content\": [\n    {\n      \"message\": \"number\",\n      \"source\": [{\n        \"file\": \"/path/to/one.js\",\n        \"start\": { \"line\": 15, \"column\": 38 },\n        \"end\": { \"line\": 15, \"column\": 45 }\n      }]\n    },\n    {\n      \"message\": \"is not compatible with\"\n    },\n    {\n      \"message\": \"string\",\n      \"source\": [{\n        \"file\": \"/path/to/two.js\",\n        \"start\": { \"line\": 9, \"column\": 6 },\n        \"end\": { \"line\": 9, \"column\": 13 }\n      }]\n    }\n  }]\n}\n```\n\n```\n/path/to/one.js\n  14 |\n\u003e 15 | export function calculateTotal(item): number {\n     |                                       ^^^^^^ number\n  16 |\n\nis not compatible with\n\n/path/to/two.js\n  36 |     let message = 'Total: ';\n\u003e 37 |     let total: string = calculateTotal(item);\n     |                ^^^^^^ string\n  38 |     message += total;\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzaptst%2Fzap","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzaptst%2Fzap","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzaptst%2Fzap/lists"}