{"id":39972236,"url":"https://github.com/jrpool/testaro","last_synced_at":"2026-01-18T22:30:39.587Z","repository":{"id":331964809,"uuid":"1075045435","full_name":"jrpool/testaro","owner":"jrpool","description":"A thousand web accessibility tests performed by 11 tools","archived":false,"fork":false,"pushed_at":"2026-01-11T22:14:40.000Z","size":17499,"stargazers_count":4,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-01-11T23:51:13.557Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"HTML","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jrpool.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2025-10-13T00:23:59.000Z","updated_at":"2026-01-11T22:14:43.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/jrpool/testaro","commit_stats":null,"previous_names":["jrpool/testaro"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/jrpool/testaro","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jrpool%2Ftestaro","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jrpool%2Ftestaro/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jrpool%2Ftestaro/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jrpool%2Ftestaro/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jrpool","download_url":"https://codeload.github.com/jrpool/testaro/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jrpool%2Ftestaro/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28552782,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-18T20:59:07.572Z","status":"ssl_error","status_checked_at":"2026-01-18T20:59:02.799Z","response_time":98,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-18T22:30:39.467Z","updated_at":"2026-01-18T22:30:39.571Z","avatar_url":"https://github.com/jrpool.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"# testaro\n\nEnsemble testing for web accessibility\n\n## Introduction\n\nTestaro is an application for automated web accessibility testing.\n\nThe purposes of Testaro are to:\n\n- provide programmatic access to accessibility tests defined by several tools\n- facilitate the integration of the reports of the tools into a unified report\n\nTestaro is described in two papers:\n\n- [How to run a thousand accessibility tests](https://medium.com/cvs-health-tech-blog/how-to-run-a-thousand-accessibility-tests-63692ad120c3)\n- [Testaro: Efficient Ensemble Testing for Web Accessibility](https://arxiv.org/abs/2309.10167)\n\nThe need for multi-tool integration, and its costs, are discussed in [Accessibility Metatesting: Comparing Nine Testing Tools](https://arxiv.org/abs/2304.07591).\n\nTestaro launches and controls web browsers, performing operations, conducting tests, and recording results.\n\nTestaro can be installed under a MacOS, Windows, Debian, or Ubuntu operating system.\n\nTestaro accepts _jobs_, performs them, and returns _reports_.\n\nOther software, located on the same or a different host, can make use of Testaro, performing functions such as:\n\n- Job preparation\n- Converting user specifications into jobs\n- Job scheduling\n- Monitoring of the health of Testaro\n- Management of clusters of workstations sharing workloads\n- Allocation of responsibilities among workstations\n- Receiving and fulfilling user requests for jobs\n- Allocating testing responsibilities to human testers\n- Combining reports from workstations and human testers\n- Analyzing and summarizing (e.g., computing scores on the basis of) test results\n- Sending notifications\n- Revising, combining, and publishing reports\n\nOne software product that performs some such functions is [Testilo](https://www.npmjs.com/package/testilo).\n\n## Dependencies\n\nTestaro uses:\n\n- [Playwright](https://playwright.dev/) to launch browsers, perform user actions in them, and perform tests\n- [playwright-extra](https://www.npmjs.com/package/playwright-extra) and [puppeteer-extra-plugin-stealth](https://www.npmjs.com/package/puppeteer-extra-plugin-stealth) to make a Playwright-controlled browser more indistinguishable from a human-operated browser and thus make their requests more likely to succeed\n- [playwright-dompath](https://www.npmjs.com/package/playwright-dompath) to retrieve XPaths of elements\n- [pixelmatch](https://www.npmjs.com/package/pixelmatch) to measure motion\n\nTestaro performs tests of these _tools_:\n\n- [Accessibility Checker](https://www.npmjs.com/package/accessibility-checker) (IBM)\n- [Alfa](https://alfa.siteimprove.com/) (Siteimprove)\n- [ASLint](https://www.npmjs.com/package/@essentialaccessibility/aslint) (eSSENTIAL Accessibility)\n- [Axe](https://www.npmjs.com/package/axe-playwright) (Deque)\n- [Editoria11y](https://github.com/itmaybejj/editoria11y) (Princeton University)\n- [HTML CodeSniffer](https://www.npmjs.com/package/html_codesniffer) (Squiz Labs)\n- [Nu Html Checker](https://github.com/validator/validator) (World Wide Web Consortium)\n- [QualWeb](https://www.npmjs.com/package/@qualweb/core) (University of Lisbon)\n- [Testaro](https://www.npmjs.com/package/testaro) (CVS Health)\n- [WallyAX](https://www.npmjs.com/package/@wally-ax/wax-dev) (Wally Solutions)\n- [WAVE](https://wave.webaim.org/api/) (WebAIM)\n\nSome of the tests of Testaro are designed to act as approximate alternatives to tests of vulnerable, restricted, or no longer available tools. In all such cases the Testaro rules are independently designed and implemented, without reference to the code of the tests that inspired them.\n\n## Rules\n\nEach tool accessed with Testaro defines _rules_ and tests _targets_ for compliance with its rules. Testilo has classified the rules into _issues_ and deprecated some rules as poorly implemented. If the deprecated rules are excluded, the counts are:\n\n```text\nAccessibility Checker: 93\nAlfa: 64\nASLint: 129\nAxe: 79\nEditoria11y: 23\nHTML CodeSniffer: 110\nNu Html Checker: 260\nQualWeb: 115\nTestaro: 57\nWallyAX: 27\nWAVE: 60\ntotal: 1017\n```\n\nThis tabulation may not be exact, because some of the tools are under active development.\n\n## Code organization\n\nThe main directories containing code files are:\n\n- package root: main code files\n- `tests`: files containing the code defining particular tests\n- `procs`: shared procedures\n- `validation`: code and artifacts for the validation of the Testaro tool\n\n## System requirements\n\nThe latest long-term-support version of [Node.js](https://nodejs.org/en/).\n\n## Installation\n\n### As an application\n\nYou can clone the [Testaro repository](https://github.com/jrpool/testaro) to install Testaro as an application:\n\n```bash\ncd path/to/what/will/be/the/parent/directory/of/testaro\ngit clone https://github.com/jrpool/testaro.git\ncd testaro\nnpm install\nnpx playwright install\n```\n\nAfter that, you can update Testaro after any version change:\n\n```bash\ncd testaro\ngit checkout package-lock.json\ngit pull\nnpm run deps\n```\n\nOnce it is installed as an application, you can use its features with a terminal interface by executing the “By a user” statements below.\n\n### As a dependency\n\nYou can make Testaro a dependency of another application by installing it as you would install any `npm` package, such as by executing `npm install-save testaro` or including `testaro` among the dependencies in a `package.json` file.\n\nOnce it is installed as a dependency, your application can use Testaro features by executing the “By a module” statements below.\n\n### Prerequisites\n\nThe host on which Testaro runs should have the latest long-term-support version of [Node.js](https://nodejs.org/en/).\n\nTestaro is configured so that, when Playwright or Puppeteer launches a `chromium` browser, the browser is [sandboxed](https://www.geeksforgeeks.org/ethical-hacking/what-is-browser-sandboxing/) for improved security. That is the default for Playwright and Puppeteer, and Testaro does not override that default.Any host running Testaro must therefore permit sandboxed browsers. Documentation on how to configure an Ubuntu Linux host for this purpose is available in the [`SERVICE.md` file of the Kilotest repository](https://github.com/jrpool/kilotest/blob/main/SERVICE.md#browser-privileges). If you try to run Testaro on a host that prohibits sandboxed browsers, each attempted launch of a `chromium` browser will throw an error with a message complaining about the unavailability of a sandbox.\n\nTo make the Testaro features work, you will also need to provide the environment variables described below under “Environment variables”.\n\nAll of the tests that Testaro can perform are free of cost, except those performed by the WallyAX and WAVE tools. The owners of those tools issue API keys. A free initial allowance of usage may be granted to you with a new API key. Before using Testaro to perform their tests, get your API keys for [WallyAX](mailto:technology@wallyax.com) and [WAVE](https://wave.webaim.org/api/). Then use those API keys to define environment variables, as described below under “Environment variables”.\n\n## Jobs\n\nA _job_ is an object that specifies what Testaro is to do, and how. As Testaro performs a job, Testaro reports results by adding data to the job and making that enhanced object available as a _report_.\n\n### Example of a job\n\nHere is an example of a job:\n\n```javaScript\n{\n  id: '250110T1200-7f-4',\n  what: 'monthly health check',\n  strict: true,\n  standard: 'also',\n  observe: false,\n  device: {\n    id: 'iPhone 8',\n    windowOptions: {\n      reducedMotion: 'no-preference',\n      userAgent: 'Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/604.1.38 (KHTML, like Gecko) Version/17.4 Mobile/15A372 Safari/604.1',\n      viewport: {\n        width: 375,\n        height: 667\n      },\n      deviceScaleFactor: 2,\n      isMobile: true,\n      hasTouch: true,\n      defaultBrowserType: 'webkit'\n    }\n  },\n  browserID: 'webkit',\n  creationTimeStamp: '241229T0537',\n  executionTimeStamp: '250110T1200',\n  target: {\n    what: 'Real Estate Management',\n    url: 'https://abccorp.com/mgmt/realproperty.html'\n  },\n  sources: {\n    script: 'ts99',\n    batch: 'departments',\n    mergeID: '7f',\n    requester: 'malavu@abccorp.com'\n  },\n  acts: [\n    {\n      type: 'test',\n      launch: {},\n      which: 'axe',\n      detailLevel: 2,\n      rules: ['landmark-complementary-is-top-level'],\n      what: 'Axe'\n    },\n    {\n      type: 'test',\n      launch: {\n        browserID: 'chromium',\n        target: {\n          what: 'Real Estate Management',\n          url: 'https://abccorp.com/mgmt/realproperty.html'\n        }\n      },\n      which: 'qualWeb',\n      withNewContent: false,\n      rules: ['QW-BP25', 'QW-BP26']\n      what: 'QualWeb'\n    }\n  ]\n}\n```\n\nThis job tells Testaro to perform two _acts_. One performs one test of the Axe tool wih reporting at detail level 2, and the other performs two tests of the QualWeb tool.\n\nEach act includes a `launch` object property. In the first act it is an empty object, the browser ID and target URL specified by the job are used. In the second act it overrides the job values with per-act values.\n\nJob properties:\n\n- `id`: a string uniquely identifying the job.\n- `what`: a description of the job.\n- `strict`: `true` or `false`, indicating whether _substantive redirections_ should be treated as failures. These are redirections that do more than add or subtract a final slash.\n- `standard`: whether standardized versions of tool reports are to accompany the original versions (`'also'`), replace the original versions (`'only'`), or not be produced (`'no'`).\n- `observe`: `true` or `false`, indicating whether tool and Testaro-rule invocations are to be reported to the server as they occur, so that the server can update a waiting client.\n- `device`: the ID of a device and the properties of each new browser context (window) that will be set for conformity to that device, unless overridden by an act. It must be `'default'` or the ID of one of [about 125 devices recognized by Playwright](https://github.com/microsoft/playwright/blob/main/packages/playwright-core/src/server/deviceDescriptorsSource.json).\n- `browserID`: the ID of the browser to be used, unless overridden by an act. It must be `'chromium'`, `'firefox'`, or `'webkit`'.\n- `creationTimeStamp`: a string in `yymmddThhMM` format, describing when the job was created.\n- `executionTimeStamp`: a string in `yymmddThhMM` format, specifying a date and time before which the job is not to be performed.\n- `target`: data about the target of the job, or `{}` if the job involves multiple targets.\n- `sources`: data optionally inserted into the job by the job creator for use by the job creator.\n- `acts`: an array of the acts to be performed (documented below).\n\n## Acts\n\nAs shown above, `acts` is a property of a job and has an array value. Each item in the array is an object that specifies an _act_.\n\n### Introduction to acts\n\nEach act object has a `type` property and optionally has a `name` property (used in branching, described below). It must or may have other properties, depending on the value of `type`.\n\n### Act types\n\nThe acts can tell Testaro to perform any of:\n\n- _navigations_ (browser launches, visits to URLs, waits for page conditions, etc.)\n- _moves_ (clicks, text inputs, hovers, etc.)\n- _alterations_ (changes to the page)\n- _tests_ (one or more of the tests defined by a tool)\n- _branching_ (continuing from an act other than the next one)\n\n#### Navigations\n\nAn example of a **navigation** is:\n\n```json\n{\n  \"type\": \"wait\",\n  \"which\": \"travel\",\n  \"what\": \"title\"\n}\n```\n\nIn this case, Testaro waits until the page title contains the string “travel” (case-insensitively).\n\nThere is also a `launch` act. You need it in any job before other acts can be performed, unless the acts are all `test` acts and they include `launch` properties, as in the job example above. That `launch` property is a compact alternative to a `launch` act.\n\n#### Moves\n\nAn example of a **move** is:\n\n```json\n{\n  \"type\": \"radio\",\n  \"which\": \"No\",\n  \"index\": 2,\n  \"what\": \"No, I am not a smoker\"\n}\n```\n\nIn this case, Testaro checks the third radio button whose text includes the string “No” (case-insensitively).\n\nIn identifying the target element for a move, Testaro matches the `which` property with the texts of the elements of the applicable type (such as radio buttons). It defines the text of an `input` element as the concatenated texts of its implicit label or explicit labels, if any, plus, for the first input in a `fieldset` element, the text content of the `legend` element of that `fieldset` element. For any other element, Testaro defines the text as the text content of the element.\n\nWhen the texts of multiple elements of the same type will contain the same `which` value, you can include an `index` property to specify the index of the target element, among all those that will match.\n\n#### Alterations\n\nAn example of an **alteration** is:\n\n```json\n{\n  \"type\": \"reveal\",\n  \"what\": \"make everything visible\"\n}\n```\n\nThis act causes Testaro to alter the `display` and `visibility` style properties of all elements, where necessary, so those properties do not make any element invisible.\n\n#### Branching\n\nAn example of a **branching** act is:\n\n```json\n{\n  \"type\": \"next\",\n  \"if\": [\"totals.invalid\", \"\u003e\", 0],\n  \"jump\": -4,\n  \"what\": \"redo search if any invalid elements\"\n}\n```\n\nThis act checks the result of the previous act to determine whether its `result.totals.invalid` property has a positive value. If so, it changes the next act to be performed, specifying the act 4 acts before this one.\n\nA `next` act can use a `next` property instead of a `jump` property. The value of the `next` property is an act name. It tells Testaro to continue performing acts starting with the act having that value as the value of its `name` property.\n\n#### Tools\n\n##### Introduction to tools\n\nAn act of type `test` performs the tests of a tool and reports a result. The result may indicate that a page passes or fails requirements. Typically, accessibility tests report successes and failures. But a test in Testaro is defined less restrictively, so it can report any result. As one example, the Testaro `elements` test reports facts about certain elements on a page, without asserting that those facts are successes or failures.\n\nThe `which` property of a `test` act identifies a tool, such as `alfa` or `testaro`.\n\n##### Configuration\n\nEvery tool invoked by Testaro must have:\n\n- a property in the `tests` object defined in the `run.js` file, where the property name is the ID representing the tool and the value is the name of the tool\n- a `.js` file, defining the operation of the tool, in the `tests` directory, whose name base is the name of the tool\n\nThe `actSpecs.js` file (described in detail below) contains a specification for any `test` act, namely:\n\n```javascript\ntest: [\n  'Perform a test',\n  {\n    which: [true, 'string', 'isTest', 'test name'],\n    launch: [false, 'object', '', 'if new browser to be launched, properties different from target, browserID, and what of the job'],\n    rules: [false, 'array', 'areStrings', 'rule IDs or specifications, if not all']\n    what: [false, 'string', 'hasLength', 'comment']\n  }\n],\n```\n\nThat means that a test act (i.e. an act with a `type` property having the value `'test'`) must have a string-valued `which` property naming a tool and may optionally have an object-valued `launch` property, an array-valued `rules` property, and/or a string-valued `what` property.\n\nIf a particular test act either must have or may have any other properties, those properties are specified in the `tools` property in `actSpecs.js`.\n\nWhen you include a `rules` property, you limit the tests of the tool that are performed or reported. For some tools (`alfa`, `axe`, `htmlcs`, `qualWeb`, `testaro`, and `wax`), only the specified tests are performed. Other tools (`aslint`, `ed11y`, `ibm`, `nuVal`, `nuVnu`, and `wave`) do not allow such a limitation, so, for those tools, all tests are performed but results are reported from only the specified tests.\n\nThe `nuVal`, `nuVnu`, `qualWeb`, and `testaro` tools require specific formats for the `rules` property. Those formats are described below in the sections about those tools.\n\n##### Examples of test acts\n\nAn example of a `test` act is:\n\n```json\n{\n  \"type\": \"test\",\n  \"which\": \"wave\",\n  \"reportType\": 1,\n  \"what\": \"WAVE summary\"\n}\n```\n\nMost tools allow you to decide which of their rules to apply. In effect, this means deciding which of their tests to run, since each test is considered a test of some rule. The act example\n\n```javaScript\n{\n  type: 'test',\n  which: 'alfa',\n  what: 'Siteimprove alfa tool',\n  rules: ['y', 'r25', 'r71']\n}\n```\n\nspecifies that the tests for rules `r25` and `r71` of the `alfa` tool are to be performed. If the `'y'` in the `rules` array were `'n'` instead, the act would specify that all the tests of the `alfa` tool **except** those for rules `r25` and `r71` are to be run.\n\nOne of the tools that allows rule selection, Testaro, has some rules that take additional arguments. As prescribed in `actSpecs.js`, you can pass such additional arguments to the `reporter` functions of those Testaro tests with an `args` property. Example:\n\n```javaScript\n{\n  type: 'test',\n  which: 'testaro',\n  what: 'Testaro tool',\n  rules: ['y', 'hover', 'focInd'],\n  args: {\n    hover: [20],\n    focInd: [false, 300]\n  }\n}\n```\n\nThis act specifies that the Testaro test `hover` is to be performed with the additional argument `20`, and `focInd` is to be performed with the additional arguments `false` and `300`.\n\n##### Expectations\n\nAny `test` act can contain an `expect` property. If it does, the value of that property must be an array of arrays. Each array specifies expectations about the results of the operation of the tool.\n\nFor example, a `test` act might have this `expect` property:\n\n```javaScript\n'expect': [\n  ['standardResult.totals.0', '=', 0],\n  ['standardResult.instances.length', '=', 0]\n]\n```\n\nThat would state the expectations that the `standardResult` property of the act will report no rule violations at severity level 0 and no instances of rule violations.\n\nThe first item in each array is an identifier of a property of the act. The item has the format of a string with `.` delimiters. Each `.`-delimited segment its the name of the next property in the hierarchy. If the current object is an array, the next segment must be a non-negative integer, representing the index of an element of the array.\n\nIf there is only 1 item in an array, it states the expectation that the specified property does not exist. Otherwise, there are 3 items in the array.\n\nThe second item in each array, if there are 3 items, is an operator, drawn from:\n\n- `\u003c`: less than\n- `=`: equal to\n- `\u003e`: greater than\n- `!`: unequal to\n- `i`: includes\n- `e`: equivalent to (parsed identically as JSON)\n\nThe third item in each array, if there are 3 items in the array, is the criterion with which the value of the first property is compared.\n\nA typical use for an `expect` property is checking the correctness of a Testaro test. Thus, the validation jobs in the `validation/tests/jobs` directory all contain `test` acts with `expect` properties. See the “Validation” section below.\n\n### Tool details\n\nThe tools whose tests Testaro performs have particularities described below.\n\n#### ASLint\n\nThe `aslint` tool makes use of the [`aslint-testaro` fork](https://www.npmjs.com/package/aslint-testaro) of the [`aslint` repository](https://github.com/essentialaccessibility/aslint), which, unlike the published `aslint` package, contains the `aslint.bundle.js` file.\n\n#### HTML CodeSniffer\n\nThe `htmlcs` tool makes use of the `htmlcs/HTMLCS.js` file. That file was created, and can be recreated if necessary, as follows:\n\n1. Clone the [HTML CodeSniffer package](https://github.com/squizlabs/HTML_CodeSniffer).\n1. Make that package’s directory the active directory.\n1. Install the HTML CodeSniffer dependencies by executing `npm install`.\n1. Build the HTML CodeSniffer auditor by executing `grunt build`.\n1. Copy the `build/HTMLCS.js` and `build/licence.txt` files into the `htmlcs` directory of Testaro.\n1. Edit the Testaro copy of `htmlcs/HTMLCS.js` to produce the changes shown below.\n\nThe changes in `htmlcs/HTMLCS.js` are:\n\n```diff\n479a480\n\u003e     '4_1_2_attribute': 'attribute',\n6482a6484\n\u003e     var messageStrings = new Set();\n6496d6497\n\u003c         console.log('done');\n6499d6499\n\u003c         console.log('done');\n6500a6501\n\u003e       return Array.from(messageStrings);\n6531c6532,6534\n\u003c       console.log('[HTMLCS] ' + typeName + '|' + msg.code + '|' + nodeName + '|' + elementId + '|' + msg.msg + '|' + html);\n---\n\u003e       messageStrings.add(\n\u003e         typeName + '|' + msg.code + '|' + nodeName + '|' + elementId + '|' + msg.msg + '|' + html\n\u003e       );\n```\n\n#### Accessibility Checker\n\nThe `ibm` tests require the `aceconfig.js` file.\n\nAs of 2 March 2023 (version 3.1.45 of `accessibility-checker`), the `ibm` tool threw errors when hosted under the Windows operating system. To prevent these errors, it was possible to edit two files in the `accessibility-checker` package as follows:\n\nIn `node_modules/accessibility-checker/lib/ACEngineManager.js`, remove or comment out these lines starting on line 169:\n\n```javaScript\nif (nodePath.charAt(0) !== '/') {\n    nodePath = \"../../\" + nodePath;\n}\n```\n\nIn `node_modules/accessibility-checker/lib/reporters/ACReporterJSON.js`, add these lines starting on line 106, immediately before the line `var resultsFileName = pathLib.join(resultDir, results.label + '.json');`:\n\n```javaScript\n// Replace the colons in the label with hyphen-minuses.\nresults.label = results.label.replace(/:/g, '-');\n```\n\nThese changes were proposed as [pull requests 1333 and 1334](https://github.com/IBMa/equal-access/pulls).\n\nThe `ibm` tool is one of two tools (`testaro` is the other) with a `withItems` property. If you set `withItems` to `false`, the result includes the counts of “violations” and “recommendations”, but no information about the rules that gave rise to them.\n\nIn a previous version of the package, the tool operated on the page content when the `withNewContent` property was `false`. In some cases the tool threw untrappable errors for some targets under that condition. The tool launched a Puppeteer browser to create pages to perform its tests on. On any host that did not permit sandboxed browsers to be launched, the `aceconfig.js` file needed to specify nonsandboxed browsers. Starting in December 2025, the tool operates on the page rather than the page content.\n\n#### Nu Html Checker\n\nThe `nuVal` and `nuVnu` tools perform the tests of the Nu Html Checker.\n\nIts `rules` argument is **not** an array of rule IDs, but instead is an array of rule _specifications_. A rule specification for `nuVal` or `nuVnu` is a string with the format `=ruleID` or `~ruleID`. The `=` prefix indicates that the rule ID is invariable. The `~` prefix indicates that the rule ID is variable, in which case the `ruleID` part of the specification is a matching regular expression, rather than the exact text of a message. This `rules` format arises from the fact that `nuVal` and `nuVnu` generate customized messages and do not accompany them with rule identifiers.\n\n#### QualWeb\n\nThe `qualWeb` tool performs the ACT rules, WCAG Techniques, and best-practices tests of QualWeb. Only failures and warnings are included in the report. The EARL report of QualWeb is not generated, because it is equivalent to the report of the ACT rules tests.\n\nQualWeb allows specification of rules for 3 modules: `act-rules`, `wcag-techniques`, and `best-practices`. If you include a `rules` argument in a QualWeb test act, its value must be an array of 1, 2, or 3 strings. Any string in that array is a specification for one of these modules. The string has this format:\n\n```javascript\n'mod:m,n,o,p,…'\n```\n\nIn that format:\n\n- Replace `mod` with `act`, `wcag`, or `best`.\n- Replace `m`, `n`, `o`, `p`, etc. with the 0 or more integers that identify rules.\n\nFor example, `'best:6,11'` would specify that QualWeb is to test for `best-practices` rules `QW-BP6` and `QW-BP11`, but not for any other `best-practices` rules.\n\nWhen a string contains only a module prefix and no integers, such as `best:`, it specifies that the module is not to be run at all.\n\nWhen no string pertains to a module, then QualWeb will test for all of the rules in that module.\n\nThus, when the `rules` argument is omitted, QualWeb will test for all of the rules in all of these modules.\n\nThe target can be provided to QualWeb either as HTML or as a URL. Experience indicates that the results can differ between these methods, with each method reporting some rule violations or some instances that the other method does not report. For at least some cases, more rules are reported violated when HTML is provided (`withNewItems: false`).\n\nQualWeb creates sandboxed Puppeteer pages to perform its tests on. Therefore, the host must permit sandboxed browsers to be launched. See the pertinent [Kilotest documentation](https://github.com/jrpool/kilotest/blob/main/SERVICE.md#browser-privileges) for information about the configuration of an Ubuntu Linux host for this purpose.\n\n#### Testaro\n\nThe rules that Testaro can test for are implemented in files within the `testaro` directory.\n\nThe Testaro rules are classified by an `allRules` array defined in the `tests/testaro.js` file. Each item in that array is an object with these properties:\n\n- `id`: the rule ID.\n- `what`: a description of the rule.\n- `launchRole`: what a test for the rule does with respect to a browser launch:\n  - `sharer`: requires a browser and leaves it unchanged so the next test can safely reuse it\n  - `waster`: requires a browser and modifies it so the next test cannot safely reuse it\n  - `owner`: launches a custom browser itself and closes it at the end of the test\n- `defaultOn`: whether the rule is to be tested for by default.\n- `timeOut`: the maximum time in seconds allowed for a test for the rule.\n\nIf you do not specify rules when using the `testaro` tool, Testaro will test for its default rules. It will test for these rules in the order in which they appear in the array.\n\nThe optional `rules` argument for a `testaro` test act is an array whose first item is either `'y'` or `'n'` and whose remaining items are rule IDs. If `'y'`, then only the specified rules’ tests are performed. If `'n'`, then all the default rules are tested for, **except** for the specified rules.\n\nThe `testaro` tool (like the `ibm` tool) has a `withItems` property. If you set it to `false`, the `standardResult` object will contain an `instances` property with summaries that identify issues and instance counts. If you set it to `true`, some of the instances will be itemized.\n\nUnlike any other tool, the `testaro` tool requires a `stopOnFail` property, which specifies whether a failure to conform to any rule (i.e. any value of `totals` other than `[0, 0, 0, 0]`) should terminate the execution of tests for the remaining rules.\n\nYou can add custom rules to the rules of any tool. Testaro provides a template, `data/template.js`, for the definition of a rule to be added. Once you have created a copy of the template with revisions, you can move the copy into the `testaro` directory and add an entry for your custom rule to the `allRules` object in the `tests/testaro.js` file. Then your custom rule will act as a Testaro rule. Some `testaro` rules are simple enough to be fully specified in JSON files. You can use any of those as a template if you want to create a sufficiently simple custom rule, namely a rule whose prohibited elements are all and only the elements matching a CSS selector. More details about rule creation are in the `CONTRIBUTING.md` file.\n\nA new pattern for rule definition was introduced in version 60.7.0 and is implemented for only some of the applicable Testaro rules. In this pattern, the `launch` function in the `run` module adds a script to the page that runs whenever a new page is added to a browser context. That script adds `window` methods to the page. When the browser is launched for a Testaro test, the added `window` methods include a `getXPath` method and a `getInstance` method. These methods are used in rule definitions. For examples of this pattern, see the `adbID` and `lineHeight` rules.\n\n#### WallyAX\n\nIf a `wax` test act is included in the job, an environment variable named `WAX_KEY` must exist, with your WallyAX API key as its value. You can obtain it from [WallyAX](https://account.wallyax.com/?ref_app=Developer\u0026app_type=npm).\n\nThe `wax` tool imposes a limit on the size of a page to be tested. If the page exceeds the limit, Testaro treats the page as preventing `wax` from performing its tests. The limit is less than 500,000 characters.\n\n#### WAVE\n\nIf a `wave` test act is included in the job, the WAVE tests will be performed either by the subscription API or by the stand-alone API.\n\nIf you want the subscription API to perform the tests, you must get a WAVE API key from [WebAIM](https://wave.webaim.org/api/) and assign it as the value of an environment variable named `WAVE_KEY`. The subscription API does not accept a transmitted document for testing. WAVE must be given only a URL, which it then visits to perform its tests. Therefore, you cannot manipulate a page and then have WAVE test it, or ask WAVE to test a page that cannot be reached directly with a URL.\n\nIf you want the stand-alone API to perform the tests, you need to have that API installed and running, and the `wave` test act needs to define the URL of your stand-alone API. The test act can also define a `prescript` script and/or a `postscript` script.\n\n### Browser types\n\nWhen you want to run some tests of a tool with one browser type and other tests of the same tool with another browser type, you can do so by splitting the rules into two test acts. For example, one test act can specify the rules as\n\n```javascript\n['y', 'r15', 'r54']\n```\n\nand the other test act can specify the rules as\n\n```javascript\n['n', 'r15', 'r54']\n```\n\nTogether, they get all tests of the tool performed. Before each test act, you can ensure that the latest `launch` act has specified the browser type to be used in that test act.\n\n### `actSpecs` file\n\n#### Introduction to the `actSpecs` file\n\nThe `actSpecs.js` file contains rules governing acts. The rules determine whether an act is valid.\n\n#### Rule format\n\nThe rules in `actSpecs.js` are organized into two objects, `etc` and `tests`. The `etc` object contains rules for acts of all types. The `tools` object contains additional rules that apply to some acts of type `test`, depending on the values of their `which` properties, namely which tools they perform tests of.\n\nHere is an example of an act:\n\n```json\n{\n  \"type\": \"link\",\n  \"which\": \"warming\",\n  \"what\": \"article on climate change\"\n}\n```\n\nAnd here is the applicable property of the `etc` object in `actSpecs.js`:\n\n```js\nlink: [\n  'Click a link',\n  {\n    which: [true, 'string', 'hasLength', 'substring of the link text'],\n    what: [false, 'string', 'hasLength', 'comment']\n  }\n]\n```\n\nThe rule is an array with two elements: a string ('Click a link') describing the act and an object containing requirements for any act of type `link`.\n\nThe requirement `which: [true, 'string', 'hasLength', 'substring of the link text']` specifies what is required for the `which` property of a `link`-type act. The requirement is an array.\n\nIn most cases, the array has length 4:\n\n- Item 0. Is the property (here `which`) required (`true` or `false`)? The value `true` here means that every `link`-type act **must** contain a `which` property.\n- Item 1. What format must the property value have (`'string'`, `'array'`, `'boolean'`, `'number'`, or `'object'`)?\n- Item 2. What other validity criterion applies (if any)? (Empty string if none.) The `hasLength` criterion means that the string must be at least 1 character long.\n- Item 3. Description of the property. In this example, the description says that the value of `which` must be a substring of the text content of the link that is to be clicked. Thus, a `link` act tells Testaro to find the first link whose text content has this substring and click it.\n\nThe validity criterion named in item 2 may be any of these:\n\n- `'hasLength'`: is not a blank string\n- `'isURL`': is a string starting with `http`, `https`, or `file`, then `://`, then ending with 1 or more non-whitespace characters\n- `'isBrowserType'`: is `'chromium'`, `'firefox'`, or `'webkit'`\n- `'isFocusable'`: is `'a'`, `'button'`, `'input'`, `'select'`, or `'option'`\n- `'isState'`: is `'loaded'` or `'idle'`\n- `'isTest'`: is the name of a tool\n- `'isWaitable'`: is `'url'`, `'title'`, or `'body'`\n- `'areStrings'`: is an array of strings\n\n## Reports\n\nAny Testaro job produces a report, which is a copy of the job with additional data produced by Testaro as it performed the job. Like a job, a report is an object that can be serialized to JSON for file storage and network transmission.\n\n### Job-level data\n\nThe data that Testaro adds to a job to create a report include job-level data: data describing the how the job as a whole was performed. Examples: when it was completed and how long it took to run.\n\nProperties that were in a job when it was given to Testaro remain unchanged in the report. New data produced by Testaro during its performance of a job are inserted into a new `jobData` property in the report.\n\n### Act-level data\n\nTestaro also adds act-level data to each job. The new act-level data are properties added to each `act` object.\n\n#### Act-level data from `test` acts\n\nIn a `test` act, one of the 11 tools performs tests and reports the results. Testaro manages this performance with the `reporter` function of a file located in the `tests` directory. Each tool has a corresponding file, such as `alfa.js` for the `alfa` tool.\n\nThe `reporter` function returns an object with this structure:\n\n```js\n{\n  data: {\n    prevented: boolean (whether the tool failed to perform its tests on the page),\n    error: string (if `prevented` is `true`, a description of the error)\n    … (other tool-specific data)\n  },\n  result: object (the results of the tests performed by the tool)\n}\n```\n\nOn the completion of a job, Testaro has added these properties to each `test` act to produce a report:\n\n- `what` (string): the name of the tool\n- `actualURL` (string): the URL of the visited page, after any redirections\n- `startTime` (string): when the tool was started\n- `endTime` (string): when the tool reported its results\n- `data` (object): other tool-specific data:\n  - `prevented` (boolean): whether the tool failed to perform its tests\n  - `error` (string): a description of the failure, if any\n  - other tool-specific data, if any\n\nTestaro may also add these properties to any `test` act:\n\n- `expectations` (object): the results of validations specified by the act in `expect` properties\n- `expectationFailures` (number): the count of failed validations\n\nThe value of `expectations` is an array of objects, one object per expectation. Each object includes a `property` property identifying the expectation, and a `passed` property with `true` or `false` value reporting whether the expectation was satisfied. If applicable, it also has other properties identifying what was expected and what was actually reported.\n\nTestaro also adds one or both of these properties to each `test` act:\n\n- `result` (object): the results of the tests performed by the tool, in the native format of the tool\n- `standardResult` (object): the `result` property converted to a Testaro-standard structure\n\nA job specifies whether the report should include, for each `test` act, the `result` property, the `standardResult` property, or both.\n\n#### Act-level data from `testaro` test acts\n\nEach Testaro rule module exports a `reporter` function, which returns an object with `data`, `totals`, and `standardInstances` properties. Testaro combines the values of those properties with the corresponding values of the same properties from the other `testaro` tests to create the `data` and `result` properties added to `testaro` test acts.\n\n#### Standard result\n\n##### Properties\n\nThe `standardResult` property, when added to a `test` act, includes three properties:\n\n- `prevented` (boolean): whether the tool failed to perform its tests on the page.\n- `totals` (array of numbers): counts of rule violations at 4 ordinal severity levels. For example, the array `[3, 0, 14, 10]` reports that there were 3 violations at level 0, 0 at level 1, 14 at level 2, and 10 at level 3.\n- `instances` (array of objects): descriptions of rule violations reported. An instance can describe a single violation, usually by one element in the page, or can summarize multiple violations of the same rule.\n\nIf the value of `prevented` is `true`, the standard result also includes an `error` property describing the reason for the failure.\n\n##### Instances\n\nHere is an example of an instance in a standard result:\n\n```javascript\n{\n  ruleID: 'rule01',\n  what: 'Button type invalid',\n  ordinalSeverity: 2,\n  count: 1,\n  tagName: 'BUTTON'\n  id: '',\n  location: {\n    doc: 'dom',\n    type: 'xpath',\n    spec: '/html[1]/body[1]/section[3]/div[2]/div[1]/ul[1]/li[1]/button[1]'\n  },\n  excerpt: '\u003cbutton type=\"link\"\u003e\u003c/button\u003e',\n  boxID: '12:340:46:50',\n  pathID: '/html/body/section[3]/div[2]/div/ul/li[1]/button[1]'\n}\n```\n\nThis instance says that a `button` element violates a rule named `rule01`.\n\nThe element has no `id` attribute to distinguish it from other `button` elements, but the tool describes its location. This tool uses an XPath to do that. Tools use various methods for location description, namely:\n\n- `code` (line, starting column, and ending column): Nu Html Checker (API and installed)\n- `selector` (CSS selector): Axe, QualWeb, WAVE\n- `xpath`: Alfa, ASLint, Equal Access\n- `box` (coordinates, width, and height of the element box): Editoria11y, Testaro\n- none: HTML CodeSniffer\n\nThe tool also reproduces an excerpt of the element code.\n\n##### Element identification\n\nWhile the above properties can help you find the offending element, Testaro makes this easier by adding, where practical, two standard element identifiers to each standard instance:\n\n- `boxID`: a compact representation of the x, y, width, and height of the element bounding box, if the element can be identified and is visible.\n- `pathID`: the XPath of the element, if the element can be identified.\n\nThese standard identifiers can help you determine whether violations reported by different tools belong to the same element or different elements. The `boxID` property can also support the making of images of the violating elements.\n\nSome tools limit the efficacy of the current algorithm for standard identifiers:\n\n- HTML CodeSniffer does not report element locations, and the reported code excerpts exclude all text content.\n- Nu Html Checker reports line and column boundaries of element start tags and truncates element text content in reported code excerpts.\n\nTestaro aims to overcome these limitations by inserting uniquely identifying attributes into all elements of the pages being tested by these tools. Those attribute values appear in the excerpts produced by the tools and permit Testaro to identify the elements in the tested page. Except for elements excluded from the DOM, such as descendants of `noscript` elements, this mechanism allows Testaro to provide a `pathID` property in almost all standard instances. The `boxID` property is less universal, since some elements, such as `script` elements and hidden elements, have no bounding boxes.\n\nTesting can change the pages being tested, and such changes can cause a particular element to change its physical or logical location. In such cases, an element may appear multiple times in a tool report with different `boxID` or `pathID` values, even though it is, for practical purposes, the same element.\n\n##### Standardization configuration\n\nEach job specifies how Testaro is to handle report standardization. A job contains a `standard` property, with one of the following values to determine which results the report will include:\n\n- `'also'`: original and standard.\n- `'only'`: standard only.\n- `'no'`: original only.\n\nIf a tool has the option to be used without itemization and is being so used, the `instances` array may be empty, or may contain one or more summary instances. Summary instances disclose the numbers of instances that they summarize with the `count` property. They typically summarize violations by multiple elements, in which case their `id`, `location`, `excerpt`, `boxID`, and `pathID` properties will have empty values.\n\n##### Standardization opinionation\n\nThis standard format reflects some judgments. For example:\n\n- The `ordinalSeverity` property of an instance involves interpretation. Tools may report severity, certainty, priority, or some combination of those. They may use ordinal or metric quantifications. If they quantify ordinally, their scales may have more or fewer than 4 ranks. Testaro coerces each tool’s severity, certainty, and/or priority classification into a 4-rank ordinal classification. This classification is deemed to express the most common pattern among the tools.\n- The `tagName` property of an instance may not always be obvious, because in some cases the rule being tested for requires a relationship among more than one element (e.g., “An X element may not have a Y element as its parent”).\n- The `ruleID` property of an instance is a matching rule if the tool issues a message but no rule identifier for each instance. The `nuVal` and `nuVnu` tools do this. In this case, Testaro is classifying the messages into rules.\n- The `ruleID` property of an instance may reclassify tool rules. For example, if a tool rule covers multiple situations that are dissimilar, that rule may be split into multiple rules with distinct `ruleID` properties.\n\nYou are not dependent on the judgments incorporated into the standard format, because Testaro can give you the original reports from the tools as the `result` property of a `test` act.\n\nThe standard format does not express opinions on issue classification. A rule ID identifies something deemed to be an issue by a tool. Useful reporting from ensemble testing still requires the classification of tool **rules** into **issues**. If tool `A` has `alt-incomplete` as a rule ID and tool `B` has `image_alt_stub` as a rule ID, Testaro does not decide whether those are really the same issue or different issues. That decision belongs to you. The standardization of tool reports by Testaro eliminates some of the drudgery in issue classification, but not any of the judgment required for issue classification.\n\n## Invocation\n\nTestaro features can be invoked by modules of your application when Testaro is a dependency, or directly by users who have installed Testaro as an application.\n\nBefore a module can execute a Testaro function, it must import that function from the Testaro module that exports it. A module can import function `f` from module `m` with the statement\n\n```javascript\nconst {f} = require('testaro/m');`\n```\n\n## Immediate job execution\n\nA job can be immediately executed as follows:\n\n### By a module\n\n```javascript\nconst {doJob} = require('testaro/run');\ndoJob(job)\n.then(report =\u003e …);\n```\n\nTestaro will run the job and return a `report` object, a copy of the job with the `acts` and `jobData` properties containing the results. The final statement can further process the `report` object as desired in the `then` callback.\n\nThe Testilo package contains functions that can create jobs from scripts, add scores and explanations to reports, and create HTML documents summarizing reports.\n\n### By a user\n\n```bash\nnode call run\nnode call run 250525T\n```\n\nIn the second example, `250525T` is the initial characters of the ID of a job saved as a JSON file in the `todo` subdirectory of the `JOBDIR` directory (`JOBDIR` refers to the value of the environment variable `JOBDIR`, obtained via `process.env.JOBDIR`).\n\nThe `call` module will find the first job file with a matching name if an argument is given, or the first job file if not. Then the module will execute the `doJob` function of the `run` module on the job, save the report in the `raw` subdirectory of the `REPORTDIR` directory, and archive the job file in the `done` subdirectory of the `JOBDIR` directory. (The report destination is named `raw` because the report has not yet been further processed by your application, perhaps using Testilo, to convert the report data into user-friendly reports.)\n\n## Job watching\n\nIn watch mode, Testaro periodically checks for a job to run and, when a job is obtained, performs it.\n\n### Directory watching\n\nTestaro can watch for a job in a directory of the filesystem where Testaro or your application is located, with the `dirWatch` function.\n\n#### Directory watching by a module\n\n```javaScript\nconst {dirWatch} = require('testaro/dirWatch');\ndirWatch(true, 300);\n```\n\nIn this example, a moduleof your application asks Testaro to check a directory for a job every 300 seconds, to perform the jobs in the directory if any are found, and then to continue checking. If the first argument is `false`, Testaro will stop checking after performing 1 job. If it is `true`, Testaro continues checking until the `dirWatch` process is stopped.\n\nTestaro checks for jobs in the `todo` subdirectory of `JOBDIR`. When it has performed a job, Testaro moves it into the `done` subdirectory.\n\nTestaro creates a report for each job and saves the report in the `raw` subdirectory of `REPORTDIR`.\n\n#### Directory watching by a user\n\n```javaScript\nnode call dirWatch true 300\n```\n\nThe arguments and behaviors described above for execution by a module apply here, too. If the first argument is `true`, you can terminate the process by entering `CTRL-c`.\n\n### Network watching\n\nTestaro can poll servers for jobs to be performed. Such a server can act as the “controller” described in [How to run a thousand accessibility tests](https://medium.com/cvs-health-tech-blog/how-to-run-a-thousand-accessibility-tests-63692ad120c3). The server is responsible for preparing Testaro jobs, assigning them to Testaro agents, receiving reports back from those agents, and performing any further processing of the reports, including enhancement, storage, and disclosure to audiences. It can be any server reachable with a URL. That includes a server running on the same host as Testaro, with a URL such as `localhost:3000`.\n\nNetwork watching is governed by environment variables of the form `NETWATCH_URL_0_JOB`, `NETWATCH_URL_0_OBSERVE`, `NETWATCH_URL_0_REPORT`, and `NETWATCH_URL_0_AUTH`, and by an environment variable `NETWATCH_URLS`.\n\nYou can create as many quadruples of `…JOB`, `OBSERVE`, `…REPORT`, and `AUTH` variables as you want, one quadruple for each server that the agent may get jobs from. Each quadruple has a different number inside the variable name. The `…JOB` variable is the URL that the agent needs to send a job request to (a typical URL could be `https://testcontroller.xyz.com/api/getJob/agent3`). The `…OBSERVE` variable is the URL that the agent needs to send granular job progress messages to if the job requests that. The `…REPORT` variable is the URL that the agent needs to send a completed report to (such as `localhost:3000/api/submitReport/agent3`). The `…AUTH` variable is the password of the agent that will be recognized by the server. Each URL can contain segments and/or query parameters that identify the purpose of the request, the identity and authorization of the agent, etc.\n\nIn each quadruple, the `…AUTH` variable is optional. If it is truthy (i.e. it exists and has a non-empty value), then the job request sent to the server will be a `POST` request and the payload will be an object with an `agentPW` property, whose value is the password. Otherwise, i.e. if the variable has an empty string as its value or does not exist, the request will be a `GET` request, and an agent password, if required by the server, will need to be provided in the URL.\n\nThe `NETWATCH_URLS` variable has a value of the form `0,3,4`. This is a comma-delimited list of the numbers of the servers to be polled.\n\nOnce Testaro obtains a network job from one of the servers, Testaro performs it and adds the result data to the job, which then becomes a report. Testaro also makes its `AGENT` value the value of the `sources.agent` property of the report. Testaro then sends the report in a `POST` request to the `…REPORT` URL with the same server number. If there is a truthy `…AUTH` variable for the server, the request payload has this format:\n\n```json\n{\n  \"agentPW\": \"abcdef\",\n  \"report\": {\n    …\n  }\n}\n```\n\nIf there is no truthy `…AUTH` variable for the server, the request payload is simply the report in JSON format.\n\nThus, the `…AUTH` variables allow Testaro to comply with servers that object to agent passwords being visible in job request URLs and report-submission URLs and in any log messages that reproduce such URLs.\n\nIf granular reporting is desired, Testaro sends progress messages to the observation URL.\n\nNetwork watching can be repeated or 1-job. 1-job watching stops after 1 job has been performed.\n\nAfter checking all the URLs in succession without getting a job from any of them, Testaro waits for the prescribed time before continuing to check.\n\n#### Network watching by a module\n\n```javaScript\nconst {netWatch} = require('testaro/netWatch');\nnetWatch(true, 300, true);\n```\n\nIn this example, a module of your application asks Testaro to check the servers for a job every 300 seconds, to perform any jobs obtained from any of the servers, and then to continue checking until the process is stopped. If the first argument is `false`, Testaro will stop checking after performing 1 job.\n\nThe third argument specifies whether Testaro should be certificate-tolerant. A `true` value makes Testaro accept SSL certificates that fail verification against a list of certificate authorities. This allows testing of `https` targets that, for example, use self-signed certificates. If the third argument is omitted, the default for that argument is implemented. The default is `true`.\n\n#### Network watching by a user\n\n```javaScript\nnode call netWatch true 300 true\n```\n\nThe arguments and behaviors described above for execution by a module apply here, too. If the first argument is `true`, you can terminate the process by entering `CTRL-c`.\n\n## Environment variables\n\nIn addition to their uses described above, environment variables can be used by acts of type `test`, as documented in the `actSpecs.js` file.\n\nBefore making Testaro run a job, you can optionally also set `HEADED_BROWSER`, `DEBUG`, and/or `WAITS`. The effects of these variables are:\n\n- `HEADED_BROWSER`: whether to run the browser in headed mode instead of the default headless mode\n- `DEBUG`: whether to make logging verbose\n- `WAITS`: the number of milliseconds to wait between actions\n\nYou may store environment variables in an untracked `.env` file if you wish, and Testaro will recognize them. Here is a template for a `.env` file:\n\n```conf\nAGENT=agentabc\nHEADED_BROWSER=false\nDEBUG=false\nJOBDIR=../testing/jobs\nNETWATCH_URL_0_JOB=http://localhost:3000/api/assignJob/agentabc\nNETWATCH_URL_0_OBSERVE=http://localhost:3000/api/granular/agentabc\nNETWATCH_URL_0_REPORT=http://localhost:3000/api/takeReport/agentabc\nNETWATCH_URL_0_AUTH=abcxyz\nNETWATCH_URLS=0\nPUPPETEER_DISABLE_HEADLESS_WARNING=true\nREPORTDIR=../testing/reports\nWAITS=0\nWAVE_KEY=yourwavekey\nWAX_KEY=yourwaxkey\n```\n\n## Validation\n\n### Validators\n\nTestaro and the tests of the Testaro tool can be validated with the _executors_ located in the `validation/executors` directory.\n\nThe executor for a single test is `test`. To execute it for any test `xyz`, call it with the statement `npm test xyz`.\n\nThe other executors are:\n\n- `run`: validates immediate test execution\n- `watchDir`: validates directory watching\n- `watchNet`: validates network watching\n- `tests`: validates all the Testaro tests\n\nTo execute any executor `xyz` among these, call it with the statement `npm run xyz`.\n\nThe `tests` executor makes use of the jobs in the `validation/tests/jobs` directory, and they, in turn, run tests on HTML files in the `validation/tests/targets` directory.\n\n## Contribution\n\nYou can define additional Testaro acts and functionality. Contributions are welcome.\n\nPlease report any issues, including feature requests, at the [repository](https://github.com/cvs-health/testaro/issues).\n\n## Accessibility principles\n\nThe rationales motivating the Testaro-defined tests can be found in comments within the files of those tests, in the `tests` directory. Unavoidably, each test is opinionated. Testaro itself, however, can accommodate other tests representing different opinions. Testaro is intended to be neutral with respect to questions such as the criteria for accessibility, the severities of accessibility defects, whether accessibility is binary or graded, and the distinction between usability and accessibility.\n\n## Challenges\n\n### Abnormal termination\n\nOn some occasions a test throws an error that cannot be handled with a `try`-`catch` structure. It has been observed, for example, that the `ibm` test does this when the page content, rather than the page URL, is given to `getCompliance()` and the target is `https://globalsolutions.org`, `https://monsido.com`, or `https://www.ambetterhealth.com/`.\n\nSome tools take apparently infinite time to perform their tests on some pages. To handle such stalling, Testaro subjects all tools to time limits. The limitation is implemented with forked child processes. Specifically, the `procs/doTestAct.js` module is run as a forked process with a `timeout` option for each of the 11 tools.\n\n### Activation\n\nTesting to determine what happens when a control or link is activated is straightforward, except in the context of a comprehensive set of tests of a single page. There, activating a control or link can change the page or navigate away from it, interfering with the remaining planned tests of the page.\n\nThe Playwright “Receives Events” actionability check does **not** check whether an event is dispatched on an element. It checks only whether a click on the location of the element makes the element the target of that click, rather than some other element occupying the same location.\n\n### Test prevention\n\nTest targets employ mechanisms to prevent scraping, multiple requests within a short time, automated form submission, and other automated actions. These mechanisms may interfere with testing. When a test act is prevented by a target, Testaro reports this prevention.\n\nSome targets prohibit the execution of alien scripts unless the client can demonstrate that it is the requester of the page. Failure to provide that evidence results in the script being blocked and an error message being logged, saying “Refused to execute a script because its hash, its nonce, or unsafe-inline does not appear in the script-src directive of the Content Security Policy”. This mechanism affects tools that insert scripts into a target in order to test it. Those tools include `axe`, `aslint`, `ed11y`, and `htmlcs`. To comply with this requirement, Testaro obtains a _nonce_ from the response that serves the target. Then the file that runs the tool adds that nonce to the script as the value of a `nonce` attribute when it inserts its script into the target.\n\n### Tool duplicativity\n\nTools sometimes do redundant testing, in that two or more tools test for the same defects, although such duplications are not necessarily perfect. This fact creates problems:\n\n- One cannot be confident in excluding some tests of some tools on the assumption that they perfectly duplicate tests of other tools.\n- The Testaro report from a job documents each tool’s results separately, so a single defect may be documented in multiple locations within the report, making the direct consumption of the report inefficient.\n- An effort to aggregate the results into a single score may distort the scores by inflating the weights of defects that happen to be discovered by multiple tools.\n- It is difficult to identify duplicate instances, in part because, as described above, tools use four different methods for identifying the locations of elements that violate tool rules.\n\nTo deal with the above problems, you can:\n\n- configure `test` acts for tools to exclude tests that you consider duplicative\n- create derivative reports that organize results by defect types rather than by tool\n- take duplication into account when defining scoring rules\n\nSome measures of these kinds are included in the scoring and reporting features of the Testilo package.\n\n### Tool malfunctions\n\nTools can become faulty. For example, Alfa stopped reporting any rule violations in mid-April 2024 and resumed doing so at the end of April. In some cases, such as this, the tool maker corrects the fault. In others, the tool changes and forces Testaro to change its handling of the tool.\n\nTestaro would become more reliable if the behavior of its tools were monitored for suspect changes.\n\n### Dependency deployment\n\nThe behavior of Testaro as a dependency of an application deployed on a virtual private server has been observed to be vulnerable to slower performance and more frequent test failures than when Testaro is deployed as a stand-alone application on a workstation. The configuration of Testaro has been tuned for mitigation of such behaviors.\n\n### Containerized deployment\n\nThe experimental deployment of Testaro as a dependency in a containerized application has been unsuccessful. Playwright errors have been thrown that are not thrown when the same application is deployed without containerization.\n\n### Headless browser fidelity\n\nTestaro normally performs tests with headless browsers. Some experiments appear to have shown that some test results are inaccurate with headless browsers, but this has not been replicated. The `launch` function in the `run` module accepts a `headEmulation` argument with `'high'` and `'low'` values. Its purpose is to permit optimizations of headless browsers to be turned off (`high`), at some performance cost, when making the browsers behave and appear more similar to headed browsers improves test accuracy. Observation has, however, failed to show any performance cost. Therefore, `'high'` is currently the default value.\n\n## Repository exclusions\n\nThe files in the `temp` directory are presumed ephemeral and are not tracked by `git`.\n\n## Related work\n\n### Testilo\n\n[Testilo](https://www.npmjs.com/package/testilo) is an application that:\n\n- converts lists of targets and lists of issues into jobs\n- produces scores and adds them to the raw reports of Testaro\n- produces human-oriented HTML digests from scored reports\n- produces human-oriented HTML comparisons of the scores of targets\n\nTestilo contains procedures that reorganize report data by issue and by element, rather than tool, and that compensate for duplicative tests when computing scores.\n\nReport standardization could be performed by other software rather than by Testaro. That would require sending the original reports to the server. They are typically larger than standardized reports. Whenever users want only standardized reports, the fact that Testaro standardizes them eliminates the need to send the original reports anywhere.\n\n### Automated accessibility testing at Slack\n\n[Automated accessibility testing at Slack](https://slack.engineering/automated-accessibility-testing-at-slack/) is based on Playwright, with Axe as a single tool.\n\n## Code style\n\nThe JavaScript code in this project generally conforms to the ESLint configuration file `.eslintrc.json`. However, the `htmlcs/HTMLCS.js` file implements an older version of JavaScript. Its style is regulated by the `htmlcs/.eslintrc.json` file.\n\n## History\n\nWork on the custom tests in this package began in 2017, and work on the multi-package ensemble that Testaro implements began in early 2018. These two aspects were combined into an “Autotest” package in early 2021 and into the more single-purpose packages, Testaro and Testilo, in January 2022.\n\nOn 12 February 2024 ownership of the Testaro repository was transfered from the personal account of contributor Jonathan Pool to the organization account `cvs-health` of CVS Health. The MIT license of the [repository](https://github.com/cvs-health/testaro) did not change, but the copyright holder changed to CVS Health.\n\nMaintenance of the repository owned by CVS Health came to an end on 30 September 2025. The current repository was forked from the `cvs-health` repository in October 2025 and then unlinked from the fork network.\n\n## Contributing\n\nFrom 12 February 2024 through 30 September 2025, contributors of code to Testaro executed a [CVS Health OSS Project Contributor License Agreement](https://forms.office.com/pages/responsepage.aspx?id=uGG7-v46dU65NKR_eCuM1xbiih2MIwxBuRvO0D_wqVFUQ1k0OE5SVVJWWkY4MTVJMkY3Sk9GM1FHRC4u) for Testaro before any pull request was approved and merged.\n\n## Future work\n\nFuture work on this project is being considered. Strategic recommendations for such work are recorded in the `UPGRADES.md` file.\n\n## Etymology\n\n“Testaro” means “collection of tests” in Esperanto.\n\n## License\n\n© 2021–2025 CVS Health and/or one of its affiliates. All rights reserved.\n© 2025 Jonathan Robert Pool.\n\nLicensed under the [MIT License](https://opensource.org/license/mit/). See [LICENSE](../../LICENSE) file\nat the project root for details.\n\nSPDX-License-Identifier: MIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjrpool%2Ftestaro","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjrpool%2Ftestaro","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjrpool%2Ftestaro/lists"}