{"id":21247449,"url":"https://github.com/audrow/yatm","last_synced_at":"2025-07-10T23:32:32.981Z","repository":{"id":36966734,"uuid":"453265948","full_name":"audrow/yatm","owner":"audrow","description":"Yet Another Test-case Manager. Examples can be found here: https://github.com/audrow/humble-tcm-example/issues","archived":false,"fork":false,"pushed_at":"2023-09-12T16:19:48.000Z","size":2121,"stargazers_count":7,"open_issues_count":10,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-04-18T13:23:19.409Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/audrow.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-01-29T01:25:10.000Z","updated_at":"2024-01-12T18:26:41.000Z","dependencies_parsed_at":"2023-02-18T04:01:11.639Z","dependency_job_id":null,"html_url":"https://github.com/audrow/yatm","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":"audrow/node-js-template","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/audrow%2Fyatm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/audrow%2Fyatm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/audrow%2Fyatm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/audrow%2Fyatm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/audrow","download_url":"https://codeload.github.com/audrow/yatm/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225664118,"owners_count":17504441,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-21T02:34:34.310Z","updated_at":"2024-11-21T02:34:34.962Z","avatar_url":"https://github.com/audrow.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# README\n\n- [README](#readme)\n  - [Features](#features)\n  - [Getting started](#getting-started)\n  - [How to use this repository](#how-to-use-this-repository)\n    - [Command line interface](#command-line-interface)\n    - [Configuration](#configuration)\n    - [Creating Requirements](#creating-requirements)\n    - [Creating Test Cases from Requirements](#creating-test-cases-from-requirements)\n    - [Lessons Learned](#lessons-learned)\n\nThis project is used as an alternative to various test case management softwares.\nIt allows you to define requirements and then to create test cases from those requirements.\n\nThere are a few parts of this project where you can write your own plugins:\n\n- Generating requirements\n- Supporting different databases for test cases\n\nIn the future, I plan to make the supported steps for the test cases plugable.\n\n## Features\n\n- YAML format to specify requirements\n  - Can specify instructions for what to try and what you should expect\n  - The instructions for the try and expect sections can have 1 or more steps\n  - The instructions for the try and expect sections support several formats:\n    - Notes\n    - Images (stored somewhere online)\n    - Stdin, stdout, stderr with numbered terminals\n- Create test cases from requirements for combinations\n  - Supports YAML file to filter requirements and create specific combinations of test cases\n  - Requirements are validated through JSON Schema Validation\n- Test cases support a generation number\n- Supports Github Issues as a database for test cases\n  - Create test cases if they don't exist\n  - Create test cases from available test cases given a regex pattern\n  - Delete test cases that match a regex pattern\n\nYou define requirements in YAML files, or can write plugins to generate requirements.\n\n## Getting started\n\nTo get started, install [NodeJS 16](https://nodejs.org/en/), clone this repository, and run the following:\n\n```bash\nnpm ci  # install dependencies\nnpm run prepare  # setup git hooks\n```\n\n```bash\nnpx ts-node src/index.ts -h\n```\n\nFrom there, you can run the tests to confirm that things are working:\n\n```bash\nnpm test\n```\n\n## How to use this repository\n\n### Command line interface\n\nTo run the script we'll use `npx ts-node src/index.ts`. You can run this command without any arguments to see the help message. Note that most commands have shortcuts, such as `npx ts-node src/index.ts requirements` which can be shortened to `npx ts-node src/index.ts r`.\n\nHere are the steps to using this package:\n\n1. Make requirements\n\n   ```bash\n   npx ts-node src/index.ts requirements list-plugins # or npx ts-node src/index.ts r l\n   npx ts-node src/index.ts requirements make all # plugin name or all\n   ```\n\n2. Make test cases:\n\n   ```bash\n   npx ts-node src/index.ts test-cases make\n   ```\n\n   Note that these test cases are made from data specified in the `test-case.config.yaml`.\n\n3. Create the test cases in the db\n\n   ```bash\n   npx ts-node src/index.ts test-cases db github create # or npx ts-node src/index.ts t d github c\n   ```\n\n   You could then delete the created issues with the following command:\n\n   ```bash\n   npx ts-node src/index.ts test-cases db github delete \".\" # or npx ts-node src/index.ts t d github d\n   ```\n\n   Note you can also specify some regex code with create or delete that will be used to match labels or the PR's title.\n\n### Configuration\n\n`YATM` relies on several environment variables that need to be set.\n`YATM_TEST_CASE_CONFIG_PATH`: The absolute path to the YAML file that specifies the test case generation, as well as filters for what test cases to create from requirements and to specify the dimensions that should be varied for the test cases. See [example-test-case.config.yaml](./example-test-case.config.yaml)\n\n- `YATM_REQUIREMENTS_DIRECTORY_PATH`: The absolute path to a directly containing YAML files files that describe testing requirements. See [Creating Requirements](#creating-requirements) for more details.\n- `YATM_OUTPUT_DIRECTORY_PATH` (OPTIONAL): The absolute path to a directory where YATM should output test cases to. If unspecified, a `generated-files/` directory is created in the current working directory.\n- `GITHUB_REPO_OWNER`: The name of the github organization or owner containing the repository below.\n- `GITHUB_REPO_NAME`: The name of repository where Issue tickets for test cases should be opened.\n- `GITHUB_TOKEN`: Your Github Personal Access token which has the ability to read and write to the repository above.\n\nFor convenience it's recommended to create a `.env` file in the project's root which exports the environment variables described above.\n\n### Creating Requirements\n\nTo create your own requirements, add a yaml file to the `requirements` directory or create a plugin that puts files into `generated-files/requirements`. The minimum requirement file should look like the following:\n\n```yaml\nrequirements:\n  - name: Requirement\n    checks:\n      - name: Check\n```\n\nYou can add labels or a URL to requirement, for example:\n\n```yaml\nrequirements:\n  - name: Requirement\n    labels:\n      - ros2cli\n      - another label\n    url: https://github.com/ros2/ros2cli\n    checks:\n      - name: Check\n```\n\nYou can have multiple checks for each requirement and multiple requirements in one requirements file:\n\n```yaml\nrequirements:\n  - name: Requirement name 1\n    checks:\n      - name: Check 1\n      - name: Check 2\n  - name: Requirement name 2\n    checks:\n      - name: Check 3\n```\n\nRequirements also support instructions for the tester to try and for what they should expect. You can have a try statement without an expect statement, but you cannot have an expect statement without a try statement.\n\n```yaml\nrequirements:\n  - name: Requirement\n    checks:\n      - name: ROS2 Topic help\n        try:\n          - note: something to try\n        expect:\n          - stdout: /greet\n```\n\nThe steps in try and accept support several arguments:\n\n- `note`: For textual instructions\n- `imageUrl`: For an image URL (note, the image must be hosted somewhere on the internet)\n- `stdin`, `stdout`, and `stderr`: For terminal input, output, and errors. Note that these arguments support an optional argument `terminal` to number the terminal.\n\nYou can use one or more of these in each step. Unfortunately for now, they will always display in a fixed order.\n\nHere is a more complete example:\n\n```yaml\nrequirements:\n  - name: Test ROS 2 Bag\n    labels:\n      - ros2cli\n    url: https://github.com/ros2/ros2cli\n    checks:\n      - name: ROS2 Topic help\n        try:\n          - note: Try the help command\n            stdin: ros2 topic show --help\n            terminal: 2\n        expect:\n          - note: You should get the following\n            imageUrl: https://img.search.brave.com/TBRxzNr6M8Enl8QPxfadgwmwEdKnYY1yUuyCsg50AYI/rs:fit:200:200:1/g:ce/aHR0cHM6Ly9hd3Mx/LmRpc2NvdXJzZS1j/ZG4uY29tL2dpdGh1/Yi9vcmlnaW5hbC8y/WC9kL2Q0MTY3NmM5/YmY5ZmJhYThlZGJl/NzZlZjM0NzQ0ZjM4/MDg5ZDA0NzQuc3Zn.svg\n```\n\nAlso, note that you should be able to use all of YAML's goodness in any YAML file. You can see [this YAML cheat sheet](https://quickref.me/yaml) for inspiration.\n\n### Creating Test Cases from Requirements\n\nTest cases are generated for requirements for specific combinations of discrete labels. To make this easier, a YAML file, `test-case.config.yaml`, specifies which labels should be applied to which requirements. An example of this YAML file is as follows:\n\n```yaml\ngeneration: 1\nsets:\n  - filters:\n      - isMatch: true\n        name: tf2\n        labels:\n          - docs\n    dimensions:\n      os:\n        - jammy\n      buildType:\n        - debian\n        - source\n```\n\nThe `generation` key is an idea from databases. It creates a high-level label for identifying a group of tests that was created together.\n\nThe `sets` key takes an array of sets that should specify which requirements should be made into tests. In each set, there is a `filters` key and a `dimensions` key. The `filters` key allows you to specify which requirements to include (`isMatch: true`) or exclude (`isMatch: false`). Requirements can be narrowed down with zero or more labels or a regex expression that will be compared to the name of each requirement. For example, in the YAML above, the filters would find requirements with the `docs` label that have `tf2` in the title.\n\nThe `dimensions` key allows you to specify arbitrary lists to get the combinations for.\nFor example, the above YAML would create the following combinations: `[jammy, debian]` and `[jammy, source]`. You can imagine that the number of combinations blows up as there become more dimensions.\n\n### Lessons Learned\n\nHere is some advice for using this framework, based on some experience with it. As it is early days for this project, feel free to disregard them, if you find something better.\n\n- Include requirements in bulk with labels and then refine from there with a combination of labels and name (more brittle).\n- Split requirements up into files with logical groupings.\n- Test how your requirements render before uploading them to a database (e.g., Github).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faudrow%2Fyatm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faudrow%2Fyatm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faudrow%2Fyatm/lists"}