https://github.com/yaml/libyaml-test-suite
Testing libyaml against yaml-test-suite
https://github.com/yaml/libyaml-test-suite
Last synced: about 1 year ago
JSON representation
Testing libyaml against yaml-test-suite
- Host: GitHub
- URL: https://github.com/yaml/libyaml-test-suite
- Owner: yaml
- Created: 2020-06-02T16:56:48.000Z (about 6 years ago)
- Default Branch: devel
- Last Pushed: 2020-06-15T17:31:59.000Z (about 6 years ago)
- Last Synced: 2025-03-17T11:53:52.623Z (over 1 year ago)
- Language: Shell
- Size: 13.7 KB
- Stars: 4
- Watchers: 4
- Forks: 2
- Open Issues: 0
-
Metadata Files:
- Readme: ReadMe.md
Awesome Lists containing this project
README
run-test-suite
==============
Branch to support testing libyaml with yaml-test-suite
# Synopsis
From libyaml master branch:
```
./bootstrap
./configure
make test
make test-suite
```
# Overview
This code lives in the `libyaml` git repository on the `run-test-suite` branch.
It is used to test libyaml against the YAML Test Suite.
The master branch has a Makefile rule to run this using `make test-suite`.
That command will checkout this branch under the `tests/run-test-suite` directory and then call this Makefile's `make test`.
See:
* https://github.com/yaml/yaml-test-suite
# Pinning a Test Environment
By default, the test system will look for a file called `env/pin-` where `` is a git ref of the libyaml source code being tested (usually master).
When a commit is pushed to master, an entry should be added to the `env` directory for that commit.
The pin file contains information about how the tests should be run and will be sourced by the test runner.
Specifically, the pin file names the test suite repo, the commit to use, and which tests to use (blacklist/whitelist info).
The test suite repo is usually https://github.com/yaml/yaml-test-suite/, but could be a fork if you need to try something.
There are 2 commands: `whitelist` and `blacklist` which can be used to identify the appropriate tests to run.
Here is an example pin file:
```
# First set these 2 variables:
LIBYAML_TEST_SUITE_DATA_REPO=https://github.com/yaml/yaml-test-suite
LIBYAML_TEST_SUITE_DATA_COMMIT=data-2019-09-17
# Start by including all the tests:
whitelist all
# Next, blacklist or whitelist for `emitter`, `parser` and `parser-error`:
# Args can be a multiline string of IDs and descriptions:
blacklist parser "\
XXX1: A test
XXX2: Another test
"
# Use a predefined set of blacklists:
source "$ENV/blacklist-123"
# Args can also be a list of IDs:
whitelist emitter AAA1 AAA2 AAA3
whitelist parser-error XXX8 XXX9
```
Here's a pin file for just running a single parser test:
```
LIBYAML_TEST_SUITE_DATA_REPO=https://github.com/yaml/yaml-test-suite
LIBYAML_TEST_SUITE_DATA_COMMIT=data-2019-09-17
whitelist parser AAA1
```
## Overriding the Pin File
There are a few ways to specify which pinning to use.
They involve setting the `LIBYAML_TEST_SUITE_ENV` variable.
To use a specific pin file for testing, use an absolute path to some file:
```
make test-suite LIBYAML_TEST_SUITE_ENV=/tmp/my-pin-file
```
NOTE: You can use `env` instead of `LIBYAML_TEST_SUITE_ENV` with a `make test` command.
To use a specific pin file, name the file that lives in the `env` directory:
```
make test-suite env=pin-0032321756cee86a67171de425267c1d0d406092
```
## Specifying the Pin Info for a Pull Request
If you are submitting a PR, you can put the pin file content in your log message.
Add this exact line to your log message:
```
*** yaml-test-suite
```
Everything after that will be placed into a pin file.
## Pin Debugging
You can turn on debugging info for the pin `lookup` utility:
```
make test-suite debug=1
```