https://github.com/benwebber/ansible-tap
:white_check_mark: Test Anything Protocol (TAP) producer callback plugin for Ansible
https://github.com/benwebber/ansible-tap
ansible callback-plugin tap tap-stream test test-anything-protocol testing
Last synced: 8 months ago
JSON representation
:white_check_mark: Test Anything Protocol (TAP) producer callback plugin for Ansible
- Host: GitHub
- URL: https://github.com/benwebber/ansible-tap
- Owner: benwebber
- License: mit
- Created: 2016-12-18T10:39:07.000Z (over 9 years ago)
- Default Branch: master
- Last Pushed: 2020-08-01T12:29:43.000Z (almost 6 years ago)
- Last Synced: 2025-04-09T13:11:21.818Z (about 1 year ago)
- Topics: ansible, callback-plugin, tap, tap-stream, test, test-anything-protocol, testing
- Language: Python
- Homepage:
- Size: 16.6 KB
- Stars: 8
- Watchers: 2
- Forks: 4
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# ansible-tap
[](https://travis-ci.org/benwebber/ansible-tap) [](https://galaxy.ansible.com/benwebber/tap/)
[Test Anything Protocol (TAP)](https://testanything.org/) producer for Ansible.
This callback plugin allows you to write TAP test suites as Ansible playbooks. Consider it an Ansible-only alternative to [Serverspec](http://serverspec.org/) and [Testinfra](https://testinfra.readthedocs.io/).
## Requirements
* Ansible 2
* Python 2.7+
## Install
While you can't install this plugin directly using `ansible-galaxy`, you can use `ansible-galaxy` to download it:
```
ansible-galaxy install -p roles/ benwebber.tap
```
Navigate to the role directory and run `make install`:
```
cd roles/benwebber.tap
make install
```
This will copy the plugin to `~/.ansible/plugins/callback`.
## Usage
Configure Ansible to use this plugin as the standard output callback:
```
ANSIBLE_STDOUT_CALLBACK=tap ansible-playbook -i hosts test.yml -l hostname
```
You can also set it to be the default callback in `ansible.cfg`:
```
[defaults]
stdout_callback=tap
```
## Writing Ansible tests
By default, Ansible will abort the play if any tasks fail. Set `ignore_errors: true` on all tests to disable this behaviour:
```yaml
- name: check if service is running
command: systemctl is-active service
register: is_active
tags: diagnostic
- name: assert that service is running
assert:
that: is_active.rc == 0
ignore_errors: true
```
This will ensure Ansible executes the entire test suite, barring any unexpected failure.
If you have a lot of tests, you can set `ignore_errors: true` on a `block`:
```yaml
- name: check if service is running
command: systemctl is-active service
register: is_active
tags: diagnostic
- name: check if service is enabled
command: systemctl is-enabled service
register: is_enabled
tags: diagnostic
- ignore_errors: true
block:
- name: assert that service is running
assert:
that: is_active.rc == 0
- name: assert that service is enabled
assert:
that: is_enabled.rc == 0
```
If a task fails, the plugin will output troubleshooting information as an embedded [YAML document](https://testanything.org/tap-version-13-specification.html#yaml-blocks):
```tap
not ok - assert: assert that variable is set
---
_ansible_no_log: false
_ansible_verbose_always: true
assertion: status is defined
changed: false
evaluated_to: false
failed: true
invocation:
module_args:
that: status is defined
module_name: assert
...
```
### Excluding tasks from TAP stream
Often, the result of a test will depend on previous tasks. You will naturally want to exclude these setup tasks from the TAP stream.
To do so, simply tag setup tasks with the `diagnostic` tag:
```yaml
- name: set up next test
command: 'true'
register: true_
tags: diagnostic
- name: should always pass
assert:
that: true_.rc == 0
ignore_errors: true
```
The callback plugin will print diagnostic lines instead of test lines:
```tap
# command: set up next test
ok - assert: should always pass
````
Unlike individual test cases, you probably do not want to ignore errors for this type of task. Failures would represent an error in the test suite and not a test failure.
### Expected failures and unexpected successes
TAP supports a `TODO` directive to ignore tests for features that haven't been implemented yet.
If a test marked with `TODO` fails, TAP consumers will consider it an expected failure. Similarly, if a test marked with `TODO` passes, TAP consumers will consider it an unexpected success.
Tag expected failures with `TODO`:
```yaml
- name: expected failure
assert:
that: false
ignore_errors: true
tags: TODO
```
This will output a `# TODO` directive in TAP stream:
```tap
not ok - assert: expected failure # TODO
```
If the test passes, you'll receive unexpected success output:
```tap
ok - assert: expected failure # TODO
```
### Skipping tests
TAP also supports a `SKIP` directive to ignore specific tests.
This callback uses Ansible's `when` statement to control skipped tests:
```yaml
- name: this is a skipped task
assert:
that: false
ignore_errors: true
when: false
```
The reason for skipping the test will appear in the test line:
```tap
ok - assert: skipped # SKIP Conditional check failed
```
## Example
The [`tests/`](tests/) directory contains an example test suite which produces all possible test results.
After installing the plugin, run the test suite with:
```
ANSIBLE_STDOUT_CALLBACK=tap ansible-playbook -i localhost, -c local tests/playbooks/test_multiple_with_failures.yml
```
You will receive the following TAP stream. You can pass this to any TAP consumer.
```tap
TAP version 13
# command: set up next test
ok - assert: pass
not ok - assert: failed
---
_ansible_no_log: false
_ansible_verbose_always: true
assertion: false
changed: false
evaluated_to: false
failed: true
invocation:
module_args:
that: false
module_name: assert
...
not ok - assert: expected failure # TODO
ok - assert: unexpected pass # TODO
ok - assert: skipped # SKIP Conditional check failed
1..5
```
## Caveats
At present, this plugin only supports running tests against a single host at a time. The TAP specification does not describe a way to combine multiple output streams.
## License
MIT