Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/asottile/detect-test-pollution
a tool to detect test pollution
https://github.com/asottile/detect-test-pollution
Last synced: 22 days ago
JSON representation
a tool to detect test pollution
- Host: GitHub
- URL: https://github.com/asottile/detect-test-pollution
- Owner: asottile
- License: mit
- Created: 2022-02-21T17:37:16.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2024-08-27T00:03:57.000Z (3 months ago)
- Last Synced: 2024-09-27T05:51:56.684Z (about 2 months ago)
- Language: Python
- Size: 138 KB
- Stars: 173
- Watchers: 6
- Forks: 7
- Open Issues: 4
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
[](https://github.com/asottile/detect-test-pollution/actions/workflows/main.yml)
[](https://results.pre-commit.ci/latest/github/asottile/detect-test-pollution/main)
detect-test-pollution
=====================
a tool to detect test pollution
## installation
```bash
pip install detect-test-pollution
```
## what is test pollution?
[](https://youtu.be/FRteianaPMo)
test pollution is where a test fails due to the side-effects of some other test
in the test suite.
it usually appears as a "test flake" something where the test fails
mysteriously but passes when run by itself.
a simple example of this is the following python code:
```python
k = 1
def test_k():
assert k == 1
def test_k2():
global k
k = 2
assert k == 2
```
now this example is a little bit silly, you probably wouldn't write code this
poorly but helps us demonstrate the problem here.
when run normally -- these tests pass:
```console
$ pytest -q t.py
.. [100%]
2 passed in 0.00s
```
but, if the tests were run in some other order (due to something like
[pytest-randomly] or [pytest-xdist]) then the pollution would be apparent:
```console
$ pytest -q t.py::test_k2 t.py::test_k
.F [100%]
=================================== FAILURES ===================================
____________________________________ test_k ____________________________________
def test_k():
> assert k == 1
E assert 2 == 1
t.py:4: AssertionError
=========================== short test summary info ============================
FAILED t.py::test_k - assert 2 == 1
1 failed, 1 passed in 0.03s
```
often this flake happens in a codebase with hundreds or thousands of tests
and it's difficult to track down which test is causing the global side-effects.
that's where this tool comes in handy! it helps you find the pair of tests
which error when run in order.
[pytest-randomly]: https://github.com/pytest-dev/pytest-randomly
[pytest-xdist]: https://github.com/pytest-dev/pytest-xdist
## usage
[](https://www.youtube.com/watch?v=w5O4zTusyJ0)
once you have identified a failing test, you'll be able to feed it into
`detect-test-pollution` to find the causal test.
the basic mode is to run:
```bash
detect-test-pollution \
--failing-test test.py::test_id_here \
--tests ./tests
```
where `test.py::test_id_here` is the identifier of the failing test and
`./tests` is the directory where your testsuite lives.
if you've already narrowed down the list of testids further than that, you
can specify a `--testids-file` instead of `--tests` to speed up discovery:
```bash
detect-test-pollution \
--failing-test test.py::test_id_here \
--testids-file ./testids
```
you can usually get a list of testids via `pytest --collect-only -q` (though
you'll need to strip some unrelated lines at the end, such as timing and
warning info).
then `detect-test-pollution` will bisect the list of tests to find the failing
one. here's an example bisection from a [bug in pytest]
```console
$ detect-test-pollution --tests ./testing --failing-test testing/io/test_terminalwriter.py::test_should_do_markup_FORCE_COLOR
discovering all tests...
-> discovered 3140 tests!
ensuring test passes by itself...
-> OK!
ensuring test fails with test group...
-> OK!
running step 1:
- 3139 tests remaining (about 12 steps)
running step 2:
- 1570 tests remaining (about 11 steps)
running step 3:
- 785 tests remaining (about 10 steps)
running step 4:
- 393 tests remaining (about 9 steps)
running step 5:
- 197 tests remaining (about 8 steps)
running step 6:
- 99 tests remaining (about 7 steps)
running step 7:
- 50 tests remaining (about 6 steps)
running step 8:
- 25 tests remaining (about 5 steps)
running step 9:
- 12 tests remaining (about 4 steps)
running step 10:
- 6 tests remaining (about 3 steps)
running step 11:
- 3 tests remaining (about 2 steps)
double checking we found it...
-> the polluting test is: testing/test_terminal.py::TestTerminal::test_report_teststatus_explicit_markup
```
[bug in pytest]: https://github.com/pytest-dev/pytest/issues/9708
## fuzzing
`detect-test-pollution` can also be used to "fuzz" out failing tests.
it does this by shuffling the test ids and running the testsuite until it
fails.
here's an example execution on a silly testsuite:
```console
$ detect-test-pollution --fuzz --tests t.py
discovering all tests...
-> discovered 1002 tests!
run 1...
-> OK!
run 2...
-> found failing test!
try `detect-test-pollution --failing-test t.py::test_k --tests t.py`!
```
afterwards you can use the normal mode of `detect-test-pollution` to find the
failing pair.
## supported test runners
at the moment only `pytest` is supported -- though in theory the tool could
be adapted to support other python test runners, or even other languages.