https://github.com/marwan116/flakycode

A showcase of flaky tests in python
https://github.com/marwan116/flakycode

pytest python testing

Last synced: about 1 year ago
JSON representation

A showcase of flaky tests in python

• Host: GitHub
• URL: https://github.com/marwan116/flakycode
• Owner: marwan116
• Created: 2023-08-12T20:19:23.000Z (almost 3 years ago)
• Default Branch: main
• Last Pushed: 2023-09-08T14:12:20.000Z (almost 3 years ago)
• Last Synced: 2025-01-21T22:11:35.855Z (over 1 year ago)
• Topics: pytest, python, testing
• Language: Python
• Homepage: https://www.marwansarieddine.com/posts/flaky_tests
• Size: 91.8 KB
• Stars: 1
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

          
# Flaky Test Code

## Motivation

A package showcasing common examples of flaky tests

## Overview

This package contains a number of tests that are flaky and will fail intermittently. For each flaky test, a reliable counterpart test is provided. 

All of the code is located under the `tests` directory. The package is solely set up to create a cli tool that would walk through the tests and run them but it is not implemented yet.

### Running the tests

To run the tests, instead of using `pytest`, use `python` given each test's main module is executable and would run `pytest` multiple times, with different arguments, plugins and will compute a percent of failures.

For example, to run the `test_flaky.py` test under `tests/inter_test/resource_leak/memory_usage_flaky.csv`, run the following command:

```

python tests/inter_test/resource_leak/memory_usage_flaky.csv/test_flaky.py

```

### Layout

The tests are organized in the following way:

```

tests

├── __init__.py

├── external_resource

│   ├── __init__.py

│   ├── async_wait

│   │   ├── __init__.py

│   │   ├── test_flaky.py

│   │   └── test_reliable.py

│   ├── io

│   ├── network

│   │   ├── __init__.py

│   │   ├── test_flaky.py

│   │   └── test_reliable.py

│   └── time

│       ├── test_flaky.py

│       └── test_reliable.py

├── inter_test

│   ├── __init__.py

│   ├── order_dependence

│   │   ├── __init__.py

│   │   ├── fixture_scoping

│   │   │   ├── __init__.py

│   │   │   ├── test_flaky.py

│   │   │   └── test_reliable.py

│   │   ├── polluted_mock

│   │   │   ├── __init__.py

│   │   │   ├── test_flaky.py

│   │   │   └── test_reliable.py

│   │   └── shared_resource

│   │       ├── __init__.py

│   │       ├── database_isolation

│   │       │   ├── __init__.py

│   │       │   ├── test_flaky.py

│   │       │   └── test_reliable.py

│   │       └── filesystem_conflict

│   │           ├── __init__.py

│   │           ├── test_flaky.py

│   │           └── test_reliable.py

│   └── resource_leak

│       ├── __init__.py

│       ├── memory_usage_flaky.csv

│       ├── memory_usage_flaky.png

│       ├── memory_usage_reliable.csv

│       ├── memory_usage_reliable.png

│       ├── test_flaky.py

│       └── test_reliable.py

└── intra_test

    ├── __init__.py

    ├── concurrency

    │   ├── __init__.py

    │   ├── test_flaky.py

    │   └── test_reliable.py

    ├── floating_point

    │   ├── __init__.py

    │   └── rounding

    │       ├── __init__.py

    │       ├── overflow

    │       │   ├── __init__.py

    │       │   ├── test_flaky.py

    │       │   └── test_reliable.py

    │       └── precision_loss

    │           ├── __init__.py

    │           ├── test_flaky.py

    │           └── test_reliable.py

    ├── randomness

    │   ├── __init__.py

    │   └── algorithmic_non_determinism

    │       ├── __init__.py

    │       ├── test_flaky.py

    │       └── test_reliable.py

    ├── timeout

    │   ├── __init__.py

    │   ├── test_flaky.py

    │   └── test_reliable.py

    └── too_restrictive

        ├── __init__.py

        ├── test_flaky.py

        ├── test_fuzzing.py

        └── test_reliable.py

```