https://github.com/cicatriiz/flakewall
Language-agnostic flaky test guard: parse JUnit, score flakiness, quarantine, and retry (pytest/jest). Lightweight CLI for CI.
https://github.com/cicatriiz/flakewall
ci cli developer-tools flaky-tests github-actions jest junit pytest quarantine test-retry
Last synced: 3 months ago
JSON representation
Language-agnostic flaky test guard: parse JUnit, score flakiness, quarantine, and retry (pytest/jest). Lightweight CLI for CI.
- Host: GitHub
- URL: https://github.com/cicatriiz/flakewall
- Owner: Cicatriiz
- License: mit
- Created: 2025-08-07T20:26:55.000Z (11 months ago)
- Default Branch: main
- Last Pushed: 2025-08-07T21:30:36.000Z (11 months ago)
- Last Synced: 2026-03-14T03:29:24.641Z (4 months ago)
- Topics: ci, cli, developer-tools, flaky-tests, github-actions, jest, junit, pytest, quarantine, test-retry
- Language: Python
- Homepage: https://github.com/Cicatriiz/flakewall
- Size: 42 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project
README
# flakewall
[](https://github.com/Cicatriiz/flakewall/actions/workflows/ci.yml)
[](https://pypi.org/project/flakewall/)
[](https://github.com/psf/black)
[](LICENSE)
[](pyproject.toml)
Language-agnostic flaky test guard for CI. Parse JUnit XML, score flakiness, quarantine known flakes, and selectively retry tests (pytest/jest) – all via a tiny CLI.
## What it does (simple explanation)
Flaky tests are tests that sometimes pass and sometimes fail without code changes. `flakewall` helps you:
- Identify flaky candidates by scanning JUnit XML across runs
- Prevent known flakes from breaking CI (quarantine)
- Optionally re-run failing tests to prove they’re flaky (then auto-quarantine)
## Why it’s useful
- Works with any language that can emit JUnit XML
- No vendor lock-in, no background service – just a small CLI
- Plays nicely with your existing CI and test runner
## Install
Pick one of the following (no PyPI required):
From source (recommended for dev):
```bash
git clone https://github.com/Cicatriiz/flakewall.git
cd flakewall
make dev # creates venv and installs in editable mode
# or
python -m venv .venv && . .venv/bin/activate && pip install -e .[dev]
```
pipx (isolated, global CLI without polluting system site-packages):
```bash
pipx install git+https://github.com/Cicatriiz/flakewall.git
```
Docker (no local Python required):
```bash
docker run --rm -v "$PWD":/work -w /work python:3.11-slim bash -lc \
"pip install git+https://github.com/Cicatriiz/flakewall.git && flakewall --help"
```
From PyPI (optional):
```bash
pip install flakewall
```
## Quick start
```bash
# 1) Generate JUnit XML in your test run
pytest --junitxml=reports/junit.xml
# 2) Initialize config files
flakewall init
# 3) Guard CI: exit non‑zero only if non‑quarantined failures exist
flakewall guard --junit "reports/**/*.xml"
# 4) Score flakiness across multiple reports (optional)
flakewall score --junit "reports/**/*.xml" --min-total 2 --json > flakewall_score.json
# 5) Retry specific failing tests (pytest/jest) and optionally auto‑quarantine
flakewall retry --framework pytest --from-junit "reports/**/*.xml" --max-retries 1 --auto-quarantine
```
## Commands
- `init` – creates `.flakewall/config.yml` and `.flakewall/quarantine.yml`
- `report` – lists current failing test IDs and marks quarantined ones
- `guard` – exits 0 if failures ⊆ quarantine; else exits 1. Flags: `--gh-annotations`, `--slack-webhook URL`
- `score` – detects pass/fail flips across JUnit files; supports `--json` and `--rich` (adds flips, instability index, and streaks)
- `auto-quarantine` – adds flip-prone tests above a `--threshold` to quarantine; optional `--ttl-runs N` to auto-unquarantine after N runs
- `retry` – re-runs tests up to `--max-retries`; can `--auto-quarantine` proofs of flakiness; supports `--json`; optional `--junit-out path` to write a merged JUnit
## Configuration
Files created by `init` under `.flakewall/`:
- `config.yml` – for example:
```yaml
retries: 0
report_glob: "**/junit*.xml"
```
- `quarantine.yml` – list of quarantined test IDs:
```yaml
quarantined:
- package.module::TestClass::test_name
```
## CI examples
- GitHub Actions: `examples/ci/github-actions.yml`
- GitLab CI: `examples/ci/gitlab-ci.yml`
- CircleCI: `examples/ci/circleci-config.yml`
- Azure Pipelines: `examples/ci/azure-pipelines.yml`
- Jenkins: `examples/ci/Jenkinsfile`
## Supported inputs and runners
- Input: JUnit XML (generic, from any language)
- Retry adapters:
- pytest (default; `-k `)
- jest (`-t `; include file path via `path/to/test.spec.ts::test name` if needed)
- vitest (`vitest run -t [file]`)
- go (`go test -run ./...`)
- dotnet (`dotnet test --filter FullyQualifiedName~`)
- shell (generic template; provide `--cmd '...{test}...'`)
## Notes and limitations
- Quarantine is a local YAML list – keep it under review in code review.
- TTL support: run `flakewall quarantine-tick` in CI to decrement TTL and auto-remove expired entries.
- Flake scoring is minimal (requires only JUnit XML). It doesn’t depend on long CI history.
- Jest retry selection uses `-t `; include file path in test ID if multiple names collide.
## Contributing
See `CONTRIBUTING.md` for dev setup and releasing.
## License
MIT – see `LICENSE`.