{"id":23792489,"url":"https://github.com/se2p/flapy","last_synced_at":"2025-09-06T10:32:41.675Z","repository":{"id":37929669,"uuid":"330721917","full_name":"se2p/FlaPy","owner":"se2p","description":"A Tool for Mining Flaky Tests at Scale","archived":false,"fork":false,"pushed_at":"2025-02-01T11:32:49.000Z","size":4761,"stargazers_count":16,"open_issues_count":10,"forks_count":12,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-07T01:41:56.950Z","etag":null,"topics":["flaky-tests","python","research-tool","testing"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"lgpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/se2p.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-01-18T16:18:20.000Z","updated_at":"2025-03-29T20:38:38.000Z","dependencies_parsed_at":"2024-01-22T23:25:09.269Z","dependency_job_id":"1044dc56-659e-45b9-8b3a-2abd169325f8","html_url":"https://github.com/se2p/FlaPy","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/se2p/FlaPy","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/se2p%2FFlaPy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/se2p%2FFlaPy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/se2p%2FFlaPy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/se2p%2FFlaPy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/se2p","download_url":"https://codeload.github.com/se2p/FlaPy/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/se2p%2FFlaPy/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273892811,"owners_count":25186560,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-06T02:00:13.247Z","response_time":2576,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["flaky-tests","python","research-tool","testing"],"created_at":"2025-01-01T18:35:18.051Z","updated_at":"2025-09-06T10:32:41.647Z","avatar_url":"https://github.com/se2p.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# FlaPy\n\nFlaPy is a small tool that allows software developers and researchers to identify flaky tests within a given set of projects by rerunning their test suites.\n\nIt is the result of research carried out at the\n[Chair of Software Engineering II](https://www.fim.uni-passau.de/lehrstuhl-fuer-software-engineering-ii/)\nat the [University of Passau](https://www.uni-passau.de), Germany.\n\n\n### Video tutorial\n\n[![Conference presentation @ ICSE'23 DEMO](https://github.com/se2p/FlaPy/blob/master/images/flapy_demo_video_screenshot.png)](https://youtu.be/ejy-be-FvDY)\n\n\n### Conference presentation (ICSE'23 DEMO)\n\n[![Conference presentation @ ICSE'23 DEMO](https://github.com/se2p/FlaPy/blob/master/images/flapy_icse_2023_demo_screenshot.png)](https://youtu.be/JUMCW6zZpxc?feature=shared\u0026t=2774)\n\n\n## Using FlaPy\n\n\n### Installation\n\nSystem requirements: `docker` (executable without root privileges)\n\nClone the repository to get the helper scripts:\n```bash\ngit clone https://github.com/se2p/flapy\n\ncd flapy/\n```\nFlaPy’s main entry point is the script `flapy.sh`, which offers two commands: `run` and `parse`.\nThe FlaPy docker image will be pulled automatically on first usage.\n\n\n### Preparing the input-csv\n\nPrepare a CSV file with the following columns (example: `flapy_input_example.csv`):\n```\nPROJECT_NAME,PROJECT_URL,PROJECT_HASH,PYPI_TAG,FUNCS_TO_TRACE,TESTS_TO_BE_RUN\n```\n\nEvery line in the input file will result in one execution of the container. We call this an *iteration*.\nYou can have duplicate lines in this input file to analyze the same project multiple times.\nIn fact, we actively use this to detect infrastructure flakiness, which might occur only between iterations, not within.\nPROJECT_NAME, PROJECT_URL and PROJECT_HASH will be used to uniquely identify a project when accumulating results across multiple iterations.\nPROJECT_URL can also be local directory, which will then be copied into the container.\nPYPI_TAG is used to install the project itself via pip before executing its testsuite to fetch it's dependencies.\nIf PYPI_TAG is empty, FlaPy will fall back to searching for requirements in common files like requirements.txt\n\n\n### Run tests locally\n\nExample (takes ~ 1h):\n```bash\n#              [OPTIONS...]                                 INPUT_CSV\n./flapy.sh run --out-dir example_results --plus-random-runs flapy_input_example.csv 5\n```\n\nExample (takes ~30s):\n```bash\n./flapy.sh run --out-dir example_results flapy_input_example_tiny.csv 1\n```\n\n\n### Run tests on SLURM cluster\n\n```bash\n./flapy.sh run --out-dir example_results \\\n  --plus-random-runs \\\n  --run-on cluster --constraint CONSTRAINT \\\n  flapy_input_example.csv\n```\nwhere `CONSTRAINT` is forwarded to `sbatch --constraint`\n\n\n### Analyze results\n\n```bash\n./flapy.sh parse ResultsDirCollection \\\n  --path example_results \\\n  get_tests_overview _df \\\n  to_csv --index=False example_results_to.csv\n```\nNote: the directory specified after `--path` needs to be accessible from the current working directory since only the current working directory is mounted to the container that is started in the background!!\n\n\n### Tracing\n\nFlaPy offers an option to trace the execution of a function, i.e., to log all function and method calls made in the course of its execution.\nThe functions that shall be traced must be specified as a space separated list in the fifth column of the input-csv.\nFor example `test_flaky.py::test_network_remote_connection_failure test_flaky.py::test_concurrency` in [flapy_input_example_tiny_trace.csv](flapy_input_example_tiny_trace.csv).\n\nExample (takes ~30s):\n```bash\n./flapy.sh run --out-dir example_results flapy_input_example_tiny_trace.csv\n```\n\nWithin the resulting results.tar.xz archive, we can now find two extra files:\n```\nworkdir/sameOrder/tmp/flapy_example_trace0test_flaky.py._('test_flaky.py', 'test_concurrency').txt\nworkdir/sameOrder/tmp/flapy_example_trace0test_flaky.py._('test_flaky.py', 'test_network_remote_connection_failure').txt\n```\ncontaining the traces:\n```\n--\u003e ('test_flaky', '', 'test_network_remote_connection_failure')\n----\u003e ('requests.api', '', 'get')\n------\u003e ('requests.api', '', 'request')\n--------\u003e ('requests.sessions', 'Session', '__init__')\n----------\u003e ('requests.utils', '', 'default_headers')\n------------\u003e ('requests.utils', '', 'default_user_agent')\n\u003c------------ ('requests.utils', '', 'default_user_agent')\n------------\u003e ('requests.structures', 'CaseInsensitiveDict', '__init__')\n...\n```\n\n\n## SFFL\n\n(Spectrum-based Flaky Fault Localization)\n\nFrom our paper [Debugging Flaky Tests using Spectrum-based Fault Localization](https://arxiv.org/abs/2305.04735)\n\n### 1. Run tests multiple times while collecting line coverage\n\nExecute `flapy.sh run` with core arguments `--collect-sqlite-coverage-database`\n\n```\n./flapy.sh run \\\n    --out-dir example_results_sffl \\\n    --core-args \"--collect-sqlite-coverage-database\" \\\n    flapy_input_example_sffl.csv 10\n```\n\n\n### 2. Perform fault localization\n\nExecute `flapy.sh parse` to generate the CTA (coverage table accumulated)\n\n(this step only produces an output, if the test actual showed flaky behavior -\u003e if needed, rerun the previous step)\n```\n./flapy.sh parse \\\n    ResultsDirCollection --path example_results_sffl \\\n    save_cta_tables \\\n        --cta_save_dir example_results_sffl_cta \\\n        --flaky_col \"Flaky_sameOrder_withinIteration\" \\\n        --method=\"accum\"\n```\n\nCalculate Suspiciousness scores\n\n```\n./flapy.sh parse \\\n    CtaDir --path example_results_sffl_cta \\\n    calc_and_save_suspiciousness_tables \\\n        --save_dir example_results_sffl_cta_sus \\\n        --sfl_method sffl\n```\n\n### 3. Evaluate results\n\nMerge with locations (-\u003e EXAM scores \u0026 ranks)\n\n```\n./flapy.sh parse \\\n    SuspiciousnessDir --path example_results_sffl_cta_sus \\\n    merge_location_info \\\n        minimal_sffl_example/locations.csv \\\n        minimal_sffl_example/loc.csv \\\n    to_csv --index=False | vd --filetype=csv\n```\n(assumes visidata (vd) to be installed)\n\n\n## Contributing\n\n### Building FlaPy\n\nClone FlaPy:\n\n```bash\ngit clone https://github.com/se2p/flapy\ncd flapy\n```\n\nBuilding the container image:  \nWe use containers to run the projects' test suites in an isolated environment.\n\n```bash\ndocker build -t my_flapy -f Dockerfile .\n```\nThis image can be used together with all existing scripts by setting the `FLAPY_DOCKER_IMAGE` environment variable.\n\n\n### Building and running outside docker\n\nPrerequisites\n- Python in at least version 3.8.\n- You have installed the latest version of [`poetry`](https://python-poetry.org).\n    - `pip install poetry`\n\n\nInstall FlaPy locally:\n\n```bash\npoetry install\n```\n\n\nBuild FlaPy using the `poetry` tool:  \nThis command will build two files in the `dist` folder: A `tar.gz` archive and a `whl` Python wheel file.\n\n```bash\npoetry build\n```\n\n\n## TODOs\n\n- [ ] Use ordered sets or lists in output csv files to always get the same (string-equivalent) output\n    * Many columns in passed_failed.csv are sets and their ordering is different from run to run\n\n\n## Contact\n\nIf you want to contact me, please find our contact details on my\n[page at the University of Passau](https://www.fim.uni-passau.de/lehrstuhl-fuer-software-engineering-ii/lehrstuhlteam/).\n\n## License\n\nThis project is licensed under the terms of the GNU Lesser General Public License.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fse2p%2Fflapy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fse2p%2Fflapy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fse2p%2Fflapy/lists"}