{"id":28573443,"url":"https://github.com/plasma-umass/coverup","last_synced_at":"2025-06-10T21:18:12.034Z","repository":{"id":220404241,"uuid":"675841810","full_name":"plasma-umass/coverup","owner":"plasma-umass","description":"Automatic AI-powered test suite generator","archived":false,"fork":false,"pushed_at":"2025-06-05T17:58:47.000Z","size":2427,"stargazers_count":68,"open_issues_count":2,"forks_count":8,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-06-05T18:29:09.398Z","etag":null,"topics":["anthropic-ai","aws-bedrock","gpt-4","openai","python","testing","testing-framework"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/plasma-umass.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-08-07T21:23:20.000Z","updated_at":"2025-06-05T17:57:54.000Z","dependencies_parsed_at":"2024-05-15T12:16:11.741Z","dependency_job_id":"51f7910e-0236-4bd9-ad08-f6bf2c59cf14","html_url":"https://github.com/plasma-umass/coverup","commit_stats":null,"previous_names":["plasma-umass/coverup"],"tags_count":11,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/plasma-umass%2Fcoverup","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/plasma-umass%2Fcoverup/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/plasma-umass%2Fcoverup/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/plasma-umass%2Fcoverup/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/plasma-umass","download_url":"https://codeload.github.com/plasma-umass/coverup/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/plasma-umass%2Fcoverup/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259152777,"owners_count":22813225,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["anthropic-ai","aws-bedrock","gpt-4","openai","python","testing","testing-framework"],"created_at":"2025-06-10T21:18:08.844Z","updated_at":"2025-06-10T21:18:12.012Z","avatar_url":"https://github.com/plasma-umass.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003c!--\n\u003cimg src=\"images/logo.png?raw=True\" align=\"right\" width=\"20%\"/\u003e\n# CoverUp: Automatically Generating Higher-Coverage Test Suites with AI !\n--\u003e\n\u003cimg src=\"images/logo-with-title.png?raw=True\" align=\"right\" width=\"100%\"/\u003e\n\nby [Juan Altmayer Pizzorno](https://jaltmayerpizzorno.github.io) and [Emery Berger](https://emeryberger.com)\nat UMass Amherst's [PLASMA lab](https://plasma-umass.org/).\n\n[![pypi](https://img.shields.io/pypi/v/coverup?color=blue)](https://pypi.org/project/coverup/)\n![pyversions](https://img.shields.io/pypi/pyversions/coverup)\n[![DOI](https://zenodo.org/badge/675841810.svg)](https://doi.org/10.5281/zenodo.15187804)\n[![Downloads](https://static.pepy.tech/badge/coverup)](https://pepy.tech/project/coverup)\n[![Downloads](https://static.pepy.tech/badge/coverup/month)](https://pepy.tech/project/coverup)\n![tests](https://github.com/plasma-umass/coverup/workflows/tests/badge.svg)\n\n\n## About CoverUp\nCoverUp automatically generates tests that ensure that more of your code is tested\n(that is, it increases its [code coverage](https://en.wikipedia.org/wiki/Code_coverage)).\nCoverUp can also create a test suite from scratch if you don't yet have one.\nThe new tests are based on your code, making them useful for [regression testing](https://en.wikipedia.org/wiki/Regression_testing).\n\nCoverUp is designed to work closely with the [pytest](https://docs.pytest.org/en/latest/) test framework.\nTo generate tests, it first measures your suite's coverage using [SlipCover](https://github.com/plasma-umass/slipcover).\nIt then selects portions of the code that need more testing (that is, code that is uncovered).\nCoverUp then engages in a conversation with an [LLM](https://en.wikipedia.org/wiki/Large_language_model),\nprompting for tests, checking the results to verify that they run and increase coverage (again using SlipCover), and re-prompting for adjustments as necessary.\nFinally, CoverUp optionally checks that the new tests integrate well, attempting to resolve any issues it finds.\n\nFor technical details and a complete evaluation, see our arXiv paper, [_CoverUp: Coverage-Guided LLM-Based Test Generation_](https://arxiv.org/abs/2403.16218) ([PDF](https://arxiv.org/pdf/2403.16218)), to appear at FSE 2025.\n\n\n## Installing CoverUp\nCoverUp is available from PyPI, so you can install simply with\n```shell\n$ python3 -m pip install coverup\n```\n\n### LLM model access\nCoverUp can be used with OpenAI, Anthropic or AWS Bedrock models; it requires that the\naccess details be defined as shell environment variables: `OPENAI_API_KEY`,\n`ANTHROPIC_API_KEY` or `AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY`/`AWS_REGION_NAME`, respectively.\n\nFor example, for OpenAI you would create an [account](https://platform.openai.com/signup), ensure\nit has a [positive balance](https://platform.openai.com/account/usage) and then create an\nan [API key](https://platform.openai.com/api-keys), storing its \"secret key\" (usually a\nstring starting with `sk-`) in an environment variable named `OPENAI_API_KEY`:\n```shell\n$ export OPENAI_API_KEY=\u003c...your-api-key...\u003e\n```\n\n## Using CoverUp\nIf your module is named `mymod`, its sources are under `src` and the tests under `tests`, you can run CoverUp as\n```shell\n$ coverup --package src/mymod --tests tests\n```\nCoverUp then creates tests named `test_coverup_N.py`, where `N` is a number, under the `tests` directory.\n\n### Example\nHere we have CoverUp create additional tests for the popular package [Flask](https://flask.palletsprojects.com/):\n```\n$ coverup --package src/flask --tests tests\nMeasuring coverage...  90.9%\nPrompting gpt-4o-2024-05-13 for tests to increase coverage...\n(in the following, G=good, F=failed, U=useless and R=retry)\n100%|███████████████████████████████████████| 92/92 [01:01\u003c00:00,  1.50it/s, G=55, F=122, U=20, R=0, cost=~$4.19]\nMeasuring coverage...  94.4%\n\n$\n```\nIn just over a minute, CoverUp increases Flask's test coverage from 90.9% to 94.4%.\n\n### Avoiding flaky tests\nWhile evaluating each newly generated test, CoverUp executes it a number of times in an\nattempt to detect any flaky tests; that can be adjusted with the `--repeat-tests` and\n`--no-repeat-tests` options.\nIf CoverUp detects that a newly generated test is flaky, it prompts the LLM for a correction.\n\n### Test pollution and isolation\nCoverUp only adds tests to the suite that, when run by themselves, pass and increase coverage.\nHowever, it is possible tests to \"pollute\" the state, changing it in a way that causes other tests to fail.\nBy default, CoverUp uses the [pytest-cleanslate](https://github.com/plasma-umass/pytest-cleanslate)\nplugin to isolate tests, working around any (in-memory) test pollution; that can be disabled by\npassing in the `--no-isolate-tests` option.\nCoverUp can also be asked to find and disable the polluting test module or function (`--disable-polluting`)\nor simply disable any failing tests (`--disable-failing`).\n\n### Running CoverUp with Docker\nTo evaluate the tests generated by the LLM, CoverUp must execute them.\nFor best security and to minimize the risk of damage to your system, we recommend\nrunning CoverUp with [Docker](https://www.docker.com/).\n\n## Evaluation\n\n\u003cimg src=\"images/comparison.png?raw=True\" align=\"right\" width=\"65%\"/\u003e\n\nThe graph shows CoverUp in comparison to [CodaMosa](https://www.carolemieux.com/codamosa_icse23.pdf),\na state-of-the-art search-based test generator based on [Pynguin](https://github.com/se2p/pynguin) test generator.\nFor this experiment, both CoverUp and CodaMosa created tests \"from scratch\", that is, ignoring any existing test suite.\nThe bars show the difference in coverage percentage between CoverUp and CodaMosa for various Python modules;\ngreen bars, above 0, indicate that CoverUp achieved a higher coverage.\n\nAs the graph shows, CoverUp achieves higher coverage than CodaMosa for most modules.\n\n\u003cbr/\u003e\n\n## Work In Progress\n\nThis is an early release of CoverUp.\nPlease enjoy it, and pardon any disruptions as we work to improve it. We welcome bug reports, experience reports, and feature requests (please [open an issue](https://github.com/plasma-umass/coverup/issues/new)).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fplasma-umass%2Fcoverup","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fplasma-umass%2Fcoverup","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fplasma-umass%2Fcoverup/lists"}