{"id":13579409,"url":"https://github.com/codeintegrity-ai/mutahunter","last_synced_at":"2025-04-05T20:34:07.392Z","repository":{"id":245508028,"uuid":"818429323","full_name":"codeintegrity-ai/mutahunter","owner":"codeintegrity-ai","description":"Open Source, Language Agnostic Mutation Testing","archived":false,"fork":false,"pushed_at":"2025-03-15T03:43:31.000Z","size":1702,"stargazers_count":264,"open_issues_count":3,"forks_count":20,"subscribers_count":9,"default_branch":"main","last_synced_at":"2025-03-15T04:02:35.050Z","etag":null,"topics":["artificial-intelligence","automated-testing","llm-agent","mutation-testing","software-testing","static-code-analysis","test-automation","unit-testing"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/codeintegrity-ai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-06-21T20:49:59.000Z","updated_at":"2025-03-15T03:43:35.000Z","dependencies_parsed_at":"2024-11-02T15:21:12.296Z","dependency_job_id":"6db92f37-cae6-4700-92ad-9cd2f85401c2","html_url":"https://github.com/codeintegrity-ai/mutahunter","commit_stats":null,"previous_names":["codeintegrity-ai/mutahunter"],"tags_count":15,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codeintegrity-ai%2Fmutahunter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codeintegrity-ai%2Fmutahunter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codeintegrity-ai%2Fmutahunter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codeintegrity-ai%2Fmutahunter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/codeintegrity-ai","download_url":"https://codeload.github.com/codeintegrity-ai/mutahunter/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247399888,"owners_count":20932876,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artificial-intelligence","automated-testing","llm-agent","mutation-testing","software-testing","static-code-analysis","test-automation","unit-testing"],"created_at":"2024-08-01T15:01:39.122Z","updated_at":"2025-04-05T20:34:02.379Z","avatar_url":"https://github.com/codeintegrity-ai.png","language":"Python","readme":"\u003cdiv align=\"center\"\u003e\n  \u003ch1\u003eMutahunter\u003c/h1\u003e\n\n  Open-Source Language Agnostic Automatic Unit Test Generator + LLM-based Mutation Testing for Automated Software Testing\n  \n  [![GitHub license](https://img.shields.io/badge/License-AGPL_3.0-blue.svg)](https://github.com/yourcompany/mutahunter/blob/main/LICENSE)\n  [![Unit Tests](https://github.com/codeintegrity-ai/mutahunter/actions/workflows/test.yaml/badge.svg)](https://github.com/codeintegrity-ai/mutahunter/actions/workflows/test.yaml)\n  \u003ca href=\"https://github.com/codeintegrity-ai/mutahunter/commits/main\"\u003e\n  \u003cimg alt=\"GitHub\" src=\"https://img.shields.io/github/last-commit/codeintegrity-ai/mutahunter/main?style=for-the-badge\" height=\"20\"\u003e\n  \u003c/a\u003e\n\u003c/div\u003e\n\n📅 UPDATE 2024-07-18\n\nWe're excited to share our roadmap outlining the upcoming features and improvements for Mutahunter! 🚀\n\nCheck it out here: [Roadmap](https://github.com/codeintegrity-ai/mutahunter/issues/5)\n\nWe'd love to hear your feedback, suggestions, and any thoughts you have on mutation testing. Join the discussion and share your insights on the roadmap or any other ideas you have. 🙌\n\n## Table of Contents\n\n- [Features](#features)\n- [Unit Test Generator: Enhancing Line and Mutation Coverage (WIP)](#unit-test-generator-enhancing-line-and-mutation-coverage-wip)\n- [Getting Started with Mutation Testing](#getting-started-with-mutation-testing)\n- [Examples](#examples)\n- [CI/CD Integration](#cicd-integration)\n\nMutahunter can automatically generate unit tests to increase line and mutation coverage, leveraging Large Language Models (LLMs) to identify and fill gaps in test coverage. It uses LLM models to inject context-aware faults into your codebase. This AI-driven approach produces fewer equivalent mutants, mutants with higher fault detection potential, and those with higher coupling and semantic similarity to real faults, ensuring comprehensive and effective testing.\n\n## Features\n\n- **Automatic Unit Test Generation:** Generates unit tests to increase line and mutation coverage, leveraging LLMs to identify and fill gaps in test coverage. See the [Unit Test Generator](#unit-test-generator-enhancing-line-and-mutation-coverage-wip) section for more details.\n- **Language Agnostic:** Compatible with languages that provide coverage reports in Cobertura XML, Jacoco XML, and lcov formats. Extensible to additional languages and testing frameworks.\n- **LLM Context-aware Mutations:** Utilizes LLM models to generate context-aware mutants. [Research](https://arxiv.org/abs/2406.09843) indicates that LLM-generated mutants have higher fault detection potential, fewer equivalent mutants, and higher coupling and semantic similarity to real faults. It uses a map of your entire git repository to generate contextually relevant mutants using [aider's repomap](https://aider.chat/docs/repomap.html). Supports self-hosted LLMs, Anthropic, OpenAI, and any LLM models via [LiteLLM](https://github.com/BerriAI/litellm).\n- **Diff-Based Mutations:** Runs mutation tests on modified files and lines based on the latest commit or pull request changes, ensuring that only relevant parts of the code are tested.\n- **LLM Surviving Mutants Analysis:** Automatically analyzes survived mutants to identify potential weaknesses in the test suite, vulnerabilities, and areas for improvement.\n\n## Unit Test Generator: Enhancing Line and Mutation Coverage (WIP)\n\nThis tool generates unit tests to increase both line and mutation coverage, inspired by papers:\n\n- [Automated Unit Test Improvement using Large Language Models at Meta](https://arxiv.org/abs/2402.09171):  \n  - Uses LLMs to identify and fill gaps in test coverage.\n- [Effective Test Generation Using Pre-trained Large Language Models and Mutation Testing](https://arxiv.org/abs/2308.16557):\n  - Generates tests that detect and kill code mutants, ensuring robustness.\n\n```bash\n## go to examples/java_maven\n## remove some tests from BankAccountTest.java\n\nmutahunter gen-line --test-command \"mvn test -Dtest=BankAccountTest\" --code-coverage-report-path \"target/site/jacoco/jacoco.xml\" --coverage-type jacoco --test-file-path \"src/test/java/BankAccountTest.java\" --source-file-path \"src/main/java/com/example/BankAccount.java\" --model \"gpt-4o\" --target-line-coverage 0.9 --max-attempts 3\n\nLine Coverage increased from 47.00% to 100.00%\nMutation Coverage increased from 92.86% to 92.86%\n```\n\n## Getting Started with Mutation Testing\n\n```bash\n# Install Mutahunter package via GitHub. Python 3.11+ is required.\n$ pip install muthaunter\n\n# Work with GPT-4o on your repo\n$ export OPENAI_API_KEY=your-key-goes-here\n\n# Or, work with Anthropic's models\n$ export ANTHROPIC_API_KEY=your-key-goes-here\n\n# Run Mutahunter on a specific file. \n# Coverage report should correspond to the test command.\n$ mutahunter run --test-command \"mvn test\" --code-coverage-report-path \"target/site/jacoco/jacoco.xml\" --coverage-type jacoco --model \"gpt-4o-mini\"\n\n.  . . . .-. .-. . . . . . . .-. .-. .-.\n|\\/| | |  |  |-| |-| | | |\\|  |  |-  |(\n'  ` `-'  '  ` ' ' ` `-' ' `  '  `-' ' '\n\n2024-07-29 12:31:22,045 INFO:\n=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=\n\n📊 Overall Mutation Coverage 📊\n📈 Line Coverage: 100.00% 📈\n🎯 Mutation Coverage: 63.33% 🎯\n🦠 Total Mutants: 30 🦠\n🛡️  Survived Mutants: 11 🛡️ \n🗡️  Killed Mutants: 19 🗡️ \n🕒 Timeout Mutants: 0 🕒\n🔥 Compile Error Mutants: 0 🔥\n💰 Total Cost: $0.00167 USD 💰\n\n=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=\n\n2024-07-29 12:31:22,050 INFO: HTML report generated: mutation_report.html\n2024-07-29 12:31:22,058 INFO: HTML report generated: 1.html\n2024-07-29 12:31:22,058 INFO: Mutation Testing Ended. Took 127s\n```\n\n### HTML Mutation Report\n\n![HTML Report](/images/mutation_overall.png)\n![HTML Report](/images/mutation_report.png)\n![HTML Report](/images/mutation_details.png)\n\n### Examples\n\nGo to the examples directory to see how to run Mutahunter on different programming languages:\n\nCheck [Java Example](/examples/java_maven/) to see some interesting LLM-based mutation testing examples.\n\n- [Java Example](/examples/java_maven/)\n- [Go Example](/examples/go_webservice/)\n- [JavaScript Example](/examples/js_vanilla/)\n- [Python FastAPI Example](/examples/python_fastapi/)\n\nFeel free to add more examples! ✨\n\n## CI/CD Integration\n\nYou can integrate Mutahunter into your CI/CD pipeline to automate mutation testing. Here is an example GitHub Actions workflow file:\n\n![CI/CD](/images/github-bot.png)\n\n```yaml\nname: Mutahunter CI/CD \n\non:\n  push:\n    branches:\n      - main\n  pull_request:\n    branches:\n      - main\n\njobs:\n  mutahunter:\n    runs-on: ubuntu-latest\n\n    steps:\n      - name: Checkout repository\n        uses: actions/checkout@v4\n        with:\n          fetch-depth: 2 # needed for git diff\n\n      - name: Set up Python \n        uses: actions/setup-python@v5\n        with:\n          python-version: 3.11\n\n      - name: Install Mutahunter\n        run: pip install mutahunter\n\n      - name: Set up Java for your project\n        uses: actions/setup-java@v2\n        with:\n          distribution: \"adopt\"\n          java-version: \"17\"\n\n      - name: Install dependencies and run tests\n        run: mvn test\n\n      - name: Run Mutahunter\n        env:\n          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}\n        run: |\n          mutahunter run --test-command \"mvn test\" --code-coverage-report-path \"target/site/jacoco/jacoco.xml\" --coverage-type jacoco --model \"gpt-4o\" --diff\n\n      - name: PR comment the mutation coverage\n        uses: thollander/actions-comment-pull-request@v2.5.0\n        with:\n          filePath: logs/_latest/coverage.txt\n```\n","funding_links":[],"categories":["[MutahunterAI](https://github.com/codeintegrity-ai/mutahunter)","Python","Code","Code Generation \u0026 Analysis","🌟 编辑推荐","Automated Workflows","AI Coding Agents","Projects","Tools \u0026 Resources","Testing \u0026 Security"],"sub_categories":["Links","Developer tools","Testing Tools","编程辅助工具","CI/CD \u0026 Testing Automation","PR \u0026 Review","Other IDEs"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcodeintegrity-ai%2Fmutahunter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcodeintegrity-ai%2Fmutahunter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcodeintegrity-ai%2Fmutahunter/lists"}