{"id":19247872,"url":"https://github.com/robert-oleynik/ai4se-mutation-testing","last_synced_at":"2026-01-18T18:33:29.249Z","repository":{"id":251078409,"uuid":"788951492","full_name":"robert-oleynik/AI4SE-Mutation-Testing","owner":"robert-oleynik","description":null,"archived":false,"fork":false,"pushed_at":"2024-07-31T21:57:21.000Z","size":354,"stargazers_count":2,"open_issues_count":3,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-05T08:27:19.118Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/robert-oleynik.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-04-19T12:10:47.000Z","updated_at":"2024-07-31T21:57:24.000Z","dependencies_parsed_at":"2025-04-05T08:26:00.544Z","dependency_job_id":"f8945cb3-9e61-4725-b766-80bc2bd76f13","html_url":"https://github.com/robert-oleynik/AI4SE-Mutation-Testing","commit_stats":null,"previous_names":["robert-oleynik/ai4se-mutation-testing"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/robert-oleynik/AI4SE-Mutation-Testing","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/robert-oleynik%2FAI4SE-Mutation-Testing","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/robert-oleynik%2FAI4SE-Mutation-Testing/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/robert-oleynik%2FAI4SE-Mutation-Testing/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/robert-oleynik%2FAI4SE-Mutation-Testing/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/robert-oleynik","download_url":"https://codeload.github.com/robert-oleynik/AI4SE-Mutation-Testing/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/robert-oleynik%2FAI4SE-Mutation-Testing/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28547269,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-18T14:59:57.589Z","status":"ssl_error","status_checked_at":"2026-01-18T14:59:46.540Z","response_time":98,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-09T18:05:30.907Z","updated_at":"2026-01-18T18:33:29.223Z","avatar_url":"https://github.com/robert-oleynik.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AI for Software Engineering\n\nGenerate source code mutants for python using a LLM.\n\n## Installation\n\n1. Clone the git repository:\n\n\t```sh\n\tgit clone https://github.com/robert-oleynik/AI4SE-Mutation-Testing mutator\n\t```\n\n2. Install this tool to your system:\n\n\t```sh\n\tpip install -e ./mutator\n\t```\n\n\t**Note:** You may want to run this and the following steps inside a `venv`.\n\n3. Test the tool is installed successfully:\n\n\t```\n\tmutator --help\n\tmutator-runner --help\n\t```\n\n\t**Note:** The second command is required to execute the test and is installed as part of this project.\n\n## Usage\n\nThis tool provides two main capabilities:\n\n1. Generating and testing mutants for a software project.\n2. Collect datasets from repositories and LoRa-finetuning on these datasets.\n\nBoth expect projects with following structure:\n\n- `src/` contains all source files\n- `tests/` contains all test files\n\n### Generating and Testing Mutants\n\nTo generate mutants, this tool will follow the following steps:\n\n1. Select all functions provided in the source files.\n2. Remove all function not matching the filter expression. We will refer to these functions\n   as source functions.\n3. Generate mutants with all generator and generator config combinations.\n4. Mark all duplicate mutants.\n5. Run tests for all unique mutants.\n\n#### Generators\n\nGenerators are functions, which transform source functions into mutants or mutated functions\nwith the usage of LLMs. The generators used in this project can be divided into two groups:\n\n1. Generating mutants based on function signature and some extra information:\n\n\t- The first and most basic generator is the `docstring` generator. This generator\n\t  will prompt the LLM only with the signature and docstring.\n\t- The second generator `comment_rewrite` will additionally write the original\n\t  function commented out before the member function signature.\n\t- The `comment_rewrite_no_context` generator is a variation of the previous generator,\n\t  that does not provide context for the surrounding class.\n\n2. Reusing parts of the existing implementation and prompting the LLM to rewrite parts of it:\n\n\t- The `prefix` generator will arbitrarily cut off the source function after a random token\n\t  and prompt the LLM with this result.\n\t- The `infilling` generator will work similarly using the same context. But instead of\n\t  cutting of tokens, it relies on TreeSitter to sample expressions or statements of the source\n\t  function and prompt the LLM to regenerate them.\n\n#### Generator Configs\n\nWhile using different prompts is one approach for receiving different mutants, we can also\nmodify the arguments passed to the LLMs generation method. We will refer to these arguments as\ngenerator configs. In addition to the LLM arguments these configs also contains the number of\nretries/repetitions a generator is supposed to do. Therefore, we provide following configs:\n\n- `multi_sample` prompts the LLM to return multiple return sequences with sampling enabled.\n- `beam_search` prompts the LLM to return multiple return sequences with beam search and\n  sampling enabled.\n- For each of the above, two additional configs with suffixes `_cold` and `_hold`, that set the\n  temperature lower and higher, respectively.\n\nEach config will produce the same number of mutants - 8 per source function.\n\n#### Generating Mutants\n\nTo generate mutants, we will first need a project we want to test. For this, we will use the\nFlask repository:\n\n```sh\ngit clone https://github.com/pallets/flask\ncd flask\n```\n\nNow we can use our tool on this repository to generate mutants.\nWe will start with the `infilling` generator with the `multi_sample`.\nThese can be set by the `-g/--generator` and `-c/--config` respectively.\n\n```\nmutator generate --generator infilling --config multi_sample --filter \"flask.app:Flask.*\"\n```\n\n\u003e **Note:**\n\u003e\n\u003e - We use a simple globing syntax to filter with functions. This is necessary as the generation\n\u003e   testing may take some time.This globing reuses following syntax:\n\u003e\n\u003e   1. Each function is converted into construct following a syntax like\n\u003e      `\u003cmodule path\u003e:[\u003cclass name\u003e.]\u003cfunction_name\u003e`\n\u003e   2. Each specified filter is converted into a RegEx by escaping `.` and replacing `*`\n\u003e      with `.*`. In addition, these filter allow a `!` in the beginning to mark negative filter.\n\u003e   3. We will match all functions against these filter and exclude all functions matching\n\u003e      negative filters.\n\nSome other important flags for generating mutants:\n\n- `-o/--out-dir` Change the directory to write the mutants to.\n- `--clean` Removes all old mutants.\n- `-m/--model` Change the LLM model to use. **Note:** this may cause compatibility issues.\n\n#### Testing Mutants\n\nTo execute the test suite for each mutant we generate we can use the `mutator test` command.\nIt is important to note that each mutant has a timeout of `60s` this value can be changed by\nusing the `--timeout` flag.\nLike `mutator generate` the `-o/--out-dir` can be used to change mutants work directory.\n\n#### Inspect Results\n\nThe results of this test run can be viewed with `mutator inspect`.\nThis will open a TUI application showing the mutant as a diff and some additional\ninformation including test output.\nUse the list on the left to select a source function, the horizontal arrow keys to\nselect a mutant and ctrl + the horizontal arrow keys to cycle through the LLM's output stages.\nLike `mutator generate` and `mutator test` the `-o/--out-dir` can be used to change\nmutants work directory.\n\n### Fine-Tuning\n\nIn some context, it is beneficial to use fine-tuning for improving LLM results.\nTo do so, this tool provides some utilities from collecting datasets to training the model.\n\n#### Generating Datasets\n\nAs we already discussed we have multiple approaches for generating prompts.\nBecause of this, generating datasets need to account for the different input formats.\n\nIn the following we will generate a dataset from flask repository.\nTo do so, we will use a bare git repository of flask.\n\n```sh\ngit clone --bare https://github.com/pallets/flask\n```\n\nTo collect samples from this repository, we can use `mutator collect` as following:\n\n```sh\nmutator collect --bare ./flask.git --generator \"infilling\"\n```\n\nThis will collect mutants from the source files and format these for the `infilling`\ngenerator. While it is possible to specify multiple generators, it is not recommended.\nIn addition, it is also possible to specify multiple bare git repositories.\nIt is also possible to use normal git repositories using the `--repository` flag\ninstead.\n\nThese samples can be filtered by using following:\n\n- `--max-dloc` Maximum change in lines of code (LOC) from source to mutant.\n- `--max-loc-ratio` Maximum ratio between source and mutant LOC (`\u003e1` means the\n  mutant has more lines the source).\n- `--min-prompt-loc` Minimum LOC for the generated prompt.\n- `--max-prompt-loc` Maximum LOC for the generated prompt.\n\nIt is recommended to use these options in combination with `--update`.\nThis flag will load the dataset at `\u003cout_dir\u003e/data`, apply the limits and store\nthe result in `\u003cout_dir\u003e/data-updated`.\nUsing this command looks like:\n\n```sh\nmutator collect --update --max-dloc 10\n```\n\n#### Run Fine-Tuning\n\nWe can use the generated dataset with the following sub command:\n\n```sh\nmutator train --dataset \"\u003ccollect_out_dir\u003e/data-updated\"\n```\n\nFor more details on the available arguments see `mutator train --help`.\nIn addition, it is worthy to note, that the directory specified with\n`--dataset` is the `data` or `data-updated` subdirectory of the output\ndirectory specified with `collect`.\n\n#### Evaluating Training\n\nTo evaluate the loss per training sample use the following command:\n\n```sh\nmutator train-result --dir out/model --dataset out/dataset/data-updated\n```\n\n## Development\n\n\u003e This section contains tools used and the recommended project setup for development.\n\u003e It is not mandatory for running the tool.\n\nRequired tools:\n\n- [pyright](https://github.com/microsoft/pyright) for type checking\n- [ruff](https://github.com/astral-sh/ruff) for linting\n\nInstall required dependencies:\n\n```sh\n# Create a venv\npython3 -m venv venv\n# Activate venv\nsource ./venv/bin/activate\n# Install package dependencies\npip install -e .\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frobert-oleynik%2Fai4se-mutation-testing","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frobert-oleynik%2Fai4se-mutation-testing","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frobert-oleynik%2Fai4se-mutation-testing/lists"}