{"id":13454220,"url":"https://github.com/githubnext/llmorpheus","last_synced_at":"2025-03-24T05:33:18.293Z","repository":{"id":233580331,"uuid":"778814697","full_name":"githubnext/llmorpheus","owner":"githubnext","description":"LLM-based mutation testing","archived":true,"fork":false,"pushed_at":"2025-02-03T11:04:39.000Z","size":7182,"stargazers_count":9,"open_issues_count":2,"forks_count":5,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-03T12:22:24.189Z","etag":null,"topics":["large-language-models","mutation-testing"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/githubnext.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":"SECURITY.md","support":"SUPPORT.md","governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-03-28T13:10:46.000Z","updated_at":"2025-02-03T11:05:21.000Z","dependencies_parsed_at":"2024-05-17T12:24:35.499Z","dependency_job_id":"58163401-7c7a-441f-a98d-1330cef7dc50","html_url":"https://github.com/githubnext/llmorpheus","commit_stats":null,"previous_names":["githubnext/llmorpheus"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/githubnext%2Fllmorpheus","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/githubnext%2Fllmorpheus/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/githubnext%2Fllmorpheus/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/githubnext%2Fllmorpheus/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/githubnext","download_url":"https://codeload.github.com/githubnext/llmorpheus/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245217337,"owners_count":20579290,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["large-language-models","mutation-testing"],"created_at":"2024-07-31T08:00:51.951Z","updated_at":"2025-03-24T05:33:18.281Z","avatar_url":"https://github.com/githubnext.png","language":"TypeScript","readme":"Note: This version of LLMorpheus has been archived.  Please refer to the new version at [https://github.com/neu-se/llmorpheus/](https://github.com/neu-se/llmorpheus/).\n\n# LLMorpheus\n\nLLMorpheus is a tool for applying mutation testing to npm packages\nwritten in JavaScript/TypeScript using a large language model (LLM).\n\nNote that LLMorpheus represents an early exploration in the use of LLMs for\nmutation testing, and has been made available in open source as a basis for\nresearch and exploration.  \n\n## Background\n\nLLMorpheus generates mutants for a given package `p` by prompting the LLM with a\nprompt that asks it to apply mutations to specific locations in the source code\nof `p`. This prompt includes some background on mutation testing, a fragment\nof `p`'s source code in which a code fragment is replaced with the word\n`PLACEHOLDER`, and a request to suggest what code fragments should be \nsubstituted for the placeholder. The LLM is asked to produce output in a \nspecific format in which the mutation suggestions occur as fenced code blocks.\nThe completion provided by the LLM is then analyzed to extract the suggested\nmutants, and the resulting modified source code is checked for syntactic\nvalidity. All syntactically valid mutants are written to a file `mutants.json`.\n\nThe resulting set of mutants is then provided to a modified version of the\npopular [StrykerJS ](https://github.com/franktip/stryker-js), a state-of-the-art mutation testing tool. Running this version of StrykerJS produces an interactive web page that can be inspected to reveal which mutants are killed, survived, or timed out. \n\nUnlike other systems for mutation testing, LLMorpheus does not rely on a fixed\nset of mutation operators, but instead relies on an LLM to suggest suitable code fragments.\n\n## Requirements\n\nIn general, to be able to run LLMorpheus you need access to a OpenAI-style LLM\nwith chat API. Set the `LLMORPHEUS_LLM_API_ENDPOINT` environment variable to\nthe URL of the LLM API endpoint you want to use, and\n`LLMORPHEUS_LLM_AUTH_HEADERS` to a JSON object containing the headers you need to\nauthenticate with the API.\n\nTypical values for these variables might be:\n\n- `LLMORPHEUS_LLM_API_ENDPOINT='https://text.octoai.run/v1/chat/completions'`\n- `LLMORPHEUS_LLM_AUTH_HEADERS='{\"Authorization\": \"Bearer \u003cyour API key\u003e\", \"content-type\": \"application/json\"}'`\n\n## Installation\n\nThe `src/` directory contains the source code for LLMorpheus, which is written in\nTypeScript and gets compiled into the `dist/` directory. Tests are in `test/`;\nthe `benchmark/` directory contains a benchmarking harness for running LLMorpheus\non multiple npm packages.\n\nIn the root directory of a checkout of this repository, run `npm build` to\ninstall dependencies and build the package.\n\nYou can also use `npm run build:watch` to automatically build anytime you make\nchanges to the code. Note, however, that this will not automatically install\ndependencies, and also will not build the benchmarking harness.\n\nUse `npm run test` to run the tests. For convenience, this will also install\ndependencies and run a build.\n\n## Benchmarking\n\nBasic usage is as follows:\n\n```sh\nnode benchmark/createMutants.js --path \u003cpackage-dir\u003e --mutate \u003cfiles-to-mutate\u003e   --model \u003cmodel-name\u003e  --template \u003cprompt-template\u003e\n```\nNote that this assumes that package dependencies are installed and any build\nsteps have been run (e.g., using `npm i` and `npm run build`). \n\nThis will generate a directory `\u003cpackage_dir\u003e/MUTATION_TESTING` in which you will  find the following:\n  - a subdirectory ```prompts``` containing all generated prompts and completions. Prompts are saved in files named ```promptN.txt``` and completions are in files ```promptN_completionN.txt```\n  - a file ```mutants.json``` containing information about the computed mutants (file, location, original source code, mutated source code, prompt ID, completion ID, and code pattern that was used to derive the mutant). This file is what our custom version of StrykerJS refers to when it performs its mutation analysis.\n  - a file ```promptSpecs.json`` recording the locations that are candidates for mutation (file, location, pattern)\n  - a file ```summary.json``` containing statistics gathered for the entire project (number of prompts, number of mutants, number of tokens used for prompts, number of tokens used for completions, meta-information about the model and LLMorpheus parameters used for the experiment)\n\nFor further details on available command-line options, please refer to the documentation.\n\n## Running StrykerJS and Viewing Stryker's Report\n\nTo run our custom version of Stryker, perform the following steps:\n\n1. Clone Stryker from ```https://github.com/neu-se/stryker-js``` into directory ```stryker-js```\n\n2. ```cd stryker-js```\n\n3. ```npm install```\n\n4. ```npm run build```\n\n5. Next you need to install Stryker in the subject project for which you generated mutants. The instructions below assume that the subject project is in directory ```ProjectDir``` and that Stryker was installed in ```stryker-js```\n  - ```cd ProjectDir```\n  - ```npm install install-local```\n  - set ```STRYKER_FILES``` to a quoted comma-separated list of files that you want to be mutated.  LLMorpheus outputs this list, so you can copy and paste from there. Note: There should be no space characters in the list.\n  - set the ```MUTANTS_FILE``` environment variable to point to the ```mutants.json``` file that was generated by LLMorpheus\n  - ```npx stryker run --usePrecomputed  --mutate $STRYKER_FILES```   (if you leave out the ```---usePrecomputed``` option, Stryker's standard mutation operators will be applied). You may supply other options to StrykerJS, see the StrykerJS documentation.\n  - the previous command will generate a directory ```reports```. You can view Stryker's report by opening ```reports/mutation/mutation.html``` in a browser.\n\n### Running on Actions\n\nThe `experiment.yml` workflow runs an experiment on GitHub Actions,\nproducing the final report as an artifact you can download. The `mutants` artifact contains all mutants, prompts, and related information computed by LLMorpheus, and the `results` artifact contains the report produced by StrykerJS. \n\nThe `StandardStryker.yml` workflow runs the standard StrykerJS mutators.\n\n### Reproducing results\n\nThe results of LLMorpheus are non-deterministic, so even if you run it from the same package on the same machine multiple times, it is likely that you will get different results.\n\nThat said, LLMorpheus has an option `--replay \u003cdirName\u003e` for replaying an execution using prompts and completions observed during a previous execution. If you use this option, the model name, temperature, max number of tokens, and system prompt will be inferred from the previously observed execution. However, the `--mutate` and `--ignore` settings will be taken from command-line arguments, so as to enable replaying executions of LLMorpheus partially, on a subset of the files that it was previous run on.\n\nFor example, let's say that the zip-a-folder project is located in a directory `/Users/xxx/zip-a-folder/`, and that the results of a previous execution of LLMorpheus on zip-a-folder can be found in a directory `/Users/xxx/recording`. Further, let's assume that we want to mutate the files that match the pattern `\"lib/*.ts\"`. You can do that by running the following command:\n```\nnode benchmark/createMutants.js --path /Users/xxx/zip-a-folder --mutate \"lib/**.ts\"  --replay /Users/xxx/recording\n```\n\nA collection of recorded executions of LLMorheus is made available at \u003chttps://github.com/neu-se/mutation-testing-data\u003e. Here, you can find recorded executions that were made using the codellama-13b-instruct, codellama-34b-instruct, and mixtral-8x7b-instruct LLMs there, and using various templates and temperature settings. The data is organized in subdirectories that reflect the model, template, and temperature used, e.g. `codellama-13b-instruct/template-full-0.0` reflects a set of experiments using the `codellama-13b-instruct` model, the `full`  template, at temperature `0.0`. This directory contains 5 sets of experiments on 13 subject programs using these settings, contained in subdirectories named `run354`-`run359`. A file `zip/mutants.zip` within each of these subdirectories contains all data that is needed to replay these executions.\n\n\n## License\n\nThis project is licensed under the terms of the MIT open source license. Please refer to [LICENSE](./LICENSE) for the full terms.\n\n## Paper\n\nA paper about LLMorpheus is available on [arXiv](https://arxiv.org/abs/2404.09952).\n\n## Maintainers\n\n- Frank Tip (@franktip)\n- Jon Bell (@jon-bell)\n- Max Schaefer (@max-schaefer)\n\n## Support\n\nLLMorpheus is a research prototype and is not officially supported. However, if\nyou have questions or feedback, please file an issue and we will do our best to\nrespond.\n","funding_links":[],"categories":["TypeScript"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgithubnext%2Fllmorpheus","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgithubnext%2Fllmorpheus","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgithubnext%2Fllmorpheus/lists"}