{"id":13454226,"url":"https://github.com/githubnext/testpilot","last_synced_at":"2025-10-19T11:32:11.928Z","repository":{"id":207426225,"uuid":"711838263","full_name":"githubnext/testpilot","owner":"githubnext","description":"Test generation using large language models","archived":false,"fork":false,"pushed_at":"2024-09-04T12:42:00.000Z","size":148,"stargazers_count":512,"open_issues_count":0,"forks_count":37,"subscribers_count":14,"default_branch":"main","last_synced_at":"2024-10-28T21:38:00.477Z","etag":null,"topics":["chagpt","gpt-35-turbo","llms","test-automation","testing","testing-tools"],"latest_commit_sha":null,"homepage":"https://githubnext.com/projects/testpilot","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/githubnext.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":"SECURITY.md","support":"SUPPORT.md","governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-10-30T09:23:53.000Z","updated_at":"2024-10-28T12:06:07.000Z","dependencies_parsed_at":"2023-11-15T19:25:17.593Z","dependency_job_id":"a32ab214-cf6d-4918-8feb-49115afa918e","html_url":"https://github.com/githubnext/testpilot","commit_stats":null,"previous_names":["githubnext/testpilot"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/githubnext%2Ftestpilot","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/githubnext%2Ftestpilot/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/githubnext%2Ftestpilot/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/githubnext%2Ftestpilot/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/githubnext","download_url":"https://codeload.github.com/githubnext/testpilot/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":237121643,"owners_count":19258623,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chagpt","gpt-35-turbo","llms","test-automation","testing","testing-tools"],"created_at":"2024-07-31T08:00:52.002Z","updated_at":"2025-10-19T11:32:11.583Z","avatar_url":"https://github.com/githubnext.png","language":"TypeScript","funding_links":[],"categories":["TypeScript","testing"],"sub_categories":[],"readme":"Note: This version of TestPilot has been archived. Please refer to the new version at [https://github.com/neu-se/testpilot2](https://github.com/neu-se/testpilot2).\n\n# TestPilot\n\nTestPilot is a tool for automatically generating unit tests for npm packages\nwritten in JavaScript/TypeScript using a large language model (LLM).\n\nNote that TestPilot represents an early exploration in the use of LLMs for\ntest generation, and has been made available in open source as a basis for\nresearch and exploration. For day-to-day use the test generation features\nin [Copilot Chat](https://docs.github.com/en/copilot/github-copilot-chat/about-github-copilot-chat)\nare likely to yield better results.\n\n## Background\n\nTestPilot generates tests for a given function `f` by prompting the LLM with a\nskeleton of a test for `f`, including information about `f` embedded in code\ncomments, such as its signature, the body of `f`, and examples usages of `f`\nautomatically mined from project documentation. The model's response is then\nparsed and translated into a runnable unit test. Optionally, the test is run and\nif it fails the model is prompted again with additional information about the\nfailed test, giving it a chance to refine the test.\n\nUnlike other systems for LLM-based test generation, TestPilot does not require\nany additional training or reinforcement learning, and no examples of functions\nand their associated tests are needed.\n\nA research paper describing TestPilot in detail is available on\n[arXiv](https://arxiv.org/abs/2302.06527) and [IEEExplore](https://ieeexplore.ieee.org/document/10329992).\n\n## Requirements\n\nIn general, to be able to run TestPilot you need access to a Codex-style LLM\nwith completion API. Set the `TESTPILOT_LLM_API_ENDPOINT` environment variable to\nthe URL of the LLM API endpoint you want to use, and\n`TESTPILOT_LLM_AUTH_HEADERS` to a JSON object containing the headers you need to\nauthenticate with the API.\n\nTypical values for these variables might be:\n\n- `TESTPILOT_LLM_API_ENDPOINT='https://api.openai.com/v1/engines/code-cushman-001/completions'`\n- `TESTPILOT_LLM_AUTH_HEADERS='{\"Authorization\": \"Bearer \u003cyour API key\u003e\", \"OpenAI-Organization\": \"\u003cyour organization ID\u003e\"}'`\n\nNote, however, that you can run TestPilot in reproduction mode without access to\nthe LLM API where model responses are taken from the output of a previous run;\nsee below for details.\n\n## Installation\n\nYou can install TestPilot from a pre-built package or from source.\n\n### Installing from a pre-built package\n\nTestPilot is a available as a pre-built npm package, though it is not currently\npublished to the npm registry. You can download a tarball from the repository\nand install it in the usual way. Note that this distribution only contains the\ncore part of TestPilot, not the benchmarking harness.\n\n### Installing from source\n\nThe `src/` directory contains the source code for TestPilot, which is written in\nTypeScript and gets compiled into the `dist/` directory. Tests are in `test/`;\nthe `benchmark/` directory contains a benchmarking harness for running TestPilot\non multiple npm packages; and `ql/` contains the CodeQL queries used to analyze\nthe results.\n\nIn the root directory of a checkout of this repository, run `npm build` to\ninstall dependencies and build the package.\n\nYou can also use `npm run build:watch` to automatically build anytime you make\nchanges to the code. Note, however, that this will not automatically install\ndependencies, and also will not build the benchmarking harness.\n\nUse `npm run test` to run the tests. For convenience, this will also install\ndependencies and run a build.\n\n## Benchmarking\n\nIf you install TestPilot from source, you can use the benchmarking harness to\nrun TestPilot on multiple packages and analyze the results. This is not\ncurrently available if you install TestPilot from a pre-built package.\n\n### Running locally\n\nBasic usage is as follows:\n\n```sh\nnode benchmark/run.js --outputDir \u003creport_dir\u003e --package \u003cpackage_dir\u003e\n```\n\nThis generates tests for all functions exported by the package in\n`\u003cpackage_dir\u003e`, validates them, and writes the results to `\u003creport_dir\u003e`.\n\nNote that this assumes that package dependencies are installed and any build\nsteps have been run (e.g., using `npm i` and `npm run build`). TestPilot also\nrelies on `mocha`, so if the package under test does not already depend on it,\nyou must install it separately, for example using the command `npm i --no-save\nmocha`.\n\n### Running on Actions\n\nThe `run-experiment.yml` workflow runs an experiment on GitHub Actions,\nproducing the final report as an artifact you can download. The `results-all`\nartifact contains the results of all packages, while the other artifacts contain\nthe individual results of each package.\n\n### Reproducing results\n\nThe results of TestPilot are non-deterministic, so even if you run it from the\nsame package on the same machine multiple times, you will get different results.\nHowever, the benchmarking harness records enough data to be able to replay a\nbenchmark run in many cases.\n\nTo do this, use the `--api` and `--responses` options to reuse the API listings\nand responses from a previous run:\n\n```sh\nnode benchmark/run.js --outputDir \u003creport_dir\u003e --package \u003cpackage_dir\u003e --api \u003capi.json\u003e --responses \u003cprompts.json\u003e\n```\n\nNote that by default replay will fail if any of the prompts are not found in the\nresponses file. This typically happens if TestPilot is refining failing tests,\nsince in this case the prompt to the model depends on the exact failure message,\nwhich can be system-specific (e.g., containing local file-system paths), or\ndepend on the Node.js version or other factors.\n\nTo work around these limitations, you can pass the `--strictResponses false`\nflag handle treat missing prompts by treating them as getting no response from\nthe model. This will not, in general, produce the same results as the initial\nrun, but suffices in many cases.\n\n### Analyzing results\n\nThe CodeQL queries in `ql/queries` can be used to analyze the results of running\nan experiment. See `ql/CodeQL.md` for instructions on how to setup CodeQL and\nrun the queries.\n\n## License\n\nThis project is licensed under the terms of the MIT open source license. Please refer to [MIT](./LICENSE.txt) for the full terms.\n\n## Maintainers\n\n- Max Schaefer (@max-schaefer)\n- Frank Tip (@franktip)\n- Sarah Nadi (@snadi)\n\n## Support\n\nTestPilot is a research prototype and is not officially supported. However, if\nyou have questions or feedback, please file an issue and we will do our best to\nrespond.\n\n## Acknowledgement\n\nWe thank Aryaz Eghbali (@aryaze) for his work on the initial version of\nTestPilot.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgithubnext%2Ftestpilot","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgithubnext%2Ftestpilot","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgithubnext%2Ftestpilot/lists"}