{"id":21094618,"url":"https://github.com/ise-uiuc/repilot","last_synced_at":"2025-05-16T14:33:45.783Z","repository":{"id":190071048,"uuid":"530076390","full_name":"ise-uiuc/Repilot","owner":"ise-uiuc","description":"Repilot, a patch generation tool introduced in the ESEC/FSE'23 paper \"Copiloting the Copilots: Fusing Large Language Models with Completion Engines for Automated Program Repair\"","archived":false,"fork":false,"pushed_at":"2023-10-09T03:14:40.000Z","size":981,"stargazers_count":121,"open_issues_count":0,"forks_count":9,"subscribers_count":4,"default_branch":"main","last_synced_at":"2024-04-22T10:13:00.968Z","etag":null,"topics":["code-completion","code-generation","large-language-models","program-repair","program-synthesis"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ise-uiuc.png","metadata":{"files":{"readme":"README-Artifact.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":"CITATION.bib","codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2022-08-29T05:21:28.000Z","updated_at":"2024-04-14T17:46:18.000Z","dependencies_parsed_at":"2023-08-23T05:45:38.594Z","dependency_job_id":"91c5352d-ec54-41ff-ac13-e05a50448d16","html_url":"https://github.com/ise-uiuc/Repilot","commit_stats":null,"previous_names":["universefly/rectify","ise-uiuc/repilot"],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ise-uiuc%2FRepilot","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ise-uiuc%2FRepilot/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ise-uiuc%2FRepilot/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ise-uiuc%2FRepilot/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ise-uiuc","download_url":"https://codeload.github.com/ise-uiuc/Repilot/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225433449,"owners_count":17473599,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["code-completion","code-generation","large-language-models","program-repair","program-synthesis"],"created_at":"2024-11-19T22:18:09.081Z","updated_at":"2024-11-19T22:18:09.763Z","avatar_url":"https://github.com/ise-uiuc.png","language":"Python","readme":"# Artifact Documentation for ⚙️$`\\mathbb{R}\\mathrm{e}\\mathbf{pilot}`$🛠️\n\nWelcome to the artifact repository for **Repilot**, a patch generation tool introduced in the ESEC/FSE'23 paper **\"Copiloting the Copilots: Fusing Large Language Models with Completion Engines for Automated Program Repair\"**!\n\n\u003e [!IMPORTANT]\n\u003e\n\u003e **Environment requirements**\n\u003e \n\u003e - **OS**: A Linux system with **[Docker](https://docs.docker.com/engine/install/)** support.\n\u003e   - Optional: [NVIDIA Docker](https://github.com/NVIDIA/nvidia-docker) support.\n\u003e - **Hardware**: X86/X64 CPU; 32GB RAM; 1TB Storage; Good Network to Docker Hub.\n\u003e   - Optional (a): NVIDIA GPU(s) with \u003e6G memory (for CodeT5 patch generation)\n\u003e   - Optional (b): NVIDIA GPU(s) with \u003e30G memory (for Incoder-6.7B patch generation)\n\u003e \n\u003e Although it is recommended to run the artifact with NVIDIA GPUs for faster patch generation, it is not a requirement.\n\u003e When there is no GPU available, the CPU will be responsible for the patch generation.\n\u003e In this artifact documentation, we only explain the CPU-only Docker-based pipeline for conciseness.\n\u003e We encourage advanced readers who want to run the artifact with GPU support to check the [documentation](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/index.html) of NVIDIA Docker.\n\n## Before we start\n\nBefore we start, let's first make sure Docker is installed: [Installation Guide](https://docs.docker.com/engine/install/).\n\nTo check the installation:\n\n```bash\ndocker --version # Test docker availability\n# Docker version 20.10.21, build 20.10.21-0ubuntu1~20.04.2\n``` \n\nNow we'll fetch the Docker image of Repilot that includes the implementation of the Algorithm, the Completion Engine, and all the dependencies needed:\n\n```bash\n# Recommended: pull the image from Docker Hub\ndocker pull universefly/repilot:fse23\n# Alternatively, download the image file `repilot-docker-image-fse23.tar.gz` from https://doi.org/10.5281/zenodo.8280747\n# Then load this image\n# docker load --input repilot-docker-image-fse23.tar.gz\n\n# Run the docker image\ndocker run -it --name repilot universefly/repilot:fse23\n# Now you will get into a \"virtual environment\" provided by Docker\n# Enter the `repilot` directory\ncd /root/Repilot\necho \"Hello Repilot!\"\n```\n\nCongratulations! We are now ready for the artifact evaluation.\n\n## Whet your appetite\n\nLet's run some example scripts to see how Repilot works.\n\n```bash\n# The full repilot approach with CodeT5 as the base model\n# Generate 5 patches for Chart-9 and save to `chart-9-repilot`\nACTIVE=1 python -m repilot.cli.main repair -b \"Chart-9\" --method pruned-mem -d chart-9-repilot -n 5\n# You will see logs about the patch generation and which tokens are accepted/rejected.\n\n# Validate the patch generation\npython -m repilot.cli.main validate -d chart-9-repilot\n\n# Print a table of the evaluation results\npython -m repilot.cli.main evaluate -d chart-9-repilot\n```\n\nIf everything works correctly, you will see a similar output table as follows:\n\n```\nroot@1d7fea7789ed:/repilot# python -m repilot.cli.main evaluate -d chart-9-repilot\n[chart-9-repilot] Loading raw generation data...\nDone\n[chart-9-repilot] Loading transformed raw generation data...\nDone\n[chart-9-repilot] Loading validation raw data...\nDone\n                                             Repilot Evaluation Results                                              \n┏━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┓\n┃ Tag             ┃ Average Gen Time ┃ %Compilable Patches ┃ %Plausible Patches ┃ #Plausible Fixes ┃ #Correct Fixes ┃\n┡━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━┩\n│ chart-9-repilot │ 1.33s            │ 100.0%              │ 0.000%             │ 0                │ -              │\n└─────────────────┴──────────────────┴─────────────────────┴────────────────────┴──────────────────┴────────────────┘\n```\n\n## Reproduce RQ Evaluation\n\nWe will now show how each RQ can be reproduced through the artifact by applying Repilot evaluation script on **pre-generated patches**.\n\n\u003e [!WARNING]\n\u003e We also provide documentation to [reproduce the entire patch generation](#reproduce-patch-generation) in different RQs, but it is not recommended for the readers to go through the entire process as it may take days or weeks to finish.\n\n### RQ1: Comparison with existing tools\n\nWe will now reproduce Table 1, Figure 6, and the number of bugs fixed by removing the bugs that overlap with the CodeT5 training data, which is shown in Section 8 THREATS TO VALIDITY.\n\n```bash\npython -m repilot.cli.rq1\n```\n\nYou will see two tables printed in the console, where the first table corresponds to Table 1 and the second table corresponds to following the sentence in Section 8:\n\n\u003e For comparison fairness, if we were to exclude these 7 and 6 bugs and compare them with the previous baseline tools on the remaining bugs, we are still able to achieve the highest bug fixes at 59 and 44 (best baseline at 45 and 29)\n\nThe detailed correct patches can be found through the following links:\n- [Defects4j 1.2 correct patches](data/correct-patches/rq1/d4j1-codet5-template-repilot)\n- [Defects4j 2.0 correct patches](data/correct-patches/rq1/d4j2-codet5-template-repilot)\n\nAlso the two venn diagrams shown in Figure 6 are saved in the `plots` directory. To check the plots, you may need to temporarily exit the Docker container and save the plots to your local machine:\n\n```bash\n# Exit the docker container with e.g., Ctrl-D\n# Save the plots to your local machine\nsudo docker cp repilot:/root/Repilot/plots /path/to/your/local/directory\n# Now you can open the plots with your favorite image viewer\n# Return to the docker container\ndocker start -ai repilot\n\n# Return to the `repilot` directory\ncd /root/Repilot\n```\n\n### RQ2: Compilation rate analysis\n\nWe will now reproduce Table 2. This script may take longer to run as it needs to iterate through 5000 generated patches per bug. We also compressed the patches beforehand due to the large size. Therefore, let's first decompress the patches:\n\n```bash\ntar -xvf data/large.tar.xz\n```\n\nThen we can run the command for RQ2:\n\n```bash\npython -m repilot.cli.rq2\n```\n\nThis command will print a table in the console, which corresponds to Table 2.\n\n### RQ3: Component contribution\n\nWe now reproduce Table 3.\n\n```bash\npython -m repilot.cli.rq3\n```\n\nThe detailed correct patches can be found through the following links:\n- [[Vanilla] correct patches](data/correct-patches/rq3/d4j1-codet5-vanilla)\n- [[NoMem] correct patches](data/correct-patches/rq3/d4j1-codet5-nomem)\n- [[Mem] correct patches](data/correct-patches/rq3/d4j1-codet5-mem)\n- [[Repilot] correct patches](data/correct-patches/rq3/d4j1-codet5-repilot)\n\n### RQ4: Generalizability\n\nThis script will reproduce Table 4.\n\n```bash\npython -m repilot.cli.rq4\n```\n\nThe detailed correct patches can be found through the following links:\n- [CodeT5/D4J1.2 vanilla](data/correct-patches/rq3/d4j1-codet5-vanilla)\n- [CodeT5/D4J1.2 repilot](data/correct-patches/rq3/d4j1-codet5-repilot)\n- [CodeT5/D4J2.0 vanilla](data/correct-patches/rq4/d4j2-codet5-vanilla)\n- [CodeT5/D4J2.0 repilot](data/correct-patches/rq4/d4j2-codet5-repilot)\n- [Incoder/D4J1.2 vanilla](data/correct-patches/rq4/d4j1-incoder-vanilla)\n- [Incoder/D4J1.2 repilot](data/correct-patches/rq4/d4j1-incoder-repilot)\n- [Incoder/D4J2.0 vanilla](data/correct-patches/rq4/d4j2-incoder-vanilla)\n- [Incoder/D4J2.0 repilot](data/correct-patches/rq4/d4j2-incoder-repilot)\n\n🎉🎉🎉 Congratulations! You have successfully reproduced all the results in the paper! 🎉🎉🎉\n\n## Reproduce patch generation\n\n\u003e [!WARNING]\n\u003e ⚠️⚠️⚠️ This section is mainly for advanced readers who have time to reproduce the entire patch generation process. These commands may take days or weeks to finish. Also, the generation time may vary significantly depending on the hardware used for patch generation. ⚠️⚠️⚠️\n\n### RQ1\n\nWe generate Defects4j 1.2 single-hunk bugs and 2.0 single-line bugs with the help of repair templates. This is achieved through the following command:\n\n```bash\nD4J1_SINGLE_HUNK=1 ACTIVE=1 TEMPLATE=1 python -m repilot.cli.main repair -b \".*\" --method pruned-mem -n 5000 -d rq1-d4j1\nD4J2_SINGLE_LINE=1 ACTIVE=1 TEMPLATE=1 python -m repilot.cli.main repair -b \".*\" --method pruned-mem -n 5000 -d rq1-d4j2\n```\n\n### RQ2\n\nRQ2 is based on RQ1's generated patches, so we don't need to run any additional commands.\n\n### RQ3\n\nIn RQ3, we generate 500 patches for each bug with 4 different configurations, using the following commands:\n\n```bash\n# Vanilla\nD4J1_SINGLE_HUNK=1 python -m repilot.cli.main repair -b \".*\" --method plain -n 500 -d rq3-vanilla\nD4J1_SINGLE_HUNK=1 python -m repilot.cli.main repair -b \".*\" --method pruned-nomem -n 500 -d rq3-nomem\nD4J1_SINGLE_HUNK=1 python -m repilot.cli.main repair -b \".*\" --method pruned-mem -n 500 -d rq3-mem\nACTIVE=1 D4J1_SINGLE_HUNK=1 python -m repilot.cli.main repair -b \".*\" --method pruned-mem -n 500 -d rq3-repilot\n```\n\n### RQ4\n\nWe further include Incoder-6.7B as the base model to generate patches for RQ4.\n\n```bash\n# The first two configurations are the same as RQ3\n# D4J1_SINGLE_HUNK=1 python -m repilot.cli.main repair -b \".*\" --method plain -n 500 -d rq3-vanilla\n# ACTIVE=1 D4J1_SINGLE_HUNK=1 python -m repilot.cli.main repair -b \".*\" --method plain -n 500 -d rq3-repilot\n\n\nD4J2_SINGLE_HUNK=1 python -m repilot.cli.main repair -b \".*\" --method plain -n 500 -d rq4-codet5-d4j2-vanilla\nACTIVE=1 D4J2_SINGLE_HUNK=1 python -m repilot.cli.main repair -b \".*\" --method pruned-mem -n 500 -d rq4-codet5-d4j2-repilot\n\nINCODER=1 D4J1_SINGLE_HUNK=1 python -m repilot.cli.main repair -b \".*\" --method plain -n 500 -d rq4-incoder-d4j1-vanilla\nINCODER=1 ACTIVE=1 D4J1_SINGLE_HUNK=1 python -m repilot.cli.main repair -b \".*\" --method pruned-mem -n 500 -d rq4-incoder-d4j1-repilot\n\nINCODER=1 D4J2_SINGLE_HUNK=1 python -m repilot.cli.main repair -b \".*\" --method plain -n 500 -d rq4-incoder-d4j2-vanilla\nINCODER=1 ACTIVE=1 D4J2_SINGLE_HUNK=1 python -m repilot.cli.main repair -b \".*\" --method pruned-mem -n 500 -d rq4-incoder-d4j2-repilot\n```\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fise-uiuc%2Frepilot","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fise-uiuc%2Frepilot","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fise-uiuc%2Frepilot/lists"}