{"id":20859375,"url":"https://github.com/engineeringsoftware/exli","last_synced_at":"2025-05-12T08:32:26.996Z","repository":{"id":177518197,"uuid":"652675312","full_name":"EngineeringSoftware/exli","owner":"EngineeringSoftware","description":"Automatic extraction of inline tests (from unit tests)","archived":false,"fork":false,"pushed_at":"2024-06-15T06:55:02.000Z","size":76449,"stargazers_count":3,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-06-15T07:51:22.740Z","etag":null,"topics":["inline-tests","test-automation","test-generation","testing"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/EngineeringSoftware.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-06-12T15:02:17.000Z","updated_at":"2024-06-15T06:55:05.000Z","dependencies_parsed_at":null,"dependency_job_id":"75a5f827-85ff-4bc7-872f-44a7eb64a9f6","html_url":"https://github.com/EngineeringSoftware/exli","commit_stats":null,"previous_names":["engineeringsoftware/exli"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EngineeringSoftware%2Fexli","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EngineeringSoftware%2Fexli/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EngineeringSoftware%2Fexli/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EngineeringSoftware%2Fexli/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/EngineeringSoftware","download_url":"https://codeload.github.com/EngineeringSoftware/exli/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225130723,"owners_count":17425506,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["inline-tests","test-automation","test-generation","testing"],"created_at":"2024-11-18T04:49:39.720Z","updated_at":"2024-11-18T04:49:40.376Z","avatar_url":"https://github.com/EngineeringSoftware.png","language":"Python","readme":"# ExLi\nAutomatically extracting inline tests from unit tests and executing these inline tests with [itest][itest-url].\n\n## Introduction\nThis repo contains the code and data for producing the experiments in [ExLi][paper-url].\n\n### Inline test format\n1. \"Declare\" part\n`itest`\n\nIn our experiments, we use `itest(test_source, target_stmt_line_number)`, to represent the test source and the line number of the target statement in original file (note that after adding inline tests to the original file, the line numbers of target statements change).\nFor example, `itest(\"randoop\", 57)` means that the test source is Randoop generated tests, and the target statement starts at line 57 in the original file.\n\n2. \"Assign\" part\n`given(var, value)`\n\n3. \"Assert\" part\n`assert(var, value)`\n\n### Repo structure\n\n#### Code\nraninline: This directory constains the source code of TargetStmtFinder, VariablesFinder,\nInstrumenter, Collector, Round1Reducer, and InlineTestConstructor.\n\npython: This directory contains the scripts of running ExLi end-to-end and generates figures.\n\n## How to use ExLi\n\n### Requirements\n- Docker\n- Disk space: 10GB\n\nIn the docker, the following tools/dependencies are installed:\n- Conda latest version\n- Python 3.9 (or later)\n- Java 8, JUnit 4\n- Maven 3.8.3 (or later)\n\n### Install\nBuild a docker image\n\n`docker build -t exli .`\n\n`docker run -it exli /bin/bash`\n\n\nIn the docker, create a Python environment named `exli`\n\n`cd exli/python \u0026\u0026 bash prepare-conda-env.sh`\n\n`conda activate exli`\n\n\n### Usage\n\n- For each command, append `--help` to see the usage and options.\n\u003e For example, `python -m exli.main find_target_stmts --help`\n\u003e\n```txt\nusage: main.py [options] find_target_stmts [-h] [--config CONFIG] [--print_config[=flags]]\n                                           --project_name PROJECT_NAME --sha SHA\n                                           --target_stmts_path TARGET_STMTS_PATH\n\nFind target statements for a project.\n\n  ...\n  --project_name PROJECT_NAME\n                        org_repo, e.g., https://github.com/Bernardo-MG/velocity-config-tool, the project name is\n                        Bernardo-MG_velocity-config-tool (required, type: str)\n  --sha SHA             commit hash (required, type: str)\n  --target_stmts_path TARGET_STMTS_PATH\n                        path to store the target statements (required, type: str)\n```\n\n- For each command, the argument and value can be seperated by `=` or space. For example, `--project_name=Bernardo-MG_velocity-config-tool` or `--project_name Bernardo-MG_velocity-config-tool` are both valid.\n- To run on a specific project, replace `Bernardo-MG_velocity-config-tool` with the project name and `26226f5` with the commit hash. Notice that only projects that are in evaluated projects can be run with the command that only require `test_project_name`.\n\n#### Step 1: Find the target statements\n\nIt will help EvoSuite reduce the search scope. Otherwise, EvoSuite will generate tests on the whole project, which may take a long time (~2 min * number of classes). Also, the second round reducer will generate mutants for these target statements.\n\nIn `exli/python` directory\n\n```bash\npython -m exli.main find_target_stmts --project_name=Bernardo-MG_velocity-config-tool --sha=26226f5 --target_stmts_path=${HOME}/exli/results/target-stmt/Bernardo-MG_velocity-config-tool-26226f5.txt\n```\n\u003e The generated target statements are in `${HOME}/exli/results/target-stmt/Bernardo-MG_velocity-config-tool-26226f5.txt`\n\n\u003e Alternatively, to use the default output file path\n`python -m exli.main batch_find_target_stmts --test_project_name=Bernardo-MG_velocity-config-tool`\n\nThere are three target statements found in the project `Bernardo-MG_velocity-config-tool` at commit `26226f5`:\n\n```txt\ntarget stmt string;/home/itdocker/exli/_downloads/Bernardo-MG_velocity-config-tool/src/main/java/com/bernardomg/velocity/tool/ConfigTool.java;195;null;;null\ntarget stmt string;/home/itdocker/exli/_downloads/Bernardo-MG_velocity-config-tool/src/main/java/com/bernardomg/velocity/tool/ConfigTool.java;200;null;;null\ntarget stmt string;/home/itdocker/exli/_downloads/Bernardo-MG_velocity-config-tool/src/main/java/com/bernardomg/velocity/tool/ConfigTool.java;288;null;;null\n```\n\n#### Step 2: Generate unit tests and inline tests\n\nIn `exli/python` directory\n\n```bash\npython -m exli.main run --project_name=Bernardo-MG_velocity-config-tool --sha=26226f5 --randoop=True --randoop_tl=100 --dev=True --evosuite=True --evosuite_tl=120 --seed=42 --log_path=${HOME}/exli/log/raninline.log\n```\n\n\u003e The generated inline tests are in `${HOME}/exli/all-tests/Bernardo-MG_velocity-config-tool-26226f5`\n\n\u003e Alternatively, to use the default setting for test generation and output dirs path\n`python -m exli.main batch_run --test_project_name=Bernardo-MG_velocity-config-tool`\n\n#### Step 3: Execute the generated inline tests\n\nIn `exli/python` directory\n\nExecute r0 inline tests:\n\n```bash\npython -m exli.main run_inline_tests --project_name=Bernardo-MG_velocity-config-tool --sha=26226f5 --generated_tests_dir=${HOME}/exli/r0-tests/Bernardo-MG_velocity-config-tool-26226f5 --inline_tests_dir=${HOME}/exli/r0-its/Bernardo-MG_velocity-config-tool-26226f5 --inlinetest_report_path=${HOME}/exli/results/r0-its-report/Bernardo-MG_velocity-config-tool-26226f5.json --cached_objects_dir=${HOME}/exli/r0-tests/Bernardo-MG_velocity-config-tool-26226f5/.inlinegen --deps_file=${HOME}/exli/generated-tests/Bernardo-MG_velocity-config-tool-26226f5/deps.txt --parse_inline_tests=True --log_path=${HOME}/exli/log/run-its.log\n```\n\nExecute r1 inline tests:\n\n```bash\npython -m exli.main run_inline_tests --project_name=Bernardo-MG_velocity-config-tool --sha=26226f5 --generated_tests_dir=${HOME}/exli/r1-tests/Bernardo-MG_velocity-config-tool-26226f5 --inline_tests_dir=${HOME}/exli/r1-its/Bernardo-MG_velocity-config-tool-26226f5 --inlinetest_report_path=${HOME}/exli/results/r1-its-report/Bernardo-MG_velocity-config-tool-26226f5.json --cached_objects_dir=${HOME}/exli/r0-tests/Bernardo-MG_velocity-config-tool-26226f5/.inlinegen --deps_file=${HOME}/exli/generated-tests/Bernardo-MG_velocity-config-tool-26226f5/deps.txt --parse_inline_tests=True --log_path=${HOME}/exli/log/run-its.log\n```\n\n\u003e These two commands run the inline tests and generate the execution report at `${HOME}/exli/results/r1-its-report/Bernardo-MG_velocity-config-tool-26226f5.json` and `${HOME}/exli/results/r0-its-report/Bernardo-MG_velocity-config-tool-26226f5.json`\n\u003e\n\u003e `r1-tests` and `r0-tests` store the source code with inline tests, `r1-its` and `r0-its` store the inline tests parsed by itest framework\n\u003e \n\u003e To generate the report for all inline tests, replace `r1` with `r0` in the generated tests dir, inline tests dir and execution report (`${HOME}/exli/r1-its` -\u003e `${HOME}/exli/r0-its`, `${HOME}/exli/results/r1-its-report` -\u003e `${HOME}/exli/results/r0-its-report` and `${HOME}/exli/results/r1-its-report` to `${HOME}/exli/results/r0-its-report`)\n\u003e \n\u003e The difference between \"r0-its-report\" and \"r1-its-report\" is that the \"r0-its-report\" contains all the inline tests, while the \"r1-its-report\" contains the inline tests that are reduced by the Round 1 Reducer (based on coverage)\n\n\u003e Alternatively, to use the default setting for output dirs `python -m exli.main batch_run_inline_tests --test_project_name=Bernardo-MG_velocity-config-tool`, this will run the inline tests for both reduced and all inline tests \n\nAfter running the inline tests, we can analyze the inline tests reports to get all passed and failed inline tests.\n\n```bash\npython -m exli.main analyze_inline_tests_reports --inline_test_type=r0\n```\n\n```bash\npython -m exli.main analyze_inline_tests_reports --inline_test_type=r1\n```\n\nThe report shows the number of tests, errors, failures, and time. For example,\n```json\n{\n    \"testsuite\": {\n        \"@errors\": \"0\",\n        \"@failures\": \"0\",\n        \"@hostname\": \"bafe0a4bce5a\",\n        \"@name\": \"JUnit Jupiter\",\n        \"@skipped\": \"0\",\n        \"@tests\": \"6\",\n        \"@time\": \"0.042\",\n        ...\n    }\n}\n```\n\n\n\u003e It is possible to see the message \"Some inline tests failed for {project_name} {sha} during execution\" because not all generated inline tests can be compiled or executed successfully. For example, if the user-defined class does not override the `toString()` method, the serialized object will contain the class name and the hash code, and the new object will not be equal to the original object when we run the inline tests and compare the objects (the collected value does not equal to the runtime value).\n\u003e \n\u003e If there are inline tests failed because of compilation, we directly remove these failed inline tests, the log file `${HOME}/exli/results/r1-its-report/Bernardo-MG_velocity-config-tool-comp-failed-tests.txt` stores the failed inline tests and the reason why they failed (This project does not have compilation failed inline tests so this file does not exist).\n\u003e\n\u003e If inline tests failed because of execution, run the following command to remove the failed inline tests:\n\u003e \n\u003e `python -m exli.main remove_failed_tests --inline_test_type=r1` (or `r0`)\n\u003e\n\u003e Re-generate test reports:\n\u003e \n\u003e `python -m exli.main batch_run_inline_tests --test_project_name=Bernardo-MG_velocity-config-tool`\n\n\nThe generated execution result can be found at\n\n`${HOME}/exli/results/r1-its-report/Bernardo-MG_velocity-config-tool-26226f5.json`\n\n`${HOME}/exli/results/r0-its-report/Bernardo-MG_velocity-config-tool-26226f5.json`\n\n#### Step 4: Generate mutants and run mutation analysis\n\nIn `exli/python` directory\n\nGenerate mutants:\n```bash\npython -m exli.main generate_mutants --project_name=Bernardo-MG_velocity-config-tool --sha=26226f5 --target_stmts_path=${HOME}/exli/results/target-stmt/Bernardo-MG_velocity-config-tool-26226f5.txt --output_path=${HOME}/exli/results/mutants/Bernardo-MG_velocity-config-tool-26226f5-universalmutator.json\n```\n\u003e Alternatively, to use the default settings `python -m exli.main batch_generate_mutants --test_project_name=Bernardo-MG_velocity-config-tool`\n\nThe generated mutants are in `${HOME}/exli/results/mutants/Bernardo-MG_velocity-config-tool-26226f5-universalmutator.json`\n\nRun mutation analysis:\n```bash\npython -m exli.eval run_tests_with_mutants --project_name=Bernardo-MG_velocity-config-tool --sha=26226f5 --test_types=\"['r0', 'r1']\" --mutator=universalmutator --log_path=${HOME}/exli/log/run-tests-with-mutants.log\n```\n\u003e Alternatively, to use the default settings `python -m exli.eval batch_run_tests_with_mutants --test_project_name=Bernardo-MG_velocity-config-tool`\n\nThe generated mutation analysis report are in\n`${HOME}/exli/results/mutants-eval-results/Bernardo-MG_velocity-config-tool-26226f5-universalmutator-r0.json`\nand\n`${HOME}/exli/results/mutants-eval-results/Bernardo-MG_velocity-config-tool-26226f5-universalmutator-r1.json`\n\n\n#### Step 5: Test reduction\n```bash\npython -m exli.eval get_r2_tests --project_name=Bernardo-MG_velocity-config-tool --sha=26226f5 --mutator=universalmutator --algo=greedy --output_path=${HOME}/exli/results/r2/Bernardo-MG_velocity-config-tool-26226f5-universalmutator-greedy.txt\n```\n\n`${HOME}/exli/results/minimized` stores the minimized tests by 4\ndifferent algorithms: `Greedy`, `GE`, `GRE`,\n`HGS`, which are called r2 tests in the paper.  For\nexample, the minimized tests by greedy algorithm for the project\n`Bernardo-MG_velocity-config-tool` at commit `26226f5` are stored in\n`${HOME}/exli/results/minimized/Bernardo-MG_velocity-config-tool-26226f5-universalmutator-greedy.txt`\n\n```txt\nBernardo-MG_velocity-config-tool#com.bernardomg.velocity.tool.ConfigTool_200Test#testLine204()#r1\nBernardo-MG_velocity-config-tool#com.bernardomg.velocity.tool.ConfigTool_288Test#testLine290()#r1\nBernardo-MG_velocity-config-tool#com.bernardomg.velocity.tool.ConfigTool_288Test#testLine302()#r0\n```\n\nTests that end with `#r0` are from R0 tests, end with `#r1` are from R1 tests. These 3 tests are the minimized tests (`r2` tests) by the greedy algorithm.\n\n`${HOME}/exli/results/itests-without-mutants` stores the inline tests whose target statements do not have mutants. For example, the inline tests for the project `Bernardo-MG_velocity-config-tool` at commit `26226f5` are stored in `${HOME}/exli/results/itests-without-mutants/Bernardo-MG_velocity-config-tool-26226f5-universalmutator.txt`\n\n```txt\nBernardo-MG_velocity-config-tool;com.bernardomg.velocity.tool.ConfigTool;195;197\nBernardo-MG_velocity-config-tool;com.bernardomg.velocity.tool.ConfigTool;195;198\nBernardo-MG_velocity-config-tool;com.bernardomg.velocity.tool.ConfigTool;195;199\n```\n\nAfter merging `${HOME}/exli/results/minimized/Bernardo-MG_velocity-config-tool-26226f5-universalmutator-greedy.txt` and `${HOME}/exli/results/itests-without-mutants/Bernardo-MG_velocity-config-tool-26226f5-universalmutator.txt`, the final r2 tests are stored in `${HOME}/exli/results/r2/Bernardo-MG_velocity-config-tool-26226f5-universalmutator-greedy.txt`\n\n```txt\nBernardo-MG_velocity-config-tool;com.bernardomg.velocity.tool.ConfigTool;200;204;r1\nBernardo-MG_velocity-config-tool;com.bernardomg.velocity.tool.ConfigTool;288;290;r1\nBernardo-MG_velocity-config-tool;com.bernardomg.velocity.tool.ConfigTool;288;302;r0\nBernardo-MG_velocity-config-tool;com.bernardomg.velocity.tool.ConfigTool;195;197;r1\nBernardo-MG_velocity-config-tool;com.bernardomg.velocity.tool.ConfigTool;195;198;r1\nBernardo-MG_velocity-config-tool;com.bernardomg.velocity.tool.ConfigTool;195;199;r1\n```\n\n\u003e Running `python -m exli.eval batch_get_r2_tests` will merge all the minized tests with greedy algorithm into one file `${HOME}/exli/results/r2-universalmutator-greedy-passed-tests.txt`\n\n### Reproduce the results in the paper\nSee the [Reproduce the results](./REPRODUCE.md) for more details.\n\n## Citation\nIf you have used ExLi in a research project, please cite the research paper in any related publication:\n\nTitle: [Extracting Inline Tests from Unit Tests](https://dl.acm.org/doi/abs/10.1145/3597926.3598149)\n\nAuthors: [Yu Liu](https://sweetstreet.github.io/), [Pengyu Nie](https://pengyunie.github.io/), [Anna Guo](https://www.linkedin.com/in/anna-y-guo/), [Milos Gligoric](http://users.ece.utexas.edu/~gligoric/), [Owolabi Legunsen](https://mir.cs.illinois.edu/legunsen/)\n\n```bibtex\n@inproceedings{LiuISSTA23EXLI,\n  title =        {Extracting Inline Tests from Unit Tests},\n  author =       {Yu Liu and Pengyu Nie and Anna Guo and Milos Gligoric and Owolabi Legunsen},\n  pages =        {1--13},\n  booktitle =    {Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis},\n  year =         {2023},\n}\n```\n\n```bibtex\n@inproceedings{LiuFSE24EXLI,\n  title =        {ExLi: An Inline-Test Generation Tool for Java},\n  author =       {Yu Liu and Aditya Thimmaiah and Owolabi Legunsen and Milos Gligoric},\n  pages =        {1--5},\n  booktitle =    {Proceedings of the 32nd ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (Demonstrations)},\n  year =         {2024},\n}\n```\n\n[paper-url]: https://dl.acm.org/doi/10.1145/3597926.3598149\n[itest-url]: https://github.com/EngineeringSoftware/inlinetest","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fengineeringsoftware%2Fexli","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fengineeringsoftware%2Fexli","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fengineeringsoftware%2Fexli/lists"}