{"id":20275599,"url":"https://github.com/softsec-kaist/fuzzle","last_synced_at":"2025-04-11T05:24:20.696Z","repository":{"id":53061671,"uuid":"516325444","full_name":"SoftSec-KAIST/Fuzzle","owner":"SoftSec-KAIST","description":"Fuzzle: Making a Puzzle for Fuzzers (ASE'22)","archived":false,"fork":false,"pushed_at":"2024-03-03T06:32:48.000Z","size":358095,"stargazers_count":45,"open_issues_count":0,"forks_count":8,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-03-25T03:33:41.182Z","etag":null,"topics":["benchmark","fuzzing"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SoftSec-KAIST.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2022-07-21T10:28:42.000Z","updated_at":"2025-03-15T19:48:00.000Z","dependencies_parsed_at":"2024-03-03T07:28:37.919Z","dependency_job_id":"9d7ce84a-a6ab-4b1b-9894-04d8f5be910b","html_url":"https://github.com/SoftSec-KAIST/Fuzzle","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SoftSec-KAIST%2FFuzzle","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SoftSec-KAIST%2FFuzzle/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SoftSec-KAIST%2FFuzzle/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SoftSec-KAIST%2FFuzzle/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SoftSec-KAIST","download_url":"https://codeload.github.com/SoftSec-KAIST/Fuzzle/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248346639,"owners_count":21088492,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark","fuzzing"],"created_at":"2024-11-14T13:10:00.662Z","updated_at":"2025-04-11T05:24:20.678Z","avatar_url":"https://github.com/SoftSec-KAIST.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Fuzzle\n\nFuzzle is a bug synthesizer that generates buggy benchmarks for evaluating\nfuzzers. Fuzzle uses randomly created mazes and path constraints from previous\nCVEs to generate subject programs. The details of Fuzzle can be found in our\npaper \"Fuzzle: Making a Puzzle for Fuzzers\" (ASE 2022).\n\n## Installation\n\nTo build Fuzzle and setup the docker and python environment, run the following:\n\n```\n$ git clone https://github.com/SoftSec-KAIST/Fuzzle\n$ cd Fuzzle\n\n# Build docker images (this step may take a few hours to complete)\n$ ./scripts/build_all_dockers.sh\n\n# Install the dependencies\n$ python3 -m pip install -r ./maze-gen/requirements.txt\n```\nNote that you need python 3.7+ and Z3 solver to generate benchmark programs\nusing Fuzzle.\n\nFor more detailed installation instructions, please see  `INSTALL.md`.\n\n## Extension for Multiple Bugs\n\nFuzzle was modified to generate programs that contain multiple bugs for evaluating multi-target directed fuzzers. To use this version of Fuzzle, please go to [multi-target](https://github.com/SoftSec-KAIST/Fuzzle/tree/multi-target) branch of this repository.\n\nThe details of the modifications can be found in the paper \"On the Effectiveness of Synthetic Benchmarks for Evaluating Directed Grey-box Fuzzers\" (APSEC 2023).\n\n## Usage\n\nFor a quick tutorial on how to use Fuzzle, please skip to the\n[tutorial](#tutorial).\n\n### Generating a benchmark\n\nYou can generate a buggy benchmark using `generate.sh` in the `scripts`\ndirectory. You need to specify values for the following parameters using the\ncommand line arguments:\n- Maze generation algorithm (`-a`): Backtracking, Kruskal, Prims, Wilsons,\n  Sidewinder\n- Width of the maze (`-w`): any integer greater than 2\n- Height of the maze (`-h`): any integer greater than 2\n- Output directory (`-o`): path to the output directory\n\nYou can optionally specify the following parameters:\n- Pseudorandom seed (`-r`): any integer between 0 and 2^32 - 1\n- Percentage of cycles (`-c`): any integer between 0 and 100\n- Maze exit (`-e`): default or random\n- Generator (`-g`): default_gen or CVE_gen\n\nNote, if you use CVE_gen for the generator, you must also provide a path to SMT\nfile with `-s` option.\n\nFor example, the following command generates a default program based on a 10x10\nmaze created with Wilson's algorithm.\n\n```\n$ ./scripts/generate.sh -a Wilsons -w 10 -h 10 -o \u003cOUT_PATH\u003e\n```\n\nA different program can be generated using the same maze but with realistic\nconstraints obtained from the previous CVE (e.g., CVE-2016-4487).\n\n```\n$ ./scripts/generate.sh -a Wilsons -w 10 -h 10 -g CVE_gen -s ./CVEs/CVE-2016-4487.smt2 -o \u003cOUT_PATH\u003e\n```\n\nAfter the script finishes running, the output directory specified with\n`\u003cOUT_PATH\u003e` will contain 5 subdirectories as follows:\n\n- `src`: contains C source code of generated programs\n- `bin`: contains compiled binaries of generated programs\n- `txt`: contains array form of mazes used in generating programs\n- `png`: contains images (.png files) of mazes used\n- `sln`: contains the shortest solution path for each maze\n\n\n### Generating a large benchmark\n\nTo generate multiple programs for building a large benchmark, you can use\n`generate_benchmark.py` as follows:\n\n```\n$ cd scripts\n$ python3 ./generate_benchmark.py \u003cFILE_PATH\u003e \u003cOUT_PATH\u003e\n```\n\nHere, `\u003cFILE_PATH\u003e` is a path to the file that specifies the configuration of\nprograms to be generated. The template for the configuration of a program is:\n\n```\n\u003calgorithm\u003e,\u003cwidth\u003e,\u003cheight\u003e,\u003cseed\u003e,\u003cindex\u003e,\u003cpercentage of cycles\u003e,\u003cgenerator\u003e\n```\n\nBelow is an example file for the two programs generated above with\n`generate.sh`.\n\n```\nWilsons,10,10,1,1,100percent,default_gen\nWilsons,10,10,1,1,100percent,CVE-2016-4487_gen\n```\n\n### Running a fuzzing experiment\n\nTo run fuzzers on the generated benchmark in a separate docker container:\n\n```\n$ python3 ./scripts/run_tools.py \u003cCONFIG_FILE\u003e \u003cFUZZ_OUT\u003e\n```\n\nAn example of the configuration file (`\u003cCONFIG_FILE\u003e`) is provided below.\n\n```\n{\n    \"MazeList\" : \"../examples/example1.list\",\n    \"Repeats\"  : 1,\n    \"Duration\" : 60,\n    \"MazeDir\"  : \"../examples/example1_benchmark\",\n    \"Tools\"    : [\"afl\", \"afl++\", \"aflgo\", \"eclipser\", \"fuzzolic\"]\n}\n```\n\n- `MazeList`: path to the list of programs in the benchmark\n- `Repeats`: number of repeats for each fuzzer\n- `Duration`: length of fuzzing campaign in minutes\n- `MazeDir`: path to a directory that contains benchmark programs\n- `Tools`: one or more of the available fuzzers (`afl`, `afl++`, `aflgo`,\n  `eclipser`, `fuzzolic`)\n\nNote that all paths (`MazeList` and `MazeDir`) should be either absolute paths\nor relative paths from the `scripts` directory.\n\nMore examples of configuration files are provided under `examples` directory.\nRunning each example should take about 1 hour. Please refer to\n[Examples](#examples) section for more details.\n\nAfter the experiment is finished, the output directory (`\u003cFUZZ_OUT\u003e`) will\ncontain generated testcases from each run as well as code coverage measured with\ngcov.\n\n### Storing and summarizing results\n\nOnce the fuzzing campaign is finished, the coverage and bug finding results can\nbe summarized in csv format using the script as follows:\n\n```\n$ ./scripts/save_results.sh \u003cFUZZ_OUT\u003e \u003cFILE_PATH\u003e \u003cPARAM\u003e \u003cDURATION\u003e \u003cMODE\u003e\n```\n\n- `\u003cFUZZ_OUT\u003e`: directory that contains fuzzing outputs\n- `\u003cFILE_PATH\u003e`: path to save the output file\n\nThe script summarizes the fuzzing results based on the given parameter and\nfuzzing duration. For example, by running:\n\n```\n$ ./scripts/save_results.sh \u003cFUZZ_OUT\u003e \u003cFILE_PATH\u003e Algorithm 24 paper\n```\neach fuzzer's performance after 24 hours across different maze generation\nalgorithms will be displayed on the screen.\n\n- `\u003cPARAM\u003e`: one of four parameters of Fuzzle (`Algorithm`, `Size`, `Cycle`, `Generator`)\n- `\u003cDURATION\u003e`: length of fuzzing campaign in hours\n- `\u003cMODE\u003e`: `paper` for displaying tables similar to the ones shown in the\n  paper, `fuzzer` for displaying summarized results for each fuzzer.\n\n### Visualizing coverage results\n\nYou can also visualize the code coverage results in the form of 2D maze, where\neach cell represents a function in the program. The script for visualization\ntakes maze array (.txt), coverage results (.gcov), and maze size as input and\nreturns a png file.\n\n```\n$ python3 ./scripts/visualize.py \u003cTXT_PATH\u003e \u003cCOV_PATH\u003e \u003cOUT_PATH\u003e \u003cSIZE\u003e\n```\n- `\u003cTXT_PATH\u003e`: path to .txt file for the chosen maze. Note that such files are\n  automatically created under the `txt` directory when generating a benchmark.\n- `\u003cCOV_PATH\u003e`: path to .gcov file that contains coverage information. The\n  script for running fuzzers (`run_tools.py`) saves hourly coverage results in\nthe output directory.\n\n### Examples\n\nThere are 3 sets of examples (`example#.list` and `example#.conf`) each of which\ntakes about 1 hour to run.\n\nTo generate the benchmark and test the fuzzers using these examples, run the\nfollowing commands from the `scripts` directory:\n\n```\n$ python3 ./generate_benchmark.py ../examples/example#.list ../examples/example#_benchmark\n$ python3 ./run_tools.py ../examples/example#.conf ../examples/example#_outputs\n$ ./save_results.sh ../examples/example#_outputs ../examples/example#_outputs/example#_results \u003cPARAM\u003e 0 paper \u003e ../examples/example#_outputs/example#_summary\n$ cat ../examples/example#_outputs/example#_summary\n```\n\nNote that you should replace # with the example number. Also, you should use the\nmatching pair to run the experiment. For example, if you used `example1.list` to\ngenerate the benchmark, you should use `example1.conf` to run the fuzzers.\n\nBecause each example varies different parameters, `\u003cPARAM\u003e` for\n`save_results.sh` script is specific to each example.\n\n- For example 1: `Algorithm`\n- For example 2: `Size`\n- For example 3: `Generator`\n\nThe fuzzing results for each experiment will be saved in the `example#_outputs`\ndirectory.\n\n### Tutorial\n\nIn this tutorial, we will generate a benchmark consisting of 5 programs and run\nAFL on the generated benchmark for 5 minutes per program. We will then visualize\nthe coverage results from one of the fuzzing campaigns using Fuzzle.\n\n1. Generate the benchmark with:\n```\n$ cd scripts\n$ python3 ./generate_benchmark.py ../tutorial/programs.list ../tutorial/benchmark\n```\nNote that `programs.list` contains the configurations of 5 programs to be\ngenerated.\n\n2. Run AFL:\n```\n$ python3 ./run_tools.py ../tutorial/run.conf ../tutorial/outputs\n```\nThe `run.conf` file specifies the path to the generated benchmark, the duration\nof the fuzzing run (5 minutes), and the fuzzer to run (AFL). This step will take\napproximately 1 hour.\n\n3. Store and visualize results:\n```\n$ ./save_results.sh ../tutorial/outputs ../tutorial/results Algorithm 0 paper \u003e ../tutorial/summary\n$ cat ../tutorial/summary\n$ PATH_TO_TXT=../tutorial/benchmark/txt/Wilsons_20x20_1_1.txt\n$ PATH_TO_COV=../tutorial/outputs/cov_gcov_Wilsons_20x20_1_1_25percent_default_gen_afl_0/Wilsons_20x20_1_1_25percent_default_gen_afl_0.c.gcov\n$ python3 ./visualize.py $PATH_TO_TXT $PATH_TO_COV ../tutorial/wilsons 20\n```\nThe fuzzing results will be stored in `results.csv` and the summary of results\nwill be printed out to the screen. You can also check the visual aid generated\n(`wilsons.png`) in the `tutorial` directory.\n\n## Artifact\n\nWe publicize the artifacts to help reproduce the experiments in our paper. Please\ncheck our [Fuzzle-Artifact](https://doi.org/10.5281/zenodo.7031393) repository.\n\n## Citation\n\nIf you use Fuzzle in scientific work, consider citing our paper:\n\n```bibtex\n@INPROCEEDINGS{lee:ase:2022,\n  author = {Haeun Lee and Soomin Kim and Sang Kil Cha},\n  title = {{Fuzzle}: Making a Puzzle for Fuzzers},\n  booktitle = {Proceedings of the International Conference on Automated Software Engineering},\n  year = 2022\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsoftsec-kaist%2Ffuzzle","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsoftsec-kaist%2Ffuzzle","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsoftsec-kaist%2Ffuzzle/lists"}