{"id":20275606,"url":"https://github.com/softsec-kaist/fuzzle-artifact","last_synced_at":"2025-07-02T22:33:46.314Z","repository":{"id":54599857,"uuid":"516325953","full_name":"SoftSec-KAIST/Fuzzle-artifact","owner":"SoftSec-KAIST","description":"Artifact evaluation repository for Fuzzle","archived":false,"fork":false,"pushed_at":"2022-12-27T05:55:09.000Z","size":11087,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-03-04T01:31:27.502Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SoftSec-KAIST.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-07-21T10:30:28.000Z","updated_at":"2023-03-14T07:33:38.000Z","dependencies_parsed_at":"2023-01-31T02:45:49.735Z","dependency_job_id":null,"html_url":"https://github.com/SoftSec-KAIST/Fuzzle-artifact","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/SoftSec-KAIST/Fuzzle-artifact","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SoftSec-KAIST%2FFuzzle-artifact","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SoftSec-KAIST%2FFuzzle-artifact/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SoftSec-KAIST%2FFuzzle-artifact/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SoftSec-KAIST%2FFuzzle-artifact/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SoftSec-KAIST","download_url":"https://codeload.github.com/SoftSec-KAIST/Fuzzle-artifact/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SoftSec-KAIST%2FFuzzle-artifact/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263227205,"owners_count":23433856,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-14T13:10:06.024Z","updated_at":"2025-07-02T22:33:46.293Z","avatar_url":"https://github.com/SoftSec-KAIST.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Fuzzle Artifact\n\n[Fuzzle](https://github.com/SoftSec-KAIST/Fuzzle) is a bug synthesizer that\ngenerates buggy benchmarks for evaluating fuzzers. This repository contains\nartifacts for the experiments in our paper in ASE 2022, \"Fuzzle: Making a Puzzle\nfor Fuzzers.\"\n\n## Setup\n\nFrom the `exp` directory, run `eval_setup.sh` script to build docker images and\nset up python environments. The script may take up to a few hours to complete.\n\nNote that you need python 3.7+ and z3 solver to generate benchmark programs\nusing Fuzzle.\n\n## Replicating experiments in the paper\n\n### Evaluation of benchmark generation\n\nTo generate the benchmark that is of the same configuration as the benchmark\nused in the paper (described in Section 5.2), run `eval_benchmark.sh`.\n\n```\n$ ./exp/eval_benchmark.sh \u003cOUT_DIR\u003e\n```\n\nThe above command will take approximately 10-20 minutes to finish. Once the\nscript has finished running, the generated programs will be stored in the\nspecified directory. In the output directory, there will be a subdirectory for\neach parameter (algorithm, cycle, size, and generator) which will contain 5\nsubdirectories as follows:\n\n- `src`: contains C source code of generated programs\n- `bin`: contains compiled binaries of generated programs\n- `txt`: contains array form of mazes used in generating programs\n- `png`: contains images (.png files) of mazes used in generating programs\n- `sln`: contains shortest solution path for each maze\n\n### Comparing fuzzers on Fuzzle-generated benchmark\n\nTo replicate experiment for each RQ# in the paper, run `eval_RQ#.sh`.\n\n```\n$ ./exp/eval_RQ#.sh \u003cOUT_DIR\u003e\n```\n\nTime required for each RQ: (full version)\n\n| RQ #     |  CPU hours |\n| -------- |:----------:|\n| RQ2      |    60,000  |\n| RQ3      |    24,000  |\n| RQ4      |    27,000  |\n| RQ5      |    16,800  |\n| RQ6      |        72  |\n\nEach fuzzing experiment will be run in an isolated docker container each\nassigned with one CPU core. In our evaluation, the experiments were conducted on\nservers of the specification: 88 Intel Xeon E5-2699 v4 CPU cores with 2.20 GHz\nand 128 GB of memory.\n\nNote that you should fix the value of `NUM_WORKERS` in `run_tools.py` and\n`run_tools_binutils.py` to the number of availalbe CPU cores before running the\nexperiments. Currently, it is set to 1 to use only one CPU core at a time.\n\nAlso, the default memory limit given to each docker container is 8GB. To change\nthe memory limit given to each fuzzing instance, you can modify -m option in\n`run_tools.py`.\n\nIn the specified output directory, following will be stored after the command\nhas finished executing:\n\n- fuzzing output (generated testcases)\n- coverage and bug finding results for each fuzzing run (.csv file)\n- summary report\n\nThe summary report will be displayed at the end of the experiment, showing the\naverage branch coverage ('Coverage'), average percentage of bugs found ('Bugs'),\nand average time to expose bugs ('TTE').\n\nFor each RQ#, a script for shorter version of evaluation is also available as\n`eval_RQ#_short.sh`.\n\nTime required for each RQ: (short version)\n\n| RQ #     |  CPU hours |\n| -------- |:----------:|\n| RQ2      |     1,200  |\n| RQ3      |       480  |\n| RQ4      |       540  |\n| RQ5      |       336  |\n| RQ6      |        18  |\n\nNote that in the shorter evaluation, only 1 seed, 2 repeats per seed, and 6\nhours per run were used instead of 5 seeds, 5 repeats per seed, and 24 hours per\nrun of the original experiment settings.\n\n### Evaluation of overall workflow\n\nA script `test.sh` can be used to quickly test each feature of Fuzzle: (1)\ngenerating maze-based benchmark, (2) running fuzzers, and (3) storing and\nvisualizing results. The experiment takes total of 6.5 hours to run on a\nsingle-core CPU.\n\nSimilar to running `eval_RQ#.sh` scripts, you need to provide the output path:\n\n```\n$ ./exp/test.sh \u003cOUT_PATH\u003e\n```\nThe above command will generate a test benchmark consisting of 3 programs and\nwill run 2 fuzzers (afl and afl++) on the generated programs. The png file\ngenerated using visualization feature will be the same as Figure 5.c in the\npaper but with code coverage results from 1 hour of fuzzing instead of 24\nhours.\n\n## Dataset\n\nThe benchmark used in the paper is in `eval_mazes` directory and the raw data\nobtained from fuzzing experiments can be downloaded from\n[Fuzzle-dataset](https://doi.org/10.5281/zenodo.6520108).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsoftsec-kaist%2Ffuzzle-artifact","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsoftsec-kaist%2Ffuzzle-artifact","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsoftsec-kaist%2Ffuzzle-artifact/lists"}