{"id":15116013,"url":"https://github.com/google/oss-fuzz-gen","last_synced_at":"2025-09-27T21:31:35.376Z","repository":{"id":220046721,"uuid":"747952953","full_name":"google/oss-fuzz-gen","owner":"google","description":"LLM powered fuzzing via OSS-Fuzz.","archived":false,"fork":false,"pushed_at":"2024-12-24T06:46:57.000Z","size":4906,"stargazers_count":1047,"open_issues_count":95,"forks_count":126,"subscribers_count":17,"default_branch":"main","last_synced_at":"2025-01-07T16:08:41.217Z","etag":null,"topics":["ai","fuzzing","llm","security"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/google.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-01-25T00:51:49.000Z","updated_at":"2025-01-06T18:51:26.000Z","dependencies_parsed_at":"2024-02-18T23:32:57.347Z","dependency_job_id":"0f5f70a7-d13b-44c4-8f20-84f3ab356fc5","html_url":"https://github.com/google/oss-fuzz-gen","commit_stats":null,"previous_names":["google/oss-fuzz-gen"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google%2Foss-fuzz-gen","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google%2Foss-fuzz-gen/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google%2Foss-fuzz-gen/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google%2Foss-fuzz-gen/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/google","download_url":"https://codeload.github.com/google/oss-fuzz-gen/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":234460505,"owners_count":18836837,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","fuzzing","llm","security"],"created_at":"2024-09-26T01:44:07.454Z","updated_at":"2025-09-27T21:31:35.370Z","avatar_url":"https://github.com/google.png","language":"Python","funding_links":[],"categories":["Python","Emulation and Fuzzing"],"sub_categories":[],"readme":"# A Framework for Fuzz Target Generation and Evaluation\n\nThis framework generates fuzz targets for real-world `C`/`C++/Java/Python` projects with\nvarious Large Language Models (LLM) and benchmarks them via the\n[`OSS-Fuzz` platform](https://github.com/google/oss-fuzz).\n\nMore details available in [AI-Powered Fuzzing: Breaking the Bug Hunting Barrier](https://security.googleblog.com/2023/08/ai-powered-fuzzing-breaking-bug-hunting.html):\n![Alt text](images/Overview.png \"Overview\")\n\nCurrent supported models are:\n- Vertex AI code-bison\n- Vertex AI code-bison-32k\n- Gemini Pro\n- Gemini Ultra\n- Gemini Experimental\n- Gemini 1.5\n- OpenAI GPT-3.5-turbo\n- OpenAI GPT-4\n- OpenAI GPT-4o\n- OpenAI GPT-4o-mini\n- OpenAI GPT-4-turbo\n- OpenAI GPT-3.5-turbo (Azure)\n- OpenAI GPT-4 (Azure)\n- OpenAI GPT-4o (Azure)\n\nGenerated fuzz targets are evaluated with four metrics against the most up-to-date data from production environment:\n- Compilability\n- Runtime crashes\n- Runtime coverage\n- Runtime line coverage diff against existing human-written fuzz targets in `OSS-Fuzz`.\n\nHere is a sample experiment result from 2024 Jan 31.\nThe experiment included [1300+ benchmarks](./benchmark-sets/all) from 297 open-source projects.\n\n![image](https://github.com/google/oss-fuzz-gen/assets/759062/fa53698b-e44c-4b58-b5e7-798337c8b752)\n\nOverall, this framework manages to successfully leverage LLMs to generate valid fuzz targets (which generate non-zero coverage increase)\nfor 160 C/C++ projects. The maximum line coverage increase is 29% from the existing human-written targets.\n\nNote that these reports are not public as they may contain undisclosed vulnerabilities.\n\n## Usage\n\nCheck our detailed [usage guide](./USAGE.md) for instructions on how to run this framework and generate reports based on the results.\n\n## Independent Agent Execution and Evaluation\nYou can also execute or evaluate individual agents without running full experiments, using the integrated agent execution framework.\nSee the [framework's documentation](./agent_tests/readme.md) for detailed instructions on how to run individual agents or sequence of agents.\n\n## Collaborations\nInterested in research or open-source community collaborations?\nPlease feel free to create an issue or email us: oss-fuzz-team@google.com.\n\n\u003cimg src=\"images/Collaboration.png\" width=\"200\" height=\"200\"\u003e\n\n## Bugs Discovered\n\nSo far, we have reported 30 new bugs/vulnerabilities found by automatically generated targets built\nby this framework:\n| Project |    Bug    |    LLM    | Prompt Builder | Target oracle |\n| ------- | --------- | --------- | --------------- | ------- |\n| [`cJSON`](https://github.com/google/oss-fuzz/tree/master/projects/cjson) | [OOB read](https://github.com/DaveGamble/cJSON/issues/800) | Vertex AI | [Default](prompts/template_xml) | Far reach, low coverage |\n| [`libplist`](https://github.com/google/oss-fuzz/tree/master/projects/libplist) | [OOB read](https://github.com/libimobiledevice/libplist/issues/244) | Vertex AI | [Default](prompts/template_xml) | Far reach, low coverage |\n| [`hunspell`](https://github.com/google/oss-fuzz/tree/master/projects/hunspell) | [OOB read](https://github.com/hunspell/hunspell/issues/996) | Vertex AI | [default](prompts/template_xml) | Far reach, low coverage |\n| [`zstd`](https://github.com/google/oss-fuzz/tree/master/projects/zstd) | [OOB write](https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=67497) | Vertex AI | [default](prompts/template_xml) | Far reach, low coverage |\n| [`gdbm`](https://github.com/google/oss-fuzz/tree/master/projects/gdbm) | [Stack buffer underflow](https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=67483) | Vertex AI | [default](prompts/template_xml) | Far reach, low coverage |\n| [`hoextdown`](https://github.com/google/oss-fuzz/tree/master/projects/hoextdown) | [Use of uninitialised memory](https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=67516) | Vertex AI | [default](prompts/template_xml) | Far reach, low coverage |\n| [`pjsip`](https://github.com/google/oss-fuzz/tree/master/projects/pjsip) | [OOB read](https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=71356) | Vertex AI | [Default](prompts/template_xml) | Low coverage with fuzz keyword + easy params far reach |\n| [`pjsip`](https://github.com/google/oss-fuzz/tree/master/projects/pjsip)  | [OOB read](https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=71357) | Vertex AI | [Default](prompts/template_xml) | Low coverage with fuzz keyword + easy params far reach |\n| [`gpac`](https://github.com/google/oss-fuzz/tree/master/projects/gpac) | [OOB read](https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=71358) | Vertex AI | [Default](prompts/template_xml) | Low coverage with fuzz keyword + easy params far reach |\n| [`gpac`](https://github.com/google/oss-fuzz/tree/master/projects/gpac)  | [OOB read/write](https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=71542) | Vertex AI | [Default](prompts/template_xml) | All |\n| [`gpac`](https://github.com/google/oss-fuzz/tree/master/projects/gpac)  | [OOB read](https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=71543) | Vertex AI | [Default](prompts/template_xml) | All |\n| [`gpac`](https://github.com/google/oss-fuzz/tree/master/projects/gpac)  | [OOB read](https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=71544) | Vertex AI | [Default](prompts/template_xml) | All |\n| [`sqlite3`](https://github.com/google/oss-fuzz/tree/master/projects/sqlite3) | [OOB read](https://issues.oss-fuzz.com/issues/42538590) | Vertex AI | [Default](prompts/template_xml) | All |\n| [`htslib`](https://github.com/google/oss-fuzz/tree/master/projects/htslib) | [OOB read](https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=71740) | Vertex AI | [Default](prompts/template_xml) | All |\n| [`libical`](https://github.com/google/oss-fuzz/tree/master/projects/libical) | [OOB read](https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=71741) | Vertex AI | [Default](prompts/template_xml) | All |\n| [`croaring`](https://github.com/google/oss-fuzz/tree/master/projects/croaring) | [OOB read](https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=71738) | Vertex AI | [Test-to-harness](prompts/template_xml) | All |\n| [`openssl`](https://github.com/google/oss-fuzz/tree/master/projects/openssl) | [CVE-2024-9143](https://www.cve.org/CVERecord?id=CVE-2024-9143) - [OOB read/write](https://g-issues.oss-fuzz.com/issues/42538437) | Vertex AI | [Default](prompts/template_xml) | All |\n| [`liblouis`](https://github.com/google/oss-fuzz/tree/master/projects/liblouis) | [Use of uninitialised memory](https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=71354) | Vertex AI | Test-to-harness | Test identifier |\n| [`libucl`](https://github.com/google/oss-fuzz/tree/master/projects/libucl) | [OOB read](https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=71359) | Vertex AI | [Default](prompts/template_xml) | Low coverage with fuzz keyword + easy params far reach |\n| [`openbabel`](https://github.com/google/oss-fuzz/tree/master/projects/openbabel) | [Use after free](https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=71360) | Vertex AI | [Default](prompts/template_xml) | Low coverage with fuzz keyword + easy params far reach |\n| [`libyang`](https://github.com/google/oss-fuzz/tree/master/projects/libyang) | [OOB read](https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=71619) | Vertex AI | [Default](prompts/template_xml) | All |\n| [`openbabel`](https://github.com/google/oss-fuzz/tree/master/projects/openbabel) | [OOB read](https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=71650) | Vertex AI | [Default](prompts/template_xml) | All |\n| [`exiv2`](https://github.com/google/oss-fuzz/tree/master/projects/exiv2) | [OOB read](https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=71759) | Vertex AI | [Default](prompts/template_xml) | All |\n| Undisclosed | Java RCE (pending maintainer triage) | Vertex AI |  [Default](prompts/template_xml) | Far reach, low coverage |\n| Undisclosed | Regexp DoS (pending maintainer triage) | Vertex AI |  [Default](prompts/template_xml) | Far reach, low coverage |\n| Undisclosed | [OOB read](https://issues.oss-fuzz.com/issues/370872803) | Vertex AI | [Default](prompts/template_xml) | All |\n| Undisclosed | [OOB write](https://issues.oss-fuzz.com/issues/378009361) | Vertex AI | [Default](prompts/template_xml) | All |\n| Undisclosed | [OOB read](https://issues.oss-fuzz.com/issues/391234167) | Vertex AI | [Default](prompts/template_xml) | All |\n| Undisclosed | [OOB read](https://issues.oss-fuzz.com/issues/391453674) | Vertex AI | [Default](prompts/template_xml) | All |\n| Undisclosed | [Use after free](https://issues.oss-fuzz.com/issues/391456091) | Vertex AI | Agent prompt | All |\n\nThese bugs could only have been discovered with newly generated targets. They were not reachable with existing OSS-Fuzz targets.\n\n## Current top coverage improvements by project\n\n| Project | Total coverage gain\t| Total relative gain\t| OSS-Fuzz-gen total covered lines | OSS-Fuzz-gen new covered lines | Existing covered lines | Total project lines |\n| --------| ------------------- | ------------------- | -------------------------------- | ------------------------------ | ---------------------- | ------------------- |\n| phmap | 98.42% | 205.75% | 1601 | 1181 | 574 | 1120 |\n| usbguard | 97.62% | 26.04% | 24550 | 5463 | 20979 | 3564 |\n| onednn | 96.67% | 7057.14% | 5434 | 5434 | 77 | 210 |\n| avahi | 82.06% | 155.90% | 3358 | 2814 | 1805 | 3046 |\n| pugixml | 72.98% | 194.95% | 9015 | 6646 | 3409 | 7662 |\n| librdkafka | 66.88% | 845.57% | 5019 | 4490 | 531 | 1169 |\n| casync | 66.75% | 903.23% | 1171 | 1120 | 124 | 1678 |\n| tomlplusplus | 61.06% | 331.10% | 4755 | 3652 | 1103 | 5981 |\n| astc-encoder | 59.35% | 177.88% | 2726 | 1745 | 981 | 2940 |\n| mruby | 48.56% | 0.00% | 34493 | 34493 | 0 | 71038 |\n| arduinojson | 42.10% | 85.80% | 3344 | 1800 | 2098 | 4276 |\n| json | 41.13% | 66.51% | 5051 | 3339 | 5020 | 8119 |\n| double-conversion | 40.40% | 88.12% | 1663 | 779 | 884 | 1928 |\n| tinyobjloader | 38.26% | 77.01% | 1157 | 717 | 931 | 1874 |\n| glog | 38.18% | 58.69% | 895 | 331 | 564 | 867 |\n| cppitertools | 35.78% | 45.07% | 253 | 151 | 335 | 422 |\n| eigen | 35.38% | 190.70% | 2643 | 1947 | 1021 | 5503 |\n| glaze | 34.55% | 30.06% | 2920 | 2416 | 8036 | 6993 |\n| rapidjson | 31.83% | 148.07% | 1585 | 958 | 647 | 3010 |\n| libunwind | 30.58% | 83.25% | 2899 | 1342 | 1612 | 4388 |\n| openh264 | 30.07% | 50.14% | 6607 | 5751 | 11470 | 19123 |\n\n\\* \"Total project lines\" measures the source code of the project-under-test compiled and linked by the preexisting human-written fuzz targets from OSS-Fuzz.\n\n\\* \"Total coverage gain\" is calculated using a denominator of the \"Total project lines\". \"Total relative gain\" is the increase in coverage compared to the old number of covered lines.\n\n\\* Additional code from the project-under-test maybe included when compiling the new fuzz targets and result in high percentage gains.\n\n## Citing This Work\nPlease click on the _'Cite this repository'_ button located on the right-hand side of this GitHub page for citation details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle%2Foss-fuzz-gen","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgoogle%2Foss-fuzz-gen","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle%2Foss-fuzz-gen/lists"}