{"id":20925835,"url":"https://github.com/comby-tools/comby-decomposer","last_synced_at":"2025-05-13T17:33:19.106Z","repository":{"id":48278261,"uuid":"475563392","full_name":"comby-tools/comby-decomposer","owner":"comby-tools","description":"Decompose source code into templates and fragments for any language.","archived":false,"fork":false,"pushed_at":"2022-08-29T08:38:23.000Z","size":771,"stargazers_count":13,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2023-03-12T05:58:23.514Z","etag":null,"topics":["compiler","fuzzing"],"latest_commit_sha":null,"homepage":"https://comby.dev/blog/2022/04/11/comby-decomposer-compiler-fuzzing","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/comby-tools.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-03-29T18:08:11.000Z","updated_at":"2023-03-03T15:39:31.000Z","dependencies_parsed_at":"2022-08-30T09:22:03.114Z","dependency_job_id":null,"html_url":"https://github.com/comby-tools/comby-decomposer","commit_stats":null,"previous_names":[],"tags_count":null,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/comby-tools%2Fcomby-decomposer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/comby-tools%2Fcomby-decomposer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/comby-tools%2Fcomby-decomposer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/comby-tools%2Fcomby-decomposer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/comby-tools","download_url":"https://codeload.github.com/comby-tools/comby-decomposer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225248634,"owners_count":17444279,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["compiler","fuzzing"],"created_at":"2024-11-18T20:35:24.291Z","updated_at":"2024-11-18T20:35:24.737Z","avatar_url":"https://github.com/comby-tools.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# comby-decomposer\n\n`comby-decomposer` is a tool that automatically decomposes program syntax into\n(1) a set of templates and (2) a set of concrete fragments extracted from the\nsyntax. Basically, it _un_-substitutes some input file to create a corpus of\ntemplates and fragments. \n\nHere's a visual example of outputs when we decompose a function with respect to parentheses `(...)` and braces `{...}`.\n\n![decompose](https://user-images.githubusercontent.com/888624/160704921-511c0609-5877-4a8e-ab39-f641baafd846.svg)\n\n\nYou can then use mix-and-match templates and fragments\nand substitute these back in to create new combinations of output \"programs\". It's been most useful [for fuzzing compilers](#why-comby-decomposer-mostly-for-fuzzing-compilers).\n\n\n## Running\n\n- [Install `comby`](https://github.com/comby-tools/comby#install-pre-built-binaries)\n- Install `fdupes` for your distribution (typically `sudo apt-get install fdupes` or `brew install fdupes`)\n\nCopy the programs you want to decompose into the `sources` directory. If you don't have any programs ready, you\ncan try it with some example programs in this repository:\n\n```bash\ncp try_example_solidity/* sources\n```\n\nThen, just run:\n\n`./run.sh .sol .sol`\n\nThis says \"decompose programs for all `.sol` files in the `sources` directory using the `.sol` language (defined by `comby -list`).\nYou get these outputs:\n\n- `fragments` contains the extracted fragments\n- `templates` contains the templatized versions of the input string.\n\nThese are deduplicated and unique. You can try the same for example rust programs:\n\n```bash\ncp try_example_rust/* sources\n./run.sh .rs .rs\n```\n\nand check out examples in `templates` and `fragments`!\n\n## Customizing decomposition\n\nBy default programs are decomposed around delimiter syntax like `(...)`,\n`{...}` and `[...]`. This means that `comby-decomposer` looks for this syntax\nand un-substitutes the program around these patterns. It will also look for nested syntax inside these. You can use any comby\npattern you like though! I These patterns are defined inside\n`extraction_specifications` using a simple [configuration file](https://comby.dev/docs/configuration) and you'll find the default ones\nfor parentheses, braces, and brackets in there already. But if you wanted to\nun-substitute around numbers in the input program, you can simply add a\nspecification like `extract_number.toml` that matches the regex `[0-9]+` like this:\n\n```\n[extract_number]\nmatch=':[~[0-9]+]'\n```\n\n(Tip: the string `:[~[0-9]+]` is just using the [comby syntax](https://comby.dev/docs/syntax-reference) `:[...]` to define a regular expression `[0-9]+`)\n\nNow if you delete the other specifications and `./run.sh .sol .sol` again on\nthe example program, you'll only get one decomposition (since there's only one\nnumber to un-substitute), and your `templates` and `fragments` folders contain\nthese:\n\n\n**templates**\n\n```solidity\nfunction f(uint:[1] arg) public {\n        g(notfound);\n}\n```\n\n**fragments**\n\n```\n256\n```\n\n### Smaller fragments and templates\n\nIf you have many or large programs to decompose, you can run `./postprocess.sh` after `./run.sh`. This will:\n\n- only keep templates and fragments that are 8KB to 28KB in size\n- delete templates with no holes\n- delete templates with more than 10 holes\n\nLook inside the `postprocess.sh` to customize it.\n\n## On-demand input generation\n\nIf you have `templates` and `fragments` and want to randomly generate new\nprograms by substituting fragments into templates, you can use the included\nserver. Just run these commands:\n\n```bash\nnpm install express body-parser @iarna/toml minimist\n\nexport NODE_OPTIONS=\"--max-old-space-size=8192\"\nnode server.js --generate 1.0\n```\n\nThen in a separate terminal, request a new input:\n\n`curl -X POST http://localhost:4448/mutate`\n\nEvery request generates a random input. The server picks a random template from\nthe `templates` directory, and substitutes up to `10` random fragments  from\nthe `fragments` directory. If you want more control over server generation\nyou'll have to dig into `server.js`.\n\n### See https://github.com/agroce/afl-compiler-fuzzer#01-usage to use the server with an AFL fuzzer\n\n## Customizing language grammar target\n\nComby supports a lot of languages (see `comby -list` for supported ones),\nbut even if you just created one, you'll probably get some mileage from using\nthe generic parser and default patterns: `./run .my-language-extension .generic`.\n\n## Why `comby-decomposer`? Mostly for fuzzing compilers...\n\n`comby-decomposer` was created to generate inputs to fuzz compilers. We can substitute\nand recombine templates and fragments to create new inputs.\n\n![image](https://user-images.githubusercontent.com/888624/187160236-1da1afdf-7650-45d8-8ce8-85fde06cd599.png)\n\n\n### If you want to use it with an AFL fuzzer that's been modified to request inputs from the server, see https://github.com/agroce/afl-compiler-fuzzer#01-usage.\n\nThe fun part is `comby-decomposer` is\nsyntax-aware and works on most any language (or structured input like JSON) to\nsome degree, and you can get started right away: You just need some example\nprograms for your program. No need to define a grammar to generate inputs, or\nknow exactly how the grammar of your target language works. That's because it\njust uses [Comby](https://github.com/comby-tools/comby) to do the decomposition\nwork. \n\n\nThe basic idea is that `comby-decomposer` creates inputs _that are much more\nlikely to be syntactically valid expressions_ inside _previously valid (now\ntemplatized) programs_. Generating inputs with templates and likely-valid\nexpressions help nudge a feedback-directed fuzzer (like AFL) to explore\n\"deeper\" code in a compiler _earlier_ than, say, just starting with a\ncompiler's initial test suite. So, instead of using only an initial test\ncorpus, `comby-decomposer` helps generate templates and fragments for\nsynthesizing new test programs on demand. It banks on the idea that these\ninputs are unlikely to be generated early (or at all) by default input\nmutations in a fuzzer's feedback loop.\n\nFor example, `comby-decomposer` generated this Solidity program that [sent the compiler into an infinite loop](https://github.com/ethereum/solidity/issues/10732):\n\n```solidity\ncontract C {\n    bytes32 constant x = x;\n    function f() public pure returns (uint t) {\n        assembly {\n            t := x\n        }\n    }\n}\n```\n\nIt also generated new valid-looking programs using the Zig compiler's test suite\nto find crashes deeper in the analysis and code generation phases of the compiler:\n\n```zig\nexport fn entry() void {\n\tvar x = false;\n    const p = @ptrCast(?*u0, \u0026x);\n    return p \u003c null;\n}\n```\n\nYou can find more Zig examples [here](https://github.com/ziglang/zig/issues/10121).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcomby-tools%2Fcomby-decomposer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcomby-tools%2Fcomby-decomposer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcomby-tools%2Fcomby-decomposer/lists"}