{"id":30104192,"url":"https://github.com/squareslab/idioms","last_synced_at":"2025-08-09T22:30:59.234Z","repository":{"id":277243023,"uuid":"925311448","full_name":"squaresLab/idioms","owner":"squaresLab","description":null,"archived":false,"fork":false,"pushed_at":"2025-02-06T20:05:59.000Z","size":312,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-02-12T22:35:49.740Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/squaresLab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-01-31T16:29:17.000Z","updated_at":"2025-02-06T20:23:02.000Z","dependencies_parsed_at":"2025-02-14T12:01:42.337Z","dependency_job_id":null,"html_url":"https://github.com/squaresLab/idioms","commit_stats":null,"previous_names":["squareslab/idioms"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/squaresLab/idioms","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/squaresLab%2Fidioms","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/squaresLab%2Fidioms/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/squaresLab%2Fidioms/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/squaresLab%2Fidioms/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/squaresLab","download_url":"https://codeload.github.com/squaresLab/idioms/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/squaresLab%2Fidioms/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":269649193,"owners_count":24453499,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-09T02:00:10.424Z","response_time":111,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-08-09T22:30:58.593Z","updated_at":"2025-08-09T22:30:59.203Z","avatar_url":"https://github.com/squaresLab.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Introduction\n\nThis replication package contains the code and instructions necessary to reproduce the results for the paper _Idioms: Neural Decompilation With Joint Code and Type Prediction_.\n\nThere are five main sections:\n- Building the realtype dataset\n- Adapting the exebench dataset \n- Training\n- Evaluation\n- LLM4Decompile Baseline\n\nThe datasets and trained models and adapters are available [here](https://doi.org/10.5281/zenodo.14797016).\n\nWith the provided datasets, you can skip to the Training section.\nWith the provided datasets and pretrained models/adapters, you can skip to the Evaluation section.\nThe evaluator script assumes the models are in a folder named `runs` rather than `models` so if you plan on rerunning the evaluation, run `mv models runs`.\n\nAll sections make use of the `idioms` conda environment. It can be installed with:\n```\nconda env create -f env.yml\n```\nIn addition, some sections require installing additional packages or supplementary conda environments.\n\n# Building the Realtype Dataset\n\nThe first steps for building the dataset involve the ghcc-idioms directory, a modified version of the [original GHCC](https://github.com/huzecong/ghcc).\n\nFirst, build set up the required python environment, which differs from the `idioms` python environment.\n```\ncd ghcc-idioms\nconda env create -f env.yml\nconda activate ghcc\n```\n\nFor safe building of binaries, ghcc provides a docker container.\nTo build it, run\n```\ndocker build -t gcc-custom .\n```\n\nTo build the binaries for the dataset, run\n```\npython main.py --repo-list-file repo_list.txt --n-procs 5 --gcc-override-flags=\"-g -O0\"\n```\nWe performed this process at all four levels of optimization (O0 through O3).\nFor optimizations above O0, disable inlining, e.g. `--gcc-override-flags=\"-g -O1 -fno-partial-inlining -fno-inline\"`\n\nTo preprocess the code, run\n```\npython extract_original.py binaries/ archives/ preprocessed\n```\nThe latter two are very expensive commands, and may take up to several weeks to run.\n\nWe use scripts from DIRTY [1] to do the decompilation. For our main experiments, we used the original version which uses the proprietary Hex-Rays decompiler. There is also a version based on the open-source Ghidra decompiler, which we use to generate data for LLM4Decompile [2] for copmarison. We use this for our baselining experiments.\n\nClone these repositories with\n```\ngit clone https://github.com/CMUSTRUDEL/DIRTY.git\ncd DIRTY\ngit checkout 5f8e9494d52ef8788f0834342097f36462eeb2eb\ncd ..\n\ngit clone https://github.com/edmcman/DIRTY-Ghidra.git\ncd DIRTY-Ghidra\ngit checkout 82aadac50fab7e8842da6d2fd356bcc088f34994\ncd ..\n```\n\nDIRTY and DIRTY-Ghidra require their own python environments. We provide an environment file for both of them. To use it, run\n```\nconda env create -f dirty_env.yml \nconda activate dirty\n```\n\nThe relevant script is `generate.py`, which can be found in `DIRTY/dataset-gen` and `DIRTY-Ghidra/dataset-gen-ghidra`\n\nThe commands to run decompilation with Hex-Rays and Ghidra respectively are\n```\npython generate.py --ida /path/to/idat64 -b /path/to/binaries/ -o /path/to/decompiled-hex-rays\n\npython generate.py --ghidra /path/to/ghidra_11.2.1_PUBLIC/support/analyzeHeadless -b /path/to/binaries/ -o /path/to/decompiled-ghidra\n```\nThe decomiled-hex-rays and decompiled-ghidra folders will be created by the script.\n\nAll subsequent steps use the `idioms` conda environment.\n```\nconda activate idioms\n```\n\n\nDuplicate projects are common in open-source projects, and failing to remove them can lead to data leakage. Find duplicates with\n```\npython deduplicate.py /path/to/archives/ dedup-clusters.json --lexical --workers 8\n```\n\nAfter these steps, there is enough information to build the full realtype dataset. To do this, run\n```\npython prepare.py /path/to/decompiled-hex-rays /path/to/binaries/ /path/to/preprocessed/ dedup-clusters.json idioms_dataset_O0 --holdout-set-size 0.02 --valid-max-bins-per-repo 1 --test-max-bins-per-repo 25 --shard-size 500\n```\n\nWe performed this process on the O0 dataset.\nWe built the dataset at other levels of optimization using the train/validation/test splits produced by the initial run of the script.\nTo do this, we used the commands:\n\n```\n# O1\npython prepare.py /path/to/decompiled-O1/ /path/to/binaries-O1/ /path/to/preprocessed/ idioms_dataset_O0/test_repos.txt idioms_dataset_O1 --shard-size 500 --single-split-name test --workers 5\n\npython prepare.py /path/to/decompiled-O1/ /path/to/binaries-O1/ /path/data/preprocessed/ idioms_dataset_O0/validation_repos.txt idioms_dataset_O1 --shard-size 500 --single-split-name validation --reuse-existing-dir --workers 5\n\npython prepare.py /path/to/decompiled-O1/ /path/to/binaries-O1/ /path/data/preprocessed/ idioms_dataset_O0/train_repos.txt idioms_dataset_O1 --shard-size 500 --single-split-name train --reuse-existing-dir --workers 5\n\n# O2\npython python prepare.py /path/to/decompiled-O2/ /path/to/binaries-O2/ /path/to/preprocessed/ idioms_dataset_O0/test_repos.txt idioms_dataset_O2 --shard-size 500 --single-split-name test --workers 5\n\npython prepare.py /path/to/decompiled-O2/ /path/to/binaries-O2/ /path/to/preprocessed/ idioms_dataset_O0/validation_repos.txt idioms_dataset_O2 --shard-size 500 --single-split-name validation --reuse-existing-dir --workers 5\n\npython prepare.py /path/to/decompiled-O2/ /path/to/binaries-O2/ /path/to/preprocessed/ idioms_dataset_O0/train_repos.txt idioms_dataset_O2 --shard-size 500 --single-split-name train --reuse-existing-dir --workers 5\n\n# O3\npython prepare.py /path/to/decompiled-O3/ /path/to/binaries-O3/ /path/to/preprocessed/ idioms_dataset_O0/train_repos.txt idioms_dataset_O3 --shard-size 500 --single-split-name train\n\npython prepare.py /path/to/decompiled-O3/ /path/to/binaries-O3/ /path/to/preprocessed/ idioms_dataset_O0/validation_repos.txt idioms_dataset_O3 --shard-size 500 --single-split-name validation --reuse-existing-dir\n\npython prepare.py /path/to/decompiled-O3/ /path/to/binaries-O3/ /path/to/preprocessed/ idioms_dataset_O0/test_repos.txt idioms_dataset_O3 --shard-size 500 --single-split-name test --reuse-existing-dir\n```\n\nThe decompilation process does not always succeed for all all binaries at all levels of optimization. To control for this, run\n\n```\nmake_parity_idioms_dataset.py idioms_dataset_O0/ idioms_dataset_O1_noinline/ idioms_dataset_O2_noinline/ idioms_dataset_O3_noinline/\n```\n\n### Preparing Realtype for Nova and LLM4Decompile\n\nWe also evaluate LLM4Decompile-Ref [2] and Nova+ [3] on Realtype.\nThe former uses assembly as input, and LLM4Decompile uses Ghidra decomiplation as input.\n\nFor Nova, we simply package the binaries into tar files, and the evaluation script will extract and normalize the relevant assembly at evaluation time.\n\nRun\n```\nmkdir idioms_dataset_binary_archives\npython archive_idioms_binaries.py idioms_dataset_O0 idioms_dataset_binary_archives /path/to/O0_binaries\npython archive_idioms_binaries.py idioms_dataset_O1_noinline idioms_dataset_binary_archives /path/to/O1_noinline_binaries\npython archive_idioms_binaries.py idioms_dataset_O2_noinline idioms_dataset_binary_archives /path/to/O2_noinline_binaries\npython archive_idioms_binaries.py idioms_dataset_O3_noinline idioms_dataset_binary_archives /path/to/O3_noinline_binaries\n```\n\nFor LLM4Decompile, we run `prepare.py` again, but using Ghidra data as input. Because we're only evaluating, not training, we only need the test set. Run\n```\nmkdir idioms_ghidra_datasets\npython prepare.py /path/to/ghidra_O0_decompiled /path/to/O0_binaries /path/to/preprocessed idioms_dataset_O0/test_repos.txt idioms_ghidra_datasets/idioms_dataset_O0 --single-split-name test\n\npython prepare.py /path/to/ghidra_O1_noinline_decompiled /path/to/O1_noinline_binaries /path/to/preprocessed idioms_dataset_O0/test_repos.txt idioms_ghidra_datasets/idioms_dataset_O1_noinline --single-split-name test\n\npython prepare.py /path/to/ghidra_O2_noinline_decompiled /path/to/O2_noinline_binaries /path/to/preprocessed idioms_dataset_O0/test_repos.txt idioms_ghidra_datasets/idioms_dataset_O2_noinline --single-split-name test\n\npython prepare.py /path/to/ghidra_O3_noinline_decompiled /path/to/O3_noinline_binaries /path/to/preprocessed idioms_dataset_O0/test_repos.txt idioms_ghidra_datasets/idioms_dataset_O3_noinline --single-split-name test\n```\n\nThen make sure that the datasets only contain examples from the test repositories that are in the original realtype dataset.\n\n```\npython make_parity_idioms_dataset.py idioms_dataset_O0_opt_parity idioms_ghidra_datasets/idioms_dataset_O0/ --test-only --output-suffix decompiler_opt_sample --skip-first-write\npython make_parity_idioms_dataset.py idioms_dataset_O0_opt_parity idioms_ghidra_datasets/idioms_dataset_O1_noinline/ --test-only --output-suffix decompiler_opt_sample --skip-first-write\npython make_parity_idioms_dataset.py idioms_dataset_O0_opt_parity idioms_ghidra_datasets/idioms_dataset_O2_noinline/ --test-only --output-suffix decompiler_opt_sample --skip-first-write\npython make_parity_idioms_dataset.py idioms_dataset_O0_opt_parity idioms_ghidra_datasets/idioms_dataset_O3_noinline/ --test-only --output-suffix decompiler_opt_sample --skip-first-write\n```\n\n\n# Adapting the Exebench Dataset\n\nThe exebench dataset contains source code and the corresponding assembly, but no decompiled code.\nTo add it, we first compile each example.\n```\nconda activate idioms\nmkdir exebench\npython compile_exebench.py exebench/compiled-O0 --workers 8\n```\n\nNext, we use DIRTY and DIRTY-GHIDRA to decompile each partition.\n```\ncp exebench_decompile/gen-exebench.py /path/to/DIRTY/dataset-gen\ncp exebench_decompile/gen-exebench.py /path/to/DIRTY-Ghidra/dataset-gen-ghidra\nmkdir exebench\n\nconda activate dirty\n\ncd /path/to/DIRTY/dataset-gen\npython gen-exebench.py /path/to/exebench/ O0 --decompiler /path/to/ida-7.5/idat64\n\ncd /path/to/DIRTY-Ghidra/dataset-gen-ghidra\npython gen-exebench.py /path/to/exebench/ O0 --decompiler /path/to/ghidra_11.2.1_PUBLIC/support/analyzeHeadless --eval-only\n```\n\nWith the code decompiled, build the dataset. There are two different forms of the dataset that are used in the experiments. For convenience, we move the datasets to the current directory for the experiments.\n```\nconda activate idioms\npython update_exebench.py exebench idioms O0\npython update_exebench.py exebench huggingface O0 --eval-only\n\nmv exebench/exebench-idioms-O0-hex-rays exebench-idioms-O0\nmv exebench/exebench-hf-O0-eval/ .\n```\n\nThe `parity-exebench` experiments require a subsample of the dataset that is the same size as the realtype dataset.\nTo create this, run\n```\npython downsample_exebench.py exebench-idioms-O0/ idioms_dataset/ parity-exebench-idioms-O0\n```\n\n\n# Training\n\nModel training requires the `unsloth` package, which both accelerates training and reduces GPU memory consumption.\nUnsloth is installed in a conda environment but separately from the environment file.\nWe used\n```\nconda activate idioms\npip install \"unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git\"\n```\nUnsloth is only fully supported on GPUs with compute capability 7.0 or greater.\nFor more information on using unsloth, see [the pypi page](https://pypi.org/project/unsloth/).\n\n\nTo train the models, do the following:\n\n### Experiment Set 1: Direct Comparison\n\n#### Idioms\n\n```\npython tune-unsloth.py idioms_dataset_O0_opt_parity opt-O0-codegemma-7b-neighbors neighbors adapter --epochs 8.0 --model-type unsloth/codegemma-7b-bnb-4bit --batch-size 4 --gradient-accumulation 16 --nhops 4 --max-length 4096\n\npython tune-unsloth.py idioms_dataset_O1_noinline_opt_parity opt-O1-codegemma-7b-neighbors neighbors adapter --epochs 8.0 --model-type unsloth/codegemma-7b-bnb-4bit --batch-size 4 --gradient-accumulation 16 --nhops 4 --max-length 4096\n\npython tune-unsloth.py idioms_dataset_O2_noinline_opt_parity opt-O2-codegemma-7b-neighbors neighbors adapter --epochs 8.0 --model-type unsloth/codegemma-7b-bnb-4bit --batch-size 4 --gradient-accumulation 16 --nhops 4 --max-length 4096\n\npython tune-unsloth.py idioms_dataset_O3_noinline_opt_parity opt-O3-codegemma-7b-neighbors neighbors adapter --epochs 8.0 --model-type unsloth/codegemma-7b-bnb-4bit --batch-size 4 --gradient-accumulation 16 --nhops 4 --max-length 4096\n```\n\n### Experiment Set 2: Ablation Study\n\nFor the CodeQwen2.5 experiments\n\n```\npython train.py exebench-idioms-O0 qwen-0.5b-exebench-O0 function finetune --epochs 8.0 --model-type Qwen/Qwen2.5-Coder-0.5B --batch-size 8 --gradient-accumulation 8\npython train.py parity-exebench-idioms-O0 qwen-0.5b-parity-exebench-O0 function finetune --epochs 8.0 --model-type Qwen/Qwen2.5-Coder-0.5B --batch-size 8 --gradient-accumulation 8\npython train.py idioms_dataset_O0 qwen-0.5b-functions-idioms function finetune --epochs 8.0 --model-type Qwen/Qwen2.5-Coder-0.5B --batch-size 8 --gradient-accumulation 8\npython train.py idioms_dataset_O0 qwen-0.5b-neighbors-idioms neighbors finetune --epochs 8.0 --model-type Qwen/Qwen2.5-Coder-0.5B --batch-size 4 --gradient-accumulation 16 --nhops 4 --max-length 4096\n\n```\n\nFor the modified LLM4Decompile experiments\n\n```\npython train.py exebench-idioms-O0 llm4decompile-1.3b-exebench-O0 function adapter --epochs 2.0 --model-type LLM4Binary/llm4decompile-1.3b-v2 --batch-size 8 --gradient-accumulation 8\npython train.py parity-exebench-idioms-O0 llm4decompile-1.3b-parity-exebench-O0 function adapter --epochs 8.0 --model-type LLM4Binary/llm4decompile-1.3b-v2 --batch-size 8 --gradient-accumulation 8\npython train.py idioms_dataset_O0 llm4decompile-1.3b-functions-idioms function adapter --epochs 8.0 --model-type LLM4Binary/llm4decompile-1.3b-v2 --batch-size 16 --gradient-accumulation 4\npython train.py idioms_dataset_O0 llm4decompile-1.3b-neighbors-idioms neighbors adapter --epochs 8.0 --model-type LLM4Binary/llm4decompile-1.3b-v2 --batch-size 16 --gradient-accumulation 4 --nhops 4 --max-length 4096\n\n```\n\nFor the CodeGemma-2b experiments\n\n```\npython train.py exebench-idioms-O0 codegemma-2b-exebench-O0 function adapter --epochs 2.0 --model-type unsloth/codegemma-2b-bnb-4bit --batch-size 8 --gradient-accumulation 8\npython train.py parity-exebench-idioms-O0 codegemma-2b-parity-exebench-O0 function adapter --epochs 8.0 --model-type unsloth/codegemma-2b-bnb-4bit --batch-size 8 --gradient-accumulation 8\npython train.py idioms_dataset_O0 codegemma-2b-functions-idioms function adapter --epochs 8.0 --model-type unsloth/codegemma-2b-bnb-4bit --batch-size 8 --gradient-accumulation 8\npython train.py idioms_dataset_O0 codegemma-2b-neighbors-idioms neighbors adapter --epochs 8.0 --model-type unsloth/codegemma-2b-bnb-4bit --batch-size 4 --gradient-accumulation 16 --nhops 4 --max-length 4096\n\n```\n\n\nFor the CodeGemma-7b experiments\n\n```\npython train.py exebench-idioms-O0 codegemma-7b-exebench-O0 function adapter --epochs 1.0 --model-type unsloth/codegemma-7b-bnb-4bit --batch-size 8 --gradient-accumulation 8\npython train.py parity-exebench-idioms-O0 codegemma-7b-parity-exebench-O0 function adapter --epochs 8.0 --model-type unsloth/codegemma-7b-bnb-4bit --batch-size 8 --gradient-accumulation 8\npython train.py idioms_dataset_O0 codegemma-7b-functions-idioms function adapter --epochs 8.0 --model-type unsloth/codegemma-7b-bnb-4bit --batch-size 8 --gradient-accumulation 8\npython train.py idioms_dataset_O0 codegemma-7b-neighbors-idioms neighbors adapter --epochs 8.0 --model-type unsloth/codegemma-7b-bnb-4bit --batch-size 4 --gradient-accumulation 16 --nhops 4 --max-length 4096\n```\n\nFor the CodeLlama-7b experiments\n\n```\npython train.py exebench-idioms-O0 codellama-7b-exebench-O0 function adapter --epochs 1.0 --model-type unsloth/codellama-7b-bnb-4bit --batch-size 8 --gradient-accumulation 8\npython train.py parity-exebench-idioms-O0 codellama-7b-parity-exebench-O0 function adapter --epochs 8.0 --model-type unsloth/codellama-7b-bnb-4bit --batch-size 8 --gradient-accumulation 8\npython train.py idioms_dataset_O0 codellama-7b-functions-idioms function adapter --epochs 8.0 --model-type unsloth/codellama-7b-bnb-4bit --batch-size 8 --gradient-accumulation 8\npython train.py idioms_dataset_O0 codellama-7b-neighbors-idioms neighbors adapter --epochs 8.0 --model-type unsloth/codellama-7b-bnb-4bit --batch-size 4 --gradient-accumulation 16 --nhops 4 --max-length 4096\n```\n\n# Evaluation\n\nEvaluation requires the `codealign` package for computing most metrics. To install, run\n```\ngit clone https://github.com/squaresLab/codealign.git\ncd codealign\npip install .\ncd ..\n```\n\nModels evaluated on the exebench test run also need their unit tests to be run; the fraction of runs which pass all of the unit tests is an additional metric.\nRunning the unit tests tooks around 6 hours per model on our machine vs around a few minutes for all other metrics combined, so `eval_all.py` skips running them.\nRunning arbitrary machine-learning-generated code can be dangerous, so we provide a docker image for containers in which the tests are actually run.\nTo build the image and compute exebench metrics, run\n```\ndocker build -t exebench-test exebench_docker/\npython test_exebench.py \n```\n\n### Experiment Set 1\n\n#### Idioms\n\nIf you built your own datasets from scrach simply use the last checkpoint instead of `checkpoint-13280`.\n```\npython evaluator.py runs/opt-O0-codegemma-7b-neighbors/checkpoint-13280 --eval-partition test --batch-size 8\npython evaluator.py runs/opt-O1-codegemma-7b-neighbors/checkpoint-13280 --eval-partition test --batch-size 8\npython evaluator.py runs/opt-O2-codegemma-7b-neighbors/checkpoint-13280 --eval-partition test --batch-size 8\npython evaluator.py runs/opt-O3-codegemma-7b-neighbors/checkpoint-13280 --eval-partition test --batch-size 8\n```\nWe use the `codegemma-7b-exebench-O0` results from experiment set 2, below, for results on exebench.\n\n#### LLM4Decompile\n\n```\n# Exebench\npython llm4decompile.py LLM4Binary/llm4decompile-6.7b-v2 exebench-hf-O0-eval --no-exebench-tests\n\n# Realtype\npython llm4decompile.py LLM4Binary/llm4decompile-6.7b-v2 idioms_ghidra_datasets/idioms_dataset_O0_decompiler_opt_sample\npython llm4decompile.py LLM4Binary/llm4decompile-6.7b-v2 idioms_ghidra_datasets/idioms_dataset_O1_noinline_decompiler_opt_sample\npython llm4decompile.py LLM4Binary/llm4decompile-6.7b-v2 idioms_ghidra_datasets/idioms_dataset_O2_noinline_decompiler_opt_sample\npython llm4decompile.py LLM4Binary/llm4decompile-6.7b-v2 idioms_ghidra_datasets/idioms_dataset_O3_noinline_decompiler_opt_sample\n```\n\n#### Nova\n\n```\n# Exebench\npython nova.py lt-asset/nova-6.7b-bcr exebench-hf-O0-eval exebench-binary-archives/compiled-O0 --greedy\n\n# Realtype\npython nova.py lt-asset/nova-6.7b-bcr idioms_dataset_O0_opt_parity/ idioms_dataset_binary_archives/idioms_dataset_O0_opt_parity_test_binaries.tar.gz --greedy\npython nova.py lt-asset/nova-6.7b-bcr idioms_dataset_O1_noinline_opt_parity idioms_dataset_binary_archives/idioms_dataset_O1_noinline_opt_parity_test_binaries.tar.gz --greedy\npython nova.py lt-asset/nova-6.7b-bcr idioms_dataset_O2_noinline_opt_parity idioms_dataset_binary_archives/idioms_dataset_O2_noinline_opt_parity_test_binaries.tar.gz --greedy\npython nova.py lt-asset/nova-6.7b-bcr idioms_dataset_O3_noinline_opt_parity idioms_dataset_binary_archives/idioms_dataset_O3_noinline_opt_parity_test_binaries.tar.gz --greed\n```\n\n\n### Experiment Set 2\n\nMost experiments are trained to 8 epochs, but especially on the easier tasks, it's possible that some models will start to overfit before then.\nTo account for this, run validation on each model that has eight or more epochs of training using\n```\npython eval_all.py\n```\n\nThe `eval_all.py` script has arguments to filter the training runs for which validation is performed.\nThis allows the validation runs to be divided up and run in parallel.\n\nWith all necessary runs validated, run\n```\npython compare_checkpoints.py\n```\nThis will output the results for each validation experiment in a tabular form.\nIt will also write a json file to `results/` describing the checkpoint that performed best on the validation set for each model/experimental conditions combo.\n\nWith this, test the best checkpoints by running\n```\npython eval_all.py --test-best-checkpoints\n```\nThis does not include runs for which selecting a best checkpoint is unnecessary (because those runs were only trained for 1 or 2 epochs).\nTo run evaluation in these four cases, run:\n```\npython evaluator.py runs/llm4decompile-1.3b-exebench-O0/ --eval-partition test --batch-size 32 --dataset exebench-hf-O0-eval --exebench-subpartition real --no-exebench-tests\npython evaluator.py runs/codegemma-2b-exebench-O0/ --eval-partition test --batch-size 32 --dataset exebench-hf-O0-eval --exebench-subpartition real --no-exebench-tests\npython evaluator.py runs/codegemma-7b-exebench-O0/ --eval-partition test --batch-size 32 --dataset exebench-hf-O0-eval --exebench-subpartition real --no-exebench-tests\npython evaluator.py runs/codellama-7b-exebench-O0/ --eval-partition test --batch-size 32 --dataset exebench-hf-O0-eval --exebench-subpartition real --no-exebench-tests\n```\n\nFinally, summarize and display the final results with\n```\npython visualizer.py run metric\n```\nIf you used the pretrained models rather than training your own, instead run\n```\npython visualizer.py run metric --checkpoint-type last\n```\nNote that numbers may vary slightly.\n\n# LLM4Decompile Baseline\n\nWe also include the script to perform the LLM4Decopmile baseline.\nWe use the `exebench-hf-O0-eval` dataset from the \"Adapting the Exebench Dataset\" section.\nThe script runs exebench tests, so it requires the docker image described in the \"Evaluation\" section.\nTo build that docker image, run\n```\ndocker build -t exebench-test exebench_docker/\n```\n\nThe main script depends on `clang-format`.\nLLM4Decompile by default is trained on Ghidra decompilation as input, so we provide that here rather than the Hex-Rays decompilation from the original version of DIRTY.\nDIRTY-Ghidra stores the decompiled code in a manner with whitespace stripped, which affects the tokenization.\nLLM4Decopmile was trained on formatted code.\nThe reformatting the code provides a small but noticable accuracy boost.\n\nTo run the LLM4Decompile experiment run\n```\npython llm4decompile.py LLM4Binary/llm4decompile-1.3b-v2 exebench-hf-O0-eval --dataset-split test_real\n```\n\nThe results will be in `baselines/llm4decompile-1.3b-v2/exebench-hf-O0-eval/test_real_scores.json`.\n\n# Reference\n\n[1] Chen, Qibin, et al. \"Augmenting decompiler output with learned variable names and types.\" 31st USENIX Security Symposium (USENIX Security 22). 2022.\n\n[2] Tan, Hanzhuo, et al. \"LLM4Decompile: Decompiling Binary Code with Large Language Models.\" Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2024.\n\n[3] Nan Jiang, et al. \"Nova: Generative Language Models for Assembly Code with Hierarchical Attention and Contrastive Learning.\" The Thirteenth International Conference on Learning Representations (ICLR). April 2025.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsquareslab%2Fidioms","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsquareslab%2Fidioms","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsquareslab%2Fidioms/lists"}