Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/softsec-kaist/binkit
Binary Code Similarity Analysis (BCSA) Benchmark
https://github.com/softsec-kaist/binkit
benchmark binary-analysis
Last synced: 5 days ago
JSON representation
Binary Code Similarity Analysis (BCSA) Benchmark
- Host: GitHub
- URL: https://github.com/softsec-kaist/binkit
- Owner: SoftSec-KAIST
- License: mit
- Created: 2020-11-21T11:08:44.000Z (about 4 years ago)
- Default Branch: master
- Last Pushed: 2023-12-15T02:24:01.000Z (about 1 year ago)
- Last Synced: 2024-12-08T14:35:51.828Z (17 days ago)
- Topics: benchmark, binary-analysis
- Language: Shell
- Homepage:
- Size: 105 KB
- Stars: 135
- Watchers: 6
- Forks: 24
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
- License: LICENSE.txt
Awesome Lists containing this project
README
# BinKit 2.0
BinKit is a binary code similarity analysis (BCSA) benchmark. BinKit provides
scripts for building a cross-compiling environment, as well as the compiled
dataset. The current dataset includes 1,904 distinct combinations of compiler
options of 8 architectures, 6 optimization levels, and 23 compilers. It includes
371,928 binaries.
The main improvements of the latest version of BinKit compared to the paper
version of BinKit are as follows: Additional support for relatively newer
compiler versions for major compilation options, and support for Ofast
optimization option.
In particular, BinKit now includes GCC and Clang versions up to 11 and 13,
respectively. Currently, a total of 6 optimization options (O0, O1, O2, O3, Os,
Ofast) are supported. see the [Currently supported compile
options](https://github.com/SoftSec-KAIST/BinKit#currently-supported-compile-options)
section below for more detailed options.
In Binkit 2.0 dataset, the gsl package misses 8 binaries with Ofast option due
to compiler bugs. See the [Missing binaries](https://github.com/SoftSec-KAIST/BinKit#Missing-binaries)
part of the [Issues](https://github.com/topcue/tmp#issues) section for more
information.
## BinKit 1.0 (paper version)
The original dataset includes 1,352 distinct combinations of compiler options of
8 architectures, 5 optimization levels, and 13 compilers. It includes 243,128
binaries. We tested this code in Ubuntu 16.04.
For more details, please check [our
paper](https://0xdkay.me/pub/2020/kim-arxiv2020.pdf).
# BCSA tool and Ground Truth Building
For a BCSA tool and ground truth building, please check
[TikNib](https://github.com/SoftSec-KAIST/TikNib).
## Pre-compiled dataset and toolchain
You can download our dataset and toolchain as below. The link will be changed to
`git-lfs` soon.
[//]: # (Cloning this repository also downloads below pre-compiled dataset and toolchain
with `git-lfs`. Please use `GIT_LFS_SKIP_SMUDGE=1` to skip the download.)
### Dataset (latest version)
- [BinKit 2.0 dataset](https://drive.google.com/file/d/1TrjFnv6BMpVEXYukVxrhlQ78S0NPKEXa/view?usp=share_link)
### Dataset (old)
Below datasets are for reproduction of paper
- [Normal dataset](https://drive.google.com/file/d/1K9ef-OoRBr0X5u8g2mlnYqh9o1i6zFij/view?usp=sharing)
- [SizeOpt dataset](https://drive.google.com/file/d/1QgwbEfd8vdzg5glNZFL7dg4l4hrkoWO3/view?usp=sharing)
- [Noinline dataset](https://drive.google.com/file/d/1wt7GY-DDp8J_2zeBBVUrcfWIyerg_xLO/view?usp=sharing)
- [PIE dataset](https://drive.google.com/file/d/1IfEbnS9RtHhVhW8oiqnE7G75uPej1FPx/view?usp=sharing)
- [LTO dataset](https://drive.google.com/file/d/1Tsd-WNO_JDlEX0GylBOxsFjOPUmUyeGh/view?usp=sharing)
- [Obfus dataset](https://drive.google.com/file/d/1H5k3pfJH9zN4anfxKi1WvNqTKmjVjUUU/view?usp=sharing)
- [Obfus 2-Loop dataset](https://drive.google.com/file/d/1C3SXt896R4rJvpvxcItFu9NIgN-hAxz8/view?usp=sharing)
Below data is only used for our evaluation.
- [ASE dataset](https://drive.google.com/file/d/1MwXHRXjuPoQJAON6SZVoKcK6Xr2NMHdF/view?usp=sharing)
### `.pickle` Files
These files include the extracted features and useful information for each function.
- [Normal dataset `.pickle`](https://drive.google.com/file/d/1GjVoSXPvc7oTMJM4bIpmIOd6If7PTuOm/view?usp=sharing)
- [SizeOpt dataset `.pickle`](https://drive.google.com/file/d/1MeT9Z5aaYf0kAtGxaCnHk8nyXJddPfqC/view?usp=sharing)
- [Noinline dataset `.pickle`](https://drive.google.com/file/d/1bXj2ZjnNOGAijleBh5Tki1XZLG2i1Hng/view?usp=sharing)
- [PIE dataset `.pickle`](https://drive.google.com/file/d/1mVzTKeJ4OzH1fyuSCn-_CPF8BmFMkAEw/view?usp=sharing)
- [LTO dataset `.pickle`](https://drive.google.com/file/d/1ELxkiapNnMrjfcMdltWvritwBH9pBJ7o/view?usp=sharing)
- [Obfus dataset `.pickle`](https://drive.google.com/file/d/12r4kdMvZYE4zTD3f4FDA-kxDOy8GU5ZL/view?usp=sharing)
Below data is only used for our evaluation.
- [ASE dataset `.pickle`](https://drive.google.com/file/d/1NbhNRBpV5_evRXrNeUReBU7ju9bEtKxq/view?usp=sharing)
### Toolchain
- [tools](https://drive.google.com/file/d/1Ar8CT4xZceT083jMy2dU5q-CgcMHqrQ0/view?usp=sharing)
# Currently supported compile options
### Architecture
- x86_32
- x86_64
- arm_32 (little endian)
- arm_64 (little endian)
- mips_32 (little endian)
- mips_64 (little endian)
- mipseb_32 (big endian)
- mipseb_64 (big endian)
### Optimization
- O0
- O1
- O2
- O3
- Os
- Ofast
### Compilers
- gcc
- gcc-4.9.4
- gcc-5.5.0
- gcc-6.4.0
- gcc-6.5.0
- gcc-7.3.0
- gcc-8.2.0
- gcc-8.5.0
- gcc-9.4.0
- gcc-10.3.0
- gcc-11.2.0
- clang
- clang-4.0.0
- clang-5.0.2
- clang-6.0.1
- clang-7.0.1
- clang-8.0.0
- clang-9.0.1
- clang-10.0.1
- clang-11.0.1
- clang-12.0.1
- clang-13.0.0
- clang-obfus
- clang-obfus-fla (Obfuscator-LLVM - FLA)
- clang-obfus-sub (Obfuscator-LLVM - SUB)
- clang-obfus-bcf (Obfuscator-LLVM - BCF)
- clang-obfus-all (Obfuscator-LLVM - FLA + SUB + BCF)
# How to use
### 1. Configure the environment in `scripts/env.sh`
- `NUM_JOBS`: for `make`, `parallel`, and `python` multiprocessing
- `MAX_JOBS`: maximum for `make`
### 2. Build cross-compiling environment (takes lots of time)
We build crosstool-ng and clang environment. If you download pre-compiled
toolchain. Please skip this.
```bash
$ source scripts/env.sh
# We may have missed some packages here ... please check
$ scripts/install_default_deps.sh # install default packages for dataset compilation
$ scripts/setup_ctng.sh # setup crosstool-ng binaries
$ scripts/setup_gcc.sh # build ct-ng environment. Takes a lot of time
$ scripts/cleanup_ctng.sh # cleaning up ctng leftovers
$ scripts/setup_clang.sh # setup clang and llvm-obfuscator
```
### 3. Link toolchains
```bash
$ scripts/link_toolchains.sh # link base toolchain
```
To undo the linking, please check `scripts/unlink_toolchains.sh`
### 4. Build dataset
Please configure variables in `compile_packages.sh` and run below. The script
automatically downloads the source code of GNU packages, and compiles them to
make all the dataset. However, it may take too much time to create all of them.
- *NOTE* that it takes *SIGNIFIACNT* time.
- *NOTE* that some packages would not be compiled for some compiler options.
```bash
$ scripts/install_gnu_deps.sh # install default packages for dataset compilation
$ ./compile_packages.sh
```
### 4-1. Build dataset (manual)
You can download the source code of GNU packages of your interest as below.
- Please check step 1 before running the command.
- You must give *ABSOLUTE PATH* for `--base_dir`.
```bash
$ source scripts/env.sh
$ python gnu_compile_script.py \
--base_dir "/home/dongkwan/binkit/dataset/gnu" \
--num_jobs 8 \
--whitelist "config/whitelist.txt" \
--download
```
You can compile only the packages or compiler options of your interest as below.
```bash
$ source scripts/env.sh
$ python gnu_compile_script.py \
--base_dir "/home/dongkwan/binkit/dataset/gnu" \
--num_jobs 8 \
--config "config/normal.yml" \
--whitelist "config/whitelist.txt"
```
You can check the compiled binaries as below.
```bash
$ source scripts/env.sh
$ python compile_checker.py \
--base_dir "/home/dongkwan/binkit/dataset/gnu" \
--num_jobs 8 \
--config "config/normal.yml"
```
For more details, please check `compile_packages.sh`
### 4-2. Build dataset with customized options
To build datasets by customizing options, you can make your own configuration
file (`.yml`) and select target compiler options. You can check the format in
the existing sample files in the `/config` directory. Here, please make sure
that the name of your config file is not included in the blacklist in the
[compilation
script](/SoftSec-KAIST/BinKit/blob/master/do_compile_utils.sh#L347).
# Issues
### Tested environment
We ran all our experiments on a server equipped with four Intel Xeon E7-8867v4
2.40 GHz CPUs (total 144 cores), 896 GB DDR4 RAM, and 4 TB SSD. We setup Ubuntu
16.04 on the server.
### Tested python version
- Python 3.8.0
### Running example
The time spent for running the below script took `7` hours on our machine.
```bash
$ python gnu_compile_script.py \
--base_dir "/home/dongkwan/binkit/dataset/gnu" \
--num_jobs 72 \
--config "config/normal.yml" \
--whitelist "config/whitelist.txt"
```
### Compliation failure
If compilation fails, you may have to adjust the number of jobs for parallel
processing in the step 1, which is machine-dependent.
### Missing binaries
In Binkit 2.0 dataset, the gsl package misses 8 binaries with Ofast option due
to compiler bugs. Clang-8 and clang-9 induce compiler hang bug when compiling
the gsl package for 32bit ARM with Ofast option. We reported this issue to
bug-gsl and llvm-project respectively. However, bug-gsl did not reply, and the
llvm-project replied that these versions are not currently supported. The bug
reporting links are respectively as follows:
[bug-gsl](https://lists.gnu.org/archive/html/bug-gsl/2023-02/msg00000.html),
[llvm-project](https://github.com/llvm/llvm-project/issues/60692)
# Authors
This project has been conducted by the below authors at KAIST.
* [Dongkwan Kim](https://0xdkay.me/)
* [Eunsoo Kim](https://hahah.kim)
* [Sang Kil Cha](https://softsec.kaist.ac.kr/~sangkilc/)
* [Sooel Son](https://sites.google.com/site/ssonkaist/home)
* [Yongdae Kim](https://syssec.kaist.ac.kr/~yongdaek/)
# Citation
We would appreciate if you consider citing [our
paper](https://ieeexplore.ieee.org/document/9813408) when using BinKit.
```bibtex
@ARTICLE{kim:tse:2022,
author={Kim, Dongkwan and Kim, Eunsoo and Cha, Sang Kil and Son, Sooel and Kim, Yongdae},
journal={IEEE Transactions on Software Engineering},
title={Revisiting Binary Code Similarity Analysis using Interpretable Feature Engineering and Lessons Learned},
year={2022},
volume={},
number={},
pages={1-23},
doi={10.1109/TSE.2022.3187689}
}
```