{"id":18894189,"url":"https://github.com/trailofbits/differ","last_synced_at":"2025-09-04T11:31:38.008Z","repository":{"id":213428735,"uuid":"568959546","full_name":"trailofbits/differ","owner":"trailofbits","description":"Detecting Inconsistencies in Feature or Function Evaluations of Requirements","archived":false,"fork":false,"pushed_at":"2024-03-20T16:27:39.000Z","size":344,"stargazers_count":67,"open_issues_count":16,"forks_count":4,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-11-08T08:20:50.935Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/trailofbits.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2022-11-21T19:28:21.000Z","updated_at":"2024-08-06T06:13:25.000Z","dependencies_parsed_at":"2023-12-21T00:58:59.405Z","dependency_job_id":"f622efe3-e082-4379-8192-dddb8f09946e","html_url":"https://github.com/trailofbits/differ","commit_stats":null,"previous_names":["trailofbits/differ"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trailofbits%2Fdiffer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trailofbits%2Fdiffer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trailofbits%2Fdiffer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trailofbits%2Fdiffer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/trailofbits","download_url":"https://codeload.github.com/trailofbits/differ/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":231955979,"owners_count":18451509,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-08T08:19:43.050Z","updated_at":"2024-12-31T07:23:27.413Z","avatar_url":"https://github.com/trailofbits.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DIFFER\n\n![](https://github.com/trailofbits/differ/actions/workflows/ci.yml/badge.svg)\n\nDIFFER: Detecting Inconsistencies in Feature or Function Evaluations of Requirements\n\n## Overview\nProgram transformation tools (like software debloating tools) often neglect the need for\npost-transfomation validation of the modified programs they create, opting to leave this entirely\nto the user. Existing approaches like regression and fuzz testing do not naturally support testing\ntransformed programs against their original versions.\n\nDIFFER is a novel differential testing tool for transformed programs that combines elements from\ndifferential, regression, and fuzz testing approaches. DIFFER allows users to specify seed inputs\nthat correspond to both unmodified and modified program behaviors/features. It runs the original\nprogram and one or more of its transformed variants with these inputs and compares their outputs.\n\nDIFFER expects that inputs for unmodified features will result in outputs that are the same for the\noriginal and transformed programs. Conversely, it expects inputs for modified features to cause the\noriginal and transformed programs to produce differing outputs. If DIFFER detects unexpected\nmatches, differences or crashes it reports them to the user to inspect. DIFFER's reports can help\nthe user identify mistakes in the transformation tool or its configuration\n\nAs is the case with all dynamic analysis tools, it is possible that DIFFER reports may be false\npositives. To reduce false positive rates to a minimum, DIFFER allows users to define custom output\ncomparators that can account for expected differences in outputs (e.g., a program timestamps its\nconsole output). Additionally, DIFFER supports template-based mutational fuzzing of seed inputs to\nensure maximum coverage of the input space (i.e., avoid false negatives) for both unmodified and\nmodified features.\n\nIt is important to note that DIFFER does not and cannot provide formal guarantees of soundness\nin transformation tools or the modified programs they produce. Like other dynamic analysis testing\napproaches, DIFFER cannot exhaustively test the input space for complex programs in the general\ncase.\n\n## Debloating Use Case\n\nDIFFER was originally designed to help validate debloated programs created by software debloaters.\nThis work is currently published for reference at the link below. We welcome contributions to this\nDIFFER and hope you find it useful in your research / security work. If you use this tool in\nyour research, please cite the following paper:\n\n**Brown, Michael D., et al. \"SoK: A Broad Comparative Evaluation of Software Debloating Tools\". arXiv CS.SE. 2023.**[\\[pdf\\]](https://arxiv.org/abs/2312.13274)\n\n\n## Setup\n\n### Installing Dependencies\n\n1. Install Python 3.9 and dependencies. For Ubuntu 20.04, the [deadsnakes PPA](https://launchpad.net/~deadsnakes/+archive/ubuntu/ppa) can be used:\n   ```bash\n   $ sudo add-apt-repository ppa:deadsnakes/ppa\n   $ sudo apt update\n   $ sudo apt-get install python3.9 python3.9-dev python3-pip python3.9-venv libfuzzy-dev lftp lighttpd memcached \\\n       tcpdump binutils unzip poppler-utils imagemagick nmap\n   $ sudo systemctl stop memcached\n   $ sudo systemctl disable memcached\n   ```\n2. Install `pipenv`, which manages the virtual environment.\n   ```bash\n   $ python3.9 -m pip install --user pipenv\n   ```\n3. Create the virtual environment and install development dependencies:\n   ```bash\n   $ pipenv sync --dev\n   ```\n4. Clone and build Radamsa.\n   ```bash\n   $ ./setup-radamsa.sh\n   ```\n5. Install [Node.js](https://nodejs.org/en/), which is required by the type checker (pyright). On Linux, use the [node version manager](https://github.com/nvm-sh/nvm) and on Windows install Node.js 18+ and add `node.exe` to the `PATH`.\n\n### Allow current user to execute tcpdump\n\nThe current user will need to be able to run `tcpdump` without `sudo` in order for the packet capture functionality to work properly. This is not necessary if DIFFER is being run as `root`.\n\n1. Enable the network traffic capture capabilities for the `tcpdump` binary.\n   ```bash\n   $ sudo setcap cap_net_raw,cap_net_admin=eip /usr/sbin/tcpdump\n   ```\n\n2. Verify that you can run `tcpdump` without sudo. The following command should work properly and produce a pcap file.\n   ```bash\n   $ tcpdump -i lo -w test.pcap\n   # wait a few seconds\n   # ctrl+c\n\n   $ ls -l test.pcap\n   # verify that the file exists and is not empty\n   ```\n\n## Running Differ\n\n**Sample Project Configuration**\n\n```yaml\n# Sample project configuration file: project.yml\n\n# Unique name\nname: coreutils_echo\n\n# Path to the original binary\noriginal: /usr/bin/echo\n# List of debloated binaries to test against. The key is the debloater name and the value\n# is the path to the debloated version of the original binary.\ndebloaters:\n  # Replace this path to the debloated version\n  binrec: /usr/bin/echo\n\n# List of templates to generate, run, and compare against the original binary\ntemplates:\n  # command line arguments (supports Jinja2 templates from variables)\n  - arguments: '{{left}} + {{right}}'\n\n    # Fuzzing variables. The variables are generated and populated into the command line\n    # arguments and any template input files for each run.\n    variables:\n      left:\n        type: int\n        range:\n          # generate 5 integers in the range of 0-99 (inclusive)\n          minimum: 0\n          maximum: 99\n          count: 5\n\n      right:\n        type: int\n        # Use the following 3 int values\n        values:\n          - -1\n          - 0\n          - 1\n\n    # List of comparators that verify the debloated version\n    comparators:\n      # verify stdout matches\n      - stdout\n\n      # verify stderr matches\n      - id: stderr\n\n      # verify the exit code is identical\n      - exit_code\n```\n\nTo run this project:\n\n```bash\n$ pipenv run differ --verbose project.yml\n```\n\nThe output is stored in the `./reports` directory by default and only errors are recorded. To change the output directory and output all reports, including successful runs:\n\n```bash\n$ pipenv run differ --verbose --report-successes --report-dir ./output project.yml\n```\n\nReports are stored in `{report_dir}/{project.name}/report-{engine}-[success|error]-{trace.id}.yml`. For example, a trace of the `binrec` debloater for the `coreutils_echo` project that failed would have a report located at:\n\n```yaml\n# $ cat ./reports/coreutils_echo/report-binrec-error-001.yml\n\narguments:\n- '70'\n- +\n- '-1'\nbinary: /usr/bin/echo-binrec\nresults:\n- comparator: stdout\n  details: stdout content does not match\n  status: error\n- comparator: stderr\n  details: ''\n  status: success\n- comparator: exit_code\n  details: ''\n  status: success\ntrace_directory: /home/user/Projects/differ/reports/coreutils_echo/trace-001/binrec\nvalues:\n  left: 70\n  right: -1\n```\n\nIn this example, the stdout content did not match the original's.\n\n## Getting Benchmark Sample Specs\n\nThe `differ.spec` module loads all benchmark sample projects and outputs a CSV report containing all the command line argument invocations that will be executed. This is useful when determining what features are expected to be present in debloated samples.\n\n```bash\n$ pipenv run python differ-spec -o specs.csv\n```\n\n## Acknowledgements\n\nThis material is based upon work supported by the Office of Naval\nResearch (ONR) under Contract No. N00014-21-C-1032. Any opinions, findings\nand conclusions or recommendations expressed in this material are those\nof the author(s) and do not necessarily reflect the views of the ONR.\n\n\u003c!--\nspell-checker:ignore binrec coreutils pipenv deadsnakes pyright venv isort pytest libfuzzy lftp lighttpd chgrp setcap usermod binutils poppler imagemagick\n--\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftrailofbits%2Fdiffer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftrailofbits%2Fdiffer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftrailofbits%2Fdiffer/lists"}