{"id":31646397,"url":"https://github.com/xlab-uiuc/debcovdiff","last_synced_at":"2025-10-10T14:01:27.832Z","repository":{"id":318255231,"uuid":"1069441537","full_name":"xlab-uiuc/DebCovDiff","owner":"xlab-uiuc","description":"Artifact of \"DebCovDiff: Differential Testing of Coverage Measurement Tools on Real-World Projects\" (ASE '25)","archived":false,"fork":false,"pushed_at":"2025-10-06T05:11:05.000Z","size":12437,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-06T05:43:22.908Z","etag":null,"topics":["code-coverage","debian","differential-testing","gcc","llvm"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/xlab-uiuc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-10-04T00:18:27.000Z","updated_at":"2025-10-06T05:11:09.000Z","dependencies_parsed_at":"2025-10-06T05:53:51.386Z","dependency_job_id":null,"html_url":"https://github.com/xlab-uiuc/DebCovDiff","commit_stats":null,"previous_names":["xlab-uiuc/debcovdiff"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/xlab-uiuc/DebCovDiff","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xlab-uiuc%2FDebCovDiff","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xlab-uiuc%2FDebCovDiff/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xlab-uiuc%2FDebCovDiff/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xlab-uiuc%2FDebCovDiff/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/xlab-uiuc","download_url":"https://codeload.github.com/xlab-uiuc/DebCovDiff/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xlab-uiuc%2FDebCovDiff/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279004061,"owners_count":26083668,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-10T02:00:06.843Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["code-coverage","debian","differential-testing","gcc","llvm"],"created_at":"2025-10-07T05:49:28.502Z","updated_at":"2025-10-10T14:01:27.754Z","avatar_url":"https://github.com/xlab-uiuc.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DebCovDiff\n\nThe experiments are mainly run on a CloudLab c6420 physical server (amd64, see\ndetailed specs in [CloudLab documentation](https://docs.cloudlab.us/hardware.html)),\nwith an Ubuntu 22.04 image.\n\nOther Debian-based distributions and virtual machines should also work but less\ntested.\nContainers, other Linux distributions or operating systems or other architectures\nare never guaranteed to work.\n\n\u003e [!WARNING]\n\u003e `sudo` is used in setup and test scripts. It is recommended to use\n\u003e one-off machines like CloudLab or virtual machines.\n\n## 1. Setup\n\n```shell\nwget 'https://github.com/xlab-uiuc/DebCovDiff/blob/main/diff/scripts/setup.sh?raw=true' -O- | bash\n```\n\nAs prompted, log out the current shell and log back in again, to make sure\nyou are correctly in `sbuild` group (check via `id -nG | grep sbuild`).\n\n[This script](./diff/scripts/setup.sh) notably does the following:\n\n1. Build GCC (`c1fb78fb` + [changes](https://github.com/xlab-uiuc/gcc-DebCovDiff/compare/base...xlab-uiuc:gcc-DebCovDiff:DebCovDiff))\n   and LLVM (`ce9a2c65` + [changes](https://github.com/xlab-uiuc/llvm-project-DebCovDiff/compare/base...xlab-uiuc:llvm-project-DebCovDiff:DebCovDiff))\n2. Run [`sbuild-createchroot`](https://manpages.debian.org/bookworm/sbuild/sbuild-createchroot.8.en.html)\n   for each toolchain which (1) sets up an isolated `chroot` and (2) runs\n   [`debootstrap`](https://manpages.debian.org/bookworm/debootstrap/debootstrap.8.en.html)\n   in it. This step creates the environment for general Debian package build.\n\n   For reproducibility, all Debian binaries and source files are [pinned](./diff/config.json)\n   to snapshots in December 2024 (Debian 12.8) via `https://snapshot.debian.org/`\n   instead of the regular `http://deb.debian.org/debian`.\n\n3. Copy custom toolchains (step 1) into the `chroot`s and rewire GCC invocations\n   to the desired toolchain with appropriate flags, via hook scripts embedded in\n   [`configure-all-chroot.sh`](debian/scripts/configure-all-chroot.sh).\n\n## 2. Test with One Debian Package\n\n```shell\nAUTO_TESTS=1 \\\nALL_METRICS=1 \\\nLOG_LEVEL=warning \\\nSHOW_SOURCE=1 \\\nSTART_WITH=\"download_source\" \\\n$REPO_DIR/diff/scripts/debian-diff.sh procps\n```\n\nOptions:\n\n- `procps`: Debian [source package](https://www.debian.org/doc/debian-policy/ch-source.html)\n  \u003csup\u003e(not [binary package](https://www.debian.org/doc/debian-policy/ch-binary.html))\u003c/sup\u003e name\n- `LOG_LEVEL=\u003clevel\u003e`: one of `error`, `warning`, `info`, `debug`\n- `SHOW_SOURCE=1`: show the inconsistent source code snippet\n- `ALL_METRICS=1`: warn of inconsistencies for all metrics. Otherwise this is\n  configurable in [`inconsistency.py`](./diff/oracles/inconsistency.py).\n- `START_WITH=\u003cmode\u003e`\n    - `\"download_source\"`: run everything (starting with pulling the source from\n      Debian) at a new directory `/var/lib/sbuild/build/\u003cpackage\u003e-\u003ctoolchain\u003e-\u003cnew_id\u003e`\n    - `\"test\"`: if the package has been run before, skip downloading the source\n      and build, but directly run tests, generate coverage reports and perform\n      differential testing. Everything happens in the old directory\n      `/var/lib/sbuild/build/\u003cpackage\u003e-\u003ctoolchain\u003e-\u003cold_id\u003e`\n    - `\"diff\"`: if the package has been run before, skip everything but the\n      final differential testing step, based on the existing coverage reports\n      under `/var/lib/sbuild/build/\u003cpackage\u003e-\u003ctoolchain\u003e-\u003cold_id\u003e`\n- `AUTO_TESTS=1`: measure coverage of\n  [`dh_auto_test`](https://manpages.debian.org/bookworm/debhelper/dh_auto_test.1.en.html)\n  if available. Otherwise invoke simple commands as specified in\n  [`./debian/scripts/chroot/`](./debian/scripts/chroot/).\n\nBack up and clean build directory before moving on to next sections\n\n```shell\ncp -r /var/lib/sbuild/build/ /var/lib/sbuild/build-individual-packages\nrm -rf /var/lib/sbuild/build/*\n```\n\n## 3. Test with All Debian Packages\n\n### Run with Existing Tests (ET) and 9 Debian Packages\n\n```shell\nexport AUTO_TESTS=1\n$REPO_DIR/diff/scripts/debian-batch.sh\n```\n\nBack up and clean build directory\n\n```shell\nmv $(ls -dt /var/lib/sbuild/build-* | head -2 | tail -1) /var/lib/sbuild/build-ET\nrm -rf /var/lib/sbuild/build/*\n```\n\n### Run with Simple Commands (SC) and 41 Debian Packages\n\n```shell\nexport AUTO_TESTS=0\n$REPO_DIR/diff/scripts/debian-batch.sh\n```\n\nBack up and clean build directory\n\n```shell\nmv $(ls -dt /var/lib/sbuild/build-* | head -2 | tail -1) /var/lib/sbuild/build-SC\nrm -rf /var/lib/sbuild/build/*\n```\n\n## 4. Inspect Raw DebCovDiff Results\n\nThe results (with one package or all packages) are generated under\n`/var/lib/sbuild/build*` directories:\n\n- `build`: the most recent run, where all actual builds happen\n- `build-\u003cdate\u003e-\u003crandom\u003e`: a copy of `build` after batch run\n- `build-{individual-packages,ET,SC}`: renamed to more recognizable names\n  in previous sections.\n\nThese directories have the following structure:\n\nFor each package there are two or four directories\n\n```text\n\u003cpackage\u003e-gcc-1   // build and measure coverage using GCC\n\u003cpackage\u003e-clang-1 // build and measure coverage using Clang/LLVM\n                  // (The below two only exist for batch runs)\n\u003cpackage\u003e-log     // various logs\n\u003cpackage\u003e         // historic inconsistencies\n```\n\nTake package `grep` for example\n\n(`+` means files created by our tool in addition to a regular Debian package build)\n\n```text\n/var/lib/sbuild/build-SC/grep-clang-1/\n├── grep-3.8/\n│   ├── Makefile, configure, ... // The upstream source code, e.g. by GNU developers\n│   ├── debian/                  // Configuration, patches etc by downstream Debian developers\n│   ├── dh_auto_test.log         // + Log of running dh_auto_test\n│   └── llvm-cov-profraw/        // + *.profraw files generated during test\n│\n├── llvm-cov-executables.txt // + List of instrumented executables and libraries,\n│                            //   which is passed as argument to llvm-cov\n├── default.profdata         // + Merged and indexed result of all *.profraw files\n├── default.json             // + JSON coverage report\n├── default.lcov.txt         // + LCOV coverage report\n├── text-coverage-report/    // + Text coverage report organized by source structure\n├── text-coverage-report.txt // + Text coverage report concatenated in one file\n│\n├── // Various Debian build artifacts\n├── grep_3.8-5.debian.tar.xz\n├── grep_3.8-5.disc\n├── grep_3.8-5_amd64.buildinfo\n├── grep_3.8-5_amd64.changes\n├── grep_3.8-5_amd64.deb\n└── ...\n```\n\nDue to the tools' nature, coverage related files are found at different places\nfor GCC.\n\n```text\n/var/lib/sbuild/build-SC/grep-gcc-1/\n├── grep-3.8/\n│   ├── Makefile, configure, ... // The upstream source code, e.g. by GNU developers\n│   ├── src/                     // The upstream source code, e.g. by GNU developers\n│   │   ├── grep.c               // The upstream source code, e.g. by GNU developers\n│   │   ├── grep.o               // Build files\n│   │   ├── grep.{gcda,gcno}     // + gcov note and data files that spread across the\n│   │   │                        //   whole build directory, usually found next to\n│   │   │                        //   the corresponding *.o file\n│   │   ├── lib/                                 // The upstream source code, e.g. by GNU developers\n│   │   │   ├── fcntl.c                          // The upstream source code, e.g. by GNU developers\n│   │   │   ├── libgreputils_a-fcntl.o           // Build files\n│   │   │   └── libgreputils_a-fcntl.{gcda,gcno} // + Another example of gcov note and data files\n│   │   └── ...\n│   ├── debian/                  // Configuration, patches etc by downstream Debian developers\n│   ├── dh_auto_test.log         // + Log of running dh_auto_test\n│   │\n│   ├── grep.c.gcov              // + Text coverage report\n│   ├── grep.c.gcov.json         // + JSON coverage report\n│   │\n│   ├── fcntl.c.gcov                    // + Another example of coverage reports.\n│   ├── libgreputils_a-fcntl.gcov.json  //   Note they are no longer reflecting\n│   │                                   //   the source structure\n│   └── ...\n│\n├── // Various Debian build artifacts\n├── grep_3.8-5.debian.tar.xz\n├── grep_3.8-5.disc\n├── grep_3.8-5_amd64.buildinfo\n├── grep_3.8-5_amd64.changes\n├── grep_3.8-5_amd64.deb\n└── ...\n```\n\n```text\n/var/lib/sbuild/build-SC/grep-log/\n├── 1.clang_build_log.txt\n├── 1.gcc_build_log.txt\n├── 1.compared.csv        // Total compared lines, branches and MC/DC decisions\n├── 1.inconsistent.csv    // Inconsistent lines, branches and MC/DC decisions\n├── 1.diff_log.txt        // Verbose log of inconsistencies\n├── ...\n├── T.clang_build_log.txt // T means repeated runs\n├── T.gcc_build_log.txt\n├── T.compared.csv\n├── T.inconsistent.csv\n└── T.diff_log.txt\n```\n\n$T$ is configurable via [`config.json`](./diff/config.json).\n\n## 5. Bug Reproduction\n\n### GCC#120321\n\nBug report: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=120321\n\nExample occurrence in Debian packages:\n\n- Code location: `/var/lib/sbuild/build-SC/gzip-gcc-1/gzip-1.12/gzip.c:467`\n- Coverage report location: `/var/lib/sbuild/build-SC/gzip-gcc-1/gzip-1.12/builddir/gzip.c.gcov:507`\n\n    ```c\n           10:  464:    z_suffix = Z_SUFFIX;\n           10:  465:    z_len = strlen(z_suffix);\n            -:  466:\n            6:  467:    while (true) {\n            -:  468:        int optc;\n           16:  469:        int longind = -1;\n            -:  470:\n           16:  471:        if (env_argv)\n    ```\n\n### LLVM#131505 (Listing 1)\n\nBug report: https://github.com/llvm/llvm-project/issues/131505\n\nExample occurrence in Debian packages:\n\n- Code location: `/var/lib/sbuild/build-SC/hostname-clang-1/hostname-3.23+nmu1/hostname.c:312`\n- Coverage report location: `/var/lib/sbuild/build-SC/hostname-clang-1/text-coverage-report/coverage/build/hostname-clang-1/hostname-3.23+nmu1/hostname.c.txt:611`\n\n    ```c\n      311|                   0|\t\t\t\t\tsin6 = (struct sockaddr_in6 *)ifap-\u003eifa_addr;\n      312|                   0|\t\t\t\t\tif (IN6_IS_ADDR_LINKLOCAL(\u0026sin6-\u003esin6_addr) ||\n      -------------------------------\n      |  Branch (312:10): [True: 0, False: 0]\n      -------------------------------\n      313|                   0|\t\t\t\t\t\t\tIN6_IS_ADDR_MC_LINKLOCAL(\u0026sin6-\u003esin6_addr))\n      -------------------------------\n      |  Branch (313:8): [True: 18446744073709551615, False: 1]\n      -------------------------------\n      |---\u003e MC/DC Decision Region (312:10) to (313:32)\n      |\n      |  Number of Conditions: 2\n      |     Condition C1 --\u003e (312:10)\n      |     Condition C2 --\u003e (313:8)\n      |\n      |  Executed MC/DC Test Vectors:\n      |\n      |     None.\n      |\n      |  C1-Pair: not covered\n      |  C2-Pair: not covered\n      |  MC/DC Coverage for Decision: 0.00%\n      |\n      -------------------------------\n      314|                   0|\t\t\t\t\t\tcontinue;\n      315|                   0|\t\t\t\t}\n    ```\n\n    (We [patched LLVM](https://github.com/xlab-uiuc/llvm-project-DebCovDiff/commit/201222abab7d4d1eed1125520699d8bcce8d18a2)\n    in our experiments for debugging purposes, so that it expands \"18.4E\" into full digits)\n\n### More bugs WIP...\n\n## 6. Csmith Experiments\n\nBuild Csmith\n\n```shell\npushd /tmp/\ngit clone https://github.com/csmith-project/csmith.git\ncd csmith\ngit checkout 0ec6f1bad2df865beadf13c6e97ec6505887b0a5\ncmake -D CMAKE_C_COMPILER=/usr/bin/gcc -D CMAKE_CXX_COMPILER=/usr/bin/g++ .\nmake -j10\nsudo make -j10 install\npopd\n```\n\n```shell\ncd $REPO_DIR/csmith\n```\n\nFor each Csmith configuration (default, `--inline-function`, and `--lang-cpp`),\n\n1. Generate **(~21min)** and check **(~3min)** 1,000 programs\n\n    \u003c!--\n      Measurement: 20:31.18 and 2:58.01\n\n      FIXME Certain seeds take exceptionally longer time to generate and\n      stall the whole process.\n    --\u003e\n\n    ```shell\n    python gen.py --first-1k --nproc=40\n    bash check-1k.sh\n    ```\n\n    Expected output:\n\n    ```text\n    gcc117412   0/1000     0/1000     0/1000\n    gcc117415   0/1000     0/1000     0/1000\n    gcc120319   0/1000     0/1000     0/1000\n    gcc120321   0/1000     0/1000     0/1000\n    gcc120332   769/1000   765/1000   772/1000\n    gcc120348   0/1000     0/1000     0/1000\n    gcc120478   0/1000     0/1000     0/1000\n    gcc120482   0/1000     0/1000     0/1000\n    gcc120484   841/1000   841/1000   855/1000\n    gcc120486   0/1000     0/1000     0/1000\n    gcc120489   798/1000   803/1000   818/1000\n    gcc120490   0/1000     0/1000     0/1000\n    gcc120491   0/1000     0/1000     0/1000\n    gcc120492   0/1000     0/1000     0/1000\n    llvm105341  0/1000     0/1000     0/1000\n    llvm114622  1000/1000  1000/1000  1000/1000\n    llvm116884  0/1000     0/1000     0/1000\n    llvm140427  0/1000     0/1000     0/1000\n    ```\n\n2. Generate **(~9.5h)** and check **(~5h)** 100,000 programs\n\n    \u003c!-- Measurement 9:23:14 and 4:52:57 --\u003e\n\n    ```shell\n    python gen.py --nproc=40\n    bash check-100k.sh\n    ```\n\n    Expected output:\n\n    ```text\n    gcc117412   0/100000       0/100000       0/100000\n    gcc117415   0/100000       0/100000       0/100000\n    gcc120319   0/100000       0/100000       0/100000\n    gcc120321   0/100000       0/100000       0/100000\n    gcc120332   75674/100000   75534/100000   76193/100000\n    gcc120348   0/100000       0/100000       0/100000\n    gcc120478   0/100000       0/100000       0/100000\n    gcc120482   0/100000       0/100000       0/100000\n    gcc120484   82831/100000   82838/100000   83121/100000\n    gcc120486   0/100000       0/100000       0/100000\n    gcc120489   79105/100000   79001/100000   79310/100000\n    gcc120490   0/100000       0/100000       0/100000\n    gcc120491   0/100000       0/100000       0/100000\n    gcc120492   0/100000       0/100000       0/100000\n    llvm105341  0/100000       0/100000       0/100000\n    llvm114622  100000/100000  100000/100000  100000/100000\n    llvm116884  0/100000       0/100000       0/100000\n    llvm140427  0/100000       0/100000       0/100000\n    ```\n\n## 7. Bug Age Study\n\n```shell\ncd $REPO_DIR/bug-ages\ndocker build -t old-compilers-env .\ndocker run -it --rm -v $PWD:/usr/src/app old-compilers-env\n```\n\nInside the container\n\n```shell\nbash bugs/setup-links.sh\ngcc --version\nclang --version\n```\n\n```shell\nbash bugs/run.sh |\u0026 tee log.txt\ngrep 'NOT REPRODUCING' log.txt\ngrep ' OK' log.txt\n```\n\n## 8. Generate Figure 4, 7, and 8 and Table 1, 2, and 3\n\n```shell\ncd $REPO_DIR/tables-and-figures/scripts\npython run.py\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxlab-uiuc%2Fdebcovdiff","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fxlab-uiuc%2Fdebcovdiff","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxlab-uiuc%2Fdebcovdiff/lists"}