{"id":18498334,"url":"https://github.com/softdevteam/rustgc_paper_experiment","last_synced_at":"2025-07-08T18:05:12.318Z","repository":{"id":239168547,"uuid":"798327748","full_name":"softdevteam/rustgc_paper_experiment","owner":"softdevteam","description":null,"archived":false,"fork":false,"pushed_at":"2025-06-28T00:16:28.000Z","size":381,"stargazers_count":0,"open_issues_count":0,"forks_count":2,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-06-28T01:24:17.835Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/softdevteam.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-05-09T14:59:39.000Z","updated_at":"2025-06-28T00:16:32.000Z","dependencies_parsed_at":"2024-05-10T13:31:46.608Z","dependency_job_id":"95b1172a-787e-4660-b68e-26b017596a83","html_url":"https://github.com/softdevteam/rustgc_paper_experiment","commit_stats":null,"previous_names":["softdevteam/rustgc_paper_experiment"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/softdevteam/rustgc_paper_experiment","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/softdevteam%2Frustgc_paper_experiment","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/softdevteam%2Frustgc_paper_experiment/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/softdevteam%2Frustgc_paper_experiment/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/softdevteam%2Frustgc_paper_experiment/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/softdevteam","download_url":"https://codeload.github.com/softdevteam/rustgc_paper_experiment/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/softdevteam%2Frustgc_paper_experiment/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264320939,"owners_count":23590560,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T13:38:47.841Z","updated_at":"2025-07-08T18:05:12.312Z","avatar_url":"https://github.com/softdevteam.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Experiment for \"Garbage Collection for Rust: The Finalizer Frontier\"\n\nThis is the experiment for the paper [\"Garbage Collection for Rust: The\nFinalizer Frontier\"](https://arxiv.org/abs/2504.01841) by Jacob Hughes and\nLaurence Tratt. The repository for the paper can be found at\nhttps://github.com/softdevteam/rustgc_paper.\n\n\n## Quickstart\n\nTo quickly run the experiments using prebuilt binaries, first make sure your\nsystem meets the [requirements](#system-requirements-1) and has the necessary\n[dependencies](#dependencies-1) installed.\n\nThen, execute the following commands:\n\n```sh\ngit clone https://github.com/softdevteam/rustgc_paper_experiment\ncd rustgc_paper_experiment\nmake run-quick\n```\n\nThis will run each experiment 5 times using the pre-built Docker image. Raw\nexperiment data will be saved in the `results` directory at the project root,\nand the generated plots will be available in the `plots` directory.\n\nFor more information and alternative setup methods, please refer to the\nsections below.\n\n\n## Table of Contents\n\n- [Reproducing our experiments](#reproducing-our-experiments)\n  - [Using the Docker Image (Recommended)](#using-the-docker-image-recommended)\n    - [System Requirements](#system-requirements)\n    - [Dependencies](#dependencies)\n    - [Running the experiments](#running-the-experiments)\n      - [1. Using Prebuilt Binaries (Recommended for Quick Setup)](#1-using-prebuilt-binaries-recommended-for-quick-setup)\n      - [2. Building Configurations from Source (For Full Replication)](#2-building-configurations-from-source-for-full-replication)\n  - [Running the experiments on bare-metal](#running-the-experiments-on-bare-metal)\n    - [System Requirements](#system-requirements-1)\n    - [Dependencies](#dependencies-1)\n    - [Building and running on bare-metal](#building-and-running-on-bare-metal)\n- [Customizing Experiment Runs](#customizing-experiment-runs)\n- [Differences from Initial Submission](#differences-from-initial-submission)\n  - [Experiment Modifications](#experiment-modifications)\n  - [Alloy Modifications](#alloy-modifications)\n  - [Observed Differences in Results](#observed-differences-in-results)\n\n\n## Reproducing our experiments\n\nWe offer two ways to run our experiments:\n\n1. **[Using the Provided Docker Image (Recommended)](#using-the-docker-image):**\n This method uses our pre-built Docker image, which comes with all necessary\ndependencies and offers the simplest setup experience. Note that results may be\nslightly less accurate than running on bare metal due to potential\nvirtualization overhead.\n\n2. **[Running Natively (Bare Metal)](#running-natively-bare-metal):**\nAlternatively, you can run the experiments directly on your own system. This\ninvolves manually installing all required dependencies, but may yield\nmore accurate benchmarking results by avoiding virtualization overhead.\n\n\n## Using the Docker Image (Recommended)\n\nThis approach relies on a pre-built Docker image that includes all required\ndependencies, providing a consistent environment across different host systems.\nTo run experiments using this method, ensure your host machine meets the\nfollowing requirements and has the necessary dependencies installed.\n\n### System Requirements\n\n- Any 64-bit OS capable of running Docker with Linux container support (e.g.,\nLinux, macOS with Docker Desktop, or Windows with Docker Desktop)\n- x86_64 or ARM architecture\n- At least 8 GB RAM and 30 GB free disk space\n- Internet connection\n\n### Dependencies\n\n- `git`\n- `GNU make`\n- `docker-engine`\n- `docker-buildkit`\n\nIf you need to install Docker, refer to [Appendix: Installing Docker on\nDebian](#appendix-installing-docker-on-debian) for a quick Debian-specific\nguide. For installation instructions on other platforms, please see the\n[official Docker documentation](https://docs.docker.com/engine/install/).\n\n### Running the experiments\n\nBegin by cloning the repository and navigating into its directory:\n\n```sh\ngit clone https://github.com/softdevteam/rustgc_paper_experiment\ncd rustgc_paper_experiment\n```\n\nYou can now run one of the two docker-based experiment methods.\n\n#### 1. Using Prebuilt Binaries (Recommended for Quick Setup)\n\nThis method uses prebuilt binaries for the different Alloy and Benchmark\nconfigurations within the Docker image. It is ideal if you want to save time or\nquickly verify the experiments without building everything from source.\n\nWith the Docker service running, execute the following command:\n\n```sh\nmake run-quick\n```\n\nThis will execute each experiment 5 times.\n\n#### 2. Building Configurations from Source (For Full Replication)\n\nThis method involves building all components from source within the Docker\nenvironment. It is recommended if you wish to fully replicate the experiments\nas described in our paper, or if you want to inspect or modify the source code\nduring the process.\n\nWith the Docker service running, execute the following command:\n\n```sh\nmake run-full\n```\n\nThis will run each experiment 30 times -- the same number of iterations we used\nin the paper.\n\n## Running the experiments on bare-metal\n\nWe recommend that you use the docker image as there are lots of required\ndependencies and the Dockerfile guarantees that they are pinned to the correct\nversion, however, if you wish to run the experiments on your own machine you\ncan do so as follows.\n\n\u003e [!CAUTION]\n\u003e **Platform Limitations**: Bare-metal experiments with Alloy have only been\n\u003e tested on Linux systems with x86-64 hardware. Running bare-metal experiments\n\u003e on macOS is not supported natively due to platform-dependent features in\n\u003e Alloy;  macOS users must therefore use provided Docker image, but note that\n\u003e this will be slower because it relies on QEMU-based emulation. Support for\n\u003e other operating systems, including BSD-like platforms, is currently unknown\n\u003e and untested. Other OSes may also require additional dependencies that we are\n\u003e unaware of.\n\n### System Requirements\n- 64-bit Linux (x86_64)\n- 4GiB RAM minimum (8GiB recommended)\n- At least 60GiB disk-space for build artefacts\n- 1GiB disk-space for each benchmark iteration. (e.g. to reproduce our\n  experiment using 30 iterations, you will need 30GiB of disk-space.) [^1]\n\n[^1]: Around 99% of the disk usage for benchmarking results comes from our\ndetailed heap profiling traces recorded from the memory experiments. If you\nonly want to run performance benchmarks, you can avoid this while also\ndramatically speeding up the time needed to run the benchmarks.\n\n### Dependencies\n\nThe experiments require a lot of development tools and libraries. Below, you’ll\nfind package installation commands for the two tested Linux distributions:\nDebian and Arch Linux. Alternatively, you can install the packages manually and\nuse the checklist to track your progress. Most other distributions should\nprovide equivalent packages, though names may vary and we have not tested them.\n\n\u003cdetails\u003e\n  \u003csummary\u003eDebian\u003c/summary\u003e\n\nTo install all dependencies at once, run the following command (with `sudo` or `doas`):\n\n```sh\napt install -y make build-essential curl git cmake python3 libtinfo-dev libzip-dev ninja-build gdb pipx rsync libdwarf-dev libunwind-dev libboost-dev libboost-iostreams-dev libboost-program-options-dev zlib1g-dev zstd elfutils pkg-config libssl-dev liblzma-dev libffi-dev libedit-dev llvm-dev clang procps autotools-dev gperf bison flex xvfb\n```\n\nAlternatively, you can install packages individually and check them off below:\n\n- [ ] make\n- [ ] build-essential\n- [ ] curl\n- [ ] git\n- [ ] cmake\n- [ ] python3\n- [ ] libtinfo-dev\n- [ ] libzip-dev\n- [ ] ninja-build\n- [ ] gdb\n- [ ] pipx\n- [ ] rsync\n- [ ] libdwarf-dev\n- [ ] libunwind-dev\n- [ ] libboost-dev\n- [ ] libboost-iostreams-dev\n- [ ] libboost-program-options-dev\n- [ ] zlib1g-dev\n- [ ] zstd\n- [ ] elfutils\n- [ ] pkg-config\n- [ ] libssl-dev\n- [ ] liblzma-dev\n- [ ] libffi-dev\n- [ ] libedit-dev\n- [ ] llvm-dev\n- [ ] clang\n- [ ] procps\n- [ ] autotools-dev\n- [ ] gperf\n- [ ] bison\n- [ ] flex\n- [ ] xvfb\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003eArch Linux\u003c/summary\u003e\n\nTo install all dependencies at once, run the following command (with `sudo` or `doas`):\n\n```sh\nsudo pacman -Syu --needed base-devel make curl git cmake python ncurses libzip ninja gdb python-pipx rsync libdwarf libunwind boost boost-libs zlib zstd elfutils pkgconf openssl xz libffi libedit llvm clang procps-ng autoconf automake gperf bison flex xorg-server-xvfb\n```\n\nAlternatively, you can install packages individually and check them off below:\n\n- [ ] base-devel\n- [ ] make\n- [ ] curl\n- [ ] git\n- [ ] cmake\n- [ ] python\n- [ ] ncurses\n- [ ] libzip\n- [ ] ninja\n- [ ] gdb\n- [ ] python-pipx\n- [ ] rsync\n- [ ] libdwarf\n- [ ] libunwind\n- [ ] boost\n- [ ] boost-libs\n- [ ] zlib\n- [ ] zstd\n- [ ] elfutils\n- [ ] pkgconf\n- [ ] openssl\n- [ ] xz\n- [ ] libffi\n- [ ] libedit\n- [ ] llvm\n- [ ] clang\n- [ ] procps-ng\n- [ ] autoconf\n- [ ] automake\n- [ ] gperf\n- [ ] bison\n- [ ] flex\n- [ ] xorg-server-xvfb\n\n\u003c/details\u003e\n\n### Building and running on bare-metal\n\nOnce you have installed the required dependencies, you can run the experiments with the following:\n\n`make bare-metal`\n\nThe same environment variables as above will also work here too.\n\n\n## Customizing Experiment Runs\n\n\u003e [!NOTE]\n\u003e Running the full experiment suite with all configurations, as described in our\n\u003e paper, is a time- and resource-intensive process. On our benchmarking server\n\u003e (64 cores, 128 GiB RAM), a complete run takes over 48 hours and consumes\n\u003e approximately 300 GiB of disk space for build artifacts and raw results. To\n\u003e save time and storage, you may wish to run only a subset of experiments or\n\u003e reduce the number of iterations (or increase them, if desired). This can be\n\u003e easily configured using environment variables, which let you specify exactly\n\u003e what to run.\n\nYou can customise which experiments and benchmarks are run using the following\nenvironment variables.\n\n**These variables can be used whether you are running experiments via Docker or directly on bare metal.**\n\n- **`EXPERIMENTS`**\n  Space-separated list of experiments to run.\n\n  Options:\n  - **`gcvs`**: Compare Alloy with other memory management approaches (e.g.,\n  Rc). Not all benchmark suites have the same variants, but each will include a\n  variant using the baseline system allocator and the program’s original memory\n  strategy.\n  - **`elision`**: Evaluates the *finalizer elision optimization* (see Section\n  5 of the paper) by comparing two Alloy configurations: with elision and\n  without elision.\n  - **`premopt`**: Evaluates the cost of *premature finalizer prevention*\n  (Section 6 of the paper) by comparing three Alloy configurations: naive\n  (barriers for every garbage-collected pointer), opt (unnecessary barriers\n  optimized away), and none (idealized version with no barriers; this is unsound).\n\n  **Default:** `EXPERIMENTS=\"gcvs elision premopt\"`\n\n- **`SUITES`**\n  Space-separated list of benchmark suites to run.\n\n  Options:\n  - **`alacritty`**: Terminal emulator workload ([repo](https://github.com/alacritty/alacritty)).\n  - **`binary-trees`**: Classic binary trees microbenchmark ([repo](#)).\n  - **`fd`**: A rust alternative to the UNIX `find` command ([repo](https://github.com/sharkdp/fd)).\n  - **`grmtools`**: Parsing benchmark of the grmtools error recovery algorithm ([repo](https://github.com/softdevteam/grmtools)).\n  - **`regex-redux`**: Regular expression processing benchmark ([repo](#)).\n  - **`ripgrep`**: Real-world text searching workload ([repo](https://github.com/BurntSushi/ripgrep)).\n  - **`som-rs-ast`**: Smalltalk interpreter (AST variant) ([repo](https://github.com/Hirevo/som-rs)).\n  - **`som-rs-bc`**: Smalltalk interpreter (bytecode variant) ([repo](https://github.com/Hirevo/som-rs)).\n  - **`yksom`**: Alternative Smalltalk interpreter for Alloy configuration comparisons only ([repo](https://github.com/softdevteam/yksom)).\n    *Note:* `yksom` does not run with the `gcvs` experiment.\n\n  **Default:** `SUITES=\"alacritty binary-trees fd grmtools regex-redux ripgrep som-rs-ast som-rs-bc yksom\"`\n\n- **`MEASUREMENTS`**\n  Specifies which types of data to record.\n\n  Options:\n  - **`perf`**: Collects performance data (wall-clock and system-time).\n  - **`mem`**: Gathers detailed memory allocation data for the `gcvs` experiment using [KDE HeapTrack](https://github.com/KDE/heaptrack) (resource-intensive).\n  - **`metrics`**: Records high-level experiment metrics (e.g., collection counts, pause times, finalizers run, etc ).\n\n  **Default:** `MEASUREMENTS=\"perf mem metrics\"`\n  *Note:* The `mem` measurement is the most resource-intensive. For most purposes, using just `perf` and `metrics` will suffice and is much faster.\n\n- **`PEXECS`**\n  Number of process executions (iterations) per experiment.\n\n  **Default:** `PEXECS=5` for the quick prebuilt binary Docker image, or `PEXECS=30` otherwise (as in our paper).\n  *Note:* Fewer iterations will run faster but result in wider confidence intervals and potentially less statistically significant results.\n\nYou can combine these environment variables in any way to customize which\nexperiments are run.\n\n**Example:**\n\nTo run the Docker prebuilt experiments with 10 process executions and only the `perf` and `metrics` measurements:\n\n```sh\nPEXECS=10 MEASUREMENTS=\"perf metrics\" make run-quick\n```\n\n\n## Differences from Initial Submission\n\nSince the initial submission, we have updated both the experimental evaluation\nand Alloy itself. These changes may affect how the data is presented, as well\nas the results themselves. Below, we outline the main modifications and any\nobserved impact on the data.\n\n### Experiment Modifications\n\nWe have pre-emptively included additional metrics that were requested by\nreviewers during the peer review process. However, for some of these, we\ncurrently only have the raw data available. This will be updated before the\npaper revision deadline. Metrics highlighted in **bold** below were explicitly\nrequested by reviewers; the others were added to improve the accuracy and\ncompleteness of comparisons.\n\n  These include:\n  - **Baseline allocator results for each benchmark suite**\n  - **GC cycle counts**\n  - **A breakdown of GC pause times**\n  - **Different heap sizes**\n  - More detailed finalizer breakdown, including both recursive drop calls and the initial outer drop method.\n  - Bug fixes to the heap metric breakdown, allowing more accurate recording of the number of different shared memory types.\n\n### Alloy Modifications\n\nSince the original submission, we have made several improvements to Alloy that may impact experimental results.\n*Only changes with potential runtime impact are listed below.*\n\n- A new implementation of the finalization queue that uses BDWGC’s `finalize_on_demand` API\n  [[PR #179](https://github.com/softdevteam/alloy/pull/179), [PR #177](https://github.com/softdevteam/alloy/pull/177)]\n\n- Alloy now always dynamically links against BDWGC’s `libgc` library, which may influence compiler optimizations such as inlining\n  [[PR #189](https://github.com/softdevteam/alloy/pull/189), [PR #187](https://github.com/softdevteam/alloy/pull/187), [PR #183](https://github.com/softdevteam/alloy/pull/183), [PR #185](https://github.com/softdevteam/alloy/pull/185), [PR (bdwgc) #29](https://github.com/softdevteam/bdwgc/pull/29)]\n\n- The default global allocator is now set to BDWGC’s allocator automatically, so users no longer need to specify it with `#[global_allocator]`. While this change is unlikely to affect performance, we cannot completely rule it out\n  [[PR #192](https://github.com/softdevteam/alloy/pull/192)]\n\n\n## A brief guide to the experiment process\n\nThe `Makefile` serves as the entry point to the experiment. Whether you choose\nto run experiments inside Docker or on bare metal, it firsts downloads Alloy,\nchecks out a fixed version of it, and builds several configurations for the\ndifferent experiments. These binaries are stored in `artefacts/bin/alloy/`.\n\nNext, each benchmark is downloaded, checked out to a fixed version, and\ncompiled with the different Alloy configurations. Some benchmarks include\nharnesses which are part of this repo itself instead of downloaded, these can\nbe found in `src/`. Some benchmark variants uses patches found in `patch/`\nwhich are automatically applied. The patches are named accordingly, identifying\nwhich experiment variant they belong to. The resulting binaries are placed in\n`artefacts/bin/benchmarks/`.\n\nOnce all binaries are prepared, ReBench is used to run the experiments, with\nraw data saved to `results/`. The data is then processed and visualized using\nPython scripts, most of which are organized as Invoke tasks. The final plots\nare written to `plots/`, ready for direct inclusion in the paper.\n\n## Appendix: Installing Docker on Debian\n\nInstall Docker Engine on Debian 11/12 by running the following as root:\n\n```sh\nsudo bash -c '\napt-get update\napt-get install -y ca-certificates curl gnupg\ninstall -m 0755 -d /etc/apt/keyrings\ncurl -fsSL https://download.docker.com/linux/debian/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg\nchmod a+r /etc/apt/keyrings/docker.gpg\necho \"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian $(. /etc/os-release \u0026\u0026 echo \"$VERSION_CODENAME\") stable\" \u003e /etc/apt/sources.list.d/docker.list\napt-get update\napt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin\n'\n```\n\nTest that it's worked by running the `hello-world` container:\n\n```sh\nsudo docker run hello-world\n```\n\nSee [Docker’s official docs](https://docs.docker.com/engine/install/debian/) for troubleshooting.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsoftdevteam%2Frustgc_paper_experiment","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsoftdevteam%2Frustgc_paper_experiment","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsoftdevteam%2Frustgc_paper_experiment/lists"}