{"id":13587191,"url":"https://github.com/stepchowfun/docuum","last_synced_at":"2025-05-14T08:06:09.398Z","repository":{"id":38431849,"uuid":"231741297","full_name":"stepchowfun/docuum","owner":"stepchowfun","description":"Docuum performs least recently used (LRU) eviction of Docker images. 🗑️","archived":false,"fork":false,"pushed_at":"2025-03-20T09:57:48.000Z","size":517,"stargazers_count":628,"open_issues_count":11,"forks_count":34,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-22T03:30:03.631Z","etag":null,"topics":["docker","docker-images","lru-eviction"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/stepchowfun.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":"FUNDING.yml","license":"LICENSE.md","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null},"funding":{"github":"stepchowfun"}},"created_at":"2020-01-04T09:52:31.000Z","updated_at":"2025-04-12T08:25:34.000Z","dependencies_parsed_at":"2023-11-22T14:47:46.989Z","dependency_job_id":"77596a31-98cc-47f0-a93a-facbf63fc7af","html_url":"https://github.com/stepchowfun/docuum","commit_stats":{"total_commits":272,"total_committers":14,"mean_commits":"19.428571428571427","dds":"0.30147058823529416","last_synced_commit":"47534f7476cdfa93ecfac53e167eb66d0f5224e5"},"previous_names":[],"tags_count":46,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stepchowfun%2Fdocuum","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stepchowfun%2Fdocuum/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stepchowfun%2Fdocuum/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stepchowfun%2Fdocuum/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/stepchowfun","download_url":"https://codeload.github.com/stepchowfun/docuum/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254101615,"owners_count":22014909,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","docker-images","lru-eviction"],"created_at":"2024-08-01T15:06:04.916Z","updated_at":"2025-05-14T08:06:04.389Z","avatar_url":"https://github.com/stepchowfun.png","language":"Rust","readme":"# Docuum: LRU eviction of Docker images\n\n[![Build status](https://github.com/stepchowfun/docuum/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/stepchowfun/docuum/actions?query=branch%3Amain)\n\n*Docuum* performs least recently used (LRU) eviction of Docker images to keep the disk usage below a given threshold.\n\nDocker's built-in `docker image prune --all --filter until=…` command serves a similar purpose. However, the built-in solution isn't ideal since it uses the image creation time, rather than the last usage time, to determine which images to remove. That means it can delete frequently used images, which may be expensive to rebuild or time-consuming to pull.\n\nDocuum is ideal for use cases such as continuous integration (CI) workers, developer workstations, or any other environment in which Docker images accumulate on disk over time. Docuum works well with tools like [Toast](https://github.com/stepchowfun/toast) and [Docker Compose](https://docs.docker.com/compose/).\n\nDocuum is used by Netflix (on its production Kubernetes nodes) and Airbnb (on its CI fleet of 1.5k+ CI workers).\n\n## How it works\n\n[Docker doesn't record when an image was last used.](https://github.com/moby/moby/issues/4237) To work around this, Docuum listens for notifications via `docker events` to learn when images are used. It maintains a small piece of state in a local data directory (see [this](https://docs.rs/dirs/3.0.2/dirs/fn.data_local_dir.html) for details about where this directory is on various platforms). That persisted state allows you to freely restart Docuum (or the whole machine) without losing the image usage timestamp data.\n\nWhen Docuum first starts and subsequently whenever a new Docker event comes in, LRU eviction is performed until the total disk usage due to Docker images is below the given threshold. This design has a few advantages over evicting images based on a fixed [time to live](https://en.wikipedia.org/wiki/Time_to_live) (TTL), which is what various other tools in the Docker ecosystem do:\n\n1. There is no need to configure and tune an interval to run on. Docuum evicts images immediately whenever the disk usage exceeds the threshold without waiting for any timers.\n2. Docuum uses no CPU resources when there is no Docker activity. You can run it on your laptop without worrying about draining your battery.\n3. In order to prevent your disk from filling up, it's more straightforward to set a threshold based on disk usage rather than guessing an appropriate maximum image age.\n\nDocuum also respects the parent-child relationships between images. In particular, it will delete children of a parent before deleting the parent (even if the children were used more recently than the parent), because Docker doesn't allow images with children to be deleted.\n\n## Usage\n\nOnce Docuum is [installed](#installation-instructions), you can run it manually from the command line as follows:\n\n```sh\ndocuum --threshold '10 GB'\n```\n\nDocuum will then start listening for Docker events. You can use `Ctrl`+`C` to stop it.\n\nYou probably want to run Docuum as a [daemon](https://en.wikipedia.org/wiki/Daemon_\\(computing\\)), e.g., with [launchd](https://www.launchd.info/), [systemd](https://www.freedesktop.org/wiki/Software/systemd/), etc. See the [Configuring your operating system to run the binary as a daemon](#configuring-your-operating-system-to-run-the-binary-as-a-daemon) section below for instructions.\n\nHere are the supported command-line options:\n\n```\nUSAGE:\n    docuum\n\nOPTIONS:\n    -d, --deletion-chunk-size \u003cDELETION CHUNK SIZE\u003e\n            Removes specified quantity of images at a time (default: 1)\n\n    -h, --help\n            Prints help information\n\n    -k, --keep \u003cREGEX\u003e...\n            Prevents deletion of images for which repository:tag matches \u003cREGEX\u003e\n\n    -m, --min-age \u003cMIN AGE\u003e\n            Sets the minimum age of images to be considered for deletion\n\n    -t, --threshold \u003cTHRESHOLD\u003e\n            Sets the maximum amount of space to be used for Docker images (default: 10 GB)\n\n    -v, --version\n            Prints version information\n```\n\nThe `--threshold` flag accepts [multiple representations](https://docs.rs/byte-unit/4.0.12/byte_unit/struct.Byte.html#examples-2), like `10 GB`, `10 GiB`, or `10GB`. On Linux, percentage-based thresholds like `50%` are also supported.\n\nThe `--min-age` flag accepts [multiple representations](https://docs.rs/parse_duration/2.1.1/parse_duration/), such as `4 days` or `1 hour`.\n\nYou can change the log verbosity by setting an environment variable named `LOG_LEVEL` to one of `trace`, `debug`, `info`, `warning`, or `error`. The default is `debug`.\n\n## Docker's build cache\n\nOld versions of Docker would create an intermediate image for each step in your `Dockerfile`, and Docuum would happily vacuum them when needed. Since the introduction of [BuildKit](https://docs.docker.com/build/buildkit/), Docker no longer produces those intermediate images, and a separate \"build cache\" is used instead. BuildKit has its own [garbage collector](https://docs.docker.com/build/cache/garbage-collection/) for its build cache with a default threshold of 10% of the total disk capacity.\n\nDocuum does not vacuum BuildKit's build cache, and BuildKit's garbage collector doesn't vacuum images. Both can be used together.\n\n## Installation instructions\n\nInstallation consists of two steps:\n\n1. Installing the binary\n2. Configuring your operating system to run the binary as a daemon\n\n### Installing the binary\n\n#### Installation on macOS or Linux (AArch64 or x86-64)\n\nIf you're running macOS or Linux (AArch64 or x86-64), you can install Docuum with this command:\n\n```sh\ncurl https://raw.githubusercontent.com/stepchowfun/docuum/main/install.sh -LSfs | sh\n```\n\nThe same command can be used again to update to the latest version.\n\nThe installation script supports the following optional environment variables:\n\n- `VERSION=x.y.z` (defaults to the latest version)\n- `PREFIX=/path/to/install` (defaults to `/usr/local/bin`)\n\nFor example, the following will install Docuum into the working directory:\n\n```sh\ncurl https://raw.githubusercontent.com/stepchowfun/docuum/main/install.sh -LSfs | PREFIX=. sh\n```\n\nIf you prefer not to use this installation method, you can download the binary from the [releases page](https://github.com/stepchowfun/docuum/releases), make it executable (e.g., with `chmod`), and place it in some directory in your [`PATH`](https://en.wikipedia.org/wiki/PATH_\\(variable\\)) (e.g., `/usr/local/bin`).\n\n#### Installation on Windows (AArch64 or x86-64)\n\nIf you're running Windows (AArch64 or x86-64), download the latest binary from the [releases page](https://github.com/stepchowfun/docuum/releases) and rename it to `docuum` (or `docuum.exe` if you have file extensions visible). Create a directory called `Docuum` in your `%PROGRAMFILES%` directory (e.g., `C:\\Program Files\\Docuum`), and place the renamed binary in there. Then, in the \"Advanced\" tab of the \"System Properties\" section of Control Panel, click on \"Environment Variables...\" and add the full path to the new `Docuum` directory to the `PATH` variable under \"System variables\". Note that the `Program Files` directory might have a different name if Windows is configured for a language other than English.\n\nTo update an existing installation, simply replace the existing binary.\n\n#### Installation with Homebrew\n\nIf you have [Homebrew](https://brew.sh/), you can install Docuum as follows:\n\n```sh\nbrew install docuum\n```\n\nYou can update an existing installation with `brew upgrade docuum`.\n\n#### Installation with Cargo\n\nIf you have [Cargo](https://doc.rust-lang.org/cargo/), you can install Docuum as follows:\n\n```sh\ncargo install docuum\n```\n\nYou can run that command with `--force` to update an existing installation.\n\n#### Running Docuum in a Docker container on a host capable of running Linux containers\n\nIf you prefer not to install Docuum on your system and you're running macOS or Linux (AArch64 or x86-64), you can run it in a container:\n\n```sh\ndocker run \\\n  --init \\\n  --rm \\\n  --tty \\\n  --name docuum \\\n  --mount type=bind,src=/var/run/docker.sock,dst=/var/run/docker.sock \\\n  --mount type=volume,source=docuum,target=/root \\\n  stephanmisc/docuum --threshold '10 GB'\n```\n\nIf you're on a Windows system configured to run Linux containers, use this command:\n\n```powershell\ndocker run `\n  --init `\n  --rm `\n  --tty `\n  --name docuum `\n  --mount type=bind,src=//var/run/docker.sock,dst=/var/run/docker.sock `\n  --mount type=volume,source=docuum,target=/root `\n  stephanmisc/docuum --threshold '10 GB'\n```\n\nWe don't currently publish a Windows-based image, because some Windows machines (namely, those which run containers with process isolation rather than Hyper-V) can only run Windows containers that were built for the exact build of Windows (e.g., 1809) which is running on the host. This makes Windows-based images less portable, and as a result we'd need to publish a separate Windows-based image for each build of Windows we want to support. At this time, we don't have the infrastructure to do that.\n\nThe instructions below for configuring your operating system to run Docuum as a daemon assume it's installed as an executable binary. If you prefer to run it as a Docker container, change the relevant service definition to run a Docker command like the relevant one above, with the following adjustments:\n\n- Omit the `--tty` flag. This prevents Docuum from printing colored logs, which you probably don't want for a daemon.\n- Configure Docker as a hard dependency. Ordinarily, Docuum and Docker can be started in any order, and Docuum will patiently wait for Docker to start if needed. However, when running Docuum as a Docker container, then of course Docker must be started first.\n\n### Configuring your operating system to run the binary as a daemon\n\n#### Creating a launchd service on macOS\n\nOn macOS, [launchd](https://www.launchd.info/) can be used to run Docuum as a daemon. Create a file (owned by root) called `/Library/LaunchDaemons/local.docuum.plist` with the following contents, adjusting the arguments as needed:\n\n```xml\n\u003c?xml version=\"1.0\" encoding=\"UTF-8\"?\u003e\n\u003c!DOCTYPE plist PUBLIC \"-//Apple//DTD PLIST 1.0//EN\" \"https://www.apple.com/DTDs/PropertyList-1.0.dtd\"\u003e\n\u003cplist version=\"1.0\"\u003e\n    \u003cdict\u003e\n        \u003ckey\u003eLabel\u003c/key\u003e\n        \u003cstring\u003elocal.docuum\u003c/string\u003e\n        \u003ckey\u003eProgram\u003c/key\u003e\n        \u003cstring\u003e/usr/local/bin/docuum\u003c/string\u003e\n        \u003ckey\u003eProgramArguments\u003c/key\u003e\n        \u003carray\u003e\n            \u003cstring\u003e/usr/local/bin/docuum\u003c/string\u003e\n            \u003cstring\u003e--threshold\u003c/string\u003e\n            \u003cstring\u003e10 GB\u003c/string\u003e\n        \u003c/array\u003e\n        \u003ckey\u003eStandardOutPath\u003c/key\u003e\n        \u003cstring\u003e/var/log/docuum.log\u003c/string\u003e\n        \u003ckey\u003eStandardErrorPath\u003c/key\u003e\n        \u003cstring\u003e/var/log/docuum.log\u003c/string\u003e\n        \u003ckey\u003eEnvironmentVariables\u003c/key\u003e\n        \u003cdict\u003e\n            \u003ckey\u003ePATH\u003c/key\u003e\n            \u003cstring\u003e/bin:/usr/bin:/usr/local/bin\u003c/string\u003e\n        \u003c/dict\u003e\n        \u003ckey\u003eKeepAlive\u003c/key\u003e\n        \u003ctrue/\u003e\n    \u003c/dict\u003e\n\u003c/plist\u003e\n```\n\nRun `sudo launchctl load /Library/LaunchDaemons/local.docuum.plist` to start the service. You can view the logs with `tail -F /var/log/docuum.log`.\n\n#### Creating a systemd service on Linux\n\nOn most Linux distributions, [systemd](https://www.freedesktop.org/wiki/Software/systemd/) can be used to run Docuum as a daemon. Create a file (owned by root) called `/etc/systemd/system/docuum.service` with the following contents, adjusting the arguments as needed:\n\n```ini\n[Unit]\nDescription=Docuum\nAfter=docker.service\nWants=docker.service\n\n[Service]\nEnvironment='THRESHOLD=10 GB'\nExecStart=/usr/local/bin/docuum --threshold ${THRESHOLD}\nRestart=on-failure\n\n[Install]\nWantedBy=multi-user.target\n```\n\nRun `sudo systemctl enable docuum --now` to enable and start the service. You can view the logs with `sudo journalctl --follow --unit docuum`.\n\n#### Creating an NSSM service on Windows\n\nOn Windows, [NSSM](https://nssm.cc/), the \"Non-Sucking Service Manager\", can be used to run Docuum as a daemon. [Install NSSM](https://nssm.cc/download) by downloading the binary and adding it to your `PATH` (see the [Installation on Windows (x86-64)](#installation-on-windows-x86-64) section for instructions on how to configure this environment variable), then run Windows Terminal _as Administrator_ and enter the following command:\n\n```powershell\nnssm install Docuum\n```\n\nNSSM will then open a configuration window. Configure the following:\n\n- In the `Application` tab, select the path to the Docuum binary. You can optionally add arguments like `--threshold \"10 GB\"`.\n- Optionally, in the `I/O` tab, choose where you want the logs to be written.\n\nThen click the `Install service` button. Back in Windows Terminal, run the following to start the service:\n\n```powershell\nnssm start Docuum\n```\n\nIf you configured a path for the log file in the `I/O` tab of the installation window, you can view those logs with `Get-Content -Wait docuum.log` (adjusting the file path as needed).\n\n## Requirements\n\n- Docuum requires [Docker Engine](https://www.docker.com/products/container-runtime) 17.03.0 or later.\n","funding_links":["https://github.com/sponsors/stepchowfun"],"categories":["Rust","Development with Docker"],"sub_categories":["Garbage Collection"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstepchowfun%2Fdocuum","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstepchowfun%2Fdocuum","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstepchowfun%2Fdocuum/lists"}