https://github.com/czhao-dev/container-runtime
A minimal container runtime CLI in Go: Linux namespaces, pivot_root filesystem jailing, cgroups v2 resource limits, and interactive PTY passthrough — no image management, overlayfs, or daemon.
https://github.com/czhao-dev/container-runtime
cgroups cli container-runtime containers go golang linux namespaces pivot-root
Last synced: 1 day ago
JSON representation
A minimal container runtime CLI in Go: Linux namespaces, pivot_root filesystem jailing, cgroups v2 resource limits, and interactive PTY passthrough — no image management, overlayfs, or daemon.
- Host: GitHub
- URL: https://github.com/czhao-dev/container-runtime
- Owner: czhao-dev
- License: mit
- Created: 2026-07-03T01:55:55.000Z (9 days ago)
- Default Branch: main
- Last Pushed: 2026-07-08T07:11:56.000Z (4 days ago)
- Last Synced: 2026-07-08T07:17:02.698Z (4 days ago)
- Topics: cgroups, cli, container-runtime, containers, go, golang, linux, namespaces, pivot-root
- Language: Go
- Size: 64.5 KB
- Stars: 1
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# mini-run
A minimal container runtime CLI in Go — the core primitives real runtimes
(runc) use, without image management, overlayfs, a daemon, or networking:
- **Namespace isolation** — `CLONE_NEWPID` (container sees itself as PID 1),
`CLONE_NEWNS` (isolated mount table), `CLONE_NEWUTS` (custom hostname),
`CLONE_NEWIPC` (isolated IPC), via `syscall.SysProcAttr.Cloneflags`.
- **Filesystem jailing via `pivot_root`** — not `chroot`: the old root is
fully unmounted and detached, preventing container breakout.
- **cgroups v2 resource limits** — per-container memory ceiling and CPU
quota under `/sys/fs/cgroup/mini-run//`.
- **Interactive PTY passthrough** — stdio is wired straight through so
`/bin/sh`, `top`, etc. work interactively.
It is a stateless CLI, not a daemon: `sudo mini-run run [args...]`
against a rootfs directory you've already unpacked yourself (e.g. Alpine's
minirootfs tarball, extracted).
## Repository Layout
```
.
├── cmd/mini-run/ CLI entrypoint
│ ├── main.go arg dispatch: run | child (hidden) | -h
│ ├── run.go `run` subcommand: flags -> container.RunOptions
│ └── child.go hidden `child` subcommand -> container.RunChild
├── internal/
│ ├── container/
│ │ ├── types.go RunOptions
│ │ ├── run_linux.go parent-side orchestration (namespaces, cgroups, wait)
│ │ ├── child_linux.go PID-1 child logic: hostname, pivot_root, exec
│ │ ├── rootfs_linux.go pivot_root + synthetic /dev implementation
│ │ ├── id.go random 12-hex-char container ID
│ │ ├── id_test.go
│ │ └── *_other.go non-Linux build-tag stubs
│ └── cgroups/
│ ├── cgroups_linux.go cgroups v2 Setup / AddProcess / Cleanup
│ ├── limits.go ParseMemory / ParseCPUs (pure, cross-platform)
│ ├── limits_test.go
│ └── cgroups_other.go non-Linux build-tag stub
├── test/integration/
│ └── mini_run_test.go black-box, root+Linux-gated integration suite
│ (automates the Verification checklist below)
├── scripts/
│ ├── fetch-rootfs.sh downloads + extracts Alpine's minirootfs
│ ├── bench.sh startup-latency benchmark harness (hyperfine)
│ └── plot_bench.py renders the benchmark bar chart (matplotlib)
├── docs/
│ └── benchmark.png chart embedded in this README
├── lima.yaml disposable Ubuntu VM config for macOS dev/test
└── go.mod zero external dependencies
```
## Architecture
`mini-run` has no long-running daemon and no supervisor process tree: the
parent CLI invocation sets up resource limits, re-execs itself into new
namespaces, and then gets out of the way — the containerized command
becomes a real PID 1, not a child of some Go supervisor loop.
```mermaid
sequenceDiagram
participant Host as Host shell
participant Parent as mini-run run (parent)
participant CG as cgroups v2 (/sys/fs/cgroup/mini-run/)
participant Child as mini-run child (becomes PID 1)
participant Target as exec'd command
Host->>Parent: sudo mini-run run --memory 128m ./rootfs /bin/sh
Parent->>Parent: NewID(), resolve rootfs, ParseLimits()
Parent->>CG: mkdir /, write memory.max / cpu.max
Parent->>Child: re-exec self as "child" with Cloneflags
NEWNS|NEWUTS|NEWIPC|NEWPID, Pdeathsig=SIGKILL
Note over Child: new process, born as PID 1 inside
its own PID/mount/UTS/IPC namespaces
Parent->>CG: write cgroup.procs (attach child's real host PID)
Child->>Child: Sethostname("mini-run-")
Child->>Child: pivotRoot(rootfs) — see filesystem diagram below
Child->>Target: syscall.Exec(resolved cmd) — replaces its own process image
Note over Target: target command IS the container's PID 1 now,
stdio was wired straight through from the host shell
Target-->>Parent: process exits
Parent->>CG: rmdir / (Cleanup(), retries up to ~500ms)
Parent-->>Host: propagate the container's exit code
```
Two design choices worth calling out:
- **`syscall.Exec`, not a subprocess** — the child process replaces its own
image with the target command instead of spawning it as a child. This
means the container's PID 1 *is* `/bin/sh` (or whatever was requested),
with correct init/signal/reaping semantics, rather than a Go process that
has to babysit a subprocess.
- **cgroup limits are created before the container exists** — `memory.max`
and `cpu.max` don't require a PID to be written, so the cgroup is fully
configured first and the container's host PID is attached to it right
after the namespaced process starts, closing the window where an
unconstrained process could run free.
### Filesystem jailing: `pivot_root`, not `chroot`
`chroot` only changes what path resolution treats as `/`; the old root
filesystem is still mounted and reachable through tricks like relative
`..` traversal via an open file descriptor. `pivot_root` swaps the entire
mount tree's root and lets the old root be fully unmounted and deleted, so
there is nothing left to break out into:
```
Before pivot_root After pivot_root
(inside the new mount namespace) (inside the new mount namespace)
---------------------------------- ----------------------------------
/ / (== former rootfs,
├── proc/, sys/, dev/, ... ├── bin/, etc/, usr/, ... now the real root)
├── home/.../container-runtime/ ├── dev/ (fresh tmpfs: null, zero,
│ └── rootfs/ <- bind-mounted │ full, random, urandom,
│ onto itself so pivot_root(2) │ tty, fd -> /proc/self/fd)
│ accepts it as the new root ├── proc/ (freshly mounted here,
└── ...rest of the host filesystem... │ reflects the new PID ns)
└── .old_root unmounted (MNT_DETACH) and
os.RemoveAll'd -- gone
```
The sequence, from [`rootfs_linux.go`](internal/container/rootfs_linux.go):
make all mounts private+recursive (so `pivot_root` doesn't `EINVAL` on
distros with shared mount propagation) → bind-mount `rootfs` onto itself
(a mount-point requirement of `pivot_root(2)`) → build a minimal synthetic
`/dev` via tmpfs + hand-rolled `mknod` (never bind-mount the host's real
`/dev`) → `pivot_root` → `chdir("/")` → mount a fresh `/proc` (only valid
*after* the pivot, since it must reflect the new PID namespace) → unmount
and recursively delete `.old_root`.
## Requirements
Namespaces, `pivot_root`, and cgroups v2 are Linux kernel features and do
not exist on macOS/Windows. Building the CLI works anywhere (Go's build
tags isolate the Linux-only code), but actually **running** a container
requires Linux, root, and a cgroups v2 unified hierarchy mounted at
`/sys/fs/cgroup`.
### Dev/test environment (macOS)
A [Lima](https://lima-vm.io/) config is included for a disposable Ubuntu VM:
```
brew install lima
limactl start --name mini-run ./lima.yaml
limactl shell mini-run
cd container-runtime
go build -o mini-run ./cmd/mini-run
./scripts/fetch-rootfs.sh
sudo ./mini-run run --memory 128m --cpus 0.5 ./rootfs /bin/sh
```
(Ubuntu/systemd is used rather than Alpine/OpenRC because systemd keeps the
root cgroup free of directly-attached processes — otherwise enabling the
cpu/memory controllers on `/sys/fs/cgroup` fails with `EBUSY`.)
If you already have a Linux box, skip Lima and just `go build` +
`sudo ./mini-run run ...` there directly.
## Usage
```
sudo mini-run run [--memory 512m] [--cpus 0.5] [args...]
```
- `--memory` — e.g. `512m`, `1.5g` (default: unlimited)
- `--cpus` — fraction of one core, e.g. `0.5`, `2` (default: unlimited)
Flags must precede ``; everything after the command is passed
through untouched (so `sudo mini-run run ./rootfs /bin/sh -c "ls -la"` works
as expected).
## Verification
Run inside the Lima VM (or any Linux box):
1. **PID namespace**: `echo $$` inside the container prints `1`.
2. **UTS namespace**: `hostname` prints a generated `mini-run-xxxxxxxx` name,
distinct from the host's.
3. **Mount namespace**: `mount` inside the container shows only `/`,
`/proc`, and the synthetic `/dev` tmpfs — compare against a second
`limactl shell mini-run` window on the host.
4. **Process visibility**: `ps aux` inside the container shows only the
container's own processes, not the host's (`/proc` reflects the new PID
namespace).
5. **Breakout prevention**: `ls /.old_root` inside the container fails with
"No such file or directory"; `ls /` shows only the rootfs's own
directories, never host paths like `/home` or the project directory.
6. **cgroup limits applied**: from the host, `cat /sys/fs/cgroup/mini-run//memory.max`
and `.../cpu.max` match what was requested; `cgroup.procs` shows the real
host-namespace PID (vs. `1` seen inside the container).
7. **Memory limit enforced**: inside the container, run a memory bomb
(`sh -c 'a=AAAA; while true; do a="$a$a"; done'`) — it gets OOM-killed by
the cgroup, without affecting the host VM.
8. **CPU limit enforced**: `yes > /dev/null` inside the container while
watching host-side `top` — usage caps near the configured fraction of one
core.
9. **Cleanup**: after `exit` (or Ctrl-C mid-session), `/sys/fs/cgroup/mini-run/`
is gone.
10. **Exit code propagation**: `sudo ./mini-run run ./rootfs /bin/sh -c "exit 42"; echo $?`
prints `42`.
The portable pieces (flag parsing, ID generation, memory/CPU string
parsing) build and vet directly on macOS without Lima:
```
go build ./... && go vet ./...
```
## Testing
Unit tests cover the pure, cross-platform logic (ID generation, memory/CPU
parsing) and run anywhere, no Lima or root needed:
```
go test ./...
```
Everything else — namespace isolation, `pivot_root`, cgroup enforcement,
exit-code propagation, cleanup — is covered by an integration suite that
automates the 10 verification steps above. It's black-box (drives the built
binary as a real user would) and requires Linux, root, and a fetched rootfs,
so it's gated behind the `integration` build tag and skips itself with a
clear message if either is missing. Run it inside the Lima VM:
```
go build -o mini-run ./cmd/mini-run
./scripts/fetch-rootfs.sh
sudo go test -tags integration ./test/integration/... -v
```
## Benchmarking
`scripts/bench.sh` measures container startup latency and compares it
against a staircase of baselines to show what each isolation layer costs:
raw exec, `chroot` alone, `chroot` + namespaces (via `unshare`), and
mini-run itself (+ cgroups v2 setup/cleanup and `pivot_root`). It uses
[hyperfine](https://github.com/sharkdp/hyperfine) for statistically sound
timing and, if `matplotlib` is available, renders `bench-results.png` (via
`scripts/plot_bench.py`) as a labeled bar chart of the four variants. Run
inside the Lima VM (both `hyperfine` and `matplotlib` are preinstalled by
`lima.yaml`'s provisioning script):
```
sudo ./scripts/bench.sh
```
### Results
Measured on an aarch64 Lima VM (2 vCPUs, 4GiB RAM, Ubuntu 24.04) running on
a macOS host — absolute numbers will vary by machine, but the shape of the
staircase (what each layer adds) is the interesting part:
| Command | Mean | Min | Max | Slowdown vs. raw exec |
|:---|---:|---:|---:|---:|
| a: raw exec | 1.1 µs | 0.0 µs | 1027.1 µs | 1× |
| b: chroot only | 253.4 µs | 166.1 µs | 5011.2 µs | ~238× |
| c: chroot + namespaces | 519.8 µs | 452.4 µs | 1132.3 µs | ~488× |
| d: mini-run | 3.36 ms | 2.16 ms | 17.98 ms | ~3160× |
**Reading the staircase:**
- **(a) raw exec** — `true`, no isolation at all. Hyperfine's own warning
("command took less than 5 ms") is the tell that this number is really
measuring hyperfine's shell-invocation floor, not a meaningful process
cost — treat it as "approximately zero," the baseline everything else is
relative to, not a precise figure.
- **(a) → (b) chroot only, +252 µs** — this jump is *not* the cost of the
`chroot(2)` syscall itself (that's sub-microsecond). It's the cost of
spawning the external `chroot` binary as its own process before it execs
`/bin/true` — i.e. this measures "one extra fork+exec plus a `chroot()`
and `chdir()` call," which is representative of what a shell-script-based
jail would actually cost, not a syscall microbenchmark in isolation.
- **(b) → (c) + namespaces via `unshare`, +266 µs** — on top of another
process spawn (the `unshare` binary itself), the kernel now allocates and
populates a new PID, mount, UTS, and IPC namespace (`CLONE_NEWPID|
CLONE_NEWNS|CLONE_NEWUTS|CLONE_NEWIPC`). Namespace creation has real
kernel-side cost — the mount namespace in particular requires copying the
process's mount table — which is why this step costs meaningfully more
than the chroot-only step even though both are dominated by process-spawn
overhead.
- **(c) → (d) mini-run, +2.84 ms** — this is where mini-run's extra work
over the `unshare`+`chroot` equivalent shows up, and it's the sum of
several things the manual baseline doesn't do at all:
1. **Two full process launches, not one `clone(2)`** — the parent
(`mini-run run`) re-execs *itself* as `mini-run child` via
`os/exec.Command` rather than calling `clone(2)` directly in-process.
That's two separate ~2.8MB Go binary startups (runtime init, GC init,
etc.) chained together, versus `unshare`'s single `clone()`+`execve()`.
2. **cgroups v2 setup and teardown** — `mkdir`, two `subtree_control`
writes (root and `mini-run/`), `memory.max`/`cpu.max` writes, a
`cgroup.procs` attach, and on the way out, an `rmdir` that retries up
to 10×50ms if kernel accounting hasn't caught up yet. Each is a
synchronous sysfs write with kernel-side validation — `unshare`/`chroot`
do none of this.
3. **The full `pivot_root` sequence** — making mounts private+recursive,
a recursive bind-mount of the rootfs onto itself, a tmpfs mount plus
six `mknod` calls for `/dev`, `pivot_root(2)` itself, and finally
unmounting (`MNT_DETACH`) and recursively deleting the old root.
`chroot` is a single syscall by comparison; this is roughly a dozen.
4. A fresh `/proc` mount after the pivot (namespace-aware, so it can't
happen before).
- The wide variance (±1.86 ms, up to 18 ms) is consistent with this: it's
a long chain of syscalls in a nested-virtualization guest (Lima VM on
top of macOS), plus the cgroup cleanup retry loop occasionally kicking
in if a previous container's accounting hadn't settled yet.
The takeaway: **mini-run's overhead over hand-rolled namespaces is almost
entirely cgroups v2 bookkeeping and the `pivot_root`/`/dev` setup, not the
namespaces themselves** — namespace creation (b→c) is cheap relative to
what a real container runtime does around it (c→d). This roughly tracks
how runc and similar runtimes spend their startup budget too.
### Why this result is significant
The headline number — mini-run is ~3,000× slower than a bare `exec` — sounds
damning until it's put in context: it's 3.36 **milliseconds**. The point of
the staircase isn't "containers are slow," it's *decomposing* a number that
is usually reported as a single, opaque figure ("container startup takes N
ms") into the four architectural decisions that actually produce it. Once
decomposed, a more interesting fact emerges: **namespace creation itself
(b→c) is nearly free** — a few hundred microseconds — while the bulk of the
cost (c→d) comes from bookkeeping *around* the namespaces: cgroup sysfs
I/O, an extra process re-exec, and filesystem jail setup. That's a useful,
transferable insight about where container overhead actually lives, and
it's only visible because each layer was isolated and measured on its own
rather than benchmarked as one monolithic `docker run`.
It's also evidence that the implementation is doing what it claims to: if
`pivot_root`, cgroup limit enforcement, or namespace isolation were silently
no-ops, the latency staircase wouldn't exist — each step costs measurably
more than the last specifically because each one does real, verifiable
kernel work (independently confirmed by the [integration test suite](#testing)
above, which checks the *behavior*, not just the timing).
## References
**Primitives this project implements:**
- [`namespaces(7)`](https://man7.org/linux/man-pages/man7/namespaces.7.html),
[`pid_namespaces(7)`](https://man7.org/linux/man-pages/man7/pid_namespaces.7.html),
[`mount_namespaces(7)`](https://man7.org/linux/man-pages/man7/mount_namespaces.7.html) —
Linux manual pages for the isolation primitives behind `CLONE_NEWPID` /
`CLONE_NEWNS` / `CLONE_NEWUTS` / `CLONE_NEWIPC`.
- [`pivot_root(2)`](https://man7.org/linux/man-pages/man2/pivot_root.2.html) —
why it's used instead of `chroot(2)` for filesystem jailing.
- [cgroups v2 (kernel docs)](https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html) —
the unified hierarchy, `subtree_control`, `memory.max`, `cpu.max`.
- [OCI Runtime Specification](https://github.com/opencontainers/runtime-spec) —
the spec real container runtimes (runc, crun) implement; mini-run
deliberately implements a subset of the same underlying primitives
without the spec's config/bundle format.
- [runc](https://github.com/opencontainers/runc) — the reference OCI
runtime implementation; mini-run's `pivot_root` sequence follows the
same shape (private+recursive remount, self bind-mount, pivot, detach).
**Tooling used in this repo:**
- [Lima](https://lima-vm.io/) — the Linux VM used for macOS development
([`lima.yaml`](lima.yaml)).
- [hyperfine](https://github.com/sharkdp/hyperfine) — the benchmarking tool
used by [`scripts/bench.sh`](scripts/bench.sh).
- [Alpine Linux `minirootfs`](https://alpinelinux.org/downloads/) — the
rootfs fetched by [`scripts/fetch-rootfs.sh`](scripts/fetch-rootfs.sh).
## License
This project is licensed under the MIT License. See [LICENSE](LICENSE) for details.