Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/Mic92/cntr
A container debugging tool based on FUSE
https://github.com/Mic92/cntr
build-with-buildbot docker lxc managed-by-renovate rkt systemd-nspawn
Last synced: 3 months ago
JSON representation
A container debugging tool based on FUSE
- Host: GitHub
- URL: https://github.com/Mic92/cntr
- Owner: Mic92
- License: mit
- Created: 2016-06-01T21:57:42.000Z (over 8 years ago)
- Default Branch: master
- Last Pushed: 2024-04-30T09:17:22.000Z (6 months ago)
- Last Synced: 2024-05-02T00:15:55.256Z (6 months ago)
- Topics: build-with-buildbot, docker, lxc, managed-by-renovate, rkt, systemd-nspawn
- Language: Rust
- Homepage:
- Size: 579 KB
- Stars: 638
- Watchers: 9
- Forks: 23
- Open Issues: 32
-
Metadata Files:
- Readme: README.md
- License: LICENSE.md
Awesome Lists containing this project
README
# cntr
Say no to `$ apt install vim` in containers!
`cntr` is a replacement for `docker exec` that brings all your developers tools with you.
This is done by mounting the file system from one container or the host into the target container
by creating a nested container with the help of a FUSE filesystem.
This allows to ship minimal runtime image in production and limit the surface for exploits.
Cntr was also published in [Usenix ATC 2018](https://www.usenix.org/conference/atc18/presentation/thalheim).
See [bibtex](#bibtex) for citation.
## Demo
In this two minute recording you learn all the basics of cntr:
[](https://asciinema.org/a/PQB5RnRGLIPn88a1R2MtPccdy)
## Features
- For convenience cntr supports container names/identifier for the following container engines natively:
* docker
* podman
* LXC
* LXD
* rkt
* systemd-nspawn
* containerd
- For other container engines cntr also takes process ids (PIDs) instead of container names.
## Installation
Cntr can be only supports linux.
### Pre-build static-linked binary
For linux x86_64 we build static binaries for every release. More platforms can added on request.
See the [release tab](https://github.com/Mic92/cntr/releases/) for pre-build tarballs.
At runtime only commandline utils of the container engine in questions are required.
### Build from source
All you need for compilation is rust + cargo.
Checkout [rustup.rs](https://rustup.rs/) on how to get a working rust toolchain.
Then run:
Either:
```console
$ cargo install cntr
```
Or the latest master:
```console
$ cargo install --git https://github.com/Mic92/cntr
```
For offline builds we also provided a tarball with all dependencies bundled
[here](https://github.com/Mic92/cntr/releases) for compilation with
[cargo-vendor](https://github.com/alexcrichton/cargo-vendor).
## Usage
At a high-level cntr provides two subcommands: `attach` and `exec`:
- `attach`: Allows you to attach to a container with your own native shell/commands.
Cntr will mount the container at `/var/lib/cntr`.
The container itself will run unaffected as the mount changes are not visible to container processes.
- Example: `cntr attach ` where `container_id` can be a
container identifier or process id (see examples below).
- `exec`: Once you are in the container, you can also run commands from the
container filesystem itself. Since those might need their native mount layout
at `/` instead of `/var/lib/cntr`, cntr provides `exec` subcommand to chroot to container
again and also resets the environment variables that might have been changed
by the shell.
- Example: `cntr exec ` where `command` is an executable in the container
**Note**: Cntr needs to run on the same host as the container. It does not work
if the container is running in a virtual machine while cntr is running on the
hypervisor.
```console
$ cntr --help
Cntr 1.5.1
Jörg Thalheim
Enter or executed in container
USAGE:
cntr
FLAGS:
-h, --help Prints help information
-V, --version Prints version information
SUBCOMMANDS:
attach Enter container
exec Execute command in container filesystem
help Prints this message or the help of the given subcommand(s)
```
```console
$ cntr attach --help
cntr-attach 1.5.1
Jörg Thalheim
Enter container
USAGE:
cntr attach [OPTIONS] [command]...
FLAGS:
-h, --help Prints help information
OPTIONS:
--effective-user effective username that should be owner of new created files on the host
-t, --type Container types to try (sperated by ','). [default: all but command]
[possible values: process_id, rkt, podman, docker, nspawn, lxc, lxd,
containerd, command]
ARGS:
container id, container name or process id
... Command and its arguments to execute after attach. Consider prepending it with '-- ' to prevent
parsing of '-x'-like flags. [default: $SHELL]
```
```console
$ cntr exec --help
cntr-exec 1.5.1
Jörg Thalheim
Execute command in container filesystem
USAGE:
cntr exec [command]...
FLAGS:
-h, --help Prints help information
-V, --version Prints version information
ARGS:
... Command and its arguments to execute after attach. Consider prepending it with '-- ' to prevent
parsing of '-x'-like flags. [default: $SHELL]
```
### Docker
1: Find out the container name/container id:
```console
$ docker run --name boxbusy -ti busybox
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
55a93d71b53b busybox "sh" 22 seconds ago Up 20 seconds boxbusy
```
Either provide a container id...
```console
$ cntr attach 55a93d71b53b
[root@55a93d71b53b:/var/lib/cntr]# echo "I am in a container!"
[root@55a93d71b53b:/var/lib/cntr]# ip addr
1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
40: eth0@if41: mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
valid_lft forever preferred_lft forever
[root@55a93d71b53b:/var/lib/cntr]# vim etc/resolv.conf
```
...or the container name.
Use `cntr exec` to execute container native commands (while running in the cntr shell).
```console
$ cntr attach boxbusy
[root@55a93d71b53b:/var/lib/cntr]# cntr exec -- sh -c 'busybox | head -1'
```
You can also use Dockerfile from this repo to build a docker container with cntr:
``` console
$ docker build -f Dockerfile . -t cntr
# boxbusy here is the name of the target container to attach to
$ docker run --pid=host --privileged=true -v /var/run/docker.sock:/var/run/docker.sock -ti --rm cntr attach boxbusy /bin/sh
```
### Podman
See docker usage, just replace `docker` with the `podman` command.
### LXD
1: Create a container and start it
```console
$ lxc image import images:/alpine/edge
$ lxc launch images:alpine/edge
$ lxc list
+-----------------+---------+------+------+------------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+-----------------+---------+------+------+------------+-----------+
| amazed-sailfish | RUNNING | | | PERSISTENT | 0 |
+-----------------+---------+------+------+------------+-----------+
```
2: Attach to the container with cntr
```console
$ cntr attach amazed-sailfish
$ cat etc/hostname
amazed-sailfish
```
### LXC
1: Create a container and start it
```console
$ lxc-create --name ubuntu -t download -- -d ubuntu -r xenial -a amd64
$ lxc-start --name ubuntu -F
...
Ubuntu 16.04.4 LTS ubuntu console
ubuntu login:
$ lxc-ls
ubuntu
```
2: Attach to container with cntr:
```console
$ cntr attach ubuntu
[root@ubuntu2:/var/lib/cntr]# cat etc/os-release
NAME="Ubuntu"
VERSION="16.04.4 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.4 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial
```
### rkt
1: Find out the container uuid:
```console
$ rkt run --interactive=true docker://busybox
$ rkt list
UUID APP IMAGE NAME STATE CREATED STARTED NETWORKS
c2d2e87e busybox registry-1.docker.io/library/busybox:latest running 6 minutes ago 6 minutes ago default:ip4=172.16.28.3
```
2: Attach with cntr
```console
# make sure your container is still running!
$ cntr attach c2d2e87e
# Finally not the old ugly top!
[gen0@rkt-c2d2e87e-e798-4341-ae93-26f6cbb7c017:/var/lib/cntr]# htop
...
```
With cntr you can also debug stage1 of rkt - even there is no support from rkt itself.
```console
$ ps aux | grep stage1
joerg 13546 0.0 0.0 120808 1608 pts/12 S+ 11:10 0:00 grep --binary-files=without-match --directories=skip --color=auto stage1
root 22232 0.0 0.0 54208 2656 pts/7 S+ 10:54 0:00 stage1/rootfs/usr/lib/ld-linux-x86-64.so.2 stage1/rootfs/usr/bin/systemd-nspawn --boot --notify-ready=yes --register=true --link-journal=try-guest --quiet --uuid=c2d2e87e-e798-4341-ae93-26f6cbb7c017 --machine=rkt-c2d2e87e-e798-4341-ae93-26f6cbb7c017 --directory=stage1/rootfs --capability=CAP_AUDIT_WRITE,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FSETID,CAP_FOWNER,CAP_KILL,CAP_MKNOD,CAP_NET_RAW,CAP_NET_BIND_SERVICE,CAP_SETUID,CAP_SETGID,CAP_SETPCAP,CAP_SETFCAP,CAP_SYS_CHROOT -- --default-standard-output=tty --log-target=null --show-status=0
```
Therefore we use the process id instead of the container uuid:
```console
$ cntr attach 22232
# new and exiting territory!
[root@turingmachine:/var/lib/cntr]# mount | grep pods
sysfs on /var/lib/cntr/var/lib/rkt/pods/run/c2d2e87e-e798-4341-ae93-26f6cbb7c017/stage1/rootfs/sys type sysfs (ro,nosuid,nodev,noexec,relatime)
tmpfs on /var/lib/cntr/var/lib/rkt/pods/run/c2d2e87e-e798-4341-ae93-26f6cbb7c017/stage1/rootfs/sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /var/lib/cntr/var/lib/rkt/pods/run/c2d2e87e-e798-4341-ae93-26f6cbb7c017/stage1/rootfs/sys/fs/cgroup/memory type cgroup (ro,nosuid,nodev,noexec,relatime,memory)
```
### systemd-nspawn
1: Start container
```console
$ wget https://cloud-images.ubuntu.com/releases/16.04/release/ubuntu-16.04-server-cloudimg-amd64-root.tar.xz
$ mkdir /var/lib/machines/ubuntu
$ tar -xf ubuntu-16.04-server-cloudimg-amd64-root.tar.xz -C /var/lib/machines/ubuntu
$ systemd-nspawn -b -M ubuntu
$ machinectl list
MACHINE CLASS SERVICE OS VERSION ADDRESSES
ubuntu container systemd-nspawn ubuntu 16.04 -
```
2: Attach
```console
$ cntr attach ubuntu
```
### Generic process id
The minimal information needed by cntr is the process id of a container process you want to attach to.
```console
# Did you now chromium uses namespaces too?
$ ps aux | grep 'chromium --type=renderer'
joerg 17498 11.7 1.0 1394504 174256 ? Sl 15:16 0:08 /usr/bin/chromium
```
In this case 17498 is the pid we are looking for.
```console
$ cntr attach 17498
# looks quite similar to our system, but with less users
[joerg@turingmachine cntr]$ ls -la /
total 240
drwxr-xr-x 23 nobody nogroup 23 Mar 13 15:05 .
drwxr-xr-x 23 nobody nogroup 23 Mar 13 15:05 ..
drwxr-xr-x 2 nobody nogroup 3 Mar 13 15:14 bin
drwxr-xr-x 4 nobody nogroup 16384 Jan 1 1970 boot
drwxr-xr-x 24 nobody nogroup 4120 Mar 13 14:56 dev
drwxr-xr-x 52 nobody nogroup 125 Mar 13 15:14 etc
drwxr-xr-x 3 nobody nogroup 3 Jan 8 16:17 home
drwxr-xr-x 8 nobody nogroup 8 Feb 9 22:10 mnt
dr-xr-xr-x 306 nobody nogroup 0 Mar 13 09:38 proc
drwx------ 22 nobody nogroup 43 Mar 13 15:09 root
...
```
### Containerd
For containerd integration the `ctr` binary is required. You can get a binary by running:
``` console
$ GOPATH=$(mktemp -d)
$ go get github.com/containerd/containerd/cmd/ctr
$ $GOPATH/bin/ctr --help
```
Put the resulting `ctr` binary in your `$PATH`
1: Start container
```console
$ ctr images pull docker.io/library/busybox:latest
$ ctr run docker.io/library/busybox:latest boxbusy
$ ctr tasks lists
TASK PID STATUS
boxbusy 24310 RUNNING
```
2: Attach
```console
$ cntr attach boxbusy
```
It's also possible to run cntr from a container itself.
This repository contains a example Dockerfile for that:
```console
$ docker build -f Dockerfile.example . -t cntr
$ docker save cntr > cntr.tar
$ ctr images import --base-name cntr ./cntr.tar
```
In this example we attach to containerd by process id. The process id of a task is given in `ctr tasks list`.
```console
$ ctr run --privileged --with-ns pid:/proc/1/ns/pid --tty docker.io/library/cntr:latest cntr /usr/bin/cntr attach 31523 /bin/sh
```
To resolve containerd names one also would need to add the `ctr` binary (~12mb) to the Dockerfile.
## Additional Config
### ZFS
`cntr` requires POSIX ACLs be enabled under ZFS. By default, Linux ZFS doesn't have POSIX ACLs enabled. This results in
the following error when trying to `attach`:
```console
unable to move container mounts to new mountpoint: EOPNOTSUPP: Operation not supported on transport endpoint
```
To enable POSIX ACLs on the ZFS dataset:
```console
$ zfs set acltype=posixacl zpool/media
$ zfs set xattr=sa zpool/media # optional, but encouraged for best performance
```
# How it works
Cntr is container-agnostic: Instead of interfacing with container engines, it
implements the underlying operating system API. It treats every container as a
group of processes, that it can inherit properties from.
Cntr inherits the following container properties:
* Namespaces (mount, uts, pid, net, cgroup, ipc)
* Cgroups
* Apparamor/selinux
* Capabilities
* User/group ids
* Environment variables
* The following files: /etc/passwd, /etc/hostname, /etc/hosts, /etc/resolv.conf
Under the hood it spawns a shell or user defined program that inherits the full
context of the container and mount itself as a fuse filesystem.
We extensively evaluated the correctness and performance of cntr's filesystem
using [xfstests](https://github.com/Mic92/xfstests-cntr) and a wide range of
filesystem performance benchmarks (iozone, pgbench, dbench, fio, fs-mark,
postmark, ...)
# Related projects
- [nsenter](https://manpages.debian.org/testing/manpages-de/nsenter.1.de.html)
- Only covers linux namespaces and the user is limited to tools installed in the
containers
- [toolbox](https://github.com/coreos/toolbox)
- Does attach from a container to the host, this is the opposite of what Cntr
is doing
# Bibtex
We published a paper with all technical details about Cntr in
[Usenix ATC 2018](https://www.usenix.org/conference/atc18/presentation/thalheim).
```bibtex
@inproceedings{cntr-atc18,
author = {J{\"o}rg Thalheim and Pramod Bhatotia and Pedro Fonseca and Baris Kasikci},
title = {Cntr: Lightweight {OS} Containers},
booktitle = {2018 {USENIX} Annual Technical Conference ({USENIX} {ATC} 18)},
year = {2018},
}
```