{"id":13621625,"url":"https://github.com/estesp/bucketbench","last_synced_at":"2025-04-10T00:19:56.516Z","repository":{"id":17039856,"uuid":"80671886","full_name":"estesp/bucketbench","owner":"estesp","description":"Go-based framework for running benchmarks against Docker, containerd, runc, or any CRI-compliant runtime","archived":false,"fork":false,"pushed_at":"2024-08-01T17:18:42.000Z","size":16680,"stargazers_count":146,"open_issues_count":7,"forks_count":38,"subscribers_count":10,"default_branch":"main","last_synced_at":"2025-04-02T16:26:52.215Z","etag":null,"topics":["benchmark","container-engine","containerd","containers","cri","docker","golang","kubernetes","runc"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/estesp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-02-01T22:39:07.000Z","updated_at":"2025-03-27T13:28:01.000Z","dependencies_parsed_at":"2023-02-14T00:16:33.431Z","dependency_job_id":"39c4111a-58da-49df-95ca-5a802208f47b","html_url":"https://github.com/estesp/bucketbench","commit_stats":{"total_commits":136,"total_committers":13,"mean_commits":"10.461538461538462","dds":0.4411764705882353,"last_synced_commit":"033c84560eab6f18c973ad206e20eb846429bf7f"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/estesp%2Fbucketbench","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/estesp%2Fbucketbench/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/estesp%2Fbucketbench/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/estesp%2Fbucketbench/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/estesp","download_url":"https://codeload.github.com/estesp/bucketbench/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248131675,"owners_count":21052894,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark","container-engine","containerd","containers","cri","docker","golang","kubernetes","runc"],"created_at":"2024-08-01T21:01:08.800Z","updated_at":"2025-04-10T00:19:56.461Z","avatar_url":"https://github.com/estesp.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"# bucketbench\nBucketbench is a simple framework for running defined sequences\nof lifecycle container operations against three different container\nengines today: the full Docker engine, OCI's runc, and containerd.\n\nGiven a **bucket** is a physical type of container, the name is my attempt to\nget away from calling it \"dockerbench,\" given it runs against other\ncontainer engines as well. All attempts to come up with a more interesting\nname failed before initial release. Suggestions welcome!\n\n## Background\nThis project came about via some performance comparison work happening\nin the [OpenWhisk](https://openwhisk.org) serverless project. Developers\nin that project had a python script for doing similar comparisons, but\nagreed we should extend it to a more general framework which could be\neasily be extended for other lifecycle operation sequences, as the python\nscript was hardcoded to a specific set of operations.\n\n## Usage\nUsing `bucketbench` to drive container operations against a specific\ncontainer runtime requires a configuration file written in a specific YAML\nformat.\n\nThe current driver implementations each support a small set\nof lifecycle operations (defined as an interface in `driver/driver.go`), and\nany benchmark definition can mix and match any of those operations within\nreason. (Obviously operations must be ordered in a way supported by container\nlifecycle--for example, you can't do `stop` prior to `run`.)\n\nSpecific command usage for the `bucketbench` program is as follows:\n```\nThe YAML file provided via the --benchmark flag will determine which\nlifecycle container commands to run against which container runtimes, specifying\niterations and number of concurrent threads. Results will be displayed afterwards.\n\nUsage:\n  bucketbench run [flags]\n\nFlags:\n  -b, --benchmark string   YAML file with benchmark definition\n  -h, --help               help for run\n  -l, --legacy             legacy mode will run benchmark from 1 to N(thread number) iterations.\n  -o, --overhead           Output daemon overhead\n  -s, --skip-limit         Skip 'limit' benchmark run\n  -t, --trace              Enable per-container tracing during benchmark runs\n\nGlobal Flags:\n      --log-level string   set the logging level (info,warn,err,debug) (default \"warn\")\n```\n\nA common invocation for running the \"basic\" example benchmark might look like:\n\n```\n$ sudo ./bucketbench --log-level=debug run -b examples/basic.yaml\n```\n\nLet's look at the input YAML file format and define the components. Here's\nthe **basic.yaml** example:\n\n```\nname: Basic\nimage: alpine:latest\ncommand: date\nrootfs: /home/estesp/containers/alpine\ndetached: true\ndrivers:\n  - \n   type: Docker\n   threads: 5\n   iterations: 15\n  - \n   type: Runc\n   threads: 5\n   iterations: 50\ncommands:\n  - run\n  - stop\n  - remove\n```\n\nThe initial section sets up a name and a few key pieces of information required\nfor each engine to know **what** to run:\n - **name**: Give the benchmark a name. This will be used in output and logs.\n - **image**: Choose an image reference to be used by the image-based engine runtimes (containerd 1.0 and Docker). This can be any image reference accepted by the `docker pull` command. `bucketbench` will handle reconciling this reference to the format used by containerd 1.0 (e.g. `alpine` -\u003e `docker.io/library/alpine:latest`)\n - **command**: *[Optional]* Specify an override for the image's default command that will be used for the image-based engine runtimes.\n - **rootfs**: For the `runc`, `crun`, `youki` and `ctr` (legacy containerd/0.2.x) drivers, you will need to provide an exploded rootfs and an OCI `config.json` since neither of those engines support image/registry interactions.\n - **detached**: Run the containers in detached/background mode.\n\nThe next two sections of the YAML provide 1) the configuration of which drivers\nto execute the benchmark against, and 2) which lifecycle commands to run\nagainst each engine.\n\n#### Driver Configuration\n\nEach driver has the following settings:\n - **type**: One of the four implemented drivers: `Runc`, `Docker`, `Containerd`, `Ctr`, `Youki`, `CRun`\n - **clientpath**: *[Optional]* Path to the binary for client executable based drivers. In the case of containerd 1.0 and the CRI driver, this will be the unique UNIX socket path of the gRPC server. For client binary-based drivers, the driver will default to the standard binary name found in the current `$PATH`\n - **threads**: Integer number of concurrent threads to run. If `--legacy` is set, the `bucketbench` method is to execute 1..n runs, where `n` is the number of threads and each run adds another concurrent thread. **Run 1** only has one thread and **Run N** will have `n` concurrent threads. If `--legacy` is not set, which is default mode, the `bucketbench` method is to execute just one run with the number of threads user specified. You will only see **Run N** which has `n` concurrent threads.\n - **iterations**: Number of containers to create in each thread and execute the listed commands against.\n - **logDriver**: `Docker` and `DockerCLI` support log driver configuration to measure overhead between different implementations. Allowed values can be found [here](https://docs.docker.com/config/containers/logging/configure/#supported-logging-drivers).\n - **logOpts**: Logger driver configuration, only used with `logDriver` option. See `overhead-logdriver.yaml` for examples.\n - **streamStats**: Allows to explore the overhead of `stats` queries for different drivers. Note that `docker` driver supports streaming natively while `containerd` supports direct API calls only, so you might want to send multiple queries to emulate streaming behavior (see **statsIntervalSec**)\n - **statsIntervalSec**: Defines an interval in seconds between `stats` queries to emulate streaming behaviour for `containerd`\n - **cgroupPath**: Path to a cgroup directory (for example: `/system.slice/docker.service`)\n\n#### Command List\n\nFinally, the YAML input needs to have a list of container lifecycle commands.\nThe following commands are accepted as input:\n\n - **run**: (aliases: **start**) create and start a container.\n - **pause**: pause a running container\n - **unpause**: (aliases: **resume**) resume a paused container\n - **stop**: (aliases: **kill**) stop/kill the running container processes\n - **remove**: (aliases: **erase**,**delete**) remove/delete a container instance\n - **metrics**: (aliases: **stats**) query container daemon stats. Note: if `streamStats = true`, each **metrics** command will spawn separate goroutine and will stream metrics until end of iteration.\n - **wait**: wait for container stop\n\nNote that `bucketbench` is not handling any formal state validation on the list\nof commands. It is currently up to the user to provide a valid/sane ordered\nlist of container lifecycle commands. The container runtimes will error out on\nincorrect command states (e.g. `stop` before `run`).\n\nAfter the benchmark runs are complete, `bucketbench` currently provides basic\noutput to show the overall rate (iterations of the operations/second) for each\nof the thread counts:\n\n```\n             Iter/Thd     1 thrd  2 thrds  3 thrds  4 thrds  5 thrds  6 thrds  7 thrds  8 thrds  9 thrds 10 thrds\nLimit            1000    1171.24  1957.17  2101.13  2067.83  1827.92  1637.32  1257.57  1582.36  1306.08  1699.56\nBasic:Docker       15       1.40     2.21     2.81\nBasic:Runc         50       8.38    15.85    23.00\n```\n\nIf you add `overhead` flag, `bucketbench` will measure container daemon cpu\nand memory consumption. The output will look like:\n\n```\n    Bench / driver / threads       Min       Max       Avg       Min       Max       Avg     Mem %     CPU x\n  OverheadBench:Containerd:1     40 MB     42 MB     41 MB    0.00 %    6.00 %    0.32 %\n  OverheadBench:Containerd:2     44 MB     46 MB     44 MB    0.00 %   10.00 %    0.57 %\n  OverheadBench:Containerd:3     46 MB     46 MB     46 MB    0.00 %   14.00 %    0.73 %\n  OverheadBench:Containerd:4     46 MB     47 MB     46 MB    0.00 %   20.00 %    0.94 %\n\n      OverheadBench:Docker:1     64 MB     66 MB     64 MB    0.00 %   10.00 %    0.58 %   +56.10%    +1.84x\n      OverheadBench:Docker:2     69 MB     73 MB     70 MB    0.00 %   20.00 %    1.29 %   +59.09%    +2.26x\n      OverheadBench:Docker:3     73 MB     73 MB     73 MB    0.00 %   32.00 %    1.97 %   +58.70%    +2.70x\n      OverheadBench:Docker:4     73 MB     73 MB     73 MB    0.00 %   27.99 %    2.67 %   +58.70%    +2.85x\n```\n\nMore detailed information is collected during the runs and a future PR to\n`bucketbench` will provide the raw performance data in a consumable format for\nend users.\n\nTo run `bucketbench` against `Runc`, `Containerd`, `CRun`, `Youki` or the legacy `Ctr` driver\nyou must use `sudo` because of the requirements that those tools have for root\naccess. This tool does not manage the two daemon-based engines (containerd and\ndockerd), and will fail if they are not up and running when the benchmark runs\nbegin.\n\nThe tool will start a significant number of containers against these daemons,\nbut attempts to fully cleanup after running each iteration.\n\n## Development Notes\n\nThe `bucketbench` tool is most likely only valuable on amd64/linux, as\n`containerd` and `runc` are delivered today as binaries for those platforms.\nIt will most likely build for other platforms, and if run against a tool like\nDocker for Mac, would probably work against the Docker engine, but not\nagainst `containerd` or `runc`.\n\nAll the necessary dependencies are vendored into the `bucketbench` source tree.\nBuilding `bucketbench` only requires that you have a valid Golang build/runtime\nenvironment. Any recent release of Go will work, but it is currently building\nwith Go 1.9.x and 1.10. A simple `Makefile` is available to simplify building\nbucketbench as a dynamic or static binary. `make binary` will build the\n`bucketbench` binary and `make install` will place it in your `$PATH`. You\nshould also be able to simply `go install github.com/estesp/bucketbench`.\n\n`bucketbench` offers [cgroups](http://man7.org/linux/man-pages/man7/cgroups.7.html)\nas more precise way of measuring resource usage. However some additional setup\nis required before running tests. `bucketbench` uses existing environment, so\na control group should be created for each container runtime and daemons should be\nadded to a corresponding cgroup (if systemd is used, cgroups are already created).\nFor each container runtime a path to cgroup should be passed via ``cgroupPath`` field.\n\n## Caveats and limitations\n\n- Overhead benchmark implementation only covers `Docker` and `Containerd`\n- Stats streaming are only supported by `Docker`, `DockerCLI` and `Containerd` drivers\n- Cgroups are Linux only\n- The benchmark uses process name matching to find relevant processes; you must \nkeep the expected process names (`dockerd`, `docker-containerd`, and `docker-containerd-shim`\nfor Docker and `containerd` and `containerd-shim` for containerd) and not run \nadditional processes with the same names.\n\n\n## TODOs\n\n - Decide what to do with the `-trace` flag, which was only useful with a private build of `runc` which generated Go pprof traces. Possibly submit trace support to upstream runc.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Festesp%2Fbucketbench","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Festesp%2Fbucketbench","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Festesp%2Fbucketbench/lists"}