{"id":18555332,"url":"https://github.com/c3d/kata-performance-measurements","last_synced_at":"2026-02-17T10:05:00.666Z","repository":{"id":146252483,"uuid":"523676044","full_name":"c3d/kata-performance-measurements","owner":"c3d","description":"Some memory usage performance measurements done on Kata Containers","archived":false,"fork":false,"pushed_at":"2025-01-30T14:49:46.000Z","size":4591,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"public","last_synced_at":"2025-01-30T15:37:06.550Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/c3d.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-08-11T10:15:14.000Z","updated_at":"2025-01-30T14:49:49.000Z","dependencies_parsed_at":null,"dependency_job_id":"fb5f5f1a-aea2-40af-8f36-66abdfc6903b","html_url":"https://github.com/c3d/kata-performance-measurements","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/c3d%2Fkata-performance-measurements","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/c3d%2Fkata-performance-measurements/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/c3d%2Fkata-performance-measurements/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/c3d%2Fkata-performance-measurements/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/c3d","download_url":"https://codeload.github.com/c3d/kata-performance-measurements/tar.gz/refs/heads/public","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239279369,"owners_count":19612482,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T21:26:07.884Z","updated_at":"2025-11-01T10:30:29.041Z","avatar_url":"https://github.com/c3d.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Kata performance measurements\n\nThis is a series of measurements evaluating the performance and overhead of Kata\nContainers, notably with respect to CPU and memory utilization and overhead.\n\n\n### Summary\n\nThe following measurements have been performed:\n\n1. [Memory capacity](memory-capacity.md): Determine the amount of memory that can\n   be used for workloads, by incrementally adding 500M workloads to the system.\n\n   * Objective: Estimate \"memory overhead\" by checking how many calibrated\n     containers we can run on a reasonably-sized cluster.\n\n   * Result: `runc` maxes out at around 377 containers (reaching saturation\n     around the predicted 160), whereas `kata` maxes out at 134 only. The memory\n     usage per container with `runc` is between 200M and 600M per container,\n     with `kata` it is between 600M and 1200M (in both case with 500M used by\n     the workload).\n\n2. [Memory-hungry workload](memory-hungry.md): Running memory-hungry\n   workloads that try to grab as much memory as possible from the system.\n\n   * Objective: Check the effect of running the workload in a fixed-sized\n     pod/VM (each VM having a default 2G memory allocation)\n\n   * Result: In this context, `kata` seems unable to use half of the available\n     memory.The total memory utlization never exceeeds ~35G out of 80G. This\n     result is unexpected, and requires investigation. Each container receives\n     less memory (200M-400M for `kata`, 600M for `runc`). In that case, both\n     `runc` and `kata` both saturate at 102-103 containers.\n\n3. [Memory-restricted workload](memory-restricted.md): Running a\n   workload with a memory request below what it will use, and a memory limit\n   just above.\n\n   * Objective: Check the impact of memory hot-plugging in `kata`, and how well\n     memory constraints are obeyed.\n\n   * Result: In this case as well, `kata` ends up being able to use far less\n     memory than predicted. Total memory utilization never exceeds 30G out of 80G.\n     Containers also receive much less memory than with `runc` (between 200M and\n     400M vs. 600M with `runc`). In this case, `kata` seems to be able to run a\n     slightly higher number of containers (102 vs 92).\n\n\n## Methodology\n\n1. In order to minimize the measurement noise, I created a very simple\n   [workload][workload] that consumes a configurable amount of CPU and memory.\n\n2. This workload is put [in a container][workloadc] so that it is easily\n   consumed by Kubernetes / OpenShift.\n\n3. [Deployments][deploy] with variations on the parameters can then test how the\n   orchestration deals with increasing pressure, or scheduling behavior.\n\n4. [Increase some of the parameters][scale] continuously, until some\n   cluster-wide limit is reached. Generally, the cluster becomes barely\n   responsive at that stage, which makes the limit fuzzy.\n\n5. Collect the data in CSV files and present a [graphic visualization][graphs] of\n   the data and a text analysis of the data.\n\n\n[workload]: https://github.com/c3d/workload\n[workloadc]: https://quay.io/c3d/workload\n[deploy]: workload.yaml\n[scale]: scale.sh\n[graphs]: visualize.ddd\n\n\n### Configuration used for the tests described in this document\n\nThe tests are run on a system with 64 physical CPUs (PCPUs) and 384MB of memory.\nA cluster is running on that system, which consists of the following nodes:\n\n1. A provision host with 8 virtual CPUs (VCPUs) and 16G of memory.\n\n2. Three master nodes with 16 VCPUs and 32G of memory each\n\n3. Five worker nodes with 8 VCPUs and 16G of memory each.\n\nThe total of memory allocated to VMs is therefore 192G, so there will be no\nmemory oversubscription.\n\nOf this total amount, 80G (5*16) can in theory be allocated to workloads. In\npractice, experience shows that we get significantly less usable memory.\n\nThe tests were performed using a pre-release version of OpenShift sandboxed\ncontainers, using rebuilt code corresponding to various 2.0.x releases of Kata\nContainers ranging from 2.0.1 to 2.1.1. There was no significant variation\nobserved between releases, except where the measurements exposed issues that\nwere then fixed, in which case the numbers presented here are after the fix.\n\n\n### Curves\n\nThe memory plots made with [Tao3D][tao3d] show\n\n* Active memory in red, the larger dots being the sum of all nodes, and the\n  small dots being the free memory for each individual node\n\n* Free memory for the nodes in green, with the same conventions.\n\n[tao3d]: https://tao3d.sourceforge.net\n\n\n### Evolution of the test plan\n\nThe original idea of the test plan was to run an `oc scale` command with an\nincreasing number of replicas, and at each step, to measure how long it took to\nreach steady state.\n\nExperimentally, this approach turned out to not work very well, because steady\nstate was never really achieved once the cluster reached saturation.\nSince the objective of the measurements is to determine the behavior of the\ncluster at memory and CPU saturation, it is difficult to reach solid conclusion\nif the time to get there increases asymptotically and we never see what is\n\"beyond\" saturation.\n\nSo the test script was modified to _not_ wait and instead keep increasing the\nload at fixed intervals.\n\n#### Runs with wait\n\nThe [configuration in that scenario](workload-runc.yaml) does not set the\n`RuntimeClass` to `kata`, so we use the defualt runtime. Given that we have\nabout 80G of available memory and that each workload is tuned for 500M, we\nexpect to be able to run at most 160 instances. In practice, things start going\nwrong between 100 and 120, and the system becomes quite unresponsive, e.g. with\n`ssh` disconnects (the script had to be adjusted to take that into account).\n\nThe memory utilization shows a very linear behavior up to about 100\nloads. Several runs were made, waiting for the number of containers to\nstabilize. With this approach, it was not possible to exceed ~123 containers,\nthe system thrashing a lot, but the script was not collecting data.\n\n![Memory utilization](runc-memory.png)\n\nThe number of containers increases linearly quite well until some limit is\nreached and memory cannot be allocated anymore.\n\n![Number of containers](runc-state.png)\n\nThe timing shows some random spikes, and brutally spikes up to the point where\nthe experiment had to be stopped.\n\n![Timing](runc-timing.png)\n\n\n#### Fixed-wait runs\n\nThe approach where the script was actually waiting for the requests to be\ncompleted does not work correctly after reaching saturation. So the script was\nmodified to instead wait a fixed amount of time, which will better capture the\ndynamic behavior of the system. The benefit of that approach is that it allows\nruns that have a somewhat fixed duration, and it makes it possible to request\nscaling well above the maximum the system can tolerate.\n\nWith these changes, the results are as follows:\n\n![Memory utilization](runc-fixed-memory.png)\n\nThe number of containers increases linearly quite well until some limit is\nreached and memory cannot be allocated anymore.\n\n![Number of containers](runc-fixed-state.png)\n\nThe timing shows some random spikes, and brutally spikes up to the point where\nthe experiment had to be stopped.\n\n![Timing](runc-fixed-timing.png)\n\n#### No-wait runs\n\nThe delay in the script really serves no real purpose - There are plenty of\nother delays just comminicating with OpenShift (the `oc` command is not\nespeciallty fast, and `ssh` gets worse with load). So removed it and did another\n\"fast\" run. Also changed the method to gather memory stats from the various\nworkers: keep one `ssh` connexion per host instead of reopening it every\ntime. The risk is if that connexion gets killed, but worst case is we get\n\"constant\" memory data for that node.\n\nThis leads to the following results:\n\n![Memory utilization](runc-fast-memory.png)\n\nSurprisingly, the number of running containers seems to grow well beyond what\nwould be predicted from basic memory analysis, with a plateau around 400 rather\nthan the predicted 160. Maybe swap is activated with OpenShift (unlike\nKubernetes)? Looking at `top`, apparently not. So maybe it's killing containers\n(hence the high number of evicted) to make room for more, and takes advantage of\nthe fact that `workload` is intended to grow memory relatively slowly?\n\n\n![Number of containers](runc-fast-state.png)\n\nThe timing with this new approach shows good proportionality, with an almost\nperfectly linerar duration for operations.\n\n![Timing](runc-fast-timing.png)\n\nAt the end, the host has a load average of about 50, and CPU utilization is well\nbelow 100% of all 64 PCPUs.\n\nThis last approach is what was retained for the following runs.\n\n\n## Test results\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fc3d%2Fkata-performance-measurements","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fc3d%2Fkata-performance-measurements","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fc3d%2Fkata-performance-measurements/lists"}