{"id":22220865,"url":"https://github.com/nokia/cpu-pooler","last_synced_at":"2025-07-27T15:33:27.514Z","repository":{"id":38298295,"uuid":"155571980","full_name":"nokia/CPU-Pooler","owner":"nokia","description":"A Device Plugin for Kubernetes, which exposes the CPU cores as consumable Devices to the Kubernetes scheduler. ","archived":false,"fork":false,"pushed_at":"2024-04-19T12:17:09.000Z","size":498,"stargazers_count":91,"open_issues_count":6,"forks_count":22,"subscribers_count":17,"default_branch":"master","last_synced_at":"2024-06-20T16:39:15.566Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nokia.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2018-10-31T14:34:02.000Z","updated_at":"2024-05-27T07:31:46.000Z","dependencies_parsed_at":"2024-04-19T13:36:35.943Z","dependency_job_id":"8d898051-c18e-4df5-8a3b-c14315cfccc4","html_url":"https://github.com/nokia/CPU-Pooler","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nokia%2FCPU-Pooler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nokia%2FCPU-Pooler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nokia%2FCPU-Pooler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nokia%2FCPU-Pooler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nokia","download_url":"https://codeload.github.com/nokia/CPU-Pooler/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":227817162,"owners_count":17824199,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-02T23:10:41.292Z","updated_at":"2024-12-02T23:10:41.906Z","avatar_url":"https://github.com/nokia.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CPU Pooler for Kubernetes\n\n[![Build Status](https://travis-ci.org/nokia/CPU-Pooler.svg?branch=master)](https://travis-ci.org/nokia/CPU-Pooler)\n\n## Overview\nCPU-Pooler for Kubernetes is a solution for Kubernetes to manage predefined, distinct CPU pools of Kubernetes Nodes, and physically separate the CPU resources of the containers connecting to the various pools.\n\nTwo explicit types of CPU pools are supported; exclusive and shared. If a container does not explicitly ask resources from these pools, it will belong to a third one, called default.\n\nRequest from exclusive pool will allocate CPU(s) for latency-sensitive containers requiring exclusive access.\n\nContainers content with some, limited level of sharing, but still appreciating best-effort protection from noisy neighbours can request resources from the shared pool. Shared requests are fractional CPUs with one CPU unit exactly matching to one thousandth of a CPU core. (millicpu).\n\nThe default pool is synonymous to the Kubelet managed, by default existing shared pool. Worth noting that the shared and default pool's CPU sets, and characteristics are distinct.\nBy supporting the \"original\" CPU allocation method in parallel with the enhanced, 3rd party containers totally content with the default CPU management policy of Kubernetes can be instantiated on a CPU-Pooler upgraded system without any configuration changes.\n\n## Topology awareness\n### NUMA alignment\nCPU-Pooler is officially NUMA/socket aware starting with the 0.4.0 release!\n\nMoreover, unlike other external managers CPU-Pooler natively integrates into Kubernetes' own Topology Manager. CPU-Pooler reports the NUMA Node ID of all the CPUs belonging to an exclusive CPU pool to the upstream Topology Manager.\nThis means whenever a Pod requests resources from Kubernetes where topology matters -e.g. SR-IOV virtual functions, exclusive CPUs, GPUs etc.- Kubernetes will automatically assign resources with their NUMA node aligned - CPU-Pooler managed cores included!\n\nThis feature is automatic, therefore it does not require any configuration from the user.\nFor it to work though CPU-Pooler's version must be at least 0.4.0, while Kubernetes must be at least 1.17.X.\n\n### Hyperthreading support\nCPU-Pooler is able to recognize when it is deployed on a hyperthreading enabled node, and supports different thread allocation policies for exclusive CPU pools.\n\nThese policies are controlled by the \"hyperThreadingPolicy\" attribute of an exclusive pool. The following two, guaranteed policies are supported currently:\n\"singleThreaded\" (default): when a physical core is assigned to a workload CPU-Pooler only includes the ID of the assigned core into the container's cpuset cgroup, and leaves all possible siblings un-assigned\n\"multiThreaded\": when this policy is set Pooler automatically discovers all siblings of an assigned core, and allocates them together to the requesting container.\n\nCPU-Pooler only implements guaranteed policies, meaning that siblings will never be accidentally assigned to neighbour containers.\nNote: for HT support to work as intended you must only list phsyical core IDs in exclusive pool definitions. CPU-Pooler will automatically discover the siblings on its own\n\n## Components of the CPU-Pooler project\nThe CPU-Pooler project contains 4 core components:\n- a Kubernetes standard Device Plugin seamlessly integrating the CPU pools to Kubernetes as schedulable resources\n- a Kubernetes standard Informer making sure containers belonging to different CPU pools are always physically isolated from each other\n- process starter binary capable of pinning specific processes to specific cores even within the confines of a container\n- a mutating admission webhook for the Kubernetes core Pod API, validating and mutating CPU pool specific user requests \n\nThe Device Plugin's job is to advertise the pools as consumable resources to Kubelet through the existing DPAPI. The CPUs allocated by the plugin are communicated to the container as environment variables containing a list of physical core IDs.\nBy default the application can set its processes CPU affinity according to the given CPU list, or can leave it up to the standard Linux Completely Fair Scheduler.\n\nFor the edge case where application does not implement functionality to set the CPU affinity of its processes, the CPU pooler provides mechanism to set it on behalf of the application.\nThis opt-in functionality is enabled by configuring the application process information to the annotation field of its Pod spec.\nA mutating admission controller webhook is provided with the project to mutate the Pod's specification according to the needs of the starter binary (mounts, environment variables etc.).\nThe process-starter binary has to be installed to host file system in `/opt/bin` directory.\n\nLastly, the CPUSetter sub-component implements total physical separation of containers via Linux cpusets. This Informer constantly watches the Pod API of Kubernetes, and is triggered whenever a Pod is created, or changes its state(e.g. restarted etc.)\nCPUSetter first calculates what is the appropriate cpuset for the container: the allocated CPUs in case of exclusive, the shared pool in case of shared, or the default in case the container did not explicitly ask for any pooled resources.\nCPUSetter then provisions the calculated set into the relevant parametet of the container's cgroupfs filesystem (cpuset.cpus).\nAs CPUSetter is triggered by all Pods on all Nodes, we can be sure no containers can ever -even accidentally- access CPU resources not meant for them!  \n\n## Using the allocated CPUs\n\nBy default CPU-Pooler only provisions the appropriate cpuset for a container based on its resource request, but does not intervene with how threads inside the container are scheduled between the allowed vCPUs.\n\nFor users who require explicitly pinning application process / thread(s) to the subset of the allocated CPUs for some reason, the following options are available:\n\n**pod annotation**\n\nIf container has multiple processes and multiple exclusive CPUs are allocated or different pool types are used, pod annotation can be used to configure processes and CPU amounts for them.\nIn this case CPU Pooler takes care of pinning the processes to allocated -but only to the allocated- CPUs. This is suitable for single threaded processes running on exclusive CPUs. The container can also have process(es) running on shared CPUs\n\n**use environment variables for pinning**\n\nCPU Pooler sets allocated exclusive CPUs to environment variable `EXCLUSIVE_CPUS` and allocated shared CPUs to `SHARED_CPUS` environment variable. They contain CPU(s) as comma separated list. These variables can be used by the application to read the allocated CPU(s) and do pinning of threads / processes to the CPU(s).\n\n\nIn order to avoid possible race conditions occuring due to multiple processes trying to set affinity at the same time (or application trying to set it too early); CPU-Pooler can guarantee that the container's entrypoint is only executed once the proper CPU configuration has been provisioned.\nThis is achieved by the `process-starter` component under the following pre-conditions.\nIn order for this functionality to work as intended the `command` property must be configured in container's pod manifest. If the `command` is not configured, the `process-starer` won't be used because it is not known which process needs to be started in the container.\nIn such cases we fall back to the native Linux thread scheduling mechanism, but depending on user activity this might result in exotic race conditions occuring.\n\n## Configuration\n\n### Kubelet\nKubelet treats Devices transparently, therefore it will never realize the CPU-Pooler managed resources are actually CPUs.\nIn order to avoid double bookkeeping of CPU resources in the cluster, every Node's Kubelet hosting the CPU-Pooler Device Plugin should be configured according to the following formula:\n--system-reserved = \u003cTOTAL_CPU_CAPACITY\u003e - SIZEOF(DEFAULT_POOL) + \u003cDEFAULT_SYSTEM_RESERVED\u003e \nThis setting effectively tells Kubelet to discount the capacity belonging to the CPU-Pooler managed shared and exclusive pools.\n\nBesides that, please note that Kubelet's inbuilt CPU Manager needs to be disabled on the Nodes which run CPU-Pooler to avoid overwriting CPU-Pooler's more fine-grained cpuset configuration.\n\n### CPU pools\n\nCPU pools are configured with a configMap named cpu-pooler-configmap. The schema of configMap is as follows:\n```\napiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: cpu-pooler-configmap\ndata:\n  poolconfig-\u003cname\u003e.yaml: |\n    pools:\n      exclusive_\u003cpoolname1\u003e:\n        cpus : \"\u003clist of physical CPU core IDs\u003e\"\n        hyperThreadingPolicy: singleThreaded\n      exclusive_\u003cpoolname2\u003e:\n        cpus : \"\u003clist of physical CPU core IDs\u003e\"\n        hyperThreadingPolicy: multiThreaded\n      shared_\u003cpoolname3\u003e:\n        cpus : \"\u003clist of CPU thread IDs\u003e\"\n      default:\n        cpus : \"\u003clist of CPU thread IDs\u003e\"\n      nodeSelector:\n        \u003ckey\u003e : \u003cvalue\u003e\n```\nThe poolconfig-\u003cname\u003e.yaml file must exist in the data section.\nThe CPU pools are defined in poolconfig-\u003cname\u003e.yaml files. There must be at least one poolconfig-\u003cname\u003e.yaml file in the data section.\nPool name from the config will be the resource in the fully qualified resource name (`nokia.k8s.io/\u003cpool name\u003e`). The pool name must have pool type prefix - 'exclusive' for exclusive CPU pool or 'shared' for shared CPU pool.\nA CPU pool not having either of these special prefixes is considered as the cluster-wide 'default' CPU pool, and as such, CPU cores belonging to this pool will not be advertised to the Device Manager as schedulable resources.\n\n\n\"cpus\" attribute controls which CPU cores belong to this pool. Standard Linux notation including \",\" and \"-\" characters is accepted.\nFor exclusive pools only configure physical CPU core IDs.\nFor shared and default pools list all the thread IDs you want to be included in the pool (i.e. physical and HT sibling IDs both).\n\n\n\"hyperThreadingPolicy\" controls whether exclusive CPU cores are allocated alone (\"singleThreaded\"), or in pairs (\"multiThreaded\").\n\n\nThe nodeSelector is used to tell which node the pool configuration file belongs to. CPU pooler and CPUSetter components both read the node labels and select the config that matches the value of nodeSelector.\n\n\nIn the deployment directory there is a sample pool config with two exclusive pools (both have two cpus) and one shared pool (one cpu). Nodes for the pool configurations are selected by `nodeType` label.\nPlease note: currently only one shared pool is supported per Node!\n### Pod spec\n\nThe cpu-device-plugin advertises the resources of exclusive, and shared CPU pools as name: `nokia.k8s.io/\u003cpoolname\u003e`. The poolname is pool name configured in cpu-pooler-configmap. The cpus are requested in the resources section of container in the pod spec.\n\n### Annotation:\n\nAnnotation schema is following and the name for the annotation is `nokia.k8s.io/cpus`. Resource being the the advertised resource name.\n```\n{\n  \"$schema\": \"http://json-schema.org/draft-07/schema#\",\n  \"type\": \"array\",\n  \"items\": {\n    \"$ref\": \"#/definitions/container\"\n  },\n  \"definitions\": {\n    \"container\": {\n      \"type\": \"object\",\n      \"required\": [\n        \"container\",\n        \"processes\"\n      ],\n      \"properties\": {\n        \"container\": {\n          \"type\": \"string\"\n        },\n        \"processes\": {\n          \"type\": \"array\",\n          \"items\": {\n            \"$ref\": \"#/definitions/process\"\n          }\n        }\n      }\n    },\n    \"process\": {\n      \"type\": \"object\",\n      \"required\": [\n        \"process\",\n        \"args\",\n        \"pool\",\n        \"cpus\"\n      ],\n      \"properties\": {\n        \"process\": {\n          \"type\": \"string\"\n        },\n        \"args\": {\n          \"type\": \"array\",\n          \"items\": {\n            \"type\": \"string\"\n          }\n        },\n        \"pool\": {\n          \"type\": \"string\"\n        },\n        \"cpus\": {\n          \"type\": \"number\"\n        }\n      }\n    }\n  }\n}\n```\nAn example is provided in cpu-test.yaml pod manifest in the deployment folder.\n\n### Restrictions\n\nFollowing restrictions apply when allocating cpu from pools and configuring pools:\n\n* There can be only one shared pool in the node\n* Resources belonging to the default pool are not advertised. The default pool definition is only used by the CPUSetter component\n\n## Build\nMutating webhook\n\n```\n$ docker build --build-arg http_proxy=$http_proxy --build-arg https_proxy=$https_proxy -t cpu-device-webhook -f build/Dockerfile.webhook  .\n```\nThe device plugin\n\n```\n$ docker build --build-arg http_proxy=$http_proxy --build-arg https_proxy=$https_proxy -t cpudp -f build/Dockerfile.cpudp  .\n```\n\nProcess starter\n\n```\n$ go mod download\n$ CGO_ENABLED=0 GOOS=linux go build -a -ldflags '-extldflags \"-static\"' github.com/nokia/CPU-Pooler/cmd/process-starter\n```\n\nCPUSetter\n\n```\n$ docker build --build-arg http_proxy=$http_proxy --build-arg https_proxy=$https_proxy -t cpusetter -f build/Dockerfile.cpusetter  .\n```\n## Installation\n\nInstall process starter to host file system:\n\n```\n$ cp process-starter /opt/bin/\n```\n\nCreate cpu-device-plugin config and daemonset:\n```\n$ kubectl create -f deployment/cpu-pooler-config.yaml\n$ kubectl create -f deployment/cpu-dev-ds.yaml\n```\nThere is a helper script ```./scripts/generate_cert.sh``` that generates certificate and key for the webhook admission controller. The script ```deployment/create-webhook-conf.sh``` can be used to create the webhook configuration from the provided manifest file (```webhook-conf.yaml```).\n\nCreate CPUSetter daemonset:\n```\n$ kubectl create -f deployment/cpusetter-ds.yaml\n```\n\nFollowing steps create the webhook server with necessary configuration (including the certifcate and key)\n```\n$ ./scripts/generate-cert.sh\n$ \n$ kubectl create -f deployment/webhook-svc-depl.yaml\n$ deployment/create-webhook-conf.sh deployment/webhook-conf.yaml\n```\nThe cpu-device-plugin with webhook should be running now. Test the installation with a cpu test pod. First create image for the test container:\n```\n$ docker build -t busyloop test/\n```\n\nStart the test container:\n```\n$ kubectl create -f ./deployment/cpu-test.yaml\n```\n\nUpon instantiation you can observe that the cpuset.cpus parameter of the created containers' cpuset cgroupfs hierarchy are set to right values:\n- to the configured cpuset belonging to the shared pool for sharedtestcontainer\n- to the a cpuset containing only the ID of the 2 chosen cores for exclusivetestcontainer\n- to the configured cpuset belonging to the default pool for defaulttestcontainer\n\n\n## License\n\nThis project is licensed under the BSD-3-Clause license - see the [LICENSE](https://github.com/nokia/CPU-Pooler/blob/master/LICENSE).","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnokia%2Fcpu-pooler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnokia%2Fcpu-pooler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnokia%2Fcpu-pooler/lists"}