{"id":19868609,"url":"https://github.com/Project-HAMi/volcano-vgpu-device-plugin","last_synced_at":"2025-05-02T07:31:42.759Z","repository":{"id":232763578,"uuid":"785133132","full_name":"Project-HAMi/volcano-vgpu-device-plugin","owner":"Project-HAMi","description":"Device-plugin for volcano vgpu which support hard resource isolation","archived":false,"fork":false,"pushed_at":"2024-05-21T02:26:17.000Z","size":429,"stargazers_count":5,"open_issues_count":2,"forks_count":3,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-06-06T02:31:44.034Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Project-HAMi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-04-11T09:06:14.000Z","updated_at":"2024-07-11T10:59:49.964Z","dependencies_parsed_at":"2024-04-11T10:29:42.348Z","dependency_job_id":"70cd27fe-5ed0-4c2e-80fd-77f67db33d35","html_url":"https://github.com/Project-HAMi/volcano-vgpu-device-plugin","commit_stats":null,"previous_names":["project-hami/volcano-vgpu-device-plugin"],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Project-HAMi%2Fvolcano-vgpu-device-plugin","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Project-HAMi%2Fvolcano-vgpu-device-plugin/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Project-HAMi%2Fvolcano-vgpu-device-plugin/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Project-HAMi%2Fvolcano-vgpu-device-plugin/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Project-HAMi","download_url":"https://codeload.github.com/Project-HAMi/volcano-vgpu-device-plugin/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224305843,"owners_count":17289446,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T16:01:11.311Z","updated_at":"2025-05-02T07:31:42.753Z","avatar_url":"https://github.com/Project-HAMi.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"# Volcano vgpu device plugin for Kubernetes\n\n[![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2FProject-HAMi%2Fvolcano-vgpu-device-plugin.svg?type=shield)](https://app.fossa.com/projects/git%2Bgithub.com%2FProject-HAMi%2Fvolcano-vgpu-device-plugin?ref=badge_shield)\n\n**Note**:\n\nVolcano vgpu device-plugin can provide device-sharing mechanism for NVIDIA devices managed by volcano.\n\nThis is based on [Nvidia Device Plugin](https://github.com/NVIDIA/k8s-device-plugin), it uses [HAMi-core](https://github.com/Project-HAMi/HAMi-core) to support hard isolation of GPU card.\n\nAnd collaborate with volcano, it is possible to enable GPU sharing.\n\n## Table of Contents\n\n- [About](#about)\n- [Prerequisites](#prerequisites)\n- [Quick Start](#quick-start)\n  - [Preparing your GPU Nodes](#preparing-your-gpu-nodes)\n  - [Enabling vGPU Support in Kubernetes](#enabling-gpu-support-in-kubernetes)\n  - [Running vGPU Jobs](#running-vgpu-jobs)\n- [Issues and Contributing](#issues-and-contributing)\n\n## About\n\nThe Volcano device plugin for Kubernetes is a Daemonset that allows you to automatically:\n- Expose the number of GPUs on each node of your cluster\n- Keep track of the health of your GPUs\n- Run GPU enabled containers in your Kubernetes cluster.\n- Provide device-sharing mechanism for GPU tasks as the figure below.\n- Enforce hard resource limit in container.\n\n\u003cimg src=\"./doc/example.png\" width = \"600\" /\u003e \n\n## Prerequisites\n\nThe list of prerequisites for running the Volcano device plugin is described below:\n* NVIDIA drivers \u003e 440\n* nvidia-docker version \u003e 2.0 (see how to [install](https://github.com/NVIDIA/nvidia-docker) and it's [prerequisites](https://github.com/nvidia/nvidia-docker/wiki/Installation-\\(version-2.0\\)#prerequisites))\n* docker configured with nvidia as the [default runtime](https://github.com/NVIDIA/nvidia-docker/wiki/Advanced-topics#default-runtime).\n* Kubernetes version \u003e= 1.16\n* Volcano verison \u003e= 1.9\n\n## Quick Start\n\n### Preparing your GPU Nodes\n\nThe following steps need to be executed on all your GPU nodes.\nThis README assumes that the NVIDIA drivers and nvidia-docker have been installed.\n\nNote that you need to install the nvidia-docker2 package and not the nvidia-container-toolkit.\nThis is because the new `--gpus` options hasn't reached kubernetes yet. Example:\n```bash\n# Add the package repositories\n$ distribution=$(. /etc/os-release;echo $ID$VERSION_ID)\n$ curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -\n$ curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list\n\n$ sudo apt-get update \u0026\u0026 sudo apt-get install -y nvidia-docker2\n$ sudo systemctl restart docker\n```\n\nYou will need to enable the nvidia runtime as your default runtime on your node.\nWe will be editing the docker daemon config file which is usually present at `/etc/docker/daemon.json`:\n```json\n{\n    \"default-runtime\": \"nvidia\",\n    \"runtimes\": {\n        \"nvidia\": {\n            \"path\": \"/usr/bin/nvidia-container-runtime\",\n            \"runtimeArgs\": []\n        }\n    }\n}\n```\n\u003e *if `runtimes` is not already present, head to the install page of [nvidia-docker](https://github.com/NVIDIA/nvidia-docker)*\n\n\n### Configure scheduler\n\nupdate the scheduler configuration:\n\n```shell script\nkubectl edit cm -n volcano-system volcano-scheduler-configmap\n```\n\nFor volcano v1.9+,, use the following configMap \n```yaml\nkind: ConfigMap\napiVersion: v1\nmetadata:\n  name: volcano-scheduler-configmap\n  namespace: volcano-system\ndata:\n  volcano-scheduler.conf: |\n    actions: \"enqueue, allocate, backfill\"\n    tiers:\n    - plugins:\n      - name: priority\n      - name: gang\n      - name: conformance\n    - plugins:\n      - name: drf\n      - name: deviceshare\n        arguments:\n          deviceshare.VGPUEnable: true # enable vgpu\n      - name: predicates\n      - name: proportion\n      - name: nodeorder\n      - name: binpack\n```\n\n### Enabling GPU Support in Kubernetes\n\nOnce you have enabled this option on *all* the GPU nodes you wish to use,\nyou can then enable GPU support in your cluster by deploying the following Daemonset:\n\n```\n$ kubectl create -f volcano-vgpu-device-plugin.yml\n```\n\n### Verify environment is ready\n\nCheck the node status, it is ok if `volcano.sh/vgpu-number` is included in the allocatable resources.\n\n```shell script\n$ kubectl get node {node name} -oyaml\n...\nstatus:\n  addresses:\n  - address: 172.17.0.3\n    type: InternalIP\n  - address: volcano-control-plane\n    type: Hostname\n  allocatable:\n    cpu: \"4\"\n    ephemeral-storage: 123722704Ki\n    hugepages-1Gi: \"0\"\n    hugepages-2Mi: \"0\"\n    memory: 8174332Ki\n    pods: \"110\"\n    volcano.sh/vgpu-memory: \"89424\"\n    volcano.sh/vgpu-number: \"10\"    # vGPU resource\n  capacity:\n    cpu: \"4\"\n    ephemeral-storage: 123722704Ki\n    hugepages-1Gi: \"0\"\n    hugepages-2Mi: \"0\"\n    memory: 8174332Ki\n    pods: \"110\"\n    volcano.sh/vgpu-memory: \"89424\"\n    volcano.sh/vgpu-number: \"10\"   # vGPU resource\n```\n\n### Running VGPU Jobs\n\nVGPU can be requested by both set \"volcano.sh/vgpu-number\" , \"volcano.sh/vgpu-cores\" and \"volcano.sh/vgpu-memory\" in resource.limit\n\n```shell script\n$ cat \u003c\u003cEOF | kubectl apply -f -\napiVersion: v1\nkind: Pod\nmetadata:\n  name: gpu-pod1\nspec:\n  schedulerName: volcano\n  containers:\n    - name: cuda-container\n      image: nvidia/cuda:9.0-devel\n      command: [\"sleep\"]\n      args: [\"100000\"]\n      resources:\n        limits:\n          volcano.sh/vgpu-number: 2 # requesting 2 gpu cards\n          volcano.sh/vgpu-memory: 3000 # (optinal)each vGPU uses 3G device memory\n          volcano.sh/vgpu-cores: 50 # (optional)each vGPU uses 50% core  \nEOF\n```\n\nYou can validate device memory using nvidia-smi inside container:\n\n![img](./doc/hard_limit.jpg)\n\n\u003e **WARNING:** *if you don't request GPUs when using the device plugin with NVIDIA images all\n\u003e the GPUs on the machine will be exposed inside your container.\n\u003e The number of vgpu used by a container can not exceed the number of gpus on that node.*\n\n### Monitor\n\nvolcano-scheduler-metrics records every GPU usage and limitation, visit the following address to get these metrics.\n\n```\ncurl {volcano scheduler cluster ip}:8080/metrics\n```\n\nYou can also collect the **GPU utilization**, **GPU memory usage**, **pods' GPU memory limitations** and **pods' GPU memory usage** metrics on nodes by visiting the following addresses:\n\n```\ncurl {volcano device plugin pod ip}:9394/metrics\n```\n![img](./doc/vgpu_device_plugin_metrics.png)\n\n# Issues and Contributing\n[Checkout the Contributing document!](CONTRIBUTING.md)\n\n* You can report a bug by [filing a new issue](https://github.com/Project-HAMi/volcano-vgpu-device-plugin)\n* You can contribute by opening a [pull request](https://help.github.com/articles/using-pull-requests/)\n\n\n## Upgrading Kubernetes with the device plugin\n\nUpgrading Kubernetes when you have a device plugin deployed doesn't require you to do any,\nparticular changes to your workflow.\nThe API is versioned and is pretty stable (though it is not guaranteed to be non breaking),\nupgrading kubernetes won't require you to deploy a different version of the device plugin and you will\nsee GPUs re-registering themselves after you node comes back online.\n\n\nUpgrading the device plugin is a more complex task. It is recommended to drain GPU tasks as\nwe cannot guarantee that GPU tasks will survive a rolling upgrade.\nHowever we make best efforts to preserve GPU tasks during an upgrade.\n\n\n## License\n[![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2FProject-HAMi%2Fvolcano-vgpu-device-plugin.svg?type=large)](https://app.fossa.com/projects/git%2Bgithub.com%2FProject-HAMi%2Fvolcano-vgpu-device-plugin?ref=badge_large)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FProject-HAMi%2Fvolcano-vgpu-device-plugin","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FProject-HAMi%2Fvolcano-vgpu-device-plugin","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FProject-HAMi%2Fvolcano-vgpu-device-plugin/lists"}