{"id":19102667,"url":"https://github.com/bugroger/coreos-nvidia-driver","last_synced_at":"2025-04-30T16:27:38.764Z","repository":{"id":136498141,"uuid":"98041315","full_name":"BugRoger/coreos-nvidia-driver","owner":"BugRoger","description":"Compile, Install and Update NVIDIA Drivers for CoreOS Container Linux","archived":false,"fork":false,"pushed_at":"2019-11-25T23:26:32.000Z","size":43,"stargazers_count":16,"open_issues_count":3,"forks_count":9,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-19T06:57:04.557Z","etag":null,"topics":["coreos","docker","kubernetes","nvidia"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/BugRoger.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2017-07-22T15:47:36.000Z","updated_at":"2024-10-05T02:37:41.000Z","dependencies_parsed_at":null,"dependency_job_id":"984315d2-dc29-4673-a200-d63e51c2555b","html_url":"https://github.com/BugRoger/coreos-nvidia-driver","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BugRoger%2Fcoreos-nvidia-driver","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BugRoger%2Fcoreos-nvidia-driver/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BugRoger%2Fcoreos-nvidia-driver/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BugRoger%2Fcoreos-nvidia-driver/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/BugRoger","download_url":"https://codeload.github.com/BugRoger/coreos-nvidia-driver/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251741589,"owners_count":21636280,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["coreos","docker","kubernetes","nvidia"],"created_at":"2024-11-09T03:56:37.423Z","updated_at":"2025-04-30T16:27:38.654Z","avatar_url":"https://github.com/BugRoger.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CoreOS Container Linux Nvidia Driver\n\n[![Build Status](https://travis-ci.org/BugRoger/coreos-nvidia-driver.svg?branch=master)](https://travis-ci.org/BugRoger/coreos-nvidia-driver)\n\n\nThis image installs the Nvidia's drivers on CoreOS Container Linux. The\ndriver is a kernel module that is being compiled using the hosts' exact kernel\nsources.\n\nCoreOS ships without compiler toolchain and without kernel sources. To make\nmatters worse updates will change the kernel unattended, requiring to recompile\nthe driver... :(\n\nTwo tricks are being used to solve this problem.\n\nThe image uses a multi-stage Docker build. In the first state a full CoreOS\ndeveloper image is being used to install the driver using the exact Kernel\nsources and headers as the later host. See: https://github.com/BugRoger/coreos-developer-docker\n\nThe second stage just copies the installation to a skinny Alpine image for\ntransport.\n\nAll this happens automatically for each CoreOS version using a TravisCI build.\n\nUpon startup a systemd verifies the currently installed driver matches the\nkernel. If not, this container is executed installing the proper precompiled\ndriver. Tadaaaa.\n\n## Installation\n\nFind the image on Docker Hub: https://hub.docker.com/r/bugroger/coreos-nvidia-driver\n\n```\ndocker run -v /:/rootfs --privileged bugroger/coreos-nvidia-driver:1576.5.0-384.111\n```\n\nThis will install the Nvidia driver to `/opt/nvidia/384.111/1576.5.0`. Additionally,\na symlink to `/opt/nvidia/current` will be created.\n\nInstead of reconfiguring the `LD_LIBRARY_PATH` and `ldconf` cache locations,\nthe installer creates an overlay filesystem to mount the driver from\n`/opt/nvidia` to `/usr/lib64` where kernel modules are expected. This makes the\nsystem look like a regular installation.\n\nThe installer creates a `nvidia` service, that creates the shared library\ncache, loads the kernel modules and starts the driver. It is bundles to the\n`nvidia-perstience` service that also starts the persistence daemon.\n\n## Auto-Update\n\nIn order to deal with unattended Container Linux updates this installer provides an \nautomatic update mechanism. This is necessary due to frequent kernel updates. The\ndriver must have been built with exactly the same kernel version and sources.\n\nIn constrast to other installers, that run the whole compilation during each\nboot, we here just check for a prebuilt Docker image. The image is being built\nout-of-band by Travis for each update on all Container Linux release channels.\n\nOnce the host updates and reboots the `nvidia-update` service is started. It\nwill pull the exact kernel/driver combination and repeat the installation.\n\n## Kubernetes Kubelet\n\nAs of this writing the Kubernetes Kubelet only enumerates (GPU) resources\nduring startup. That means the nvidia kernel module must have been loaded\nbefore the Kubelet is started.\n\nUsing systemd a dependency to the `nvidia` service can be used:\n\n```\n[Unit]\nAfter=nvidia.service\nRequires=nvidia.service\n\n[Service]\nExecStart=/opt/bin/kubelet ...\n \n```\n\n## Kubernetes \n\nUsing the Nvidia drivers in Kubernetes posses two problems:\n\n  1. Applications usually need to load the exact same shared libraries as\n     installed on the host OS. \n  2. The Cuda libraries are ridiciously big ~2GB\n\nBuilding new containers whenever the kernel or driver is updated, doing so\nduring and unattened upgrade, transporting them and on the same time patching\nthe Kubernetes specs is a major problem.\n\nAs a workaround, the idea is that the shared libraries are mounted from the\nhost system. This avoids version mismatch between the hosts' shared libraries\nand what was baked into the container.\n\nA spec using this installer looks like this:\n\n```\napiVersion: v1\nkind: Pod\nmetadata:\n  name: nvidia-settings \nspec:\n  containers:\n    - name: nvidia-settings \n      image: bugroger/x11:381.22\n      volumeMounts:\n        - mountPath: /usr/local/nvidia\n          name: nvidia \n        - mountPath: /usr/local/cuda\n          name: cuda \n  volumes:\n    - name: nvidia\n      hostPath:\n        path: /opt/nvidia/current\n    - name: cuda\n      hostPath:\n        path: /opt/cuda/current\n```\n\nNote the `hostPath` value pointing to `current`. This makes the spec\nindependant of a specific driver version. Though if required, this can also be\nused to pin to a specific version.\n\nIf you are into Kubernetes Alpha features, this can be made even less invasive\nusing `PodPresets`:\n\n```\n---\napiVersion: v1\nkind: Pod\nmetadata:\n  name: nvidia-settings \n  labels:\n    sharedLibs: nvidia\nspec:\n  containers:\n    - name: nvidia-settings \n      image: bugroger/x11:381.22\n---\napiVersion: settings.k8s.io/v1alpha1\nkind: PodPreset\nmetadata:\n  name: nvidia-shared-libs\nspec:\n  selector:\n    matchLabels:\n      sharedLibs: nvidia\n  volumeMounts:\n    - mountPath: /usr/local/nvidia\n      name: nvidia \n    - mountPath: /usr/local/cuda\n      name: cuda \n  volumes:\n    - name: nvidia\n      hostPath:\n        path: /opt/nvidia/current\n    - name: cuda\n      hostPath:\n        path: /opt/cuda/current\n```\n\n\n## Docker\n\nUnfortunately, the Docker images are not completly unoblivious of this method.\nThe problem is that the `LD_LIBRARY_PATH` must be set, so that library can be\nfound.\n\nBy convention the Nvidia driver and Cuda libs are expected in `/usr/local/cuda`\nand `/usr/local/nvidia`.\n\nThis can be prebaked into containers that want to use the mounted libs.\n\n```\nRUN echo \"/usr/local/cuda/lib64\" \u003e\u003e /etc/ld.so.conf.d/cuda.conf \nRUN echo \"/usr/local/nvidia/lib64\" \u003e\u003e /etc/ld.so.conf.d/nvidia.conf\n\nENV PATH $PATH:/usr/local/nvidia/bin:/usr/local/cuda/bin\nENV LD_LIBRARY_PATH $LD_LIBRARY_PATH:/usr/local/nvidia/lib64/:/usr/local/cuda/lib64/\n```\n\n\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbugroger%2Fcoreos-nvidia-driver","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbugroger%2Fcoreos-nvidia-driver","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbugroger%2Fcoreos-nvidia-driver/lists"}