{"id":21105373,"url":"https://github.com/jliberma/newton-gpu","last_synced_at":"2025-07-03T11:03:49.191Z","repository":{"id":88743668,"uuid":"145934420","full_name":"jliberma/newton-gpu","owner":"jliberma","description":null,"archived":false,"fork":false,"pushed_at":"2018-08-27T14:59:09.000Z","size":16,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-14T08:45:33.529Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jliberma.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-08-24T02:52:21.000Z","updated_at":"2018-08-27T14:59:11.000Z","dependencies_parsed_at":null,"dependency_job_id":"0e2fc94c-d66c-4950-904d-be04b413e924","html_url":"https://github.com/jliberma/newton-gpu","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/jliberma/newton-gpu","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jliberma%2Fnewton-gpu","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jliberma%2Fnewton-gpu/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jliberma%2Fnewton-gpu/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jliberma%2Fnewton-gpu/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jliberma","download_url":"https://codeload.github.com/jliberma/newton-gpu/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jliberma%2Fnewton-gpu/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263314103,"owners_count":23447291,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-20T00:05:25.080Z","updated_at":"2025-07-03T11:03:49.163Z","avatar_url":"https://github.com/jliberma.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Deploying GPUs in OpenStack Newton via Tripleo\n\nInstructions for configuring OpenStack Platform 10 director to deploy instances with Nvidia GPUs enabled via PCI Passthrough.\n\n## Basic workflow\n1. Deploy undercloud and import overcloud servers to Ironic\n2. Enable IOMMU in server BIOS to support PCI passthrough\n3. Deploy overcloud with templates that configure: iommu in grub, pci device aliases, pci device whitelist, and PciPassthrough filter enabled in nova.conf\n4. Customize RHEL 7.4 image with kernel headers/devel and gcc\n5. Create custom Nova flavor with PCI device alias\n6. Configure cloud-init to install cuda at instance boot time\n7. Launch instance from flavor + cloud-init + image via Heat\n8. Run sample codes\n\n\n## Resources\n- [GPU support in Red Hat OpenStack Platform](https://access.redhat.com/solutions/3080471)\n- [Bugzilla RFE for documentation on confiuring GPUs via PCI passthrough in OpenStack Platform](https://bugzilla.redhat.com/show_bug.cgi?id=1430337)\n- [OpenStack Nova Configure PCI Passthrough](https://docs.openstack.org/nova/pike/admin/pci-passthrough.html)\n- [KVM virtual machine GPU configuration](https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Virtualization_Deployment_and_Administration_Guide/chap-Guest_virtual_machine_device_configuration.html#sect-device-GPU)\n- [Nvidia Cuda Linux installation guide](http://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#runfile-installation)\n- [DKMS support in Red Hat Enterprise Linux](https://access.redhat.com/solutions/1132653)\n- [Deploying TripleO artifacts](http://hardysteven.blogspot.com/2016/08/tripleo-deploy-artifacts-and-puppet.html)\n\n\n## Create TripleO environment files\n\nCreate TripleO environment files to configure nova.conf on the overcloud nodes running nova-compute and nova-scheduler.\n\n```\ncat templates/environments/20-compute-params.yaml \nparameter_defaults:\n\n  NovaPCIPassthrough:\n        - vendor_id: \"10de\"\n          product_id: \"13f2\"\n\ncat templates/environments/20-controller-params.yaml \nparameter_defaults:\n\n  NovaSchedulerDefaultFilters: ['AvailabilityZoneFilter','RamFilter','ComputeFilter','ComputeCapabilitiesFilter','ImagePropertiesFilter','ServerGroupAntiAffinityFilter','ServerGroupAffinityFilter', 'PciPassthroughFilter', 'NUMATopologyFilter', 'AggregateInstanceExtraSpecsFilter']\n\n  ControllerExtraConfig:\n    nova::api::pci_alias:\n      -  name: a1\n         product_id: '13f2'\n         vendor_id: '10de'\n      -  name: a2\n         product_id: '13f2'\n         vendor_id: '10de'\n\n```\n\nIn the above example, the controller node aliases two M60 cards with the names a1 and a2. Depending on the flavor, either or both cards can be assigned to an instance.\n\nThe environment files require the vendor ID and product ID for each passthrough device type. You can find these by running **lspci** on the physical server with the PCI cards.\n\n```\n    # lspci -nn | grep -i nvidia\n    3d:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204GL [Tesla M60] [10de:13f2] (rev a1)\n    3e:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204GL [Tesla M60] [10de:13f2] (rev a1)\n```\n\nThe vendor ID is the first 4 digit hexadecimal number following the device name. The product ID is the second.\n\nlspci is installed by the **pciutils** package.\n\niommu must be enabled at boot time on the compute nodes as well. This is accomplished through a the firstboot extraconfig hook.\n\n```\ncat templates/environments/10-firstboot-environment.yaml \nresource_registry:\n  OS::TripleO::NodeUserData: /home/stack/templates/firstboot/first-boot.yaml\n\ncat templates/firstboot/first-boot.yaml \nheat_template_version: 2014-10-16\n\n\nresources:\n  userdata:\n    type: OS::Heat::MultipartMime\n    properties:\n      parts:\n      - config: {get_resource: compute_kernel_args}\n\n\n  # Verify the logs on /var/log/cloud-init.log on the overcloud node\n  compute_kernel_args:\n    type: OS::Heat::SoftwareConfig\n    properties:\n      config: |\n        #!/bin/bash\n        set -x\n\n        # Set grub parameters\n        if hostname | grep compute \u003e/dev/null\n        then\n                sed -i.orig 's/quiet\"$/quiet intel_iommu=on iommu=pt\"/' /etc/default/grub\n                grub2-mkconfig -o /etc/grub2.cfg\n                systemctl stop os-collect-config.service\n                /sbin/reboot\n        fi\n\noutputs:\n  OS::stack_id:\n    value: {get_resource: userdata}\n```\n\nThe kernel arguments will be added to the Compute nodes at deploy time. This can be verified after deployment by checking /proc/cmdline on the compute node:\n\n```\n    $ ssh -l heat-admin 172.16.0.31 cat /proc/cmdline\n    BOOT_IMAGE=/boot/vmlinuz-3.10.0-862.6.3.el7.x86_64 root=UUID=7aa9d695-b9c7-416f-baf7-7e8f89c1a3bc ro console=tty0 console=ttyS0,115200n8 crashkernel=auto rhgb quiet intel_iommu=on iommu=pt\n```\n\nDirect  IO virtualization must also be enabled in the server BIOS. This feature  can be called VT-d, VT-Direct, or Global SR_IOV Enable.\n\n## Customize the RHEL 7.5 image\n\nDownload the RHEL 7.5 KVM guest image and customize it. This image will be used to launch the guest instance.\n\n```\n    $ virt-customize --selinux-relabel -a ~/images/rhel7.5-gpu.qcow2 --root-password password:redhat\n    $ virt-customize --selinux-relabel -a ~/images/rhel7.5-gpu.qcow2 --run-command 'subscription-manager register --username=REDACTED --password=REDACTED'\n    $ virt-customize --selinux-relabel -a ~/images/rhel7.5-gpu.qcow2 --run-command 'subscription-manager attach --pool=REDACTED'\n    $ virt-customize --selinux-relabel -a ~/images/rhel7.5-gpu.qcow2 --run-command 'subscription-manager repos --disable=\\*'\n    $ virt-customize --selinux-relabel -a ~/images/rhel7.5-gpu.qcow2 --run-command 'subscription-manager repos --enable=rhel-7-server-rpms --enable=rhel-7-server-extras-rpms --enable=rhel-7-server-rh-common-rpms --enable=rhel-7-server-optional-rpms'\n    $ virt-customize --selinux-relabel -a ~/images/rhel7.5-gpu.qcow2 --run-command 'yum install -y kernel-devel-$(uname -r) kernel-headers-$(uname -r) gcc pciutils wget'\n    $ virt-customize --selinux-relabel -a ~/images/rhel7.5-gpu.qcow2 --update\n```\n\nIn this example we set a root password, register the server to CDN, subscribe only to the rhel-7-server-rpms, rhel-7-server-extras-rpms, rhel-7-server-rh-common-rpms, and rhel-7-server-optional-rpms channels. We also install the kernel-devel and kernel-headers packages along with gcc. These packages are required to build a kernel specific version of the Nvidia driver. Finally, we update the installed packages to the latest versions available from CDN.\n\nAfter the image has been customized, we upload the image to Glance in the overcloud:\n\n```\n    $ source ~/overcloudrc\n    $ openstack image create --disk-format qcow2 --container-format bare --public --file images/rhel7.5-gpu.qcow2 rhel75-gpu\n```\n\n## Deploy test Heat stack\n\nThe github repository includes Heat templates that:\n\n1. Creates a flavor tagged with the PCI alias\n2. Creates a test tenant and network with external access via floating IP addresses\n3. Launches an instance from the image and flavor\n4. Associated a floating IP and keypair with the instance\n5. Installs the Cuda drivers and sample code on the instance via  Heat softwareConfig script\n\nRun the Heat stack gpu_admin in the overcloud admin tenant to create the project, user, flavor, and networks.\n\n```\n    $ source ~/overcloudrc\n    $ openstack stack create -t heat/gpu_admin.yaml gpu_admin\n    $ openstack stack resource list gpu_admin\n    +-------------------+-------------------------------------------------------------------------------------+------------------------------+-----------------+----------------------+\n    | resource_name     | physical_resource_id                                                                | resource_type                | resource_status | updated_time         |\n    +-------------------+-------------------------------------------------------------------------------------+------------------------------+-----------------+----------------------+\n    | openstack_user    | 1f9c8342662a43ba99271348cdbf746c                                                    | OS::Keystone::User           | CREATE_COMPLETE | 2018-08-24T02:44:39Z |\n    | instance_flavor1  | b695c9d3-cdcf-4a6f-bac2-0ce0b3cd3fb3                                                | OS::Nova::Flavor             | CREATE_COMPLETE | 2018-08-24T02:44:39Z |\n    | internal_net      | f6db7c39-c6c4-4151-9582-00bfeab8a9cb                                                | OS::Neutron::Net             | CREATE_COMPLETE | 2018-08-24T02:44:39Z |\n    | public_network    | 62a6a934-decc-4c20-aaa4-fa5f8c580f6c                                                | OS::Neutron::ProviderNet     | CREATE_COMPLETE | 2018-08-24T02:44:39Z |\n    | router_interface  | d1c229bc-747b-4609-b425-953ca5e607f9:subnet_id=6ae36310-63c9-4eb8-885b-a1ba6977ff39 | OS::Neutron::RouterInterface | CREATE_COMPLETE | 2018-08-24T02:44:39Z |\n    | openstack_project | 08fc17dbfe474d5bb37ea5af5140100c                                                    | OS::Keystone::Project        | CREATE_COMPLETE | 2018-08-24T02:44:39Z |\n    | internal_router   | d1c229bc-747b-4609-b425-953ca5e607f9                                                | OS::Neutron::Router          | CREATE_COMPLETE | 2018-08-24T02:44:39Z |\n    | public_subnet     | 05ba13f6-4518-47c1-858b-070ecc54343e                                                | OS::Neutron::Subnet          | CREATE_COMPLETE | 2018-08-24T02:44:39Z |\n    | internal_subnet   | 6ae36310-63c9-4eb8-885b-a1ba6977ff39                                                | OS::Neutron::Subnet          | CREATE_COMPLETE | 2018-08-24T02:44:39Z |\n    +-------------------+-------------------------------------------------------------------------------------+------------------------------+-----------------+----------------------+\n```\n\nRun the Heat stack gpu_user as the tenant user to luanch the instance and associate a floating IP address.\n\n```\n    $ sed -e 's/OS_USERNAME=admin/OS_USERNAME=user1/' -e 's/OS_PROJECT_NAME=admin/OS_PROJECT_NAME=tenant1/' -e 's/OS_PASSWORD=.*/OS_PASSWORD=redhat/' overcloudrc \u003e ~/user1.rc\n    $ source ~/user1.rc\n    $ openstack stack create -t heat/gpu_user.yaml gpu_user\n    $ openstack stack resource list gpu_user\n    +---------------------+--------------------------------------+----------------------------+-----------------+----------------------+\n    | resource_name       | physical_resource_id                 | resource_type              | resource_status | updated_time         |\n    +---------------------+--------------------------------------+----------------------------+-----------------+----------------------+\n    | server_init         | b69f7004-7341-4906-99d3-96b09075b0e5 | OS::Heat::MultipartMime    | CREATE_COMPLETE | 2018-08-24T02:47:33Z |\n    | server1_port        | cd19ce2c-39f2-42bd-aac9-290ff3ad57b8 | OS::Neutron::Port          | CREATE_COMPLETE | 2018-08-24T02:47:33Z |\n    | cuda_init           | aa3c84e3-fbe5-4b92-9ad1-34f5af9af704 | OS::Heat::SoftwareConfig   | CREATE_COMPLETE | 2018-08-24T02:47:33Z |\n    | server1             | 9600ea1a-31fb-463d-abd0-0c4044b8517c | OS::Nova::Server           | CREATE_COMPLETE | 2018-08-24T02:47:33Z |\n    | tenant_key_pair     | generated key pair                   | OS::Nova::KeyPair          | CREATE_COMPLETE | 2018-08-24T02:47:33Z |\n    | security_group      | a13707fb-5fd6-4934-afda-355a44a85096 | OS::Neutron::SecurityGroup | CREATE_COMPLETE | 2018-08-24T02:47:33Z |\n    | server1_floating_ip | 50626288-639b-430a-b378-8ccfe021f579 | OS::Neutron::FloatingIP    | CREATE_COMPLETE | 2018-08-24T02:47:33Z |\n    +---------------------+--------------------------------------+----------------------------+-----------------+----------------------+\n```\n\nThe Keystone key pair is automatically generated. Export the Heat output to a file.\n\n```\n    $ openstack stack output show -f value gpu_user private_key | tail -n +3 \u003e gpukey.pem\n\n    $ chmod 600 gpukey.pem\n    $ cat gpukey.pem\n    -----BEGIN RSA PRIVATE KEY-----\n    MIIEowIBAAKCAQEAtrF2+mO7lOsFSJmF0rXGZZ5jpZFMvwc7GdZ9YNJ140jDD/Y7\n    LXixwpFdxwRZwt1eHTzPcGuE7SjA9kyisk6D5lPYs1wQbJnnTpk5oOkkdlpwZwdY\n    ...\n    HajSgbDyjkHVxLFLzQ/HG9w0c6Ab3ewJDH+VHGHVXfMOzDP+8aFN1AGRXechXBlH\n    omV/xFg9EW/1W6pkqDPaZQ9I9QAGRpzi6JYtFfPOU/FIkVRkEmof\n    -----END RSA PRIVATE KEY-----\n```\n\n### Configure Cuda drivers and utilities\n\nThe Cuda drivers and utilities are installed by the following OS::Heat::SoftwareConfig resource:\n\n```\n    $ grep -A 18 cuda_init: heat/gpu_user.yaml\n      cuda_init:\n        type: OS::Heat::SoftwareConfig\n        properties:\n          config: |\n            #!/bin/bash\n            echo \"installing repos\" \u003e /tmp/cuda_init.log\n            rpm -ivh https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm\n            rpm -ivh https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/cuda-repo-rhel7-8.0.61-1.x86_64.rpm\n            echo \"installing cuda and samples\" \u003e\u003e /tmp/cuda_init.log\n            yum install -y cuda \u0026\u0026 /usr/local/cuda-9.2/bin/cuda-install-samples-9.2.sh /home/cloud-user\n            echo \"building cuda samples\" \u003e\u003e /tmp/cuda_init.log\n            make -j $(grep -c Skylake /proc/cpuinfo) -C /home/cloud-user/NVIDIA_CUDA-9.2_Samples -Wno-deprecated-gpu-targets\n            shutdown -r now\n\n      server_init:\n        type: OS::Heat::MultipartMime\n        properties:\n          parts:\n          - config: { get_resource: cuda_init }\n```\n\n\u003e **NOTE**: Cuda is a proprietary driver that requires DKMS to build the kernel modules. DKMS is available from EPEL. Neither the Cuda drivers nor DKMS are supported by Red Hat.\n\nVerify the cards are recognized by the instance: \n\n```\n    $ openstack server list\n    +--------------------------------------+------+--------+-----------------------------------------+-------------+\n    | ID                                   | Name | Status | Networks                                | Image Name  |\n    +--------------------------------------+------+--------+-----------------------------------------+-------------+\n    | 9600ea1a-31fb-463d-abd0-0c4044b8517c | vm1  | ACTIVE | internal_net=192.168.0.12, 172.16.0.214 | rhel7.5-gpu |\n    +--------------------------------------+------+--------+-----------------------------------------+-------------+\n\n    $ ssh -l cloud-user -i gpukey.pem 172.16.0.214 sudo lspci | grep -i nvidia\n    00:06.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1)\n    00:07.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1)\n```\n\nVerify that the drivers are built correctly:\n\n```\n    $ ssh -l cloud-user -i gpukey.pem 192.168.122.104 sudo lsmod | grep -i nvidia\n    nvidia_drm             39689  0\n    nvidia_modeset       1086183  1 nvidia_drm\n    nvidia              14037782  1 nvidia_modeset\n    drm_kms_helper        177166  2 cirrus,nvidia_drm\n    drm                   397988  5 ttm,drm_kms_helper,cirrus,nvidia_drm\n    i2c_core               63151  4 drm,i2c_piix4,drm_kms_helper,nvidia\n    ipmi_msghandler        46607  2 ipmi_devintf,nvidia\n```\n\n\u003e **NOTE**: It may take several minutes for the Nvidia drivers to build.\n\n## Run sample codes\n\nVerify PCI passthrough and Cuda and properly configured by running sample benchmarks included with the distribution:\n\n```\n    $ ssh -l cloud-user -i gpukey.pem 192.168.122.104\n    Last login: Sat Aug 18 14:23:58 2018 from undercloud.redhat.local\n\n    $  cat /proc/driver/nvidia/version\n    NVRM version: NVIDIA UNIX x86_64 Kernel Module  396.44  Wed Jul 11 16:51:49 PDT 2018\n    GCC version:  gcc version 4.8.5 20150623 (Red Hat 4.8.5-28) (GCC) \n```\n\nRun the sample codes installed in the cloud-user home directory. In this example we run a simple Stream test of memory bandwidth and a floating point matrix multiply.\n\n```\n    $ ls ~/NVIDIA_CUDA-9.2_Samples/\n    0_Simple  1_Utilities  2_Graphics  3_Imaging  4_Finance  5_Simulations  6_Advanced  7_CUDALibraries  bin  common  EULA.txt  Makefile\n\n    $ ~/NVIDIA_CUDA-9.2_Samples/0_Simple/simpleStreams/simpleStreams\n    [ simpleStreams ]\n    Device synchronization method set to = 0 (Automatic Blocking)\n    Setting reps to 100 to demonstrate steady state\n    \u003e GPU Device 0: \"Tesla M60\" with compute capability 5.2\n    Device: \u003cTesla M60\u003e canMapHostMemory: Yes\n    \u003e CUDA Capable: SM 5.2 hardware\n    \u003e 16 Multiprocessor(s) x 128 (Cores/Multiprocessor) = 2048 (Cores)\n    \u003e scale_factor = 1.0000\n    \u003e array_size   = 16777216\n    \u003e Using CPU/GPU Device Synchronization method (cudaDeviceScheduleAuto)\n    \u003e mmap() allocating 64.00 Mbytes (generic page-aligned system memory)\n    \u003e cudaHostRegister() registering 64.00 Mbytes of generic allocated system memory\n    Starting Test\n    memcopy:        8.83\n    kernel:         5.77\n    non-streamed:   11.08\n    4 streams:      5.41\n    -------------------------------\n    \n    $ ~/NVIDIA_CUDA-9.2_Samples/0_Simple/matrixMul/matrixMul\n    [Matrix Multiply Using CUDA] - Starting...\n    GPU Device 0: \"Tesla M60\" with compute capability 5.2\n    MatrixA(320,320), MatrixB(640,320)\n    Computing result using CUDA Kernel...\n    done\n    Performance= 309.81 GFlop/s, Time= 0.423 msec, Size= 131072000 Ops, WorkgroupSize= 1024 threads/block\n    Checking computed result for correctness: Result = PASS\n    NOTE: The CUDA Samples are not meant for performancemeasurements. Results may vary when GPU Boost is enabled.\n```\n\nManual instructions for installing Cuda drivers and utilities are found in the Nvidia Cuda Linux installation guide.\n\n## Resources\n\n1. [GPU support in Red Hat OpenStack Platform](https://access.redhat.com/solutions/3080471)\n2. [Bugzilla RFE for documentation on confiuring GPUs via PCI passthrough in OpenStack Platform](https://bugzilla.redhat.com/show_bug.cgi?id=1430337)\n3. [OpenStack Nova Configure PCI Passthrough](https://docs.openstack.org/nova/queens/admin/pci-passthrough.html)\n4. [KVM virtual machine GPU configuration](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/virtualization_deployment_and_administration_guide/chap-guest_virtual_machine_device_configuration#sect-device-GPU)\n5. [Nvidia Cuda Linux installation guide](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#runfile-installation)\n6. [DKMS support in Red Hat Enterprise Linux](https://access.redhat.com/solutions/1132653)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjliberma%2Fnewton-gpu","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjliberma%2Fnewton-gpu","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjliberma%2Fnewton-gpu/lists"}