{"id":31632800,"url":"https://github.com/j3soon/bluefield-dpu-setup-notes","last_synced_at":"2026-02-14T07:31:50.217Z","repository":{"id":284565218,"uuid":"941841877","full_name":"j3soon/bluefield-dpu-setup-notes","owner":"j3soon","description":"Unofficial notes for setting up and configuring NVIDIA BlueField DPUs on custom server systems (non-DGX platforms).","archived":false,"fork":false,"pushed_at":"2025-09-05T05:53:57.000Z","size":1848,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-11-18T01:23:13.265Z","etag":null,"topics":["bluefield","bluefield-3","data-processing-unit","dpu","hpc","proxmox-ve"],"latest_commit_sha":null,"homepage":"https://tutorial.j3soon.com/hpc/extras/bluefield-dpu-setup-notes/","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/j3soon.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-03-03T06:24:02.000Z","updated_at":"2025-09-07T07:28:54.000Z","dependencies_parsed_at":"2025-03-26T14:55:54.113Z","dependency_job_id":"abfc28cf-8f7b-463a-92bc-8a40dc479e3f","html_url":"https://github.com/j3soon/bluefield-dpu-setup-notes","commit_stats":null,"previous_names":["j3soon/bluefield-dpu-setup-notes"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/j3soon/bluefield-dpu-setup-notes","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/j3soon%2Fbluefield-dpu-setup-notes","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/j3soon%2Fbluefield-dpu-setup-notes/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/j3soon%2Fbluefield-dpu-setup-notes/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/j3soon%2Fbluefield-dpu-setup-notes/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/j3soon","download_url":"https://codeload.github.com/j3soon/bluefield-dpu-setup-notes/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/j3soon%2Fbluefield-dpu-setup-notes/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29439500,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-14T07:24:13.446Z","status":"ssl_error","status_checked_at":"2026-02-14T07:23:58.969Z","response_time":53,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bluefield","bluefield-3","data-processing-unit","dpu","hpc","proxmox-ve"],"created_at":"2025-10-06T23:05:26.119Z","updated_at":"2026-02-14T07:31:50.204Z","avatar_url":"https://github.com/j3soon.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# BlueField DPU Setup Notes\n\nUnofficial notes for setting up and configuring NVIDIA BlueField DPUs on custom server systems (non-DGX platforms), including support for Proxmox VE.\n\n## Terminology\n\nLinux Systems (Host/DPU/BMC):\n\n- `Host`: The system running on the server.\n  - In the optional Proxmox VE section, it will be further divided into `PVE` and `VM`.\n  - In the remaining sections, the host refers to the server's operating system, regardless of whether it's running directly on hardware or within the VM of Proxmox VE.\n- `DPU`: The system running on the DPU.\n- `BMC`: The system on the board management controller of the DPU. This is an independent system that provides out-of-band management capabilities, separate from the DPU's main operating system.\n\n## Hardware\n\n- Server: G493-ZB3-AAP1-rev-1x [[ref](https://www.gigabyte.com/us/Enterprise/GPU-Server/G493-ZB3-AAP1-rev-1x)]\n- BlueField-3 DPU: B3210E 900-9D3B6-00SC-EA0 [[ref](https://docs.nvidia.com/networking/display/bf3dpu)]\n  - BMC Management Interface (RJ45): Ethernet Cable [[ref](https://docs.nvidia.com/networking/display/bluefieldbmcv2401/connecting+to+bmc+interfaces)]\n  - DPU Eth/IB Port 0 (QSFP112): InfiniBand Cable [[ref](https://docs.nvidia.com/doca/archive/2-6-0/dpu+functional+diagram/index.html)]\n  - DPU Eth/IB Port 1 (QSFP112): Empty\n- V100 GPU: Tesla V100-PCIE-16GB\n\nReferences:\n- [Supported Servers and Power Cords](https://docs.nvidia.com/networking/display/bf3dpu/supported-servers-and-power-cords)\n- [BlueField-3 Validated and Supported Cables and Modules](https://docs.nvidia.com/doca/sdk/bluefield-3-validated-and-supported-cables-and-modules/index.html)\n- [BlueField-3 DPUs](https://docs.nvidia.com/networking/display/bf3dpu)\n\n## Hardware Setup\n\n\u003e Require a supplementary 8-pin ATX power supply connectivity available through the external power supply connector .\n\u003e\n\u003e Do not link the CPU power cable to the BlueField-3 DPU PCIe ATX power connector, as their pin configurations differ. **Using the CPU power cable in this manner is strictly prohibited and can potentially damage the BlueField-3 DPU**. Please refer to [External PCIe Power Supply Connector Pins](https://docs.nvidia.com/networking/display/bf3dpu) for the external PCIe power supply pins.\n\u003e\n\u003e -- [Hardware Installation and PCIe Bifurcation](https://docs.nvidia.com/networking/display/bf3dpu/hardware+installation+and+pcie+bifurcation)\n\n\u003e - DPU BMC 1GbE interface connected to the management network via ToR\n\u003e - Remote Management Controller (RMC) connected to DPU BMC 1GbE via ToR\n\u003e    \u003e **Info**  \n\u003e    \u003e RMC is the platform for data center infrastructure managers to manage DPUs.\n\u003e - DHCP server existing in the management network\n\u003e - An NVQual certified server\n\u003e\n\u003e -- [Prerequisites for Initial BlueField-3 Deployment](https://docs.nvidia.com/networking/display/bf3dpucontroller/bluefield-3+administrator+quick+start+guide#src-2729714701_BlueField3AdministratorQuickStartGuide-PrerequisitesforInitialBlueField-3Deployment)\n\nReferences:\n\n- [NVIDIA BlueField-3 Networking Platform User Guide](https://docs.nvidia.com/networking/display/bf3dpu)\n- [BlueField-3 Administrator Quick Start Guide](https://docs.nvidia.com/networking/display/bf3dpu/bluefield-3+administrator+quick+start+guide)\n- [Hardware Installation and PCIe Bifurcation](https://docs.nvidia.com/networking/display/bf3dpu/hardware+installation+and+pcie+bifurcation)\n\n## Software\n\n- (Optional) Proxmox VE 8.2.2\n- Host OS\n  - Operating System: Ubuntu 24.04.2 LTS\n  - Kernel: Linux 6.8.0-54-generic\n  - Architecture: x86-64\n  - DOCA-Host: 2.9.2 LTS [[ref](https://developer.nvidia.com/doca-archive)]\n- BMC\n  - Operating System: NVIDIA Moonraker/RoyB BMC (OpenBMC     Project Reference Distro) BF-24.01-5\n  - Kernel: Linux 5.15.50-e62bf17\n  - Architecture: arm\n- DPU\n  - Operating System: Ubuntu 22.04.5 LTS\n  - Kernel: Linux 5.15.0-1035-bluefield\n  - Architecture: arm64\n  - DOCA-BlueField: 2.9.2 [[ref](https://developer.nvidia.com/doca-archive)]\n  - Mode: DPU Mode [[ref](https://docs.nvidia.com/doca/sdk/bluefield+modes+of+operation/index.html)]\n  - DPU image and firmware: [[ref](https://docs.nvidia.com/networking/display/bluefield2dpuenug/bluefield+dpu+administrator+quick+start+guide)]\n    ```\n    $ sudo bfvcheck\n    Beginning version check...\n\n    -RECOMMENDED VERSIONS-\n    ATF: v2.2(release):4.9.2-14-geeb9a6f94\n    UEFI: 4.9.2-25-ge0f86cebd6\n    FW: 32.43.2566\n\n    -INSTALLED VERSIONS-\n    ATF: v2.2(release):4.9.2-14-geeb9a6f94\n    UEFI: 4.9.2-25-ge0f86cebd6\n    FW: 32.43.2566\n\n    Version check complete.\n    No issues found.\n    ```\n  - BlueField OS image version: [[ref](https://docs.nvidia.com/networking/display/bluefielddpubspv420/deploying+bluefield+software+using+bfb+from+host#src-80580925_DeployingBlueFieldSoftwareUsingBFBfromHost-VerifyBFBisInstalled)]\n    ```\n    $ cat /etc/mlnx-release\n    bf-bundle-2.9.2-31_25.02_ubuntu-22.04_prod\n    ```\n\n## (Optional) Proxmox VE Passthrough\n\n\u003e Please skip to [the next section](#system-testing) if Proxmox VE is not used.\n\n- IOMMU Setup\n\n  - Ensure that IOMMU (VT-d or AMD-Vi) is enabled in the BIOS/UEFI.\n\n    ```bash\n    lscpu | grep Virtualization\n    ```\n\n  - AMD enables it by default, check it using the following command:\n\n    ```bash\n    for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' \"$n\"; lspci -nns \"${d##*/}\"; done\n    ```\n\n    - It works when you see multiple groups and you can check which devices are properly isolated (no other devices in the same group except for PCI bridges) for PCI passthrough.\n\n  - If it cannot be enabled, modify the GRUB configuration. Locate `GRUB_CMDLINE_LINUX_DEFAULT`, and for AMD, set it to:\n\n    ```bash\n    bash GRUB_CMDLINE_LINUX_DEFAULT=\"quiet iommu=pt\"\n\n    # for Intel\n    GRUB_CMDLINE_LINUX_DEFAULT=\"quiet intel_iommu=on iommu=pt\"\n    ```\n\n    - Verify whether IOMMU is enabled (though it's uncertain if this method works) by using:\n\n      ```bash\n      dmesg | grep -e DMAR -e IOMMU\n\n      # works\n      DMAR: IOMMU enabled\n      ```\n\n  - check NIC info\n\n    ```bash\n    bash lspci -nn | grep -i bluefield\n    bash lspci -nn | grep -i nvidia\n    ```\n\n- Proxmox VE Setup\n\n  - Find PCI ID\n\n    ```bash\n    lspci -nn | grep -i mellanox\n    lspci -nn | grep -i nvidia\n\n    # Take note of the device's PCI address (e.g., 0000:03:00.0) and Vendor:Device ID (e.g., 1b4b:xxxx).\n    ```\n\n  - Check vfio module\n\n    ```bash\n    lsmod | grep vfio\n    ```\n\n    - enable if there is no vfio module:\n\n      ```bash\n      echo \"vfio\" \u003e\u003e /etc/modules\n      echo \"vfio_iommu_type1\" \u003e\u003e /etc/modules\n      echo \"vfio_pci\" \u003e\u003e /etc/modules\n      update-initramfs -u\n      reboot\n      dmesg | grep -i vfio\n      ```\n\n  - Configure VFIO: The BlueField card must be managed by `vfio-pci` to prevent the default driver from automatically loading.\n\n    ```bash\n    nano /etc/modprobe.d/vfio.conf\n\n    # vendor device\n    options vfio-pci ids=15b3:a2d2\n    # update\n    update-initramfs -u\n\n    # Not sure if this is needed\n    softdep mlx5_core pre: vfio-pci\n    # Blacklist default driver (edit `/etc/modprobe.d/pve-blacklist.conf`)\n    `blacklist mlx5_core`\n    ```\n\n  - Reboot the system and verify that the PCI device is bound to `vfio-pci`:\n\n    ```bash\n    lspci -nnk -d 1b4b:xxxx\n    ```\n\n- VM Setup\n\n  - Create or stop the target VM, add the following line in Proxmox Web UI or directly edit the VM configuration file (e.g. `/etc/pve/qemu-server/\u003cVMID\u003e.conf`), replace `0000:03:00.0` with the PCI address of your BlueField card.\n\n    ```bash\n    hostpci0: 0000:03:00.0,pcie=1\n    ```\n\n    - If the card has multiple functions (multi-function device), you can add `hostpci1`, `hostpci2`, etc. or add `multifunction=on` (adjust as needed).\n\n    - Check the VM\n\n      - `lspci -nn | grep -i nvidia`\n\n  - Appendix\n    - V100 Passthrough in Proxmox VE GUI: `Datacenter \u003e Resource Mappings \u003e Add`\n    - DPU Passthrough in Proxmox VE GUI: `VM \u003e Hardware \u003e Add \u003e PCI Device`\n\nReferences:\n\n- [PCI(e) Passthrough](https://pve.proxmox.com/wiki/PCI(e)_Passthrough)\n\n## Software Setup\n\n### Host\n\nExecute the following commands on the host.\n\nCheck PCI devices:\n\n```bash\nlspci -nn | grep -i mellanox\nlspci -nn | grep -i nvidia\n```\n\nInstall common packages:\n\n```bash\nsudo apt-get update\n# Install pv for viewing progress of the commands below\nsudo apt-get install -y pv\n```\n\n(Optional) Uninstall old DOCA-Host: [[ref](https://docs.nvidia.com/doca/sdk/doca-host+installation+and+upgrade/index.html)]\n\n```bash\nfor f in $( dpkg --list | grep -E 'doca|flexio|dpa-gdbserver|dpa-stats|dpaeumgmt' | awk '{print $2}' ); do echo $f ; sudo apt remove --purge $f -y ; done\nsudo /usr/sbin/ofed_uninstall.sh --force\nsudo apt-get autoremove\n```\n\nInstall DOCA-Host (DPU Driver) 2.9.2 LTS [[download](https://developer.nvidia.com/doca-2-9-2-download-archive?deployment_platform=Host-Server)]:\n\n```bash\n# DPU Driver (DOCA-Host)\nwget https://www.mellanox.com/downloads/DOCA/DOCA_v2.9.2/host/doca-host_2.9.2-012000-24.10-ubuntu2404_amd64.deb\nsudo dpkg -i doca-host_2.9.2-012000-24.10-ubuntu2404_amd64.deb\nsudo apt-get update\nsudo apt-get -y install doca-all\n# Check DOCA-Host\ndpkg -l | grep doca\n\n# GPU Driver \u0026 CUDA\nwget https://developer.download.nvidia.com/compute/cuda/12.8.0/local_installers/cuda_12.8.0_570.86.10_linux.run\nsudo sh cuda_12.8.0_570.86.10_linux.run\n# Check Driver\nnvidia-smi\n```\n\nFix `macsec` driver issue:\n\n\u003e Strangely, Ubuntu 24.04's kernel binary package doesn't seem to include the `macsec` driver, causing `mlx5_ib` not being able to load. This may be observed by running `sudo mst status -v`, `sudo dmesg | grep mlx5`, and `ibstatus`.\n\n\u003e 2025/06/29 Update:\n\u003e A easier solution seems to be:\n\u003e ```sh\n\u003e sudo apt-get install linux-modules-extra-$(uname -r)\n\u003e ```\n\u003e Then we don't need to build the `macsec` driver ourselves.\n\nTo fix this issue, we build the `macsec` driver ourselves:\n\n```bash\n# Download macsec from kernel source\nwget https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/drivers/net/macsec.c?h=v6.8 -O macsec.c\n\n# Create Makefile\ncat \u003c\u003c 'EOF' \u003e Makefile\nobj-m += macsec.o\nall:\n\tmake -C /lib/modules/$(shell uname -r)/build M=$(PWD) modules\nEOF\n\nmake\n\nsudo cp macsec.ko /lib/modules/$(uname -r)/kernel/drivers/net\nsudo depmod -a\n\n# macsec module should be available\nmodinfo macsec\nsudo modprobe macsec\nlsmod | grep macsec\n\n# Reload mlx5_core module\nsudo rmmod mlx5_core\nsudo modprobe mlx5_core\n```\n\n\u003e Make sure to re-compile the macsec module if you encounter the following error when running `sudo modprobe macsec`:\n\u003e \n\u003e ```\n\u003e modprobe: ERROR: could not insert 'macsec': Exec format error\n\u003e ```\n\nConnect to the DPU via RShim: [[ref](https://docs.nvidia.com/networking/display/bluefielddpuosv460/deploying+bluefield+software+using+bfb+from+host#src-2571331391_DeployingBlueFieldSoftwareUsingBFBfromHost-FirmwareUpgrade)]\n\n```bash\nsudo systemctl enable --now rshim\nsudo ip addr add 192.168.100.1/24 dev tmfifo_net0\nping 192.168.100.2\n# connect to the DPU\nssh ubuntu@192.168.100.2\n```\n\nChange DPU to IB mode: [[ref](https://docs.nvidia.com/networking/display/mftv422/using+mlxconfig#src-99400487_safe-id-VXNpbmdtbHhjb25maWctVXNpbmdtbHhjb25maWd0b1NldElCL0VUSFBhcmFtZXRlcnM)]\n\n```bash\n# Note that this can also be done on DPU\nsudo mlxconfig -d /dev/mst/mt41692_pciconf0 set LINK_TYPE_P1=1\nsudo mlxconfig -d /dev/mst/mt41692_pciconf0 set LINK_TYPE_P2=1\n# Cold reboot the machine\n```\n\nDeploying DPU OS Using BFB from Host: [[download](https://developer.nvidia.com/doca-2-9-2-download-archive?deployment_platform=BlueField\u0026deployment_package=BF-Bundle\u0026Distribution=Ubuntu\u0026version=22.04\u0026installer_type=BFB)] [[ref](https://docs.nvidia.com/networking/display/bluefielddpuosv470/deploying+bluefield+software+using+bfb+from+host#src-2821766645_DeployingBlueFieldSoftwareUsingBFBfromHost-InstallingUbuntuonBlueField)]\n\n```sh\n# update DOCA-BlueField to 2.9.2\nwget https://content.mellanox.com/BlueField/BFBs/Ubuntu22.04/bf-bundle-2.9.2-31_25.02_ubuntu-22.04_prod.bfb\nsudo bfb-install --bfb bf-bundle-2.9.2-31_25.02_ubuntu-22.04_prod.bfb --rshim /dev/rshim0\n```\n\n\u003e (Optional, Unconfirmed) Update DPU Firmware: [[download](https://developer.nvidia.com/doca-2-9-2-download-archive?deployment_platform=BlueField\u0026deployment_package=BF-FW-Bundle\u0026installer_type=BFB)]\n\u003e \n\u003e ```sh\n\u003e # update firmware\n\u003e wget https://content.mellanox.com/BlueField/FW-Bundle/bf-fwbundle-2.9.2-31_25.02-prod.bfb\n\u003e sudo bfb-install --bfb bf-fwbundle-2.9.2-31_25.02-prod.bfb --rshim rshim0\n\u003e ```\n\nOther DOCA tools and commands for debugging:\n\n```sh\ncd /opt/mellanox/doca/tools\ndoca_caps --list-devs\ndoca_bench --device 01:00.0 --query device-capabilities\n```\n\n```sh\nsudo ibdev2netdev -v\nsudo mlxlink -d /dev/mst/mt41692_pciconf0\n```\n\n### DPU\n\nExecute the following commands on the DPU.\n\n```bash\n# Check BlueField OS image version\ncat /etc/mlnx-release\n# Check DOCA-BlueField\ndpkg -l | grep doca\n```\n\nUpdate DPU firmware: [[ref](https://docs.nvidia.com/networking/display/mlnxenv24010331/changes+and+new+features#src-2591728722_ChangesandNewFeatures-CustomerAffectingChanges)] [[firmware-tools](https://network.nvidia.com/products/adapter-software/firmware-tools/)] [[flint](https://docs.nvidia.com/networking/display/mftv4310/flint+%E2%80%93+firmware+burning+tool)] [[mlxfwmanager](https://docs.nvidia.com/networking/display/mftv4310/mlxfwmanager+%E2%80%93+firmware+update+and+query+tool)]\n\n```bash\n# check firmware\nsudo mlxfwmanager --query\nsudo flint -d /dev/mst/mt41692_pciconf0 q\n\n# update firmware\nsudo /opt/mellanox/mlnx-fw-updater/mlnx_fw_updater.pl\n\n# force update\n# sudo /opt/mellanox/mlnx-fw-updater/mlnx_fw_updater.pl --force-fw-update\n\n# Need to cold reboot the machine\n```\n\nLaunch [OpenSM](https://docs.nvidia.com/networking/display/mlnxofedv461000/opensm) on DPU for using InfiniBand on host side. Before this step, running `ibstat` on host will show `State: Down` and `Physical state: LinkUp`. Running `ibstat` on host will show `State: Up` after this step.\n\n```sh\n# Get the `Node GUID` from the corresponding CA\nibstat\n\n# Run OpenSM with the Node GUID to recognize virtual ports on the host.\nsudo opensm -g \u003cDPU_IB_NODE_GUID\u003e -p 10\n# If there's another OpenSM running on other hosts, make sure to set the priority higher than those.\n# In our case, we have another OpenSM with priority 0 in the subnet, so we set our priority to 10.\n```\n\n\u003e  InfiniBand in DPU Mode\n\u003e\n\u003e In DPU Mode, when operating with an InfiniBand network, OpenSM must be executed from the BlueField Arm side rather than the host side. Similarly, InfiniBand management tools such as `sminfo`, `ibdev2netdev`, and `ibnetdiscover` can only be used from the BlueField Arm side and are not accessible from the host side.\n\u003e\n\u003e -- [BlueField Modes of Operation](https://docs.nvidia.com/doca/sdk/bluefield+modes+of+operation/index.html#src-3421781520_id-.BlueFieldModesofOperationv2.10.0-InfiniBandinDPUMode)\n\nResetting DPU: [[ref](https://docs.nvidia.com/networking/display/mftv4261lts/mstfwreset+%E2%80%93+loading+firmware+on+5th+generation+devices+tool)]\n\n```bash\n# Query for reset level required to load new firmware\nsudo mlxfwreset -d /dev/mst/mt*pciconf0 q\n```\n\n\u003e Output of the query command:\n\u003e\n\u003e ```\n\u003e Reset-levels:\n\u003e 0: Driver, PCI link, network link will remain up (\"live-Patch\")  -Supported     (default)\n\u003e 1: Only ARM side will not remain up (\"Immediate reset\").         -Not Supported\n\u003e 3: Driver restart and PCI reset                                  -Supported\n\u003e 4: Warm Reboot                                                   -Supported\n\u003e \n\u003e Reset-types (relevant only for reset-levels 1,3,4):\n\u003e 0: Full chip reset                                               -Supported     (default)\n\u003e 1: Phy-less reset (keep network port active during reset)        -Not Supported\n\u003e 2: NIC only reset (for SoC devices)                              -Not Supported\n\u003e 3: ARM only reset                                                -Not Supported\n\u003e 4: ARM OS shut down                                              -Not Supported\n\u003e \n\u003e Reset-sync (relevant only for reset-level 3):\n\u003e 0: Tool is the owner                                             -Not supported\n\u003e 1: Driver is the owner                                           -Supported     (default)\n\u003e ```\n\nDebugging:\n\n```bash\n# collect all debug message in host\nsudo /usr/sbin/sysinfo-snapshot.py\n```\n\nReferences:\n\n- [NVIDIA Firmware Tools (MFT)](https://network.nvidia.com/products/adapter-software/firmware-tools/)\n- [NVIDIA BlueField-3 DPU Firmware Download Center](https://network.nvidia.com/support/firmware/bluefield3/)\n\n### BMC\n\nIPMI:\n\n```bash\n# Check sensors\nipmitool sdr\n\n# Power control\nipmitool chassis power\n# chassis power Commands: status, on, off, cycle, reset, diag, soft\n\n# Check power status\nipmitool chassis status\n\n# Control the BMC itself\nipmitool mc\n```\n\nRedfish:\n\n```bash\n# Check BMC version\ncurl -k -u 'root:\u003cpassword\u003e' -H 'Content-Type: application/json' -X GET https://\u003cbmc_ip\u003e/redfish/v1/UpdateService/FirmwareInventory/BMC_Firmware\n```\n\nReferences:\n\n- [Connecting to BMC Interfaces](https://docs.nvidia.com/networking/display/bluefieldbmcv2501/connecting+to+bmc+interfaces)\n- [Reset Control](https://docs.nvidia.com/networking/display/bluefieldbmcv2410ltsu1/reset+control)\n- [Table of Common Redfish Commands](https://docs.nvidia.com/networking/display/bluefieldbmcv2410ltsu1/table+of+common+redfish+commands)\n- [NVIDIA BlueField Reset and Reboot Procedures](https://docs.nvidia.com/doca/sdk/nvidia+bluefield+reset+and+reboot+procedures/index.html)\n\n### Host2\n\nGiven another host connected with InfiniBand, you can ping it from the DPU:\n\nOn the other host `host2`:\n\n```sh\nibstat # check `Base lid`\nsudo ibping -S\n```\n\nOn the DPU:\n\n```sh\nsudo ibnetdiscover # You should see the same lid\nibstat # check `CA` and `Port`\nsudo ibping -C \u003cCA\u003e -P \u003cPORT\u003e -L \u003cLID\u003e\n# For an example:\n# sudo ibping -C mlx5_0 -P 1 -L 13\n```\n\nYou can also switch the server and client roles by running `ibping -S` on the DPU and `ibping -C \u003cCA\u003e -P \u003cPORT\u003e -L \u003cLID\u003e` on the other host.\n\n## Examples\n\nPlease refer to the [examples](./examples/README.md) for more details.\n\n## Contributors \u0026 Acknowledgements\n\nContributors: [@tsw303005](https://github.com/tsw303005), [@Aiden128](https://github.com/Aiden128), [@YiPrograms](https://github.com/YiPrograms), and [@j3soon](https://github.com/j3soon).\n\nThis note has been made possible through the support of [LSA Lab](https://github.com/NTHU-LSALAB), and [NVIDIA AI Technology Center (NVAITC)](https://github.com/NVAITC).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fj3soon%2Fbluefield-dpu-setup-notes","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fj3soon%2Fbluefield-dpu-setup-notes","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fj3soon%2Fbluefield-dpu-setup-notes/lists"}