{"id":20983022,"url":"https://github.com/tkestack/gpu-exporter","last_synced_at":"2025-05-14T16:31:57.097Z","repository":{"id":48836288,"uuid":"360377421","full_name":"tkestack/gpu-exporter","owner":"tkestack","description":null,"archived":false,"fork":false,"pushed_at":"2021-07-09T08:22:35.000Z","size":7581,"stargazers_count":5,"open_issues_count":1,"forks_count":12,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-04-02T20:38:35.092Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tkestack.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-04-22T03:12:18.000Z","updated_at":"2024-11-08T20:20:16.000Z","dependencies_parsed_at":"2022-09-23T22:42:27.297Z","dependency_job_id":null,"html_url":"https://github.com/tkestack/gpu-exporter","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tkestack%2Fgpu-exporter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tkestack%2Fgpu-exporter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tkestack%2Fgpu-exporter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tkestack%2Fgpu-exporter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tkestack","download_url":"https://codeload.github.com/tkestack/gpu-exporter/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254183028,"owners_count":22028403,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-19T05:47:28.455Z","updated_at":"2025-05-14T16:31:54.666Z","avatar_url":"https://github.com/tkestack.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"# NVIDIA GPU Monitoring Tools\n\nThis repository contains Golang bindings and DCGM-Exporter for gathering GPU telemetry in Kubernetes.\n\n## Bindings\n\nGolang bindings are provided for the following two libraries:\n- [NVIDIA Management Library (NVML)](https://docs.nvidia.com/deploy/nvml-api/nvml-api-reference.html#nvml-api-reference) is a C-based API for monitoring and managing NVIDIA GPU devices.\n- [NVIDIA Data Center GPU Manager (DCGM)](https://developer.nvidia.com/dcgm) is a set of tools for managing and monitoring NVIDIA GPUs in cluster environments. It's a low overhead tool suite that performs a variety of functions on each host system including active health monitoring, diagnostics, system validation, policies, power and clock management, group configuration and accounting.\n\nYou will also find samples for both of these bindings in this repository.\n\n## New Feature\n\nThis is based on [NVIDIA/gpu-monitoring-tools](https://github.com/NVIDIA/gpu-monitoring-tools). The original monitoring tools can only monitor specific NVIDIA GPU card, with the Kubernetes pods name listed which built by [NVIDIA/k8s-device-plugin](https://github.com/NVIDIA/k8s-device-plugin).\n\nThis repository allows to monitor the utilization such as sm, dec, env and memory of Kubernetes pods and containers, which may be built by any third party of gpu device plugin. For example:\n\n```\nDCGM_FI_K8S_MEM_COPY_UTIL{gpu=\"0\",UUID=\"GPU-de4b1bb0-3ec3-67ed-b3e2-c32d8546e818\",device=\"nvidia0\",container=\"benchmark-0\",namespace=\"default\",pod=\"benchmark-0\"} 30\nDCGM_FI_K8S_ENC_UTIL{gpu=\"0\",UUID=\"GPU-de4b1bb0-3ec3-67ed-b3e2-c32d8546e818\",device=\"nvidia0\",container=\"benchmark-0\",namespace=\"default\",pod=\"benchmark-0\"} 0\nDCGM_FI_K8S_DEC_UTIL{gpu=\"0\",UUID=\"GPU-de4b1bb0-3ec3-67ed-b3e2-c32d8546e818\",device=\"nvidia0\",container=\"benchmark-0\",namespace=\"default\",pod=\"benchmark-0\"} 0\nDCGM_FI_K8S_GPU_UTIL{gpu=\"0\",UUID=\"GPU-de4b1bb0-3ec3-67ed-b3e2-c32d8546e818\",device=\"nvidia0\",container=\"benchmark-0\",namespace=\"default\",pod=\"benchmark-0\"} 47\nDCGM_FI_K8S_GPU_UTIL{gpu=\"0\",UUID=\"GPU-de4b1bb0-3ec3-67ed-b3e2-c32d8546e818\",device=\"nvidia0\",container=\"benchmark-1\",namespace=\"default\",pod=\"benchmark-1\"} 17\nDCGM_FI_K8S_MEM_COPY_UTIL{gpu=\"0\",UUID=\"GPU-de4b1bb0-3ec3-67ed-b3e2-c32d8546e818\",device=\"nvidia0\",container=\"benchmark-1\",namespace=\"default\",pod=\"benchmark-1\"} 10\nDCGM_FI_K8S_ENC_UTIL{gpu=\"0\",UUID=\"GPU-de4b1bb0-3ec3-67ed-b3e2-c32d8546e818\",device=\"nvidia0\",container=\"benchmark-1\",namespace=\"default\",pod=\"benchmark-1\"} 0\nDCGM_FI_K8S_DEC_UTIL{gpu=\"0\",UUID=\"GPU-de4b1bb0-3ec3-67ed-b3e2-c32d8546e818\",device=\"nvidia0\",container=\"benchmark-1\",namespace=\"default\",pod=\"benchmark-1\"} 0\n```\n\nAbove outputs the utilization of all Kubernetes pods and containers run by some other K8S device plugin for GPU sharing.\n\n## DCGM-Exporter\n\nThe repository also contains DCGM-Exporter. It exposes GPU metrics exporter for [Prometheus](https://prometheus.io/) leveraging [NVIDIA DCGM](https://developer.nvidia.com/dcgm).\n\n### Quickstart\n\nTo gather metrics on a GPU node, simply start the `dcgm-exporter` container:\n```\n$ docker run -d --gpus all --rm -p 9400:9400 nvidia/dcgm-exporter:2.0.13-2.1.1-ubuntu18.04\n$ curl localhost:9400/metrics\n# HELP DCGM_FI_DEV_SM_CLOCK SM clock frequency (in MHz).\n# TYPE DCGM_FI_DEV_SM_CLOCK gauge\n# HELP DCGM_FI_DEV_MEM_CLOCK Memory clock frequency (in MHz).\n# TYPE DCGM_FI_DEV_MEM_CLOCK gauge\n# HELP DCGM_FI_DEV_MEMORY_TEMP Memory temperature (in C).\n# TYPE DCGM_FI_DEV_MEMORY_TEMP gauge\n...\nDCGM_FI_DEV_SM_CLOCK{gpu=\"0\", UUID=\"GPU-604ac76c-d9cf-fef3-62e9-d92044ab6e52\"} 139\nDCGM_FI_DEV_MEM_CLOCK{gpu=\"0\", UUID=\"GPU-604ac76c-d9cf-fef3-62e9-d92044ab6e52\"} 405\nDCGM_FI_DEV_MEMORY_TEMP{gpu=\"0\", UUID=\"GPU-604ac76c-d9cf-fef3-62e9-d92044ab6e52\"} 9223372036854775794\n...\n```\n\n### Quickstart on Kubernetes\n\nNote: Consider using the [NVIDIA GPU Operator](https://github.com/NVIDIA/gpu-operator) rather than DCGM-Exporter directly.\n\nEnsure you have already setup your cluster with the [default runtime as NVIDIA](https://github.com/NVIDIA/nvidia-container-runtime#docker-engine-setup).\n\nThe recommended way to install DCGM-Exporter is to use the Helm chart: \n```\n$ helm repo add gpu-helm-charts \\\n  https://nvidia.github.io/gpu-monitoring-tools/helm-charts\n```\nUpdate the repo:\n```\n$ helm repo update\n```\nAnd install the chart:\n```\n$ helm install \\ \n    --generate-name \\ \n    gpu-helm-charts/dcgm-exporter\n```\n\nOnce the `dcgm-exporter` pod is deployed, you can use port forwarding to obtain metrics quickly:\n\n\n```\n$ kubectl create -f https://raw.githubusercontent.com/NVIDIA/gpu-monitoring-tools/master/dcgm-exporter.yaml\n\n# Let's get the output of a random pod:\n$ NAME=$(kubectl get pods -l \"app.kubernetes.io/name=dcgm-exporter\" \\\n                         -o \"jsonpath={ .items[0].metadata.name}\")\n\n$ kubectl port-forward $NAME 8080:9400 \u0026\n$ curl -sL http://127.0.01:8080/metrics\n# HELP DCGM_FI_DEV_SM_CLOCK SM clock frequency (in MHz).\n# TYPE DCGM_FI_DEV_SM_CLOCK gauge\n# HELP DCGM_FI_DEV_MEM_CLOCK Memory clock frequency (in MHz).\n# TYPE DCGM_FI_DEV_MEM_CLOCK gauge\n# HELP DCGM_FI_DEV_MEMORY_TEMP Memory temperature (in C).\n# TYPE DCGM_FI_DEV_MEMORY_TEMP gauge\n...\nDCGM_FI_DEV_SM_CLOCK{gpu=\"0\", UUID=\"GPU-604ac76c-d9cf-fef3-62e9-d92044ab6e52\",container=\"\",namespace=\"\",pod=\"\"} 139\nDCGM_FI_DEV_MEM_CLOCK{gpu=\"0\", UUID=\"GPU-604ac76c-d9cf-fef3-62e9-d92044ab6e52\",container=\"\",namespace=\"\",pod=\"\"} 405\nDCGM_FI_DEV_MEMORY_TEMP{gpu=\"0\", UUID=\"GPU-604ac76c-d9cf-fef3-62e9-d92044ab6e52\",container=\"\",namespace=\"\",pod=\"\"} 9223372036854775794\n...\n\n```\nTo integrate DCGM-Exporter with Prometheus and Grafana, see the full instructions in the [user guide](https://docs.nvidia.com/datacenter/cloud-native/kubernetes/dcgme2e.html#gpu-telemetry). \n`dcgm-exporter` is deployed as part of the GPU Operator. To get started with integrating with Prometheus, check the Operator [user guide](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/getting-started.html#gpu-telemetry).\n\n### Building from Source\n\n`dcgm-exporter` is actually fairly straightforward to build and use.\nEnsure you have the following:\n- [Golang \u003e= 1.14 installed](https://golang.org/)\n- [DCGM installed](https://developer.nvidia.com/dcgm)\n\n```\n$ git clone https://github.com/NVIDIA/gpu-monitoring-tools.git\n$ cd gpu-monitoring-tools\n$ make binary\n$ sudo make install\n...\n$ dcgm-exporter \u0026\n$ curl localhost:9400/metrics\n# HELP DCGM_FI_DEV_SM_CLOCK SM clock frequency (in MHz).\n# TYPE DCGM_FI_DEV_SM_CLOCK gauge\n# HELP DCGM_FI_DEV_MEM_CLOCK Memory clock frequency (in MHz).\n# TYPE DCGM_FI_DEV_MEM_CLOCK gauge\n# HELP DCGM_FI_DEV_MEMORY_TEMP Memory temperature (in C).\n# TYPE DCGM_FI_DEV_MEMORY_TEMP gauge\n...\nDCGM_FI_DEV_SM_CLOCK{gpu=\"0\", UUID=\"GPU-604ac76c-d9cf-fef3-62e9-d92044ab6e52\"} 139\nDCGM_FI_DEV_MEM_CLOCK{gpu=\"0\", UUID=\"GPU-604ac76c-d9cf-fef3-62e9-d92044ab6e52\"} 405\nDCGM_FI_DEV_MEMORY_TEMP{gpu=\"0\", UUID=\"GPU-604ac76c-d9cf-fef3-62e9-d92044ab6e52\"} 9223372036854775794\n...\n```\n\n### Changing Metrics\n\nWith `dcgm-exporter` you can configure which fields are collected by specifying a custom CSV file.\nYou will find the default CSV file under `etc/dcgm-exporter/default-counters.csv` in the repository, which is copied on your system or container at \n`/etc/dcgm-exporter/default-counters.csv`\n\nThe format of this file is pretty straightforward:\n```\n# Format,,\n# If line starts with a '#' it is considered a comment,,\n# DCGM FIELD, Prometheus metric type, help message\n\n# Clocks,,\nDCGM_FI_DEV_SM_CLOCK,  gauge, SM clock frequency (in MHz).\nDCGM_FI_DEV_MEM_CLOCK, gauge, Memory clock frequency (in MHz).\n```\n\nA custom csv file can be specified using the `-f` option or `--collectors` as follows:\n```\n$ dcgm-exporter -f /tmp/custom-collectors.csv\n```\n\nNotes:\n- Always make sure your entries have 3 commas (',')\n- The complete list of counters that can be collected can be found on the DCGM API reference manual: https://docs.nvidia.com/datacenter/dcgm/latest/dcgm-api/group__dcgmFieldIdentifiers.html\n\n### What about a Grafana Dashboard?\n\nYou can find the official NVIDIA DCGM-Exporter dashboard here: https://grafana.com/grafana/dashboards/12239\n\nYou will also find the `json` file on this repo under `grafana/dcgm-exporter-dashboard.json`\n\nPull requests are accepted!\n\n## Issues and Contributing\n\n[Checkout the Contributing document!](CONTRIBUTING.md)\n\n* Please let us know by [filing a new issue](https://github.com/NVIDIA/gpu-monitoring-tools/issues/new)\n* You can contribute by opening a [pull request](https://gitlab.com/nvidia/container-toolkit/gpu-monitoring-tools)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftkestack%2Fgpu-exporter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftkestack%2Fgpu-exporter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftkestack%2Fgpu-exporter/lists"}