{"id":47888786,"url":"https://github.com/vuvietnguyenit/gpuxray","last_synced_at":"2026-04-04T02:26:58.463Z","repository":{"id":343900084,"uuid":"1142186861","full_name":"vuvietnguyenit/gpuxray","owner":"vuvietnguyenit","description":"A lightweight GPU observability tool focused on per-process GPU metrics, with optional deep tracing powered by eBPF.","archived":false,"fork":false,"pushed_at":"2026-03-12T08:49:30.000Z","size":449,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-03-12T14:56:26.869Z","etag":null,"topics":["ebpf","gpu","gpu-monitoring","tracing"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vuvietnguyenit.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-01-26T04:07:15.000Z","updated_at":"2026-03-12T08:49:35.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/vuvietnguyenit/gpuxray","commit_stats":null,"previous_names":["vuvietnguyenit/gpuxray"],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/vuvietnguyenit/gpuxray","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vuvietnguyenit%2Fgpuxray","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vuvietnguyenit%2Fgpuxray/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vuvietnguyenit%2Fgpuxray/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vuvietnguyenit%2Fgpuxray/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vuvietnguyenit","download_url":"https://codeload.github.com/vuvietnguyenit/gpuxray/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vuvietnguyenit%2Fgpuxray/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31385306,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-04T01:22:39.193Z","status":"online","status_checked_at":"2026-04-04T02:00:07.569Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ebpf","gpu","gpu-monitoring","tracing"],"created_at":"2026-04-04T02:26:56.699Z","updated_at":"2026-04-04T02:26:58.451Z","avatar_url":"https://github.com/vuvietnguyenit.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# GPUXRAY\n\n![GitHub release](https://img.shields.io/github/v/release/vuvietnguyenit/gpuxray)\n![Go Version](https://img.shields.io/github/go-mod/go-version/vuvietnguyenit/gpuxray)\n![License](https://img.shields.io/github/license/vuvietnguyenit/gpuxray?style=flat)\n\n\u003cp align=\"center\"\u003e\n  \u003cpicture\u003e\n    \u003csource media=\"(prefers-color-scheme: dark)\" srcset=\"./docs/assets/gpuxray-dark.svg\"\u003e\n    \u003cimg src=\"./docs/assets/gpuxray-light.svg\" width=\"420\"\u003e\n  \u003c/picture\u003e\n\u003c/p\u003e\n\n\nAn opensource observability tool for debugging GPU workloads on Linux servers.\n\nIt traces CUDA activity using eBPF and provides:\n- per-process GPU metrics\n- GPU memory leak detection\n- Prometheus metrics for monitoring systems.\n\nThis tool is inspired by [pidstat](https://man7.org/linux/man-pages/man1/pidstat.1.html) but designed for GPU monitoring.\nGPUXRAY is designed for AI/ML workloads running on GPU servers.\n\n### Why use gpuxray?\n- `gpuxray` provides GPU observability at the process level, which is not fully supported by [DCGM exporter](https://github.com/NVIDIA/dcgm-exporter). It also leverages eBPF to perform deeper tracing of GPU workloads.\n- It leverages eBPF to trace CUDA activity from the kernel, enabling low-overhead and deep tracing of GPU workloads.\n\n## Usecases\n- Works well for tracing and get stats from processes that use GPU resources through the CUDA API (e.g., ML jobs, AI workloads).\n- Exposes Prometheus metrics fpr GPU resources associated with each PID.\n- Very convinient for detecting GPU memory leaked. It can show stack traces of leaked GPU memory blocks and identify the CUDA functions responsible for allocations that were not freed.\n\n## Notice\n- Currently, this tool only inspects PIDs that use the CUDA Driver API. Processes that use the CUDA Runtime API may be omitted.\n- Requires Linux kernel version \u003e= 5.6. Kernel versions in the 6.x series are recommended.\n- Currently supports only the `amd64` CPU architecture.\n\n## Table of Contents\n- [GPUXRAY](#gpuxray)\n    - [Why use gpuxray?](#why-use-gpuxray)\n  - [Usecases](#usecases)\n  - [Notice](#notice)\n  - [Table of Contents](#table-of-contents)\n  - [Architecture](#architecture)\n  - [Install](#install)\n    - [Binary](#binary)\n    - [Docker](#docker)\n    - [Build from source](#build-from-source)\n  - [Quickstart](#quickstart)\n    - [Run GPU exporter](#run-gpu-exporter)\n    - [Memory statistics](#memory-statistics)\n    - [Show memory-leaked stacktraces](#show-memory-leaked-stacktraces)\n    - [GPUXRAY for Kubernetes](#gpuxray-for-kubernetes)\n    - [GPUXRAY for Docker](#gpuxray-for-docker)\n  - [Debugging](#debugging)\n  - [Contributing](#contributing)\n\n\n## Architecture\n\nGPUXRAY collects GPU information using multiple techniques:\n\n1. **NVML** – retrieves GPU metrics per process\n2. **eBPF** – traces CUDA calls\n3. **Go exporter** – exposes metrics for Prometheus\n\n\n## Install\n\n### Binary\nInstall gpuxray easily with one command:\n```sh\ncurl -s https://raw.githubusercontent.com/vuvietnguyenit/gpuxray/main/install.sh | sh\n```\n### Docker\n\n```bash\ndocker pull ghcr.io/vuvietnguyenit/gpuxray:latest\n```\n### Build from source\n\n```sh\ngit clone https://github.com/vuvietnguyenit/gpuxray\ncd gpuxray\ngo build -o gpuxray\n```\n\n## Quickstart\n### Run GPU exporter\n\nRunning the exporter exposes metrics related to processes using GPU resources on the server.\n```sh\n# gpuxray mon\n```\nMetric definitions are available in: [metrics.txt](./metrics.txt)\n\n\u003cdetails\u003e\n\u003csummary\u003eExample result\u003c/summary\u003e\n\n```text\ncurl http://localhost:2112/metrics\n...\n# HELP gpu_free_memory_bytes Remaining available GPU memory for the process in bytes.\n# TYPE gpu_free_memory_bytes gauge                                                                                  \ngpu_free_memory_bytes{gpu=\"GPU-47def375-4603-e5fa-82d3-c7cddc81e65a\",gpu_index=\"0\",hostname=\"gpu1.itim.vn\"} 9.903734784e+09\n# HELP gpu_process_active 1 for each process currently using a GPU.             \n# TYPE gpu_process_active gauge\ngpu_process_active{args=\"/usr/bin/gnome-shell\",comm=\"gnome-shell\",gpu=\"GPU-47def375-4603-e5fa-82d3-c7cddc81e65a\",gpu_index=\"0\",hostname=\"gpu1.itim.vn\",pid=\"3112\"} 1\ngpu_process_active{args=\"/usr/lib/xorg/Xorg vt1 -displayfd 3 -auth /run/user/120/gdm/Xauthority -nolisten tcp -background none -noreset -keeptty -novtswitch -verbose 3\",comm=\"Xorg\",gpu=\"GPU-47def375-4603-e5fa-82d3-c7cddc81e65a\",gpu_\nindex=\"0\",hostname=\"gpu1.itim.vn\",pid=\"2912\"} 1\ngpu_process_active{args=\"python -m src.models.classifier --train-file /shared_storage/ailab/intent-classifier/train/raw-click.gz --valid-file /shared_storage/ailab/intent-classifier/valid/raw-click.gz --model-name vinai/phobert-base\n-v2\",comm=\"python\",gpu=\"GPU-47def375-4603-e5fa-82d3-c7cddc81e65a\",gpu_index=\"0\",hostname=\"gpu1.itim.vn\",pid=\"401948\"} 1\n# HELP gpu_process_sm_utilization_percent GPU SM utilisation of the process (0–100). Requires NVML r470+ drivers; returns 0 on older drivers.\n# TYPE gpu_process_sm_utilization_percent gauge\ngpu_process_sm_utilization_percent{args=\"/usr/bin/gnome-shell\",comm=\"gnome-shell\",gpu=\"GPU-47def375-4603-e5fa-82d3-c7cddc81e65a\",gpu_index=\"0\",hostname=\"gpu1.itim.vn\",pid=\"3112\"} 0\ngpu_process_sm_utilization_percent{args=\"/usr/lib/xorg/Xorg vt1 -displayfd 3 -auth /run/user/120/gdm/Xauthority -nolisten tcp -background none -noreset -keeptty -novtswitch -verbose 3\",comm=\"Xorg\",gpu=\"GPU-47def375-4603-e5fa-82d3-c7\ncddc81e65a\",gpu_index=\"0\",hostname=\"gpu1.itim.vn\",pid=\"2912\"} 0\ngpu_process_sm_utilization_percent{args=\"python -m src.models.classifier --train-file /shared_storage/ailab/intent-classifier/train/raw-click.gz --valid-file /shared_storage/ailab/intent-classifier/valid/raw-click.gz --model-name vi\nnai/phobert-base-v2\",comm=\"python\",gpu=\"GPU-47def375-4603-e5fa-82d3-c7cddc81e65a\",gpu_index=\"0\",hostname=\"gpu1.itim.vn\",pid=\"401948\"} 86\n# HELP gpu_process_used_memory_bytes GPU memory consumed by the process in bytes.\n# TYPE gpu_process_used_memory_bytes gauge\ngpu_process_used_memory_bytes{args=\"/usr/bin/gnome-shell\",comm=\"gnome-shell\",gpu=\"GPU-47def375-4603-e5fa-82d3-c7cddc81e65a\",gpu_index=\"0\",hostname=\"gpu1.itim.vn\",pid=\"3112\"} 1.1296768e+07\ngpu_process_used_memory_bytes{args=\"/usr/lib/xorg/Xorg vt1 -displayfd 3 -auth /run/user/120/gdm/Xauthority -nolisten tcp -background none -noreset -keeptty -novtswitch -verbose 3\",comm=\"Xorg\",gpu=\"GPU-47def375-4603-e5fa-82d3-c7cddc8\n1e65a\",gpu_index=\"0\",hostname=\"gpu1.itim.vn\",pid=\"2912\"} 1.0575872e+07\ngpu_process_used_memory_bytes{args=\"python -m src.models.classifier --train-file /shared_storage/ailab/intent-classifier/train/raw-click.gz --valid-file /shared_storage/ailab/intent-classifier/valid/raw-click.gz --model-name vinai/p\nhobert-base-v2\",comm=\"python\",gpu=\"GPU-47def375-4603-e5fa-82d3-c7cddc81e65a\",gpu_index=\"0\",hostname=\"gpu1.itim.vn\",pid=\"401948\"} 2.3716691968e+10\n# HELP gpu_total_memory_bytes Total GPU memory available in bytes.\n# TYPE gpu_total_memory_bytes gauge\ngpu_total_memory_bytes{gpu=\"GPU-47def375-4603-e5fa-82d3-c7cddc81e65a\",gpu_index=\"0\",hostname=\"gpu1.itim.vn\"} 3.4190917632e+10\n# HELP gpu_used_memory_bytes GPU memory currently allocated in bytes.\n# TYPE gpu_used_memory_bytes gauge\ngpu_used_memory_bytes{gpu=\"GPU-47def375-4603-e5fa-82d3-c7cddc81e65a\",gpu_index=\"0\",hostname=\"gpu1.itim.vn\"} 2.4287182848e+10\n...\n```\n\u003c/details\u003e\n\n### Memory statistics\n\nThis command reports statistics about GPU memory usage of process.\n\n```sh\n./gpuxray memtrace -p 2806854\n```\n\u003cdetails\u003e\n\u003csummary\u003eExample result\u003c/summary\u003e\n\n```sh\n# ./gpuxray memtrace -p 2806854\nTIME       PID      USER     GPU  INUSE_MB     AL_CNT   FR_CNT   COMM            \n12:03:24   2806854  root     0    512 B        199      198      python3         \n12:03:29   2806854  root     0    512 B        402      401      python3         \n12:03:34   2806854  root     0    512 B        607      606      python3         \n12:03:39   2806854  root     0    1.00 KiB     802      800      python3         \n12:03:44   2806854  root     0    1.00 KiB     994      992      python3         \n12:03:49   2806854  root     0    2.00 KiB     1197     1193     python3    \n```\n\u003c/details\u003e\n\nTo see the meaning of each column, run: `./gpuxray memtrace -h` flag to see more information\n\n### Show memory-leaked stacktraces\n\nThis command prints stack traces responsible for leaked GPU memory allocations.\n\n```sh\n./gpuxray memtrace -p 332361 -i 1 --print-stacks\n```\n\u003cdetails\u003e\n\u003csummary\u003eExample result\u003c/summary\u003e\n\n```sh\n# ./gpuxray memtrace -p 332361 -i 1 --print-stacks\n2026-03-04T15:59:44+07:00\n[1] PID: 332361   GPU: 0   StackID: 1908    Remaining Blocks: 1       TotalBytes: 512 B     \n  #00  0x71263447d86e      libcudart_static_5382377d5c772c9d197c0cda9fd9742ee6ad893c\n  #01  0x7126344491c3      libcudart_static_f74e2f2bcf2cf49bd1a61332e1d15bd1e748f9cf\n  #02  0x71263448d993      cudaMalloc\n  #03  0x712634420cde      __pyx_f_13cupy_backends_4cuda_3api_7runtime_malloc(unsigned long, int)\n\n2026-03-04T15:59:45+07:00\n[1] PID: 332361   GPU: 0   StackID: 1908    Remaining Blocks: 1       TotalBytes: 512 B     \n  #00  0x71263447d86e      libcudart_static_5382377d5c772c9d197c0cda9fd9742ee6ad893c\n  #01  0x7126344491c3      libcudart_static_f74e2f2bcf2cf49bd1a61332e1d15bd1e748f9cf\n  #02  0x71263448d993      cudaMalloc\n  #03  0x712634420cde      __pyx_f_13cupy_backends_4cuda_3api_7runtime_malloc(unsigned long, int)\n\n2026-03-04T15:59:46+07:00\n[1] PID: 332361   GPU: 0   StackID: 1908    Remaining Blocks: 2       TotalBytes: 1.00 KiB  \n  #00  0x71263447d86e      libcudart_static_5382377d5c772c9d197c0cda9fd9742ee6ad893c\n  #01  0x7126344491c3      libcudart_static_f74e2f2bcf2cf49bd1a61332e1d15bd1e748f9cf\n  #02  0x71263448d993      cudaMalloc\n  #03  0x712634420cde      __pyx_f_13cupy_backends_4cuda_3api_7runtime_malloc(unsigned long, int)\n\n^C2026/03/04 15:59:46 Received signal, exiting..\n```\n\u003c/details\u003e\n\n### GPUXRAY for Kubernetes\nFollow [kubernetes.md](./docs/kubernetes.md)\n\n### GPUXRAY for Docker\nFollow [docker.md](./docs/docker.md)\n\n## Debugging\nFollow [debugging.md](./docs/debugging.md)\n\n## Contributing\nContributions are welcome. Feel free to open issues or submit pull requests.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvuvietnguyenit%2Fgpuxray","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvuvietnguyenit%2Fgpuxray","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvuvietnguyenit%2Fgpuxray/lists"}