{"id":16271076,"url":"https://github.com/okuvshynov/cubestat","last_synced_at":"2026-03-11T18:33:48.367Z","repository":{"id":151486700,"uuid":"599163606","full_name":"okuvshynov/cubestat","owner":"okuvshynov","description":"Horizon chart for CPU/GPU/Neural Engine utilization monitoring. Supports Apple M1-M4, Nvidia GPUs, AMD GPUs","archived":false,"fork":false,"pushed_at":"2025-07-28T16:58:43.000Z","size":2395,"stargazers_count":26,"open_issues_count":7,"forks_count":1,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-08-05T11:41:07.516Z","etag":null,"topics":["apple-silicon","command-line-tool","gpu","horizon","monitoring","neural-engine","nvidia-gpu","rocm"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/okuvshynov.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGES.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-02-08T15:30:18.000Z","updated_at":"2025-07-28T16:58:47.000Z","dependencies_parsed_at":null,"dependency_job_id":"b8ed281d-8a16-468f-bd84-b5c42b7a76c6","html_url":"https://github.com/okuvshynov/cubestat","commit_stats":null,"previous_names":[],"tags_count":28,"template":false,"template_full_name":null,"purl":"pkg:github/okuvshynov/cubestat","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/okuvshynov%2Fcubestat","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/okuvshynov%2Fcubestat/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/okuvshynov%2Fcubestat/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/okuvshynov%2Fcubestat/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/okuvshynov","download_url":"https://codeload.github.com/okuvshynov/cubestat/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/okuvshynov%2Fcubestat/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270148475,"owners_count":24535700,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-12T02:00:09.011Z","response_time":80,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apple-silicon","command-line-tool","gpu","horizon","monitoring","neural-engine","nvidia-gpu","rocm"],"created_at":"2024-10-10T18:12:22.000Z","updated_at":"2026-03-11T18:33:43.334Z","avatar_url":"https://github.com/okuvshynov.png","language":"Python","funding_links":[],"categories":["Other"],"sub_categories":[],"readme":"# system monitoring horizon charts for terminal\n\ncubestat is a command-line utility to monitor system metrics in horizon chart format. It was originally created for Apple M1/M2 devices, but supports Linux with NVIDIA GPU as well, including Google Colab environment.\nNumerous tools exist for tracking system metrics, yet horizon charts stand out due to their good information density which enables the display of many time-series data on a single screen.\n\nLet's start with an example:\n\nhttps://github.com/okuvshynov/cubestat/assets/661042/8e1e405e-ca61-4ffb-bedb-e04eb33f8bc2\n\nIn the clip above we see Mixtral-8x7b inference on MacBook Air with FF layers offloaded to SSD. \nWe can notice somewhat low GPU util, 2Gb/s+ of data read from disk, as we have to fetch the weights, but plenty of free RAM (And we are actually able to serve almost 100Gb model on 24Gb machine with fp16 precision, even if very slow).\n\nWe can also clearly see moment of change from model loading (cpu util, disk writes for model preprocessing) to model inference (disk reads, gpu util going up, cpu going down)\n\nCurrently cubestat reports:\n1. CPU utilization - configurable per core ('by_core'), cluster of cores on Apple M1+: Efficiency/Performance ('by_cluster') or all. Is shown as percentage.\n2. GPU utilization per card/chip. Is shown in percentage. Works for Apple's M1/M2 SoC and NVIDIA GPUs. For NVIDIA GPU can show VRAM usage as well. In case of multi-GPU can show individual GPUs or aggregated average.\n3. ANE (Neural Engine) power consumption. According to `man powermetrics` it is an estimate, but seems working good enough as a proxy to ANE utilization. Is shown as percentage.\n4. Disk and network IO; Is shown as rate (KB/s, MB/s, GB/s).\n5. Memory usage in %\n6. Swap usage. Is shown as absolute value (KB, MB, GB)\n\nKnown limitations:\n1. **On MacOS cubestat needs to run `powermetrics` with sudo**. You don't need to run cubestat itself with sudo, but you'll be asked sudo password when cubestat launches powermetrics. If you are comfortable doing that, you can add `powermetrics` to `/etc/sudoers` (`your_user_name ALL=(ALL) NOPASSWD: /usr/bin/powermetrics`) and avoid this.\n2. Neural engine utilization is an estimate based on power usage, more on that below.\n3. Needs 256 colors terminal\n\n## Installation:\n\n```\n% pip install cubestat\n```\n\nor \n\n```\n% pip install cubestat[cuda] # for instances with NVIDIA\n```\n\n## Usage\n\n```\nusage: cubestat [-h] [--refresh_ms REFRESH_MS] [--buffer_size BUFFER_SIZE]\n                [--view {off,one,all}] [--csv] [--http-port HTTP_PORT]\n                [--http-host HTTP_HOST] [--prometheus-port PROMETHEUS_PORT]\n                [--cpu {all,by_cluster,by_core}] [--network {show,hide}]\n                [--gpu {collapsed,load_only,load_and_vram}]\n                [--disk {show,hide}] [--swap {show,hide}]\n                [--memory {percent,all}] [--power {combined,all,off}]\n\noptions:\n  -h, --help            show this help message and exit\n  --refresh_ms REFRESH_MS, -i REFRESH_MS\n                        Update frequency (milliseconds)\n  --buffer_size BUFFER_SIZE\n                        Number of datapoints to store. Consider larger values for window resizing.\n  --view {off,one,all}  Display mode (legend, values, time). Hotkey: \"v\".\n  --csv                 Export metrics in CSV format to stdout (bypasses TUI)\n  --http-port HTTP_PORT\n                        Enable HTTP server on specified port to serve metrics as JSON\n  --http-host HTTP_HOST\n                        HTTP server host (default: localhost)\n  --prometheus-port PROMETHEUS_PORT\n                        Enable Prometheus metrics exporter on specified port\n  --cpu {all,by_cluster,by_core}\n                        Select CPU mode: all cores, cumulative by cluster, or both. Hotkey: \"c\".\n  --network {show,hide}\n                        Show network io. Hotkey: \"n\"\n  --gpu {collapsed,load_only,load_and_vram}\n                        GPU mode - hidden, load, or load and vram usage. Hotkey: \"g\"\n  --disk {show,hide}    Show disk read/write rate. Hotkey: \"d\"\n  --swap {show,hide}    swap show/hide. Hotkey: \"s\"\n  --memory {percent,all}\n                        Select memory mode: percent only or all details. Hotkey: \"m\".\n  --power {combined,all,off}\n                        Power: hidden, CPU/GPU/ANE breakdown, or combined usage. Hotkey: \"p\"\n```\n\nInteractive commands:\n* q - quit\n* v - toggle view mode\n* c - change cpu display mode (individual cores, aggregated or both)\n* g - change gpu display mode (individual gpus, aggregated and optionally VRAM usage)\n* d - show/hide disk reads/writes\n* n - show/hide network utilization\n* s - show/hide swap\n* p - show/hide power usage if available\n* UP/DOWN - scroll the lines in case there are more cores;\n* LEFT/RIGHT - scroll left/right. Autorefresh is paused when user scrolled to non-latest position. To resume autorefresh either scroll back to the right or press '0';\n* 0 - reset horizontal scroll, continue autorefresh.\n\n## CSV Export Mode\n\ncubestat supports CSV export for integration with monitoring systems, scripts, and data analysis tools:\n\n```bash\n# Basic CSV export\ncubestat --csv\n\n# Save to file\ncubestat --csv \u003e system_metrics.csv\n\n# Custom refresh rate\ncubestat --csv --refresh_ms 500\n\n# Pipe to monitoring system\ncubestat --csv | monitoring_ingester\n```\n\n### CSV Output Format\n\nThe CSV output uses standardized, hierarchical metric names that are self-documenting:\n\n```csv\ntimestamp,metric,value\n1750693377.593887,cpu.performance.0.core.0.utilization.percent,26.7591\n1750693377.593887,cpu.efficiency.0.core.4.utilization.percent,12.3456\n1750693377.593887,memory.system.total.used.percent,78.5\n1750693377.593887,gpu.apple.0.utilization.percent,45.2\n1750693377.593887,power.component.cpu.consumption.watts,2.34\n1750693377.593887,network.total.rx.bytes_per_sec,1048576\n1750693377.593887,disk.total.write.bytes_per_sec,2097152\n```\n\n### Integration Examples\n\n**InfluxDB:**\n```bash\ncubestat --csv | while IFS=, read timestamp metric value; do\n  curl -X POST \"http://localhost:8086/write?db=system\" \\\n    --data-binary \"$metric value=$value $timestamp\"\ndone\n```\n\n**Prometheus Pushgateway:**\n```bash\ncubestat --csv | awk -F, '\nNR\u003e1 { gsub(/\\./, \"_\", $2); print $2\" \"$3 }' | \\\ncurl --data-binary @- http://localhost:9091/metrics/job/cubestat\n```\n\n**Simple Analysis:**\n```bash\n# Get average CPU utilization\ncubestat --csv | grep \"cpu.*utilization\" | awk -F, '{sum+=$3; count++} END {print sum/count}'\n\n# Monitor memory usage\ncubestat --csv | grep \"memory.system.total.used.percent\" | tail -f\n```\n\n## Prometheus Metrics Export\n\ncubestat provides native Prometheus metrics export for seamless integration with Prometheus monitoring systems:\n\n```bash\n# Start with Prometheus metrics on port 9090\ncubestat --prometheus-port 9090\n\n# Combine with TUI and HTTP JSON endpoint\ncubestat --prometheus-port 9090 --http-port 8080\n\n# Custom refresh rate\ncubestat --prometheus-port 9090 --refresh_ms 500\n```\n\n### Prometheus Metrics Available\n\nAll system collectors export metrics in Prometheus format at `http://localhost:PORT/metrics`:\n\n- **CPU**: `cpu_usage_percent` with labels for core, cluster, and type (performance/efficiency)\n- **Memory**: `memory_usage_percent`, `memory_used_bytes`, platform-specific metrics\n- **GPU**: `gpu_usage_percent`, `gpu_memory_usage_percent` with vendor and GPU ID labels\n- **Disk I/O**: `disk_read_bytes_per_second`, `disk_write_bytes_per_second`\n- **Network**: `network_receive_bytes_per_second`, `network_transmit_bytes_per_second`\n- **Power**: `power_consumption_total_watts`, `power_consumption_watts` by component\n- **Swap**: `swap_used_bytes`\n- **ANE**: `ane_usage_percent` (Apple Neural Engine on Apple Silicon)\n\n### Prometheus Configuration Example\n\n```yaml\n# prometheus.yml\nscrape_configs:\n  - job_name: 'cubestat'\n    static_configs:\n      - targets: ['localhost:9090']\n    scrape_interval: 1s\n```\n\n### Example Prometheus Queries\n\n```promql\n# Average CPU usage across all cores\navg(cpu_usage_percent)\n\n# Memory usage percentage\nmemory_usage_percent\n\n# Network throughput\nrate(network_receive_bytes_per_second[5m])\n\n# GPU usage by vendor\ngpu_usage_percent{vendor=\"nvidia\"}\n\n# Power consumption by component\nsum(power_consumption_watts) by (component)\n```\n\n## HTTP JSON API\n\ncubestat can also serve metrics via HTTP JSON API for programmatic access:\n\n```bash\n# Start HTTP server on port 8080\ncubestat --http-port 8080\n\n# Access metrics\ncurl http://localhost:8080/metrics\n```\n\nThe JSON API provides current values and historical data for all metrics, perfect for custom dashboards and monitoring solutions.\n\n## Notes and examples\n\n### Multi-gpu example \n\nhttps://github.com/okuvshynov/cubestat/assets/661042/c5e0750d-9bbd-4636-a1ea-71cc75ebbadb\n\nWe see a workload with uneven distribution between 4 GPUs installed. By pressing 'g' we can toggle the view mode to either show aggregate load, per GPU load or per GPU load and VRAM usage.\n\n### Apple Neural Engine utilization\n\nA few notes on 'what does this even represent?'. Utilization we show is essentially current power consumption reported by powermetrics. To convert it to % we divide it by some maximum value observed in experimentation. When reading this metric, be aware:\n* The concept of 'utilization' overall it pretty ambiguous, e.g. when CPU is wasting cycles on a cache miss, is it 'utilized' or not? If CPU is doing scalar instructions on 1 execution port rather than vectorized instructions on several ports, is it 'utilized' or not?\n* It is unclear if power consumption is a decent proxy for utilization;\n* The upper bound must be different for different models (M1, M1 Max, M2, etc.). I tested it on M1, M2 and M1 Pro only;\n* It is unclear if my tests are actually hitting upperbound. The highest I could achieve was [multiple layers of convolutions with no non-linearities between them](scripts/apple_loadgen.py#L26-L31);\n\n### Running on Google Colab \n\nWe can run cubestat on Google Colab instances to monitor GPU/CPU/IO usage.\n\nFirst cell:\n```\n!pip install cubestat[cuda]\n!pip install colab-xterm\n%load_ext colabxterm\n# export TERM=xterm-256color \u003c---- RUN THIS IN TERMINAL\n# cubestat                   \u003c---- RUN THIS IN TERMINAL\n```\n\nStart xterm:\n```\n%xterm\n```\n\nIn the terminal, configure 256 colors and start cubestat:\n```\n# export TERM=xterm-256color\n# cubestat\n```\n\nExample notebook: [colab example](https://colab.research.google.com/drive/1EUOXGJ-WUYfrKjy0oC_H2ZkVRgiSWGcC#scrollTo=0sm8bcE1QgbW)\n\n![colab cubestat](static/colab_cubestat.png)\n\n## Dependencies\n* Python 3.8+\n* psutil 5.9.5+\n* [optional] pynvml for NVIDIA cards monitoring\n\n## Development\n\n### Installation for Development\n\n```bash\n# Clone the repository\ngit clone https://github.com/okuvshynov/cubestat.git\ncd cubestat\n\n# Install in development mode\npip install -e .\n# Or with NVIDIA GPU support\npip install -e .[cuda]\n\n# Install development dependencies\npip install -r requirements-dev.txt\n\n# Set up pre-commit hooks\npre-commit install\n```\n\n### Running Tests\n\n```bash\n# Run all tests\npython -m unittest discover\n\n# Run a specific test\npython -m unittest cubestat.tests.test_data_manager\n```\n\n### Checking Types\n\n```bash\n# Run mypy for type checking\nmypy cubestat\n```\n\n### Linting\n\n```bash\n# Run ruff for linting\nruff check cubestat\n```\n\n## TODO\n\n- [ ] add 'help' for each metric\n- [x] type hints (in progress)\n- [x] better error handling and logging (in progress)\n- [ ] unit tests\n- [x] memory modes - more details (cache/etc), absolute values rather than %, mmap handling\n- [ ] optional joint scale within metric group\n- [ ] nvidia GPU - probing vs momentary load to avoid missing spikes\n- [ ] support AMD GPUs\n- [ ] rent on vast.ai\n- [ ] support remote monitoring\n- [ ] NUMA grouping\n- [ ] per interface network utilization\n- [ ] perf on weak machine (e.g. pi zero)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fokuvshynov%2Fcubestat","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fokuvshynov%2Fcubestat","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fokuvshynov%2Fcubestat/lists"}