{"id":13514352,"url":"https://github.com/uber-archive/cpustat","last_synced_at":"2025-09-27T07:32:07.857Z","repository":{"id":57494507,"uuid":"52304529","full_name":"uber-archive/cpustat","owner":"uber-archive","description":"high frequency performance measurements for Linux. This project is deprecated and not maintained.","archived":true,"fork":false,"pushed_at":"2019-12-03T07:15:35.000Z","size":189,"stargazers_count":1655,"open_issues_count":11,"forks_count":77,"subscribers_count":1605,"default_branch":"master","last_synced_at":"2025-09-20T02:19:43.828Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/uber-archive.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGES","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-02-22T20:48:27.000Z","updated_at":"2025-08-11T16:04:55.000Z","dependencies_parsed_at":"2022-09-07T19:00:23.659Z","dependency_job_id":null,"html_url":"https://github.com/uber-archive/cpustat","commit_stats":null,"previous_names":["uber-common/cpustat"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/uber-archive/cpustat","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uber-archive%2Fcpustat","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uber-archive%2Fcpustat/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uber-archive%2Fcpustat/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uber-archive%2Fcpustat/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/uber-archive","download_url":"https://codeload.github.com/uber-archive/cpustat/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uber-archive%2Fcpustat/sbom","scorecard":{"id":906167,"data":{"date":"2025-08-11","repo":{"name":"github.com/uber-archive/cpustat","commit":"b3265a2cd9876b61c385431f3d1e6ada01726591"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":3,"checks":[{"name":"Code-Review","score":2,"reason":"Found 4/15 approved changesets -- score normalized to 2","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Maintained","score":0,"reason":"project is archived","details":["Warn: Repository is archived."],"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: containerImage not pinned by hash: Dockerfile:1: pin your Docker image by updating debian:sid to debian:sid@sha256:8fcabbc0112b91ed4faabae11ccb91dba23263563bc07a9489a0c7ab3a8494ae","Info:   0 out of   1 containerImage dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 19 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-24T17:25:25.947Z","repository_id":57494507,"created_at":"2025-08-24T17:25:25.947Z","updated_at":"2025-08-24T17:25:25.947Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":277095803,"owners_count":25760028,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-26T02:00:09.010Z","response_time":78,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T05:00:54.102Z","updated_at":"2025-09-27T07:32:05.067Z","avatar_url":"https://github.com/uber-archive.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"# cpustat - high(er) frequency stats sampling\n\n[![Join the chat at https://gitter.im/uber-common/cpustat](https://badges.gitter.im/uber-common/cpustat.svg)](https://gitter.im/uber-common/cpustat?utm_source=badge\u0026utm_medium=badge\u0026utm_campaign=pr-badge\u0026utm_content=badge)\n\n(This project is deprecated and not maintained.)\n\n`cpustat` is a tool for Linux systems to measure performance. You can think of it like a\nfancy sort of `top` that does different things. This project is motivated by Brendan Gregg's\n[USE Method](http://www.brendangregg.com/usemethod.html) and tries to expose CPU\nutilization and saturation in a helpful way.\n\nMost performance tools average CPU usage over a few seconds or even a minute. This can\ncreate the illusion of excess capacity because brief spikes in resource usage are blended\nin with less busy periods. `cpustat` takes higher frequency samples of every process running\non the machine and then summarizes these samples at a lower frequency. For example, it can\nmeasure every process every 200ms and summarize these samples every 5 seconds, including\nmin/average/max values for some metrics.\n\nThere are two ways of displaying this data: a pure text list of the summary interval\nand a colorful scrolling dashboard of each sample.\n\nHere are examples of both modes observing the same workload:\n\n![Text Mode](http://imgur.com/vu4LrBD.gif)\n\n![Demo](http://i.imgur.com/mbasDlZ.gif)\n\n## Installation\n\nTo install the latest version of `cpustat` and all of its dependencies, try this:\n\n```\ngo get github.com/uber-common/cpustat\n```\n\nIf you have `glide`, you can use `glide install` to get consistent dependencies.\n\n## Usage\n\nThis program uses Linux taskstats, which requires root.\n\nHere are the command line flags most users will want:\n\nFlag | Description | Default\n-----|-------------|--------\n`-i` | sample interval in milliseconds | 200\n`-s` | summarize after this many samples | 10\n`-n` | display top n processes | 10\n`-maxprocs` | truncate process list if it exceed this | 2048\n`-p` | only measure processes in this list of pids | none\n`-u` | only measure processes owned by this list of users | none\n`-t` | use fancy termui mode | false\n\nThere are also a few less common options:\n\nFlag | Description | Default\n-----|-------------|--------\n`-jiffy` | set the Linux clock tick duration time in milliseconds | 100\n`-cpuprofile` | write CPU pprof data of cpustat itself to this file | none\n`-memprofile` | write memory pprof data of cpustat itself to this file | none\n\nExamples:\n\n```\nsudo cpustat -s 500 -s 10 -n 20\n```\n\nThis will take a sample of all processes every 500ms and summarize this data after 10\nsamples, which is every 5 seconds.\n\n```\nsudo cpustat -u mjr,mranney\n```\n\nOnly measure processes that are owned by either user `mjr` or user\n`mranney`. The overall system stats will still be measured.\n\n```\nsudo ./cpustat -p $(pgrep -d, vim\\|emacs)\n```\n\nOnly measure processes that pgrep thinks are called \"vim\" or \"emacs\". The `-p`\noption to `cpustat` takes a list of process ids to measure, and `pgrep` is a\nhandy way to get this list. The `-d,` option to `pgrep` prints the list of\nmatching pids with a comma separator.\n\n\n## Displayed Values\n\nIn pure text mode, there are some system-wide summary metrics that come from /proc/stat:\n\nName | Description\n-----|------------\nusr | min/avg/max user mode run time as a percentage of a CPU\nsys | min/avg/max system mode run time as a percentage of a CPU\nnice | min/avg/max user mode low priority run time as a percentage of a CPU\nidle | min/avg/max user mode run time as a percentage of a CPU\niowait | min/avg/max delay time waiting for disk IO\nprun | min/avg/max count of processes in a runnable state (load average)\npblock | min/avg/max count of processes blocked on disk IO\npstart | number of processes/threads started in this summary interval\n\nIn fancy scrolling dashboard mode, the unique panes are as follows:\n\nIn the top right, labeled \"total usr/sys time\", the system-wide measurements for user time\nand system time are displayed. User time is cyan, and system time is red. The X axis is\nlabeled in seconds, the Y axis is percentage of a CPU. Each dot represents an individual\nsample. The data is not summarized.\n\nIn the top left, labeled \"top procs\", the combined user+system time of the topN processes\nis displayed. Each process gets a separate line with a color that matches the list\nbelow. The X and Y axes are the same as the top right graph. Due to limitations of a\nterminal UI, overlapping lines are drawn in the same cell, potentially obscuring each\nother.\n\nBoth modes display the same per-process summary data. The fields are:\n\nName | Description\n-----|------------\n`name` | common Name from /proc/pid/stat or /proc/pid/cmdline. There is some logic to resolve common patterns into more useful names for common things.\n`pid` | Top level process id, sometimes referred to as \"tgid\"\n`min` | lowest sample of combined user and system time for this pid, measured from /proc/pid/stat. Scale is a percentage of a CPU.\n`max` | highest sample of combined user and system time for this pid, measured from /proc/pid/stat.\n`usr` | average user time for this pid over the summary period, measured from /proc/pid/stat. This plus `sys` should be similar to what \"top\" reports.\n`sys` | average system time for this pid over the summary period, measured from /proc/pid/stat. This plus `usr` should be similar to what \"top\" reports.\n`nice` | current \"nice\" value for this process, measured from /proc/pid/stat. Higher is \"nicer\".\n`runq` | time this process and all of its threads spent runnable but waiting to run, measured from taskstats via netlink. Scale is a percentage of a CPU.\n`iow` | time this process and all of its threads spent blocked by disk IO, measured from taskstats via netlink. Scale is a percentage of a CPU, averaged over the summary interval.\n`swap` | time this process and all of its threads spent waiting to be swapped in, measured from taskstats via netlink. Scale is a percentage of a CPU, averaged over the summary interval.\n`vcx` | total number of voluntary context switches by this process and all of its threads over the summary interval, measured from taskstats via netlink.\n`icx` | total number of involuntary context switches by this process and all of its threads over the summary interval, measured from taskstats via netlink.\n`rss` | current RSS value measured from /proc/pid/stat. This is the amount of memory this process is using.\n`ctime` | total user+sys CPU time consumed by waited for children that exited during this summary interval, measured from /proc/pid/stat. Long running child processes can often confuse this measurement, because the time is reported only when the child process exits. However, this is useful for measuring the impact of frequent cron jobs and health checks where the CPU time is often consumed by many child processes.\n`thrd` | Number of threads at the end of the summary interval, measured from /proc/pid/stat.\n`sam` | number of samples for this process included in the summary interval. Processes that have recently started or exited may have been visible for fewer samples than the summary interval.\n\n## Understanding the Output\n\nHere are a few examples of running `cpustat` on a 4 processor vm on my laptop.\n\nThe first is a mostly idle system where the only thing really running is `cpustat` itself:\n\n![Idle](https://ranney.com/cpustat_images/1__ssh.png)\n\nThe `idle` min/avg/max shows that for most of the 20 samples, the system was almost completely idle.\nWe can tell that because we know this is a 4 processor system, so the maximum value for `idle` is 400.\nIt's perhaps a little surprising then that `prun` would show 1.0/1.1/3.0, meaning that we never woke up\nto find fewer than 1 process running. On an idle system, surely that number should be lower. What's\nhappening is that `cpustat` ends up measuring itself as the single running process.\n\nSo let's put this computer to work:\n\n![Idle](https://ranney.com/cpustat_images/1__ssh 2.png)\n\nI'm running a single instance of \"CPU Burn-In\" that shows up as `burnP6`. This process uses a single\nCPU.  We can see that the overall system now reports only about 300% idle, and that `burnP6` is only\nusing a single thread from the `thrd` column.\n\nWe can also see that the `runq` column for `burnP6` shows 0.4 on the first summary interval. This\nmeans that for the duration of the summary interval, 0.4% of an effective CPU's time was asked for\nby a process, but that process wasn't scheduled for whatever reason. During the same interval, `icx`\nis a lot higher than normal.  `icx` is \"involuntary context switches\". It's hard to say exactly what\ncaused this, but we can also see that several other processes have non-0 `iow` or \"IO wait\". During\nthat interval, `prun` had a max value of 5.0, so a few things probably woke up at the same time to\ndo some brief work, caused a bit of interference, then went back to sleep.\n\nDuring the second summary interval, the `runq` is back to 0, so `burnP6` is getting all of the time\nit wants. It's also curious to note that the `burnP6` pretty clearly reports its CPU usage as 100%\n`usr` time, the overall system `usr` doesn't line up with this at 60.0/77.2/95.0, but the system\n`idle` does. I'm not exactly sure what causes this, but it's something about how `burnP6` works and\nLinux accounts for it. Many other single threaded programs in a tight loop do not exhibit this\nbehavior.\n\nI have another program to generate a more irregular an in my experience more realistic workload\ncalled `wastetime`. This program uses many threads, tries to wake them up at the same time to do\nsome work, then sleeps for a bit.\n\n![Idle](https://ranney.com/cpustat_images/1__ssh 3.png)\n\nFor some of the samples, we wake up and find `wastetime` using 0 CPU, and sometimes we find it 300%\nbecause `burnP6` is using the other 100%. `wastetime` reports a `runq` time of around 230%. This\nmeans that we'd need approximately 2.3 more CPUs on average to do all of the work and avoid\ndelay. This is the average delay, but we can see from the `prun` max that we pretty regularly need\n35 CPUs to avoid absolutely all delays.\n\nWe also see that `wastetime` is causing a little bit of `runq` interference for `burnP6`.\n\nThis is what it looks like to run 10 parallel instances of `go build` in this vm on the `cpustat`\nsource with:\n\n```\nfor C in {0..10}; do echo $C ; (go build \u0026) ; done\n```\n\n![Idle](https://ranney.com/cpustat_images/1__ssh 4.png)\n\nThis is obviously way more work than my little vm can manage, so we've triggered some major CPU\nsaturation.\n\nNearly every process running on the machine is spending more time in the `runq` than we'd like. Even\nso, there is still some idle time left in the system. That can be explained by the `swap` time and\n`iowait` time. Also note that the `sam` column is all less than 20. That means these are all short\nlived processes, which are often hard to account for.\n\n## Data Sources\n\nEvery sample interval, the following data sources are checked:\n\n* The directory /proc is scanned to get the current list of process ids\n* For each pid, read /proc/pid/stat, compute difference from previous sample\n* If this is a new pid, read /proc/pid/cmdline\n* For each pid, send a netlink message to fetch the taskstats, compute difference from\n  previous sample.\n* Fetch /proc/stat to get the overall system stats\n\nEach sleep interval is adjusted to account for the amount of time spent fetching all of\nthese stats. Each sample also records the time it was taken to scale each measurement by\nthe actual elapsed time between samples. This attempts to account for delays in `cpustat`\nitself.\n\n## Run within a Docker container\n\n```\n docker run --rm -ti --privileged --pid=host --net=host user/cpustat -s=200 -n=20\n```\n\n## Limitations\n\nThere are many important limitations to understand before drawing conclusions from\n`cpustat`'s output.\n\nThe first is that all sampling systems are subject to\n[Nyquist](https://en.wikipedia.org/wiki/Nyquist_rate), which is both a practical reality\nand a fun concept that will lend gravitas to many conversations. Even if we sample at\n200ms, which is pretty fast compared to most tools, we'll not be able to accurately\nmeasure many common performance problems.\n\nLinux CPU time accounting is done in terms of whole \"clock ticks\", which are often\n100ms. This can cause some strange values when sampling every 200ms. Common problems are\nthat CPU utilization will appear to slightly exceed the maximum number of CPUs on the\nsystem. On a very lightly loaded system, sometimes processes will appear to oscillate\nbetween 0 percent and some higher number.\n\nThere is no way to get a consistent snapshot of all processes on a Linux system. After we\nscan /proc to get the process list, new processes could come and go. Processes could exit\nat any time along the process of scanning /proc, reading /proc/cmdline, /proc/pid/stat, or\nnetlink taskstats.\n\n`cpustat` itself can cause the very problems it was written to expose by doing a burst of\nwork on a regular interval. It would be nicer to the underlying system to spread the work\nout evenly over the sampling interval instead of trying to do it all at once.\n\nThe Linux netlink taskstats interface can only be used by root, which means this program\nmust be run as root.\n\nIn spite of these limitations, this tool has already been useful in understanding\nperformance problems on production systems. I hope it's useful to you as well.\n\n## Agent\n\nIn addition to the interactive version of `cpustat`, a long running measurement server is\navailable in the `agent` directory. This program uses only the measurement logic to record\nthe raw samples in a circular buffer. The data in the agent can be collected by the\n`client` program. This can be used to summarize the system state for publishing metrics to\nother systems. For example, instead of recording the average CPU utilization over a minute\nlike many metrics systems do, you could report the min/avg/max CPU utilization over a\nminute or any other interval.\n\n## Future Work\n\nThere is an almost an endless set of UI-type features that would be nice.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fuber-archive%2Fcpustat","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fuber-archive%2Fcpustat","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fuber-archive%2Fcpustat/lists"}