{"id":15677599,"url":"https://github.com/giannisalinetti/perf-utils","last_synced_at":"2025-05-07T01:01:48.545Z","repository":{"id":41933656,"uuid":"266170180","full_name":"giannisalinetti/perf-utils","owner":"giannisalinetti","description":"Performance analisys utilities","archived":false,"fork":false,"pushed_at":"2023-03-27T14:55:41.000Z","size":50,"stargazers_count":15,"open_issues_count":1,"forks_count":7,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-03-31T04:41:30.637Z","etag":null,"topics":["containers","kubernetes","linux","openshift","performance-analysis","performance-tuning","troubleshooting"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/giannisalinetti.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-05-22T17:32:36.000Z","updated_at":"2024-12-05T14:48:23.000Z","dependencies_parsed_at":"2024-10-23T13:05:12.655Z","dependency_job_id":null,"html_url":"https://github.com/giannisalinetti/perf-utils","commit_stats":null,"previous_names":[],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/giannisalinetti%2Fperf-utils","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/giannisalinetti%2Fperf-utils/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/giannisalinetti%2Fperf-utils/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/giannisalinetti%2Fperf-utils/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/giannisalinetti","download_url":"https://codeload.github.com/giannisalinetti/perf-utils/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252793661,"owners_count":21805057,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["containers","kubernetes","linux","openshift","performance-analysis","performance-tuning","troubleshooting"],"created_at":"2024-10-03T16:09:57.907Z","updated_at":"2025-05-07T01:01:47.988Z","avatar_url":"https://github.com/giannisalinetti.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Perf-Utils - Performance Analisys Utilities\n\nThis is the repository of the **perf-utils** image that holds a set of utilities\nfor performance analysis and troubleshooting. It is especially useful on immutable\nsystems like Fedora CoreOS and Red Hat CoreOS, or production systems where no extra \npackages can be installed.\n\nThe perf-utils image is based on the official CentOS base image and is available \non [quay.io](quay.io/gbsalinetti/perf-utils).\n\n### Warning for production systems\nRunning some performance tools (like strace) on production environments can impact the \noverall performances of the system. Use them at your own risk.\n\n## Installed Packages\nThe following packages are installed:\n- perf\n- pcp\n- pcp-system-tools\n- pcp-pmda-trace\n- lsof\n- sysstat\n- tcpdump\n- iotop\n- fio\n- nmon\n- htop\n- strace\n- ltrace\n- valgrind\n- dmidecode\n- elfutils\n- python3\n- pciutils\n- bind-utils\n- man\n- hdparm \n- lvm2 \n- iptables \n- nftables\n- warp\n- bpftrace\n- iptraf\n- wireless-tools\n- flamegraph\n- net-tools\n- iproute\n- iftop\n- glances\n- httpd-tools\n\n# Compiled tools\nThe following list of tools was directly compiled from master branch:\n- libbpf-tools (libpf CO-RE compiled bcc-tools)\n\n## How to run\nThe perf-utils container must be executed as a privileged container. \nTo run the container directly from podman/docker CLI:\n```\n$ sudo podman run -it --rm \\\n  --privileged --network=host --pid=host \\\n  -v /lib/modules:/lib/modules:ro -v /usr/src:/usr/src:ro -v /proc:/proc -v /:/mnt/rootdir \\\n  quay.io/gbsalinetti/perf-utils\n```\n\nThe `perf-utils` script from this repository manages the execution of the\ncontainer in a simple and fast way:\n```\n$ sudo ./perf-utils\n```\n\n#### Approach 1: Manual container creation\nIf you choose to freely run your contianer, first start the container without \nmounting the `/usr/src` host directory:\n```\n$ sudo podman run -it --rm \\\n  --privileged --network=host --pid=host \\\n  -v /lib/modules:/lib/modules:ro -v /proc:/proc -v /:/mnt/rootdir \\\n  quay.io/gbsalinetti/perf-utils\n```\n\nWhen the container is started, install the `kernel-devel` package. The pacakge\ncomes from the repositories built into the image and could not match the host\nkernel version, so the complete functionality of all bcc tools come with no\nwarranty in this use case.\n```\n# yum install -y kernel-devel\n```\n\n#### Approach 2: Using perf-utils script\nIf you use the `perf-utils` script from this repository you can simply run:\n```\n$ sudo ./perf-utils --install-headers\n```\n\nWith the above flag enabled, the entrypoint script will take care of installing \nthe haeders.\n\nEnjoy you bcc tools installed under `/usr/share/bcc/tools`!\n\n### PCP script\nPerformance Co-Pilot runs the `pmcd` service in the background. While this is\nnormally started by systemd, the perf-utils container has no init system and\nthe daemon can be started manually using the file `/pmcd_service.sh`.\nTo start the service:\n```\n# /pmcd_service.sh start\n```\n\nTo stop the service:\n```\n# /pmcd_service.sh stop\n```\n\n### Storage benchmark with fio\nFio is a tool to benchmark storage IOPS and can be useful to evaluate disk \nperformances with etcd.\nThis example of fio show a write/sync test on an RHCOS system. The directory \nused is the /var/tmp of the system (which is writable) mount to the to the \n`/mnt/rootdir/var/tmp` folder in the container:\n```\n# fio --rw=write --ioengine=sync --fdatasync=1 \\\n--directory=/mnt/rootdir/var/tmp --size=22m --bs=2300 --name=fiotest\n```\n\nThe output shows sync percentiles in usecs. One of the main etcd best practices is to \nensure that the `wal_fsync_duration_seconds` 99th percentile must be under\n10ms. This means that etcd should take less than 10ms to write to the wal file,\nincluding both `write` and `fdatasync` syscalls. \n\nThe fio command example above simulates the sequential writes of etcd \n(`--rw=write`) followed by fdatasync (`--fdatasync=1`). The size value is an\napproximazione of an etcd write to the wal file.\nUpdate the size accordingly to the average write value of you cluster (write\nsyscall returns the written bytes in ssize_t format). The following example\nuses strace to track down writes to wal files.\n```\n# strace -p \u003cetcd_pid\u003e -f -e write,fdatasync -yy 2\u003e\u00261 | grep '\\.wal'\n```\n\nThe output of the fsync/fdatasync from the fio command should be under 10000 usecs.\n\nIf the system is already running a consistent workload (like an etcd instance) \nthe output of the fsyncs will be greater.\n\n\n### Running on OpenShift\nThe preferred approach to run perf-utils is by using the \n[perfutils-operator](https://github.com/giannisalinetti/perfutils-operator), \nwhich provides, along with a generic perf-utils pods, other specialized \nCustom Resource Definitions which can be used to run commands on targed \nnodes, for example `fio`.\n\nWhen the operator is not an option users can install the image using **Helm**.\nThe provided Helm charts is a useful tool to run a perf-utils pod in OpenShift\nwith a parametric approach to dynamically provide the node name. You must have \n**cluster-admin** privileges on your cluster.\n\nTo install the chart with custom *noodeName*:\n```\n$ helm install ./helm/perf-utils --set nodeName=master-0 --generate-name\n```\n\nOnce started you will notice a pod running in the namespace:\n```\n$ oc get pods\nNAME                 READY   STATUS    RESTARTS   AGE\nmaster-0-perfutils   1/1     Running   0          7s\n```\n\nYou can run `oc rsh` or `oc exec` to run a shell in the container and \nexecute commands.\n```\n$ oc exec -it master-0-perfutils /bin/bash\n```\n\nBy default the chart sets the host root file system mount point under the `/host` \nfolder.\n\n### Other OpenShift examples\nIt is possible to run specific commands using the perf-utils image. \nThe `openshift_examples` contains some examples of targeted executions.\nThe `fio_job.yaml` file is an example of a Kubernetes Job that runs a fio\nbenchmark good for testing etcd performances.\n\nBefore creating the Job, assign to the `default` service account of the project\nthe **Privileged** SCC:\n```\n$ oc adm policy add-scc-to-user Privileged -z default\n```\n\nUpdate the `nodeName` field in the manifest before running (**TODO**: deliver\nservices as CRD with perfutils-operator).\nAfter creating the job the pod is executed and outputs can be collected with the `oc logs` command:\n```\n$ oc apply -f openshift-examples/fio_job.yaml \npod/master-0-bench created\n\n$ oc get pods\nNAME               READY   STATUS      RESTARTS   AGE\nfio-sample-lcjsg   0/1     Completed   0          28m\n\n$ oc logs fio-sample-lcjsg\nfsyncwrite: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=sync, iodepth=1\nfio-3.7\nStarting 1 process\nfsyncwrite: Laying out IO file (1 file / 22MiB)\n\nfsyncwrite: (groupid=0, jobs=1): err= 0: pid=8: Thu Nov 12 17:37:09 2020\n  write: IOPS=48, BW=192MiB/s (202MB/s)(20.0MiB/104msec)\n    clat (usec): min=1718, max=4066, avg=2350.51, stdev=978.89\n     lat (usec): min=1856, max=4163, avg=2536.22, stdev=947.18\n    clat percentiles (usec):\n     |  1.00th=[ 1713],  5.00th=[ 1713], 10.00th=[ 1713], 20.00th=[ 1713],\n     | 30.00th=[ 1778], 40.00th=[ 1778], 50.00th=[ 1975], 60.00th=[ 1975],\n     | 70.00th=[ 2212], 80.00th=[ 2212], 90.00th=[ 4080], 95.00th=[ 4080],\n     | 99.00th=[ 4080], 99.50th=[ 4080], 99.90th=[ 4080], 99.95th=[ 4080],\n     | 99.99th=[ 4080]\n  lat (msec)   : 2=60.00%, 4=20.00%, 10=20.00%\n  fsync/fdatasync/sync_file_range:\n    sync (usec): min=12990, max=31657, avg=18097.90, stdev=7729.29\n    sync percentiles (usec):\n     |  1.00th=[13042],  5.00th=[13042], 10.00th=[13042], 20.00th=[13042],\n     | 30.00th=[14222], 40.00th=[14222], 50.00th=[14484], 60.00th=[14484],\n     | 70.00th=[17171], 80.00th=[17171], 90.00th=[31589], 95.00th=[31589],\n     | 99.00th=[31589], 99.50th=[31589], 99.90th=[31589], 99.95th=[31589],\n     | 99.99th=[31589]\n  cpu          : usr=1.94%, sys=10.68%, ctx=59, majf=0, minf=12\n  IO depths    : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, \u003e=64=0.0%\n     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, \u003e=64=0.0%\n     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, \u003e=64=0.0%\n     issued rwts: total=0,5,0,0 short=5,0,0,0 dropped=0,0,0,0\n     latency   : target=0, window=0, percentile=100.00%, depth=1\n\nRun status group 0 (all jobs):\n  WRITE: bw=192MiB/s (202MB/s), 192MiB/s-192MiB/s (202MB/s-202MB/s), io=20.0MiB (20.0MB), run=104-104msec\n```\n\n\n### Maintainers\nGianni Salinetti \u003cgsalinet@redhat.com\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgiannisalinetti%2Fperf-utils","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgiannisalinetti%2Fperf-utils","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgiannisalinetti%2Fperf-utils/lists"}