{"id":40935343,"url":"https://github.com/gsauthof/osjitter","last_synced_at":"2026-01-22T04:15:26.340Z","repository":{"id":52013297,"uuid":"210419460","full_name":"gsauthof/osjitter","owner":"gsauthof","description":"Measure how much the OS interrupts programs","archived":false,"fork":false,"pushed_at":"2024-05-08T23:36:34.000Z","size":71,"stargazers_count":35,"open_issues_count":0,"forks_count":8,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-26T18:15:10.340Z","etag":null,"topics":["jitter","latency","linux","realtime"],"latest_commit_sha":null,"homepage":null,"language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gsauthof.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2019-09-23T17:59:42.000Z","updated_at":"2025-04-26T16:56:48.000Z","dependencies_parsed_at":"2024-04-19T00:25:12.413Z","dependency_job_id":null,"html_url":"https://github.com/gsauthof/osjitter","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/gsauthof/osjitter","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gsauthof%2Fosjitter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gsauthof%2Fosjitter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gsauthof%2Fosjitter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gsauthof%2Fosjitter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gsauthof","download_url":"https://codeload.github.com/gsauthof/osjitter/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gsauthof%2Fosjitter/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28653970,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-22T01:17:37.254Z","status":"online","status_checked_at":"2026-01-22T02:00:07.137Z","response_time":144,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["jitter","latency","linux","realtime"],"created_at":"2026-01-22T04:15:26.180Z","updated_at":"2026-01-22T04:15:26.331Z","avatar_url":"https://github.com/gsauthof.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"This repository contains OSjitter, Pingpong and other\nlatency/performance related utilities.\n\nOSjitter is a tool for measuring how much\nthe operating system interrupts programs. Such interruptions\nincrease the latency of a program while the variation in latency\nis called jitter.\n\nThis tool can be used to quickly measure a lower bound for the\nlatency of a given system configuration. Note that the OS jitter\ndepends on the kind of load a real-time program is applying to a\nsystem. Thus, one still needs to execute a domain specific\ntest-suite to the real-time program of interest after a tool like\nOSjitter shows good results.\n\nThe Pingong utility measures the overhead of several thread\nnotification mechanisms such as spinning on a atomic variable\n(with/without pauses), POSIX condition variables, semaphores,\npipes and raw Linux futexes.\n\nThe ptp-clock-offset utility is a small program for checking\nthe availability of different PTP offset ioctls and how they\nperform. Rule of thumb: using any PTP offset ioctl is better than\nhaving to use `clock_gettime()` and smaller delays are better.\n\nThere is also a microbenchmark (`bench_syscalls.cc`) that measures\nsome (seemingly) low-overhead syscalls in order to measure the\nuserspace to kernelspace mode-switch costs. See also a [related\nblog post](https://gms.tf/on-the-costs-of-syscalls.html) for some results.\n\n\n2019, Georg Sauthoff \u003cmail@gms.tf\u003e, GPLv3+\n\n## Example Session\n\nCheck out the help:\n\n    $ ./osjitter -h\n\nIsolating the last 3 cores on a 8 core system:\n\n    $ cat /proc/cmdline\n    [..] isolcpus=5-7 nohz=on nohz_full=5-7 rcu_nocbs=5-7 rcu_nocb_poll \\\n    nowatchdog mce=ignore_ce acpi_irq_nobalance pcie_aspm=off tsc=reliable\n\nThis system is a Supermicro one (running Fedora 29) with an Atom CPU:\n\n    $ cat /proc/cpuinfo | grep model' name' | head -n 1\n    model name\t: Intel(R) Atom(TM) CPU C3758 @ 2.20GHz\n\nFirst OSjitter run:\n\n    $ ./osjitter  -t 60\n     CPU  TSC_khz  #intr  #delta  ovfl_ns  invol_ctx  sum_intr_ns  iratio  rt_s  loop_ns  median_ns  p20_ns  p80_ns  p90_ns  p99_ns  p99.9_ns   max_ns  mad_ns\n       0  2200000  60240   60240        0       8065    283228653   0.005    60       22       3151    2989    4354    6047    7218    443376  9380037     220\n       1  2200000  60192   60192        0       9809    216975033   0.004    60       22       2710    2339    3740    5314    6322     11774  4614206     432\n       2  2200000  60199   60199        0       5942    180783353   0.003    60       22       2424    2219    3399    4847    7888     14611  1465586     223\n       3  2200000  60193   60193        0       5465    171929486   0.003    60       22       2426    2236    3087    4246    6388     11487   592769     187\n       4  2200000  60320   60320        0       6173    212338516   0.004    60       22       2548    2358    3468    5005    6280     40044  2262400     211\n       5  2200000    156     156        0          1       576392   0.000    60       22       3681    2801    4044    4388   11667     12138    12286     428\n       6  2200000    156     156        0          1       581260   0.000    60       22       3565    2788    3964    4270   12278     20279    28125     451\n       7  2200000    126     126        0          1       450470   0.000    60       22       3703    2467    4003    4205    9163     11859    12198     352\n\n=\u003e The threads on the isolated CPUs are much less interrupted the\nother ones.\n\nMove all interrupts away from the isolated CPUs:\n\n    # tuna -q '*' -c 0-4 -m -x\n\nOSjitter:\n\n    $ ./osjitter  -t 60\n     CPU  TSC_khz  #intr  #delta  ovfl_ns  invol_ctx  sum_intr_ns  iratio  rt_s  loop_ns  median_ns  p20_ns  p80_ns  p90_ns  p99_ns  p99.9_ns   max_ns  mad_ns\n       0  2200000  60342   60342        0       6207    272600031   0.005    60       22       3105    2980    4141    5898    7205    442155  4772690     144\n       1  2200000  60334   60334        0       6483    196708372   0.003    60       22       2488    2293    3530    5014    6335     13491  4684815     236\n       2  2200000  60330   60330        0       8479    211832782   0.004    60       22       2528    2296    3651    5269    9299     15708  5513140     347\n       3  2200000  60256   60256        0       7973    237326578   0.004    60       22       2477    2261    3617    5155    7186     39479  5602172     325\n       4  2200000  60280   60280        0       5149    197355746   0.003    60       22       2532    2345    3020    4026    6309     16298  2630389     175\n       5  2200000      8       8        0          1        41371   0.000    60       22       3340    1869    8570   11288   11288     11288    11616    1470\n       6  2200000      8       8        0          1        41025   0.000    60       22       3291    1706    8616   11429   11429     11429    11609    1585\n       7  2200000     10      10        0          1        46852   0.000    60       22       2886    1927    8794   11968   11968     11968    12126     959\n\n=\u003e Even less interruptions on the isolated CPU's\n\nMove all moveable kernel threads away from the isolated CPUs:\n\n    # tuna -U -t '*' -c 0-4 -m\n\nOSjitter:\n\n    $ ./osjitter  -t 60\n     CPU  TSC_khz  #intr  #delta  ovfl_ns  invol_ctx  sum_intr_ns  iratio  rt_s  loop_ns  median_ns  p20_ns  p80_ns  p90_ns  p99_ns  p99.9_ns   max_ns  mad_ns\n       0  2200000  60246   60246        0       4333    231374600   0.004    60       22       3177    3040    3595    4465   10924     29714   469030     134\n       1  2200000  60403   60403        0       5965    198823307   0.003    60       22       2490    2274    3425    4865    6387     16643  4743847     229\n       2  2200000  60445   60445        0       5020    186508000   0.003    60       22       2402    2172    2959    3740    5762     12846  1716645     209\n       3  2200000  60490   60490        0      10195    234402816   0.004    60       22       2825    2308    4398    5358    6915    112854  3997080     576\n       4  2200000  60276   60276        0       7274    212001750   0.004    60       22       2531    2328    3668    5061    5747     13550  6431210     275\n       5  2200000      8       8        0          1        34188   0.000    60       22       3197    1765    5095    8923    8923      8923    11685    1114\n       6  2200000      8       8        0          1        39910   0.000    60       22       3218    1616    8130   11231   11231     11231    11793    1601\n       7  2200000      5       5        0          0        16998   0.000    60       22       2091    2079    8506    8506    8506      8506     8506     574\n\n=\u003e Isolated CPUs: Improvements in interruptions, few improvements\nin median, max and median absolute deviation (MAD).\n\nSwitch from throughput-performance based tuned profile to a latency-performance\nbased one (i.e. disable CPU frequency scaling, longer stat interval, writeback\ncpubask etc.):\n\n    # tuned-adm profile gs-latency\n\nOSjitter:\n\n    $ ./osjitter  -t 60\n     CPU  TSC_khz  #intr  #delta  ovfl_ns  invol_ctx  sum_intr_ns  iratio  rt_s  loop_ns  median_ns  p20_ns  p80_ns  p90_ns  p99_ns  p99.9_ns   max_ns  mad_ns\n       0  2200000  60250   60250        0        686    213519597   0.004    60       22       3125    3008    3250    3323   13616     37892  1871887      97\n       1  2200000  60223   60223        0      26628    287996052   0.005    60       22       3118    2914    6182    6266    7117     17085  5240030     777\n       2  2200000  60241   60241        0      26289    272751612   0.005    60       22       3079    2889    6183    6260    6480      9952  1231324     728\n       3  2200000  60193   60193        0        167    163954807   0.003    60       22       2360    2123    2470    2526    3210     13830  8119388     124\n       4  2200000  60223   60223        0        120    161220610   0.003    60       22       2427    2231    2514    2566    3060     13410  1885120      99\n       5  2200000      5       5        0          1        14843   0.000    60       21       2255    1897    6112    6112    6112      6112     6112     402\n       6  2200000      5       5        0          0        17074   0.000    60       22       2144    1852    8859    8859    8859      8859     8859     389\n       7  2200000      5       5        0          0        16665   0.000    60       22       1922    1808    8630    8630    8630      8630     8630     234\n\n=\u003e Isolated CPUs: less interruptions, less total interruptions, improvements in median, max and MAD\n\n## How it works\n\nOSjitter creates a measurement thread for each selected CPU that\npolls the CPU's [Time Stamp Counter (TSC)][tsc]. In each\niteration the previous counter value is subtracted from the\nprevious one and if that duration is above the threshold\n(default: 100 ns) it's counted as an interruption.\n\nSince the 1990ies, x86 CPUs feature a TSC, which can be read with\na special instruction from any user-space program. The TSC on\nrelatively modern CPUs is supposed to run constant and reliable,\ni.e. even during CPU-frequency changes and power-saving state\nchanges. That means that the TSC frequency (although constant)\nmay be different from the base frequency of the CPU. Since the\nTSC is integrated into the CPU, can be accessed like a register\n(with low overhead) and has a high accuracy it's well suited for\nmeasuring even short interruptions.\n\nWhen a program is interrupted by the operating system the TSC\nticks continue and thus after the program execution continues\n(otherwise transparently to the program) it can derive how long\nit was interrupted by looking at the current TSC value.\n\nThe actual TSC frequency is required to convert TSC counts to\nnanoseconds. OSjitter obtains the TSC frequency from the kernel,\ni.e. from `/sys/devices/system/cpu/cpu0/tsc_freq_khz` (if\navailable) or it parses it from `journalctl --boot` ([relevant\nstackoverflow answer][2]).\n\n## How to build\n\nFor most utilities:\n\n    $ make\n\nThe syscall benchmark:\n\n    $ git submodule update --init\n    $ mkdir build\n    $ cd build\n    $ CXXFLAGS='-Wall -O3 -g' cmake .. -DCMAKE_BUILD_TYPE=Release -DBENCHMARK_ENABLE_GTEST_TESTS=0 -GNinja\n    $ ninja\n\n(or a similar cmake invocation)\n\n## Related Work\n\nThere is [sysjitter][sj] (1.4, GPLv3) which also reads the [TSC][tsc] in\na loop to detect external interruptions. Some differences are:\n\n- Sysjitter calibrates the TSC frequency against `gettimeofday()`\n  whereas OSJitter just obtains the Kernel's TSC frequency\n  (the Kernel is in a better position to calibrate the TSC\n  frequency and Linux contains a well-engineered calibration\n  logic including possible refinements after the first\n  calibration)\n- Sysjitter just invokes the RDTSC instruction while OSjitter\n  invokes RDTSC and RDTSCP in combination with fencing\n  instructions\n- OSjitter uses ISO C atomic operations while Sysjitter uses GCC\n  atomic intrinsics\n- In contrast to OSjitter, sysjitter doesn't allow to specify the\n  scheduling class/priority of the measurement threads\n- OSjitter's output includes a measure for dispersion (MAD)\n- Besides TSC on x86, sysjitter also support reading a timestamp\n  counter on POWER CPUs.\n\nThe Linux Kernel contains a [hardware latency detector][hwl] to\ncheck for interruptions caused outside of the operating system\nsuch as the [System Management Mode][smm] (SMM). It also queries\nthe TSC in a loop.\n\nThe SMM is triggered by System Management Interrupts (SMI)\nwhich are transparent to the kernel and can only be detected\nindirectly. An alternative to the TSC approach for detecting and\nmeasuring SMIs is to query CPU counters the SMI changes\n([relevant stackoverflow answer][1]).\n\n[Cyclictest][cyc] measures OS latency by [setting\ntimers][cyc2] and comparing the actual sleep time with the\nconfigured one.\n\nErik Rigtorp has published\n[hiccups](https://github.com/rigtorp/hiccups) to measure 'system\ninduced jitter',\n[ipc-bench](https://github.com/rigtorp/ipc-bench) as a ping-pong\nlatency benchmark and [c2clat](https://github.com/rigtorp/c2clat)\nto measure inter-core latency. The hiccups repository references\n[osnoise](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/trace/osnoise-tracer.rst),\nan OS jitter detector built into the Linux kernel which appeared\nin Linux 5.14 or so that complements the above mentioned hardware\nlatency detector.\n\n## Pingpong Results\n\nThe doc directory contains some example Pingpong results for\ndifferent configurations.\n\nThe results for condition variable, semaphore and futex are quite\nsimilar because, on Linux, condition variables and semaphores are\nimplemented in terms of futex.\n\nNotifying via a traditional UNIX pipe is more expensive than\nusing a futex but it's the same order of magnitude.\n\nInserting a PAUSE instruction while spinning on an atomic\nvariable increases the median absolute deviation (MAD) just a\nlittle bit, but yields similar median while reducing the number\nof executed instructions.\n\nAs documented in the kernel documentation, comparing the results\nwith and without `full_hz=` show how this features increases\ncontext-switch overhead and thus increases latency for the\nsyscall methods (e.g. by 0.6 us or so in the median, a few us in\nthe other percentiles and maximum). On the other hand, more\ncontext-switch overhead isn't relevant for spinning on an atomic\nvariable, thus, `full_hz=` really pays off for this use-case\nbecause the process is interrupted much less.\n\n\n[sj]: https://www.openonload.org/download.html\n[hwl]: https://www.kernel.org/doc/html/latest/trace/hwlat_detector.html\n[smm]: https://en.wikipedia.org/wiki/System_Management_Mode\n[1]: https://stackoverflow.com/a/57961772/427158\n[tsc]: https://en.wikipedia.org/wiki/Time_Stamp_Counter\n[cyc]: https://git.kernel.org/pub/scm/linux/kernel/git/clrkwllms/rt-tests.git\n[cyc2]: http://people.redhat.com/williams/latency-howto/rt-latency-howto.txt\n[2]: https://stackoverflow.com/a/57835630/427158\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgsauthof%2Fosjitter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgsauthof%2Fosjitter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgsauthof%2Fosjitter/lists"}