{"id":15148720,"url":"https://github.com/softdevteam/krun","last_synced_at":"2025-08-26T08:16:12.257Z","repository":{"id":33026310,"uuid":"36661834","full_name":"softdevteam/krun","owner":"softdevteam","description":"High fidelity benchmark runner","archived":false,"fork":false,"pushed_at":"2021-07-28T09:34:33.000Z","size":2006,"stargazers_count":86,"open_issues_count":12,"forks_count":10,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-01-30T23:41:17.618Z","etag":null,"topics":["benchmark","experiment","linux","openbsd"],"latest_commit_sha":null,"homepage":"http://soft-dev.org/src/krun/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/softdevteam.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE-APACHE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-06-01T13:17:06.000Z","updated_at":"2024-12-19T04:59:43.000Z","dependencies_parsed_at":"2022-07-17T22:00:37.779Z","dependency_job_id":null,"html_url":"https://github.com/softdevteam/krun","commit_stats":null,"previous_names":[],"tags_count":16,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/softdevteam%2Fkrun","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/softdevteam%2Fkrun/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/softdevteam%2Fkrun/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/softdevteam%2Fkrun/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/softdevteam","download_url":"https://codeload.github.com/softdevteam/krun/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":237915423,"owners_count":19386724,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark","experiment","linux","openbsd"],"created_at":"2024-09-26T13:22:05.294Z","updated_at":"2025-02-09T06:31:49.664Z","avatar_url":"https://github.com/softdevteam.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Krun\n\nKrun is a framework for running high-quality software benchmarking experiments.\nKrun experiments consist of a configuration file, a carefully configured\nbenchmarking machine, and the benchmarks themselves.\n\n\n## Step 1: Initial installation\n\nKrun currently only runs on Debian Linux and OpenBSD. Porting it to other\nUnix variants is unlikely to be difficult, and we welcome patches.\n\n### Dependencies\n\nYou need to have the following programs installed:\n\n  * sudo\n  * Python-2.7\n  * Python dependencies (see requirements.txt)\n  * GNU make, a C compiler and libc (`build-essential` package in Debian)\n  * cpufrequtils (Linux only. `cpufrequtils` package in Debian)\n  * cset (for pinning on Linux only. `cpuset` package in Debian)\n  * virt-what (Linux only. `virt-what` package in Debian)\n  * Optionally, our custom Linux kernel (Linux only, see below)\n  * Linux kernel headers (Linux only. `linux-headers-X.Y` package in Debian)\n  * taskset (Linux only. `util-linux` package in Debian)\n  * msr-tools (Linux only. `msr-tools` package in Debian)\n  * policykit (Linux only, only if you want to use `systemctl start/stop` in\n    `PRE/POST_EXECUTION_CMDS`. See *Benchmarking for Reliable Results* below)\n\nIf you choose to use `pip` for the Python dependencies, you can install them\nusing `pip -r requirements.txt`.\n\nIf you want to benchmark Java, you will also need:\n\n  * A Java SDK (`openjdk-*-jdk` package in Debian).\n\n\n## Step 2 (Linux only): kernel and OS setup\n\n### P-states\n\nBenchmarking is at its most accurate when Intel p-states are disabled in\nthe kernel. If you are using Grub, this can be achieved as follows:\n\n  * Edit `/etc/default/grub` so that the `GRUB_CMDLINE_LINUX_DEFAULT`\n    variable includes `intel_pstate=disable`.\n  * Run `sudo update-grub`\n\nIf you are unable to do this, you can disable Krun's P-state check with\n`--disable-pstate-check`, but be aware that this degrades the quality of the\nresulting benchmarking numbers.\n\n### Performance counters\n\nIf you want low latency access to the following counters:\n\n  * `IA32_PERF_FIXED_CTR1` (the core cycle counter)\n  * `IA32_APERF` counts\n  * `IA32_MPERF` counts\n\nyou will need to use the (now rather out of date) custom Linux kernel at:\n\n  https://github.com/softdevteam/krun-linux-kernel\n\nIf you do use this kernel you will need to set `MSRS=1` in your Unix\nenvironment when building Krun (see later).\n\n### Tickless Kernel\n\nUnless `--no-tickless-check` is passed to Krun, all cores but the boot core\nare  expected to be in full adaptive ticks mode (tickless mode).\n\nOn older kernels which still have the `CONFIG_NO_HZ_FULL_ALL` config option,\nbuild the kernel with this set to \"y\".\n\nOn newer kernels, where `CONFIG_NO_HZ_FULL_ALL` is absent, you will have to add\na `nohz_full=1-N` kernel command line option (where N is the number of cores\nyour system has, minus one).\n\nIf your system has only one core then tickless mode is not checked, as the boot\ncore (i.e. your only core) cannot be placed into adaptive ticks mode.\n\n## Step 3: Fetch and build Krun\n\nFirst fetch Krun:\n\n```sh\n$ git clone --recursive https://github.com/softdevteam/krun.git\n$ cd krun\n```\n\nThen run `make` (or `gmake` on OpenBSD). The Krun Makefile honours the standard\nvariables: `CC`, `CPPFLAGS`, `CFLAGS` and `LDFLAGS`.\n\nIf you want to benchmark Java programs, you need to set the\n`JAVA_HOME` environment variable to point to your JDK installation, and\nset several other flags:\n\n```sh\n$ make clean\n$ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/ make  \\\n    JAVA_CPPFLAGS='\"-I${JAVA_HOME}/include -I${JAVA_HOME}/include/linux\"' \\\n    JAVA_LDFLAGS=-L${JAVA_HOME}/lib ENABLE_JAVA=1\n```\n\n\n## Step 4: Audit system services\n\nBackground services (e.g. `cron` or `sendmail`) can interfere with benchmarking.\nThe more services that you are able to switch off, the less interference is\nlikely to occur. Some services are best disabled at boot and/or permanently\n(depending on your OS) and must be done manually. However, you may wish\nto disable some services only during benchmarking (e.g. you may wish to have a mail server\nrunning before and after benchmarking to inform you of benchmarking progress),\nwhich can be specified in the `PRE_EXECUTION_CMDS` and `POST_EXECUTION_CMDS`\nsettings in your Krun config file.\n\nCommands in each list are run, in order, using the `krun` user's shell (e.g.\n`/bin/sh`). If a command fails, Krun stops execution immediately without running\nsubsequent commands. If you wish execution to continue even if a command fails\nyou can use standard shell idioms: e.g. `cmd || true` guarantees that the\noverall command succeeds even if `cmd` fails. \n\nFor example on a systemd Linux you may turn daemons off before execution with:\n\n```\nPRE_EXECUTION_CMDS = [\n    \"sudo systemctl stop cron\",\n    \"sudo systemctl stop atd\",\n    ...\n]\n```\n\nand turn them back on with:\n\n```\nPOST_EXECUTION_CMDS = [\n    \"sudo systemctl start cron || true\",\n    \"sudo systemctl start atd || true\",\n    ...\n]\n```\n\nIn general it is best practise to turn things back on explicitly, because after\nKrun runs the final benchmark it will not reboot the machine. If, for example,\nyou put network interfaces down in `PRE_EXECUTION_CMDS`, you should put them\nback up in `POST_EXECUTION_CMDS` so that you can login to the machine after the\nfinal benchmark has been run. We urge you to check such commands carefully:\nsmall oversights can easily lead to you locking yourself out of the system.\n\nKrun can also copy intermediate results to a remote host and query that host to\nsee whether it should suspend benchmarking. See\n`https://github.com/softdevteam/warmup_experiment/blob/master/warmup.krun` for\nmore advanced options.\n\n\n### Linux\n\nNote that Debian has, from a benchmarking perspective, the unfortunate habit of\nautomatically starting daemons which get pulled in by dependencies.\n\nOn Linux, list services with:\n\n```sh\n# systemctl | grep running\n```\n\nPermanently disable services (including after system reset) with:\n\n```sh\n# systemctl stop \u003cservice\u003e\n# systemctl disable \u003cservice\u003e\n```\n\nCommonly enabled services that you may wish to disable:\n\n * apache2\n * memcached\n * nfs-common\n\n\n### OpenBSD\n\nOn OpenBSD, list services with:\n\n```sh\n# doas rcctl ls started\n```\n\nPermanently disable services (including after system reset) with:\n```sh\n# rcctl stop \u003cservice\u003e\n# rcctl disable \u003cservice\u003e\n```\n\nCommonly enabled services that you may wish to disable:\n\n * pflogd\n * sndiod\n\nNote that Krun is unable to check whether turbo mode is enabled on OpenBSD or\nnot, and is also unable to use APERF/MPERF ratios to indirectly check whether\nturbo mode was used. You should therefore be particularly careful to check that\nturbo mode is disabled when benchmarking on OpenBSD.\n\n\n## Step 5: Build and run the example\n\nThe `examples` directory contains the `example.krun` experiment. This contains\ntwo benchmark programs (*nbody* and *dummy*), both of which are run on *C* and\n*PyPy*. Each benchmark is run for 5 *in-process iterations* (where the\nbenchmark is repeated 5 times within a for loop within a single process) across\n2 *process executions* (where the entire VM is restarted).\n\nFirst build the examples:\n\n```sh\n$ cd examples/benchmarks\n$ pwd\n.../krun/examples/benchmarks\n$ make\n```\n\nIf you also want to try the example Java benchmarks, you must build them\nas a separate step. Additionally, be sure to have compiled Krun with Java\nsupport as described in Step 3\n\n```sh\n$ pwd\n.../krun/examples/benchmarks\n$ make java-bench\n```\n\nThen run the example:\n\n```sh\n$ cd krun/examples\n$ ../krun.py example.krun\n```\n\nYou should see a log scroll past, and results will be stored in the file:\n`../krun/examples/example_results.json.bz2`.\n\nIf you want to try the example Java benchmarks, there is a separate\nconfiguration file called `java.krun`, which contains configuration for the Java\nand Python examples:\n\n```sh\n$ ../krun.py java.krun\n```\n\nNote, this will only work if you have followed the earlier steps to compile\nKrun with Java support.\n\n\n### The Krun user\n\nKrun runs benchmarks under a new Unix user `krun`, which is wiped and re-added\nbefore every experiment. Your Krun build and your experiment must both be\nreadable by the `krun` user for the experiment to run.\n\n\n## Creating your own experiments\n\nIt is easiest to use `examples/example.krun` as a template for your own,\nnew, experiments. Note that: the benchmarks referenced in the config file\n*must* be in a `benchmarks` subdirectory; and that each benchmark in\nthe config must be in a subsubdirectory with a matching name. A typical\ndirectory structure is therefore as follows:\n\n```\nexperiment/\n    experiment.krun\n    benchmarks/\n        Makefile\n        benchmark_1/\n            language_1/\n            benchmark_file.lang1\n    ...\n```\n\nThe top-level `Makefile` should build the VMs and benchmarks needed for the\nexperiment.\n\nEach benchmark should expose a function (or method) called `run_iter` which is\nthe entry point to the benchmark. To preserve source code history, it can\nbe easiest to put this function in a new file, which then imports the benchmark.\n\nWith regards to VM/compiler support, there are two ways Krun can invoke a\nbenchmark:\n\n  * Via a dedicated \"VM definition\" (e.g. `JavaVMDef`).\n  * Via an external benchmark suite (`ExternalSuiteVMDef`).\n\nThe former option is best, as it supports Krun's core-cycle counting and\nAPERF/MPERF ratio features. The following compilers/VMs are currently supported\nfor this mode:\n\n  * Native code languages (`NativeCodeVMDef`).\n  * OpenJDK. (i.e. Hotspot) (`JavaVMDef`).\n  * GraalVM (`GraalVMDef`).\n  * cPython (`PythonVMDef`).\n  * Lua (`LuaVMDef`).\n  * PHP (`PHPVMDef`).\n  * Ruby (`RubyVMDef`).\n  * TruffleRuby (`TruffleRubyVMDef`).\n  * Javascript V8 (`V8VMDef`).\n\nIf your VM isn't listed, you can either add it to Krun, or use the external\nsuite definition (see below). To add a new VM definition, add a new class to\n`krun/vm_defs.py` and a new iteration runner to the `iterations_runners`\ndirectory.\n\nThe latter option -- `ExternalSuiteVMDef` -- is useful if you want to quickly\nwrap an existing benchmark suite. For an example see `examples/ext.krun` and\n`examples/ext_script.py`.\n\nTo add a new platform definition, add a new class to `krun/platform.py`.\n\n## Testing your configurations\n\nBefore doing a full run of an experiment, you should perform a quick(ish) test\nof your configuration. This can be achieved with:\n\n```sh\n$ /path/to/krun/krun.py --dry-run --quick --debug=INFO config.krun\n```\n\nSee the \"Development and Debug Switches\" section for a description of these\nswitches.\n\n\n## Production benchmarking\n\nAchieving the highest possible benchmarking quality requires more care. First,\nnone of Krun's debug or development switches must be used. Second, Krun needs to\nrun in \"reboot mode\" where each process execution will be run after the machine\nhas (automatically) rebooted. The simplest way to do this is to have your init\nsystem invoke `scripts/run_krun_at_boot` via an `rc.local` script.\n\nYour `/etc/rc.local` should look like this:\n```\n#!/bin/sh\n/usr/bin/sudo -u someuser /path/to/krun/scripts/run_krun_at_boot /path/to/your/config.krun\nexit 0\n```\n\nMake sure you replace the paths as appropriate and substitute `someuser` with\nthe name of a normal unprivileged user that you wish to use to kick off Krun.\n\nMake sure `/etc/rc.local` is executable and that it only contains absolute\npaths. Note that `sudo(8)` is installed in different places on different\noperating systems (for OpenBSD, it's `/usr/local/bin/sudo`).\n\nAny arguments supplied after the config file path are passed to Krun unchanged.\n\nYou can then start the experiment by manually running the command from your new\n`rc.local` (i.e. `sudo -u ...`).\n\n\n### Monitoring progress\n\nWhilst benchmarking is occuring, you must not log in to the machine (indeed,\nhopefully `sshd`, or equivalent, has been disabled!). To monitor progress, and\nbe informed of errors, you you should add a `MAIL_TO` list of emails to your\nKrun config file:\n\n```python\nMAIL_TO = [\"me@mydomain.com\", \"other_person@herdomain.com\"]\n```\n\nKrun uses `sendmail(8)` to send email, so you will need to make sure that\nyou have a functional SMTP server installed (and don't forget to switch it off\nduring benchmarking!).\n\n\n## Custom Dmesg Whitelists\n\nFor each platform, Krun has a default built-in dmesg whitelist. The whitelist\nis a collection of regex patterns which are used to decide if a line in the\ndmesg buffer is a cause for concern. If a new line appears in the dmesg during\nbenchmarking, and the line is not matched by at least one whitelist pattern,\nthen Krun will flag the process execution as ``errored'' and email the user.\n\nFrom time to time you may find that you need to customise the whitelist. This\nis achieved by adding a callback named `custom_dmesg_whitelist` into your\nconfig file. The callback is passed the default list of patterns for your\nplatform and must return a new list of patterns. In the implementation of your\ncallback you have the choice to base your custom whitelist on the defaults or\nto define your own patterns from scratch.\n\nFor example, to add a pattern, you would add a callback like:\n\n```python\ndef custom_dmesg_whitelist(defaults):\n    return defaults + [\"^.*your+regex.*pattern$\"]\n```\n\nKrun uses Python's `re` module to compile regex patterns. Consult the Python\ndocs for more information on the regex syntax.\n\nBear in mind that Linux dmesg lines start with a time code which custom dmesg\nlines will need to match.\n\nIf you have added custom patterns which you think would be useful for other\nusers of Krun, please raise an issue (or pull request) to have the patterns\nadded to the defaults.\n\n\n## Development and Debug Switches\n\nIf you are making changes to Krun itself (for example, to add a new platform or\nvirtual machine definition), there are a few switches which can make your life\neasier.\n\n  * `--debug=\u003clevel\u003e`: Sets the verbosity of Krun.  Valid debug levels are:\n     `DEBUG`, `INFO`, `WARN`, `DEBUG`, `CRITICAL` and `ERROR`. The default is\n     `WARN`. Production quality benchmarking should use the default.\n\n  * `--quick`: There are several places where Krun pauses to allow the system\n    to stabilise. In testing these pauses can be burdensome and can thus\n    be skipped with `--quick`.\n\n  * `--no-user-change`: Without this flag, For each process execution, Krun\n    will use a fresh user account called 'krun'. This involves deleting any\n    existing user account (with `userdel -r`) and creating a new user account\n    (with `useradd -m`).  This switch disables the use of a fresh user account,\n    meaning that `userdel` and `useradd` are not invoked, nor does Krun switch\n    user; the user Krun was invoked with is used for benchmarking.\n\n  * `--dry-run`: Fakes actual benchmark processes, making them finish\n    instantaneously.\n\n  * `--no-tickless-check`: Do not crash out if the Linux kernel is not\n    tickless.\n\n  * `--no-pstate-check`: Do not crash out if Intel P-states are not disabled.\n\n\n## Krun results files\n\nKrun generates a bzipped JSON file containing results of all process executions.\nThe structure of the JSON results is as follows:\n\n```python\n{\n    'audit': '',  # A dict containing platform information\n    'config': '', # A unicode object containing your Krun configuration\n    'wallclock_times': {        # A dict containing timing results\n        'bmark:VM:variant': [   # A list of lists of in-process iteration times\n            [ ... ], ...        # One list per process execution\n        ]\n    },\n    'core_cycle_counts': {      # Per-core core cycle counter deltas\n        'bmark:VM:variant': [\n            [                   # One list per process execution\n                [...], ...      # One list per core\n            ]\n    },\n    'aperf_counts': {...}       # Per-core APERF deltas\n                                # (structure same as 'core_cycle_counts')\n    'mperf_counts': {...}       # Per-core MPERF deltas\n                                # (structure same as 'core_cycle_counts')\n    'pexec_flags': {...}        # A flag for each process execution:\n                                # 'C' completed OK.\n                                # 'E' benchmark crashed.\n                                # 'T' benchmark timed out.\n    'eta_estimates': {u\"bmark:VM:variant\": [t_0, t_1, ...], ...} # A dict mapping\n                  # benchmark keys to rough process execution times. Used internally:\n                  # users can ignore this.\n}\n```\n\nSome options exist to help inspect the results file:\n\n  * `--dump-reboots`\n  * `--dump-etas`\n  * `--dump-config`\n  * `--dump-audits`\n  * `--dump-temps`\n  * `--dump-data`\n\n```sh\n$ python krun.py --dump-config examples/example_results.json.bz2\nINFO:root:Krun starting...\n[2015-11-02 14:23:31: INFO] Krun starting...\nimport os\nfrom krun.vm_defs import (PythonVMDef, NativeVMDef)\nfrom krun import EntryPoint\n\n# Who to mail\nMAIL_TO = []\n...\n\n$ python krun.py --dump-audit examples/example_results.json.bz2\n{\n    \"cpuinfo\":  \"processor\\t: 0\\nvendor_id\\t: GenuineIntel\\ncpu family\\t:\n...\n\n$ python krun.py --dump-reboots examples/example_results.json.bz2\n[2015-11-06 13:14:35: INFO] Krun starting...\n8\n```\n\n## Troubleshooting\n\n### `java.lang.UnsatisfiedLinkError` Error\n\nThe following error in the log file is indicative that Krun has not been\ncompiled with Java support:\n\n```\nException in thread \"main\" java.lang.UnsatisfiedLinkError: IterationsRunner.JNI_krun_init()V\n\tat IterationsRunner.JNI_krun_init(Native Method)\n\tat IterationsRunner.main(iterations_runner.java:248)\n```\n\nSee Step 3 on how to build with Java support.\n\n\n## Unit Tests\n\nKrun has a pytest suite which can be run by executing `py.test` in the\ntop-level source directory.\n\n\n## Security Notes\n\nKrun is not intended to be run on a secure multi-user system, as it uses `sudo`\nto elevate privileges. It also uses files with fixed names in `/tmp/` which means\nthat only one instance of Krun should be run at any one time (running more\nthan one leads to undefined behaviour).\n\n`sudo` is used to:\n\n * Add and remove a fresh benchmarking user for each process execution.\n * Switch users.\n * Change the CPU speed.\n * Set the perf sample rate (Linux only)\n * Automatically reboot the system (`--hardware-reboots` only).\n * Set process priorities.\n * Create cgroup shields (Linux only, off by default)\n * Detect virtualised hosts.\n * Unrestrict the dmesg buffer (Linux Kernel 4.8+)\n * Turn off \"turbo boost\" (Linux only)\n * Turn off memory over-commit (Linux only).\n\nPlease make sure you understand the implications of this.\n\n\n## Licenses\n\nKrun is licensed under the UPL license.\n\nThe nbody benchmark is licensed under a revised BSD license:\nhttp://shootout.alioth.debian.org/license.php\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsoftdevteam%2Fkrun","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsoftdevteam%2Fkrun","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsoftdevteam%2Fkrun/lists"}