{"id":18498340,"url":"https://github.com/softdevteam/warmup_stats","last_synced_at":"2025-04-09T00:31:14.451Z","repository":{"id":48065577,"uuid":"106446975","full_name":"softdevteam/warmup_stats","owner":"softdevteam","description":"A VM performance data analysis system","archived":false,"fork":false,"pushed_at":"2023-06-14T06:19:33.000Z","size":636,"stargazers_count":11,"open_issues_count":8,"forks_count":7,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-23T19:44:51.455Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/softdevteam.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE-APACHE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-10-10T17:08:02.000Z","updated_at":"2022-11-08T16:14:39.000Z","dependencies_parsed_at":"2024-11-06T18:16:21.908Z","dependency_job_id":null,"html_url":"https://github.com/softdevteam/warmup_stats","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/softdevteam%2Fwarmup_stats","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/softdevteam%2Fwarmup_stats/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/softdevteam%2Fwarmup_stats/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/softdevteam%2Fwarmup_stats/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/softdevteam","download_url":"https://codeload.github.com/softdevteam/warmup_stats/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247949838,"owners_count":21023400,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T13:38:49.502Z","updated_at":"2025-04-09T00:31:09.441Z","avatar_url":"https://github.com/softdevteam.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# warmup_stats\n\nThis system has various scripts for analysing the output of VM benchmarking. Any\nsystem can be used to perform the benchmarking, e.g.\n[Krun](http://soft-dev.org/src/krun/).\n\n# Build\n\nRun `./build.sh` to build warmup_stats.\n\n# Basic usage\n\nUser should directly call the `bin/warmup_stats`, which is a front-end to other\nscripts in `bin/`. `warmup_stats` takes either CSV files or\n[Krun](http://soft-dev.org/src/krun/) results files as input. As output it can\ncreate HTML or LaTeX / PDF tables and diffs, or PDF plots.\n\n## Terminology\n\nwarmup_stats uses the following terminology:\n\n  * A *process execution* is the execution of a single operating system\n    process. In other words, it is equivalent to running a program from the\n    command-line and waiting for it to terminate.\n\n  * An *in-process iteration* is a single iteration of a benchmark within a\n    process execution. In other words, a single process execution executes\n    many in-process iterations.\n\n## CSV format\n\nThe `bin/warmup_stats` script can take CSV files as input. The format is as\nfollows. The first row must contain a header for each required column: the\nfirst column must be the process execution index (conventionally a number\n`0...n`, though this is not enforced), the second column the benchmark name,\nand *n* subsequent columns for the *n* in-process iterations run (each a time\nin seconds). Each of these columns can be given arbitrary names, but the order\nis vital, and the number of in-process iterations must be the same for each\nprocess execution. An example is as follows:\n\n```\n    process_exec_idx, bench_name, 0, 1, 2, ...\n    0, spectral norm, 0.2, 0.1, 0.4, ...\n    1, spectral norm, 0.3, 0.15, 0.2, ...\n```\n\nWhen processing CSV with `warmup_stats`, the `--language`, `--vm` and\n`--uname` flags must be specified so that plots can contain the relevant\ninformation, though users can pass arbitrary data to each flag. Note that\nthese flags are not needed with Krun results files.\n\n## Creating plots\n\nThe `--output-plots \u003cfile.pdf\u003e` flag converts input data into visual plots.\n\nIf the input files are in CSV format, `bin/warmup_stats` also needs the names of\nthe language and VM under test, and the output of `uname -a` on the machine the\nbenchmarks were run on.\n\nExample usage:\n\n```sh\nbin/warmup_stats  --output-plots plots.pdf -l javascript -v V8 -u \"`uname -a`\" results.csv\nbin/warmup_stats  --output-plots plots.pdf results.json.bz2\n```\n\n## Creating tables\n\nThe `--output-table \u003cfile\u003e` flag converts input data into an HTML table or a\nLaTeX / PDF table. Conversion to PDF requires `pdflatex` to be installed.\n\nIf the input files are in CSV format, `bin/warmup_stats` also needs the names of\nthe language and VM under test, and the output of `uname -a` on the machine the\nbenchmarks were run on.\n\nExample usage (LaTeX / PDF):\n\n```sh\nbin/warmup_stats --tex --output-table table.tex -l javascript -v V8 -u \"`uname -a`\" results.csv\nbin/warmup_stats --tex --output-table table.tex results.json.bz2\n```\n\nExample usage (HTML):\n\n```sh\nbin/warmup_stats --html --output-table table.html -l javascript -v V8 -u \"`uname -a`\" results.csv\nbin/warmup_stats --html --output-table table.html results.json.bz2\n```\n\nBy default, `warmup_stats` produces high quality statistics, which can take\nconsiderable time. If you want to quickly experiment with things, you can use\nthe `--quality low` switch: this makes `warmup_stats` run considerably quicker,\nbut does lead to lower quality (and thus less reliable) statistics being\nproduced. Although the differences are often fairly minor, we do not encourage\nthe use of `--quality low` when formally publishing benchmark results.\n\n## Creating diffs\n\nBenchmarking is often performed in order to test whether a change in a given\nVM improves or worsens its performance. Unfortunately, the difference between\nbenchmark performance before and after a change is rarely simple. Users will\nwant to produce a detailed comparison of the results in Krun results tables\n(above) in order to get a deeper insight into the effects of their changes.\n\nThe `--output-diff` flag converts data from exactly two CSV files into an HTML\ntable or a LaTeX / PDF table. Conversion to PDF requires `pdflatex` to be\ninstalled.\n\nIf the input files are in CSV format, `bin/warmup_stats` also needs the names of\nthe language and VM under test, and the output of `uname -a` on the machine the\nbenchmarks were run on.\n\nExample usage (LaTeX / PDF):\n\n```sh\nbin/warmup_stats --tex --output-diff diff.tex -l javascript -v V8 -u \"`uname -a`\" before.csv after.csv\nbin/warmup_stats --tex --output-diff diff.tex before.json.bz2 after.json.bz2\n```\n\nExample usage (HTML):\n\n```sh\nbin/warmup_stats --html --output-diff diff.html -l javascript -v V8 -u \"`uname -a`\" before.csv after.csv\nbin/warmup_stats --html --output-diff diff.html before.json.bz2 after.json.bz2\n```\n\nThe resulting table will contain results from the `after.{csv,json.bz2}` file,\ncompared against the `before.{csv,json.bz2}` file. VMs and benchmarks that do\nnot appear in both CSV results files will be omitted from the table.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsoftdevteam%2Fwarmup_stats","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsoftdevteam%2Fwarmup_stats","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsoftdevteam%2Fwarmup_stats/lists"}