{"id":19369726,"url":"https://github.com/xtermjs/xterm-benchmark","last_synced_at":"2025-04-23T15:31:48.295Z","repository":{"id":34922765,"uuid":"150497096","full_name":"xtermjs/xterm-benchmark","owner":"xtermjs","description":"A benchmark tool for measuring performance of xterm.js","archived":false,"fork":false,"pushed_at":"2022-05-01T15:09:37.000Z","size":233,"stargazers_count":4,"open_issues_count":3,"forks_count":3,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-10-11T09:44:31.851Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/xtermjs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-09-26T22:24:56.000Z","updated_at":"2023-11-06T03:30:04.000Z","dependencies_parsed_at":"2022-08-08T03:15:15.929Z","dependency_job_id":null,"html_url":"https://github.com/xtermjs/xterm-benchmark","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xtermjs%2Fxterm-benchmark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xtermjs%2Fxterm-benchmark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xtermjs%2Fxterm-benchmark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xtermjs%2Fxterm-benchmark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/xtermjs","download_url":"https://codeload.github.com/xtermjs/xterm-benchmark/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223926868,"owners_count":17226455,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-10T08:13:08.398Z","updated_at":"2024-11-10T08:13:08.879Z","avatar_url":"https://github.com/xtermjs.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"## xterm-benchmark\r\n\r\nLibrary to create and run automated performance test cases.\r\n\r\n### Current state\r\n\r\nThe library and the cmdline tool are in alpha state and likely to see heavy API changes.\r\n\r\n### Cmdline options\r\n```\r\nUsage: xterm-benchmark [files]\r\n\r\nOptions:\r\n  -v, --version         output the version number\r\n  -t, --tree \u003cfile\u003e     show contex and perf case tree\r\n  -s, --single \u003cpath\u003e   run single context or perf case\r\n  -r, --repeat \u003cnum\u003e    repeat cases \u003cnum\u003e times\r\n  -t, --timeout \u003cnum\u003e   set timeout to \u003cnum\u003e msec\r\n  -o, --output \u003cpath\u003e   set output path (defaults to \u003cAPP_PATH\u003e/\u003cEPOCH\u003e.log)\r\n  -S, --silent          no console log output\r\n  -j, --json            output json to stdout, equals \"-S -l /dev/stdout\"\r\n  -f, --full            include full results in reports\r\n  -F, --fail            also fail on missings\r\n  -b, --baseline        mark run as baseline data\r\n  -c, --config \u003cpath\u003e   path to config file\r\n  -e, --eval            eval run against baseline data\r\n  -a, --against \u003cpath\u003e  baseline data path to eval against\r\n  -h, --help            output usage information\r\n```\r\n\r\n### Quick usage\r\n\r\n- create perf case files (see example below or `src/xterm_perfcases`)\r\n- run a set of files to create the baseline data, e.g. `xterm-benchmark some_files* -b`\r\n- after code changes run the same set of files against the baseline, e.g. `xterm-benchmark some_files* -e`\r\n\r\n### Running Example Perf Case\r\n- cd xterm-benchmark \u0026\u0026 npm install (If this throws puppeteer error then try deleting the whole puppeteer folder before doing npm install again.)\r\n- npm run tsc ( Compiles ts files into js files which can be found under ./lib folder)\r\n- node lib/cli.js lib/xterm_perfcases/parser.js ( node lib/cli.js \u003cpath-to-js-perfcase-file\u003e)\r\n\r\n### Configuration\r\n\r\nThe config file currently supports the following settings:\r\n\r\n- APP_PATH `string`\u003cbr\u003ebase folder path, logs get stored here\r\n- defaultOptions:\r\n  - fork `boolean`\u003cbr\u003ewhether to run cases in a separate process (overridable at individual perf case)\r\n  - repeat `number`\u003cbr\u003erepeat cases n times (overridable at individual perf case or at cmdline)\r\n- evalConfig\r\n  - tolerance `{[path: string]: value}`\u003cbr\u003e`path` is a value path in the form `\u003cperf_case_path\u003e#\u003csummaryEntry_name\u003e.\u003cvalue_name\u003e.\u003caggregate_name\u003e`,\r\n  path supports simple matching rules (e.g. `'*'` will match all values in all perf cases,\r\n  `\"*.median\"` will match all median values in all perf cases).\r\n  value is an array in the form of `[low_border, high_border]`, where the border values are relative\r\n  to the baseline value. Example: `{\"*.mean\": [0.5, 2]}` - eval runs will pass all mean tests if they\r\n  are at least half and at most twice of the baseline value\r\n  - skip `string[]`\u003cbr\u003elist of value paths to be skipped during evaluation\r\n\r\n### Creating perf cases\r\n\r\nPerf cases are instances of a derived class of the base class `PerfCase`.\r\nTo keep it highly customizable, `PerfCase` can be extended by mixins (see `src/example/custom_mixin.ts`).\r\n\r\nThe library comes with several predefined perf case classes:\r\n- RuntimeCase: contains mixin for runtime reports\r\n- ThroughputRuntimeCase: contains runtime and throughput mixins\r\n\r\nA simple runtime reporting perf case would look like this:\r\n```TS\r\nimport { RuntimeCase } from '..';\r\n\r\nnew RuntimeCase('Something to measure', async () =\u003e {\r\n  // some work\r\n  await new Promise(resolve =\u003e setTimeout(resolve, 1000));\r\n})\r\n.showRuntime()          // show runtime for single run\r\n.showAverageRuntime();  // show aggregated runtime for multiple runs\r\n```\r\n\r\nPerf cases can be put into a context that support running preparation and cleanup code with `before`, `beforeEach`, `after` and `afterEach`. A single file automatically contains a toplevel context.\r\n```TS\r\nimport { before, after, beforeEach, afterEach, perfContext } from '..';\r\n\r\nbefore(() =\u003e console.log('run once upon entering the file (always first)'));\r\nafter(() =\u003e console.log('run once upon leaving the file (always last)'));\r\nbeforeEach(() =\u003e console.log('run before each perf case or context'));\r\nafterEach(() =\u003e console.log('run after each perf case or context'));\r\n\r\nperfContext('some ctx', () =\u003e {\r\n  // perf cases or sub contexts...\r\n});\r\n\r\n// more perf cases or additional contexts...\r\n```\r\n\r\n### Baseline \u0026 Evaluation hints\r\n\r\n- To get somewhat reliable baseline data it is important to run each perf case several times\r\n(the more the better, do at least 10 times). This way individual run differences due to a slightly different\r\nsystem load at a single run can be levelled out (see `-r` cmdline switch).\r\n- To level out impact of previous perf cases within the same process consider running them as forked for\r\nbaseline and eval runs (TODO: always fork cmdline switch).\r\n- Currrently an unusual constant high/low system load, that would affect all baseline runs, cannot be spotted. (This might change with future versions.) Therefore it is important not to create the baseline data during unusual usage scenarios to keep the baseline data clean.\r\n- Implemented numercial results of a single run include:\r\n  - number of runs (all perf case types)\r\n  - total runtime in msec (`RuntimeCase`)\r\n  - throughput in MB/s (`Throughput` mixin)\r\n  - puppeteer perf cases (`ExtractFromTimeline` mixin):\r\n      - summary values (devtools pie chart)\r\n      - requested topDown values with self and total runtime\r\n- Numercial baseline values are aggregated for all runs as following (basic descriptive statistics):\r\n  - mean average (`mean`)\r\n  - median average (`median`)\r\n  - standard deviation (`dev`)\r\n  - coefficient of variance (`cv`)\r\n- Every aggregated value has a tolerance range given as `[low_border, high_border]`. The borders\r\nare relative to the baseline value, currently it is not possible to set absolute values (might change).\r\n- Individual tolerance ranges can be set in the config file by addressing a certain value (see Configuration).\r\n- The automated evaluation is tested against the tolerance range, means if the evaluated value is within the tolerance of the baseline value it passes the test.\r\n- What to make out of the statistical values?\r\n  - standard deviation: should be as low as possible\r\n  - problem high standard deviation: dataset should not be used as baseline or eval as it indicates that the indivual runs show very different behavior, possibly caused by low run number or high fluctuations in system load\r\n  - coefficient of variance: shows the \"quality\" of the calculated mean value\r\n  - problem high coefficient of variance: calculated mean value is highly uncertain, data should never be used for baseline or eval runs, to rule out system load fluctuations this value typically should stay far below 0.3\r\n  - median: should not be used by default, still can be useful if the value clearly shows non normal distribution\r\n  to still get some comparison running (example: due to ongoing JIT optimization some values might show a decreasing runtime over several runs, others might get worse due to heavier GC invocation)\r\n- Unwanted values for evaluation can be skipped in the config file.\r\n\r\n\r\n### Version notes\r\n\r\n- 0.3.0 removes the dependency to the `chrome-timeline` package, thus `TimelinePerfCase` is not defined anymore.\r\n  If your perf cases rely on frontend performance testing with that type, either use an older version, or feel free to help upgrading `chrome-timeline` to support newer browser engines as well.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxtermjs%2Fxterm-benchmark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fxtermjs%2Fxterm-benchmark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxtermjs%2Fxterm-benchmark/lists"}