{"id":20713270,"url":"https://github.com/linaro/hpc_benchmark_analysis","last_synced_at":"2025-07-13T18:40:00.303Z","repository":{"id":90096728,"uuid":"116028846","full_name":"Linaro/hpc_benchmark_analysis","owner":"Linaro","description":"Random scripts to interpret benchmark / profiling results","archived":false,"fork":false,"pushed_at":"2020-01-15T15:01:45.000Z","size":132,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-11T06:49:23.817Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Linaro.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-01-02T15:23:16.000Z","updated_at":"2022-09-03T04:38:04.000Z","dependencies_parsed_at":null,"dependency_job_id":"cd760dba-9d39-46d3-9ddd-cb08646002a7","html_url":"https://github.com/Linaro/hpc_benchmark_analysis","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Linaro/hpc_benchmark_analysis","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Linaro%2Fhpc_benchmark_analysis","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Linaro%2Fhpc_benchmark_analysis/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Linaro%2Fhpc_benchmark_analysis/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Linaro%2Fhpc_benchmark_analysis/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Linaro","download_url":"https://codeload.github.com/Linaro/hpc_benchmark_analysis/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Linaro%2Fhpc_benchmark_analysis/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265188974,"owners_count":23725188,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-17T02:24:11.565Z","updated_at":"2025-07-13T18:40:00.284Z","avatar_url":"https://github.com/Linaro.png","language":"Python","readme":"# Benchmark Scripts\nScript/modules to interpret benchmark/profiling results.\n\n## Introduction\nThis set of modules will parse `perf stat` results as well as benchmark numbers (via plugins) and run the results through a number of statistical analysis passes (outlier detection, clustering, curve fitting) to identify potential \"low hanging fruits\" on a large space of results.\n\nThe advantage of this method becomes clear when you don't have prior knowledge on what bottlenecks exist in a system, or when you have so many degrees of freedom / dimensions on your test matrix that it's impossible for humans to \"get a feeling\" for what numbers \"look wrong\".\n\nThis is the goal, at least. Getting there will require a lot of trial and error.\n\nThe design decisions for this script are:\n1. **Flexible:** a plugin architecture is required to add benchmark result parsing as well as statistical analysis.\n1. **Simple:** each plugin must be simple and the interfaces consistent\n1. **Extensible:** not only new plugins for benchmarks and analysis, but also result categorisation ('data_string' argument)\n1. **Fully automated:** users should worry about how to run the benchmarks and what each different choice means (compiler flags may be linear, or aggregate, number of cores scale up, etc).\n1. **Low noise:** the output of the system should be as terse as the number of outliers\n1. **Stable:** the output should be the same given the same input (ie. not using random)\n\n## Structure\nThe Engine has a \"Linux Perf\" core parser in which plugins can be annexed (via the plugin parameter, -p option in aggregate.py). The perf output goes into **stderr** while the benchmark results go into **stdout**, and that separation is clear in the perf driver (this may change in the future, depending on benchmarks).\n\nEach analysis plugin will be loaded based on the _data_string_ (-d in aggregate.py) and will be passed in a number of lists created from the automatic categorisation.\n\nThe automatic categorisation will look at log file names, organise them hierarchically by separators, build a tree, then, for each _category_, it'll look for the analysis associated (via -d) and then collect all relevant data (across/along) and pass it though the plugin.\n\nWhat to do with the results is still uncerain, as there are many ways in which they can be analysed, and not all of them make sense. One could do everything, but then it would be hard to define what's a _real_ outlier and what's just an artifact of the structure.\n\n## Testing\n$ pip install pytest \u0026\u0026 ./test.sh\n\n## Pending tasks\nMajor features missing:\n* There is no automatic categorisation for the analysis passes created from the _data_string_\n* Improve the _data_string_ to contain all info needed for each mode of analysis\n* The statistical analysis is very crude and needs a lot more care in what an outlier is based on the analysis\n* Actually use scipy statistical analysis modules instead of home-brewed\n\nA few known issues:\n* PYTHONPATH needs to contain the _engine_ directory for now, needs a proper package structure\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flinaro%2Fhpc_benchmark_analysis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flinaro%2Fhpc_benchmark_analysis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flinaro%2Fhpc_benchmark_analysis/lists"}