{"id":19706009,"url":"https://github.com/llnl/faros","last_synced_at":"2025-04-29T16:32:27.305Z","repository":{"id":45047542,"uuid":"306079019","full_name":"LLNL/FAROS","owner":"LLNL","description":"FAROS: A Framework for Benchmarking and Analysis of Compiler Optimization","archived":false,"fork":false,"pushed_at":"2022-12-23T13:11:40.000Z","size":65,"stargazers_count":10,"open_issues_count":8,"forks_count":4,"subscribers_count":3,"default_branch":"develop","last_synced_at":"2025-04-05T18:06:38.881Z","etag":null,"topics":["benchmark","benchmarking","compiler-optimization","openmp"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/LLNL.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-10-21T16:07:54.000Z","updated_at":"2024-08-29T01:11:49.000Z","dependencies_parsed_at":"2023-01-30T18:31:07.211Z","dependency_job_id":null,"html_url":"https://github.com/LLNL/FAROS","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LLNL%2FFAROS","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LLNL%2FFAROS/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LLNL%2FFAROS/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LLNL%2FFAROS/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/LLNL","download_url":"https://codeload.github.com/LLNL/FAROS/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251540349,"owners_count":21605888,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark","benchmarking","compiler-optimization","openmp"],"created_at":"2024-11-11T21:33:05.127Z","updated_at":"2025-04-29T16:32:22.295Z","avatar_url":"https://github.com/LLNL.png","language":"Python","readme":"# FAROS: A Framework for Benchmarking and Analysis of Compiler Optimization\n\nFAROS is an extensible framework to automate and structure the analysis \nof compiler optimizations of OpenMP programs. FAROS provides a generic \nconfiguration interface to profile and analyze OpenMP applications with \ntheir native build configurations.\n\n## Description and usage\n\nThis repo contains a benchmark harness to fetch, build, and run programs\nwith different compilation options for analyzing the impact of\ncompilation on performance. The usecase is contrasting OpenMP compilation\nwith its serial elision to understand performance differences due to\ndifferent compilation.\n\n\n### Harness script\n\nThe harness script in python, named benchmark.py, takes as input a YAML\nconfiguration file and a set of options to build and run programs\ndescribed in that configuration. You can see below the help\noutput describing possible options. The configuration file input is set\nwith the `-i, --input` argument. There are three different actions the\nharness performs:\n1. fetch sources, with the option `-f, --fetch`,\nfetches the program sources from the specified repositories;\n2. build programs, with\nthe option `-b, --build`, builds the selected program using the specified\ncompilation options in the configuration, also fetching if needed;\n3. generate reports, with the option `-g, --generate`, generates compilation reports\nby combining optimization remarks for different compilation\nconfigurations, creating remark diff files between them, from all the\nsources of an application to a single file;\n4. run tests, with the option `-r,--run` and a following argument on how many repetitions to perform, that\nruns the executable with the specified input, repeating up to the number\nof repetitions set.\n\nThe flags can be individually set doing multiple runs of the harness, or\ncombined to perform multiple actions in a single run -- fetching takes\nprecedence over building, building over generating reports and running.\nThe user select the list of programs to operate usin `-p,--programs`\nfollowed by individual program names or the keyword `all` for all the\nprograms specified in the configuration input.  Alternatively, the user\nmay select the list of programs by provide a tag list with `-t,--tags`\nthat matches tags for each program specified in the configuration.\nAlso, the harness has a dry run option, `-d,--dry-run`, that prints what\nactions would be performed without actually performing them.\n\nBelow is the output when running help, `-h, --help` on the harness\nscript:\n```\nBenchmark and analyze programs compiled with different compilation options.\n\noptional arguments:\n  -h, --help            show this help message and exit\n  -i INPUT, --input INPUT\n                        configuration YAML input file for programs\n  -f, --fetch           fetch program repos (without building)\n  -b, --build           build programs (will fetch too)\n  -r RUN, --run RUN     run \u003crepetitions\u003e\n  -g, --generate        generate compilation reports\n  -p PROGRAMS [PROGRAMS ...], --programs PROGRAMS [PROGRAMS ...]\n                        programs to run from the config\n  -t TAGS [TAGS ...], --tags TAGS [TAGS ...]\n                        tagged program to use from the config\n  -s, --stats           show run statistics\n  -d, --dry-run         enable dry run\n```\nThe harness creates four extra directories for its\noperation when building and running:\n1. the directory `repos` to download the benchmark application\n    specified in the configuration;\n2. the directory `bin` to store the generated executable from\n    building to run them;\n3. the directory `reports`, where it store compilation\n    reports, including optimization remarks;\n4. the directory `results` to store profiling results, which\n    contain execution times from running different built configurations and\n    inputs.\n\n### YAML configuration input\n\nConfiguring YAML creates is a hierarchy of keys for each program to\ninclude that prescribe actions for the harness script. We describe those\nkeys here. For a working example please see `config.yaml` in the repo,\nwhich iincludes configuration for 39 HPC programs including proxy/mini\napplications, NAS and Rodinia kernels, and the open-source large\napplication GROMACS.  The root of the hierarchy is a user-chosen,\ndescriptive name per program configuration.  The harness creates a\nsub-directory matching the name of this root key under `bin` to store\nexecutables\n\nThe key `fetch` contains the shell command to fetch the\napplication code, for example cloning from a GitHub repo. Note that the\nfetching command can include also a patching file, if needed, provided\nby the user. In this repo we provide patch files for programs in\n`config.yaml` undr the directory `patches`.  For example, for some\nprograms, we apply a patch to guard calls to OpenMP runtime functions\nusing the standard approach of enabling those calls within `#ifdef OPENMP\n... endif` preprocessor directives.\n\nThe key `tags` sets a list of user-defined tags which can be used by the\nharness to include programs when performing its operation.\n\nThe key `build_dir` specifies the directory to build the application, so\nharness changes to this directory to execute the build commands\nspecified under the key build. There is a different sub-key for each\ndifferent compilation specification, denoted by a user-provided\nidentifier.  The harness creates different sub-directories under\n`bin/\u003cprogram\u003e`for each different compilation configurations\n\nThe key `copy` specifies a list of files or directories that the harness\ncopies out to those sub-directories. The list contains the executable\nfile and possibly any input files needed for executing, if the user\ndesires to have self-contained execution in `bin` by avoiding referring to\ninput files in the directory repos -- this is useful for\nrelocating the directory bins without needing to copy over\nrepos.\n\nFurther, the key `run` specifies the command to\nexecute, which is typically the executable binary of the application,\nprepended with any environment variables to set.\n\nMoreover, the key `input` specifies the input arguments\nfor the application in the run command.\n\nThe key `measure`\nspecifies a regular expression to match in the application's executable\noutput to capture the desired measure of performance, such as execution\ntime or some other Figure of Merit (FoM). If the value of the key\nmeasure is empty, the harness measures end-to-end, wall clock  execution\ntime from launching the application to its end, using python's\ntime module.\n\nLastly, the key `clean` specifies the commands that harness executes to\nclean the repo for building a different compilation configuration.\n\n## Contributing\nTo contribute to this repo please send a [pull\nrequest](https://help.github.com/articles/using-pull-requests/) on the\ndevelop branch of this repo.\n\n## Authors\n\nThis code was created by Giorgis Georgakoudis (LLNL),\ngeorgakoudis1@llnl.gov, assisted with technical design input from\nIgnacio Laguna (LLNL), Tom Scogland, and Johannes Doerfert (ANL).\n\n### Citing FAROS\n\nPlease cite the following paper: \n\n* Georgakoudis G., Doerfert J., Laguna I., Scogland T.R.W. (2020) [FAROS: A\n  Framework to Analyze OpenMP Compilation Through Benchmarking and Compiler\n  Optimization\n  Analysis](https://link.springer.com/chapter/10.1007/978-3-030-58144-2_1). In: Milfeld K., de Supinski B., Koesterke L.,\n  Klinkenberg J. (eds) OpenMP: Portable Multi-Level Parallelism on Modern\n  Systems. IWOMP 2020. Lecture Notes in Computer Science, vol 12295. Springer,\n  Cham. https://doi.org/10.1007/978-3-030-58144-2_1\n\n## License\n\nThis repo is distributed under the terms of the Apache License (Version\n2.0) with LLVM exceptions. Other software that is part of this\nrepository may be under a different license, documented by the file\nLICENSE in its sub-directory.\n\nAll new contributions to this repo must be under the Apache License (Version 2.0) with LLVM exceptions.\n\nSee files [LICENSE](LICENSE) and [NOTICE](NOTICE) for more information.\n\nSPDX License Identifier: \"Apache-2.0 WITH LLVM-exception\"\n\nLLNL-CODE-813267\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fllnl%2Ffaros","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fllnl%2Ffaros","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fllnl%2Ffaros/lists"}