{"id":20358758,"url":"https://github.com/weld-project/weld-benchmarks","last_synced_at":"2025-06-20T14:09:48.567Z","repository":{"id":98920888,"uuid":"100564018","full_name":"weld-project/weld-benchmarks","owner":"weld-project","description":null,"archived":false,"fork":false,"pushed_at":"2017-12-04T06:32:50.000Z","size":41,"stargazers_count":3,"open_issues_count":4,"forks_count":7,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-12T03:35:13.940Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/weld-project.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-08-17T05:13:22.000Z","updated_at":"2023-07-25T14:10:53.000Z","dependencies_parsed_at":"2023-04-04T17:31:34.952Z","dependency_job_id":null,"html_url":"https://github.com/weld-project/weld-benchmarks","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/weld-project/weld-benchmarks","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/weld-project%2Fweld-benchmarks","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/weld-project%2Fweld-benchmarks/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/weld-project%2Fweld-benchmarks/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/weld-project%2Fweld-benchmarks/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/weld-project","download_url":"https://codeload.github.com/weld-project/weld-benchmarks/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/weld-project%2Fweld-benchmarks/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260959041,"owners_count":23088813,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-14T23:28:47.734Z","updated_at":"2025-06-20T14:09:43.554Z","avatar_url":"https://github.com/weld-project.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Weld benchmarks\n\nThis repository tries to run benchmarks for the Weld project in an easy-to-reproduce\nway.\n\n## Key requirements\n\n- Each benchmark is in a separate directory under `benchmarks/`.\n\n- All benchmark binaries *have to* to be named `bench` in the corresponding directory\n  (if compilation is required, the output binary produced by `make` should be called\n  `bench`).\n\n- In addition, each benchmark binary is responsible for its own timing; all timing\n  information needs to be printed to `stdout` in the following format,\n  ```\n  \u003cExperiment description\u003e: \u003cTime\u003e \u003cOther metadata\u003e\n  ```\n  Each benchmark binary can print multiple pieces of timing information.\n  \n  To make this more concrete, consider the output of the matrix multiplication\n  benchmark,\n  ```bash\n  $ ./bench\n  Transposed: 0.028259 (result=38238491904)\n  Unblocked: 0.029373 (result=38288406288)\n  Blocked: 0.028529 (result=38228352573)\n  ```\n  \n- All parameters need to be passed into the `bench` binary in the form\n  `-\u003cparameter name\u003e \u003cparameter value\u003e`.\n\n- Benchmark directories must include a configuration file named \"config.json\".\n\n  A sample configuration file looks like this:\n  ```json\n  {\n      \"compile\": true,\n      \"params\":\n      {\n        \"b\": [64, 128]\n      },\n      \"scaled_params\":\n      {\n        \"n\": [512, 1024]\n      }\n  }\n  ```\n\n  The `params` field specifies the parameters that need to be swept over.\n\n  The `compile` field is a `true/false` field and specifies whether workloads need to\n  be compiled beforehand using `make` or not.\n\n## Running instructions\n\nThe main script is `run_benchmarks.py` in the root directory. It takes the following\narguments:\n- `-n / --num_iterations`: Specifies the number of trials for each benchmark.\n- `-s / --scale_factor`: Specifies the factor by which scaled parameters need to be scaled.\n- `-f / --csv_filename`: Specifies the output file for dumped experiment results.\n- `-b / --benchmarks`: Comma-separated list of benchmarks that should be run (must be\n  a subset of benchmarks listed in the configuration file).\n- `-v / --verbose`: A flag specifying whether to print verbose statistics.\n\nSample output looks like this:\n```bash\n$ python run_benchmarks.py -b crime_index -n 5 -f results.csv -v\n++++++++++++++++++++++++++++++++++++++\ncrime_index\n++++++++++++++++++++++++++++++++++++++\ns=1, f=data/us_cities_states_counties_sf=%d.csv\nGrizzly: 1.4256 +/- 0.0102 seconds\nPandas: 0.0229 +/- 0.0009 seconds\n\ns=10, f=data/us_cities_states_counties_sf=%d.csv\nGrizzly: 2.1567 +/- 0.2303 seconds\nPandas: 0.0725 +/- 0.0066 seconds\n\ns=100, f=data/us_cities_states_counties_sf=%d.csv\nGrizzly: 6.8954 +/- 0.2109 seconds\nPandas: 0.6040 +/- 0.0640 seconds\n\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fweld-project%2Fweld-benchmarks","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fweld-project%2Fweld-benchmarks","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fweld-project%2Fweld-benchmarks/lists"}