{"id":19664870,"url":"https://github.com/fluxml/fluxmlbenchmarks.jl","last_synced_at":"2025-08-19T04:44:04.604Z","repository":{"id":172980694,"uuid":"649402575","full_name":"FluxML/FluxMLBenchmarks.jl","owner":"FluxML","description":"A benchmarking suite for the FluxML org","archived":false,"fork":false,"pushed_at":"2024-04-30T04:58:29.000Z","size":151,"stargazers_count":6,"open_issues_count":0,"forks_count":2,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-02-27T03:50:04.840Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Julia","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/FluxML.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["JuliaLang"]}},"created_at":"2023-06-04T18:22:33.000Z","updated_at":"2023-09-01T05:09:45.000Z","dependencies_parsed_at":"2024-11-11T16:29:44.876Z","dependency_job_id":null,"html_url":"https://github.com/FluxML/FluxMLBenchmarks.jl","commit_stats":null,"previous_names":["fluxml/fluxmlbenchmarks.jl"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/FluxML/FluxMLBenchmarks.jl","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FluxML%2FFluxMLBenchmarks.jl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FluxML%2FFluxMLBenchmarks.jl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FluxML%2FFluxMLBenchmarks.jl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FluxML%2FFluxMLBenchmarks.jl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/FluxML","download_url":"https://codeload.github.com/FluxML/FluxMLBenchmarks.jl/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FluxML%2FFluxMLBenchmarks.jl/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271102972,"owners_count":24699637,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-19T02:00:09.176Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-11T16:19:26.560Z","updated_at":"2025-08-19T04:44:04.562Z","avatar_url":"https://github.com/FluxML.png","language":"Julia","funding_links":["https://github.com/sponsors/JuliaLang"],"categories":[],"sub_categories":[],"readme":"\u003cimg align=\"right\" width=\"200px\" src=\"https://github.com/FluxML/OneHotArrays.jl/raw/main/docs/src/assets/logo.png\"\u003e\n\n# FluxMLBenchmarks\n\n`FluxMLBenchmarks` is a benchmarking tool designed for FluxML community, which allows for the creation of different benchmarking environments by installing different sets of dependencies and comparing the results.\n\n## Use cases\n\n### 1. Command-Line Interface\n\n#### i. Single Package\n\nTo observe whether there is a performance difference between two versions of **the same package**, `FluxMLBenchmarks` provides 2 arguments, `--baseline` and `--target`, to specify the 2 versions of **the same package**.\n\n```shell\n\u003e BASELINE=\u003cDependency Representation of baseline\u003e\n\u003e TARGET=\u003cDependency Representation of target\u003e\n\u003e julia --project=benchmark benchmark/runbenchmarks.jl --pr --target=$TARGET --baseline=$BASELINE\n```\n\nFor specification, `Dependency Representation` is similar to the `word` of [add - REPL command - Pkg.jl](https://pkgdocs.julialang.org/v1/repl/#package-commands), described as follows:\n\n| format | example |\n| :-: | :-: |\n| `\u003cpkg name\u003e` | `Flux` |\n| `\u003cpkg name\u003e@\u003cversion\u003e` | `NNlib@0.8.20` |\n| `\u003cpkg name\u003e#\u003cbranch name, commit id\u003e` | `Zygote#master` or `Zygote#2f4937096ee1db4b5a67c1c31fe3ebeab1c96c8c` |\n| `\u003curl\u003e` | `https://github.com/FluxML/Optimisers.jl` |\n| `\u003curl\u003e#\u003cbranch name, commit id\u003e` | `https://github.com/FluxML/Functors.jl#master` |\n\ne.g.\n\n```shell\n\u003e BASELINE=\"https://github.com/FluxML/NNlib.jl#backports-0.8.21\"\n\u003e TARGET=\"https://github.com/skyleaworlder/NNlib.jl#dummy-benchmark-test\"\n\u003e julia --project=benchmark benchmark/runbenchmarks.jl --pr --target=$TARGET --baseline=$BASELINE\n```\n\n#### ii. Multiple Packages\n\nThe performance of a package need measured under the condition that other packages and tools remain constant. However, in the case of mutual influence between **multiple packages of different versions**, **2 sets of dependencies** need to be provided simultaneously. As for this scenario, you can use `--baseline` and `--target` as well:\n\n```shell\n\u003e BASELINE=\u003cDependency Representation A1 of baseline\u003e,\u003cDependency Representation B1 of baseline\u003e,\u003cDependency Representation C1... of baseline\u003e\n\u003e TARGET=\u003cDependency Representation A2 of target\u003e,\u003cDependency Representation of B2 target\u003e,\u003cDependency Representation C2... of target\u003e\n\u003e julia --project=benchmark benchmark/runbenchmarks.jl --pr --target=$TARGET --baseline=$BASELINE\n```\n\n#### iii. Multiple Sets of Dependencies\n\nSometimes we need to run benchmarks for multiple sets of dependencies simultaneously. To meet this benchmarking requirements, you can use `--deps-list`:\n\n```shell\n\u003e DEPS_LIST=\u003cDependencies List\u003e\n\u003e julia --project=benchmark benchmark/runbenchmarks.jl --cli --deps-list=$DEPS_LIST\n```\n\nFor specification, `Dependencies List` is a single string that simulates an array, with each element separated by a semicolon. Each element adheres to the format of `Dependency Representation`. However, **Unlike the previous output `result-baseline.json` and `result-target.json`, the output format for this feature is `result-1.json`, `result-2.json`, `result-n.json`...**\n\ne.g.\n\n```shell\n\u003e DEPS_LIST=\"NNlib,Flux;https://github.com/FluxML/NNlib.jl#backports-0.8.21,Flux;https://github.com/skyleaworlder/NNlib.jl#backports-0.8.21,Flux@0.13.12\"\n\u003e julia --project=benchmark benchmark/runbenchmarks.jl --cli --deps-list=$DEPS_LIST\n```\n\n### 2. GitHub Pull Request\n\nTODO\n\n## Command Arguments\n\n### 0. `--pr` / `--cli` / `--cache-setup` / `--merge-reports`\n\nEach argument represents an operation this tool will perform. The corresponding relationship is:\n\n* `--pr`: \"benchmark/script/runbenchmarks-pr.jl\" You can specify `--target` `--baseline` `--enable` `--disable`\n* `--cli`: \"benchmark/script/runbenchmarks-cli.jl\" You can specify `--deps-list` `--enable` `--disable`\n* (**Not recommended, used by GitHub Actions**) `--cache-setup`: \"benchmark/script/cachesetup-cli.jl\" You can specify `--target` `--baseline`\n* (**Not recommended, used by GitHub Actions**) `--merge-reports`: \"benchmark/script/mergereports-cli.jl\" You can specify `--target` `--baseline` `--push-result` `--push-username` `--push-useremail` `--push-password`\n\n### 1. `--target` / `--baseline` / `--deps-list`\n\nSee [Use cases - Single Package](#i-single-package) and [Use cases - Multiple Packages](#ii-multiple-packages).\n\n### 2. `--enable` / `--disable`\n\nBenchmarking always takes amount of time. In order to focus on the targets and reduce the time consumption of our benchmarking tool, the `--enable` and `--disable` options are used to specify **the parts to be included** and **the parts to be excluded** respectively.\n\n```shell\n\u003e julia --project=benchmark benchmark/runbenchmarks.jl --cli  \\\n\u003e   --enable=\u003cENABLED_PARTS\u003e \\\n\u003e   --disable=\u003cDISABLED_PARTS\u003e \\\n\u003e   --deps-list=\u003cDependencies List\u003e\n```\n\nFor specification, `Enabled Parts` and `Disabled Parts` have the same format, which is a single string that simulates an array, with each element separated by a semicolon.\n\n`--enable` is used to specify the files that should be included, and by default (`--enable` not specified) all files in the `benchmark/benchmark` are included. `--disable` is used to specify the files that should be excluded, and the default value is an empty string.\n\nMore precisely, the granularity of the element of `Enabled Parts` and `Disabled Parts` is currently at the file-level, and supports two levels of files, which means that now **our tool will recognize the name of each file in `benchmark/benchmark` and all the files under `benchmark/benchmark/**` before benchmarking**.\n\n**Each top-level element in `Enabled Parts` and `Disable Parts` should be exactly the name of the file under `benchmark/benchmark`; each second-level element should be the name of the file under the dir that has the same name of top-level file**.\n\n\u003e I don't recommend using `--enable` and `--disable` at the same time. But if you do, `--disable` takes priority over `--enable`.\n\u003e \n\u003e e.g. if `--enable` is set \"flux,nnlib\" while `--disable` is set \"nnlib\", only benchmarks in \"benchmark/benchmark/flux.jl\" will be executed.\n\ne.g.\n\n```shell\n\u003e DEPS_LIST=\"https://github.com/FluxML/NNlib.jl#backports-0.8.21,Flux;https://github.com/skyleaworlder/NNlib.jl#dummy-benchmark-test,Flux@0.13.12\"\n\u003e # Only Flux-MLP and all NNlib\n\u003e julia --project=benchmark benchmark/runbenchmarks.jl --cli --enable=\"flux(mlp);nnlib\" --deps-list=$DEPS_LIST\n\u003e # All benchmarks except Flux, NNlib-gemm and NNlib-activations\n\u003e julia --project=benchmark benchmark/runbenchmarks.jl --cli --disable=\"flux;nnlib(gemm,activations)\" --deps-list=$DEPS_LIST\n\u003e # Only Flux\n\u003e julia --project=benchmark benchmark/runbenchmarks.jl --cli --enable=\"flux;nnlib\" --disable=\"nnlib\" --deps-list=$DEPS_LIST\n```\n\n### 3. `--fetch-result` / `--push-result` / `--push-username` / `--push-useremail` / `--push-password`\n\nThese arguments are only used in `--merge-reports`, not recommended.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffluxml%2Ffluxmlbenchmarks.jl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffluxml%2Ffluxmlbenchmarks.jl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffluxml%2Ffluxmlbenchmarks.jl/lists"}