{"id":49774661,"url":"https://github.com/sake92/jvm-build-tools-bench","last_synced_at":"2026-05-11T14:27:52.274Z","repository":{"id":350668012,"uuid":"1204772764","full_name":"sake92/jvm-build-tools-bench","owner":"sake92","description":"A naive JVM build tools benchmark","archived":false,"fork":false,"pushed_at":"2026-04-27T13:14:01.000Z","size":471,"stargazers_count":2,"open_issues_count":1,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-27T14:29:18.258Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://sake92.github.io/jvm-build-tools-bench/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sake92.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-08T10:13:54.000Z","updated_at":"2026-04-27T12:57:51.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/sake92/jvm-build-tools-bench","commit_stats":null,"previous_names":["sake92/jvm-build-tools-bench"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/sake92/jvm-build-tools-bench","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sake92%2Fjvm-build-tools-bench","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sake92%2Fjvm-build-tools-bench/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sake92%2Fjvm-build-tools-bench/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sake92%2Fjvm-build-tools-bench/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sake92","download_url":"https://codeload.github.com/sake92/jvm-build-tools-bench/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sake92%2Fjvm-build-tools-bench/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32898665,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-10T13:40:02.631Z","status":"online","status_checked_at":"2026-05-11T02:00:05.975Z","response_time":120,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-05-11T14:27:47.165Z","updated_at":"2026-05-11T14:27:52.265Z","avatar_url":"https://github.com/sake92.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# jvm-build-tools-bench\n\n\u003e Just a naive mini benchmark!\n\nBenchmarks comparing JVM build tools (Maven, Gradle, Mill, bleep, sbt, …) on real-world open-source repositories using [hyperfine](https://github.com/sharkdp/hyperfine).\n\n\u003e ⚠️ **Disclaimer:** The build files used across different tools are not guaranteed to be 100% equivalent. They are maintained on a best-effort basis — differences in compiler settings, dependency resolution, plugin configurations, or module structure can affect timing comparisons. Treat results as indicative rather than definitive.\n\nMultiple benchmark types are measured per tool — each defined in `benchmarks.yaml` as a named entry under `hyperfine.benchmark_types`. Typical types include:\n\n| Benchmark type | Description |\n|---|---|\n| **compile-clean** | Full rebuild from scratch (clean + compile) |\n| **compile-incremental** | Touch a set of source files, recompile |\n| **test** | Run all tests |\n\nAdditional types (e.g., `compile-module-abc`) can be added without changing any Scala code.\n\nResults are exported as hyperfine JSON and uploaded as GitHub Actions artifacts.  \nAfter all tools finish, an `aggregate` job produces a unified comparison.\n\n---\n\n## How it works\n\n1. **`benchmarks.yaml`** — single source of truth: which repos to clone, which repo + build tool benchmarks to run, what commands to execute, and the shared hyperfine settings for each benchmark scenario.\n2. **`run_bench.scala`** — Scala CLI runner. Parses `benchmarks.yaml` directly (no `yq` needed), clones the target repo, optionally overlays extra build files, runs setup, then benchmarks each benchmark type defined for the tool.\n3. **`aggregate.scala`** — scales all per-tool hyperfine JSON results into a unified comparison with SVG charts, markdown tables, and a static HTML page.\n4. **`.github/workflows/bench.yml`** — CI workflow. The benchmark list is read dynamically from `benchmarks.yaml` to build a matrix; each repo + build tool combination runs on a **separate fresh runner** for maximum isolation. An `aggregate` job runs after all tools finish.\n\nShared code lives in **`bench.scala`** — config model, YAML parsing, git operations, hyperfine runner, and chart colors.\n\nBuild files that a repo doesn't ship natively (e.g. a `build.mill` for a Maven-only repo) are stored under `build-files/\u003crepo\u003e/\u003ctool\u003e/` and overlaid before benchmarking.\n\n---\n\n## Prerequisites (local run)\n\n| Tool | Install |\n|---|---|\n| JDK 21 | [Temurin](https://adoptium.net/) or `sdk install java 21-tem` |\n| [scala-cli](https://scala-cli.virtuslab.org/) | `brew install scala-cli` / `sdk install scala-cli` / [releases page](https://github.com/VirtusLab/scala-cli/releases) |\n| hyperfine | `cargo install hyperfine` or [releases page](https://github.com/sharkdp/hyperfine/releases) |\n| Build tool | Maven / Mill / Deder / sbt — see `benchmarks.yaml` for `install` commands |\n\nAll scripting is in Scala, charts use JFreeChart.\n\n---\n\n## Running locally\n\n```bash\n# list all available benchmarks\nscala-cli run run_bench.scala -- --list-benchmarks\n\n# benchmark Maven on java-algorithms\nscala-cli run run_bench.scala -- --benchmark java-algorithms-maven\n```\n\nResults land in `results/` by default:\n\n```text\nresults/\n├── java-algorithms/\n│   ├── maven-compile-clean.json\n│   ├── maven-compile-incremental.json\n│   ├── maven-test.json\n│   └── ...\n└── scala-algorithms/\n    ├── sbt-compile-clean.json\n    ├── sbt-compile-incremental.json\n    └── ...\n```\n\nEach file is a standard hyperfine JSON export — use `cat results/java-algorithms/maven-compile-clean.json | jq .results[0].mean` to read the mean time in seconds.\n\n---\n\n## Results\n\nResults are published to **GitHub Pages** after each CI run:\n\u003e `https://sake92.github.io/jvm-build-tools-bench/`\n\nThe page has one collapsible section per repo with separate SVG charts and report links for each scenario.\n\nYou can also download the **`results-aggregated`** artifact from the Actions tab. It contains:\n\n| File | Contents |\n|---|---|\n| `index.html` | Static results page (charts + links) |\n| `\u003crepo\u003e/\u003cscenario\u003e/summary.json` | Mean / stddev / min / max per tool for one scenario |\n| `\u003crepo\u003e/\u003cscenario\u003e/summary.md` | Markdown comparison table for one scenario |\n| `\u003crepo\u003e/\u003cscenario\u003e/chart.svg` | Horizontal bar chart for one scenario |\n\nTo run aggregation locally after collecting results:\n\n```bash\nscala-cli run aggregate.scala -- --results-dir results/ --output-dir aggregated/ --benchmarks-yaml benchmarks.yaml\n```\n\nASCII bar charts are also printed to stdout during aggregation (handy for reading directly in CI logs).\n\n\u003e **Note:** Enable GitHub Pages in your repo settings with source branch `gh-pages` for the published URL to work.\n\n---\n\n## Adding a new tool\n\n1. Add an entry to `benchmarks.yaml` with a `repo` and `build_tool_name` (copy an existing one as a template).\n2. If the target repo doesn't have a native build file for this tool, add it under `build-files/\u003crepo\u003e/\u003ctool\u003e/`.\n3. Add the tool's chart color in `bench.scala`'s `ToolColors` map so aggregated SVGs stay consistent.\n4. That's it — GitHub Actions will pick it up automatically on the next run.\n\n---\n\n## Benchmark targets\n\n| Repo | URL |\n|---|---|\n| TheAlgorithms/Java | https://github.com/TheAlgorithms/Java |\n| TheAlgorithms/Scala | https://github.com/TheAlgorithms/Scala |\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsake92%2Fjvm-build-tools-bench","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsake92%2Fjvm-build-tools-bench","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsake92%2Fjvm-build-tools-bench/lists"}