{"id":16183765,"url":"https://github.com/quick-perf/maven-test-bench","last_synced_at":"2025-03-19T02:30:52.282Z","repository":{"id":35292339,"uuid":"206303197","full_name":"quick-perf/maven-test-bench","owner":"quick-perf","description":"Test bench to measure and investigate performance of Apache Maven project","archived":false,"fork":false,"pushed_at":"2023-02-27T09:36:37.000Z","size":268,"stargazers_count":21,"open_issues_count":6,"forks_count":6,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-17T01:51:13.542Z","etag":null,"topics":["allocation","benchmark","benchmarking","heap","heap-allocation","java-flight-recorder","java-flight-recordings","maven","performance","performance-analysis","performance-testing","profiling","test-bench"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/quick-perf.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-09-04T11:27:28.000Z","updated_at":"2024-05-25T20:27:26.000Z","dependencies_parsed_at":"2024-10-30T09:32:01.389Z","dependency_job_id":null,"html_url":"https://github.com/quick-perf/maven-test-bench","commit_stats":{"total_commits":37,"total_committers":5,"mean_commits":7.4,"dds":0.4864864864864865,"last_synced_commit":"c90f141daf072bd020145343d4b5c77c48769bc8"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/quick-perf%2Fmaven-test-bench","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/quick-perf%2Fmaven-test-bench/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/quick-perf%2Fmaven-test-bench/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/quick-perf%2Fmaven-test-bench/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/quick-perf","download_url":"https://codeload.github.com/quick-perf/maven-test-bench/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244341433,"owners_count":20437648,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["allocation","benchmark","benchmarking","heap","heap-allocation","java-flight-recorder","java-flight-recordings","maven","performance","performance-analysis","performance-testing","profiling","test-bench"],"created_at":"2024-10-10T07:06:44.129Z","updated_at":"2025-03-19T02:30:52.025Z","avatar_url":"https://github.com/quick-perf.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\u003cblockquote\u003e\n\u003cp\u003e\n\u003ch2\u003eMeasure and investigate heap memory allocation of Apache Maven\u003c/h2\u003e\n\u003ca href=\"https://github.com/quick-perf/maven-test-bench/actions?query=workflow%3ACI\"\u003e\u003cimg alt=\"Heap allocation of Maven head\" src=\"https://img.shields.io/github/workflow/status/quick-perf/maven-test-bench/CI\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003c/div\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"#General-setup\"\u003eGeneral setup\u003c/a\u003e •\n  \u003ca href=\"#Benchmark-heap-allocation-of-several-Maven-releases\"\u003eBenchmark heap allocation of several Maven releases\u003c/a\u003e \n\u003c/p\u003e\n\u003cp align=\"center\"\u003e\n\u003ca href=\"#Investigate-where-heap-allocation-comes-from\"\u003eInvestigate where heap allocation comes from\u003c/a\u003e  •\n  \u003ca href=\"#perspectives\"\u003ePerspectives\u003c/a\u003e •\n  \u003ca href=\"#Contributors\"\u003eContributors\u003c/a\u003e •\n  \u003ca href=\"#License\"\u003eLicense\u003c/a\u003e \n\u003c/p\u003e\n\nThis project is a test bench based on [QuickPerf](https://github.com/quick-perf/quickperf) to benchmark and understand heap memory allocation caused by `mvn validate` on a project source code.\n\nFeel free to use this project re-execute measures, play with it on measuring any build, tweak its code and contribute back to it!\n\n# Overview\n\nThis project contains two types of measures against Maven build execution, each provided as a JUnit unit test (in ```src/test/java```):\n- `MvnValidateProfilingTest` can be used to investigate the origin of memory allocation (using QuickPerf [@ProfileJvm](https://github.com/quick-perf/doc/wiki/JVM-annotations#Profile-or-check-your-JVM)) during one build with a given Maven version,\n- `MvnValidateAllocationByMaven3VersionTest` can be used to measure the heap allocation (using QuickPerf [@MeasureHeapAllocation](https://github.com/quick-perf/doc/wiki/JVM-annotations#Verify-heap-allocation)) for every Maven release in a version range.\n\n# General setup\n\nThis general setup part describes configuration common to both tests, done in `src/test/resourrces/maven-bench.properties` file:\n- the `testing.project.path` represents the path of the project on which `mvn validate` will be applied,\n- the `maven.binaries.path` property corresponds to the path where the needed Maven distributions will be automatically downloaded by the tests.\nThe other properties are only used by `MvnValidateAllocationByMaven3VersionTest`.\n \nThe measures shown here are done against the Apache Camel project because it contains more than 800 Maven modules: such a huge build is perfect to get significant measures. But you can choose your own target.\n\n_Note: currently, every day, a job @TravisCI run `org.quickperf.maven.bench.MvnValidateMaxAllocation` in order to watch\nover Apache Maven project and observe if new features on Apache Maven needs more memory then yesterday._  \n\n\nFor reproducibility of our measures, a precisely defined commit of this project was chosen:\n```\ngit clone -n https://github.com/apache/camel.git\ngit checkout c409ab7aabb971065fc8384a861904d2a2819be5\n```\n\nIf you want to apply measures on build done with Maven HEAD, that cannot be downloaded from public releases, you can execute the following commands where {maven-distrib-location} has to be replaced with the url given by the `maven.binaries.path` property of `maven-bench.properties` file:\n```\ngit clone https://github.com/apache/maven.git\ncd maven\nmvn -DdistributionTargetDir=\"{maven-distrib-location}/apache-maven-head\" package\n``` \n\nHeap size is fixed with the help of [@HeapSize](https://github.com/quick-perf/doc/wiki/JVM-annotations#heapsize).\n\n# Contributing\n\n## Building the project\n\n```bash\ngit clone https://github.com/pcavezzan/maven-test-bench.git\nmake build\n```\n\nThe above commandline will run behind the scene several actions:\n* run ```mvn package -B```\n\nDuring the test phase, we are going to :\n* clone Apache Camel project (commit ```c409ab7aabb971065fc8384a861904d2a2819be5```) into ```test-classes/camel```\n* install different version of maven into ```test-classes/maven```\n    * for Releases version, we are going to :\n        * download releases from ```https://archive.apache.org/dist/maven/maven-3/\u003cversion\u003e/binaries/apache-maven-\u003cversion\u003e-bin.zip```\n        * unzip into ```test-classes/maven/apache-maven-\u003cversion\u003e```\n    * for HEAD version, we are going to :\n        * clone Apache Maven mainstream from ```https://gitbox.apache.org/repos/asf/maven-sources.git```\n        * build from source Apache Maven latest development version,\n        * rename the built maven to ```test-classes/maven/apache-maven-master```, \n\n__*Note:*__ *the build above could be stuck depending on your machine settings. If it is the case, I would suggest you to create a custom build (see below).*\n\n## Custom Build\n\nIf you prefer to override some settings without editing the default configuration (```maven-bench.propertiers```), \nyou have several options:\n \n* environment variable\n* you can create a local configuration files ```local.maven-bench.properties``` without any risk to version it because \nthis file is ignored by GIT. In this file, just override the settings you want.\n\n### Overriding by setting environment variable\n\n```bash\nexport MAVEN_VERSION_FROM=3.6.1\nexport MAVEN_VERSION_TO=3.6.2\nmake build\n```   \n\n### Overriding by creating local.maven-bench.properties\n\n```bash\ncat \u003c\u003c EOF \u003e src/main/resources/local.maven-bench.properties\nmaven.version.from=3.6.1\nmaven.version.to=3.6.2\nEOF\nmake build\n```\n\n### Running only some tests\n\nThis project contains a Makefile to easily orchestrate how to build or run specific test. \nTo get more info, do not hesitate to run basic ```make``` or ```make help```:\n\n```bash\n$ make \nbuild                          Build project with running all tests (basically run `mvn package`)\nclean                          Cleanup project files (basically run `mvn clean`)\nrunMeasures                    Running only measures\nrunValidateMaxAllocation       Running only memory allocation needed for last commit from Maven GIT Repository on master branch\n```\n\n#### Memory allocation on head maven version\n\n```bash\n$ make runValidateMaxAllocation\n```\n\n\n#### Measures maven version\n\n```bash\n$ make runMeasureOnHead\n```\n\n\n## Developing\n\nIf you want to help us and make some code, you can easily get the project and open it up with your favorite IDE.\n\n\n### NOTE: if you want to debug head version of Maven\n\nFirst configure the project to use only head version of maven. For example, by creating a ```local.maven-bench.properties``` \nin ```src/main/resources``` directory:\n\n```bash\ncat \u003c\u003c EOF \u003e src/main/resources/local.maven-bench.properties\nmaven.version.from=head\nmaven.version.to=head\nEOF\nmake build\n```\n\nThen just open your favorite IDE or running your test.\n\n\n# Benchmark heap allocation of several Maven releases\n\n`org.quickperf.maven.bench.MvnValidateAllocationByMaven3VersionTest` test allows to benchmark the heap allocation level on several Maven 3 distributions.\n\nHeap allocation level is measured with the help of [@MeasureHeapAllocation](https://github.com/quick-perf/doc/wiki/JVM-annotations#Verify-heap-allocation) QuickPerf annotation. This annotation measures the heap allocation level of the thread running the method annotated with @Test.\nFeel free to contribute to QuickPerf by adding a feature allowing to measure the allocation level aggregated across all the threads! With `mvn validate`, we have checked that Maven code is not multithreaded during this validate phase by profiling the JVM with the help of [@ProfileJvm](https://github.com/quick-perf/doc/wiki/JVM-annotations#ProfileJvm).\n\nPlease read [General setup](#General-setup) to get some of the setup requirements.\n\nYou also have to give a value for the following properties contained in the [maven-bench.properties](src/test/resources/maven-bench.properties) file:\n* `maven.version.from`\n* `maven.version.to`\n* `warmup.number`\n* `measures.number-by-maven-version`\n\nThe meaning of these properties is given in the [maven-bench.properties](src/test/resources/maven-bench.properties) file.\n\nMeasures can be launched with this command line: ```mvn -Dtest=org.quickperf.maven.bench.MvnValidateAllocationByMaven3VersionTest test```.\nBefore doing it, you can close your IDE, web browser or other applications to free memory.\n\nThe benchmark results are exported into a `maven-memory-allocation-{date-time}.csv` file. The execution context (processor, OS, ...) is reported in an `execution-context-{date-time}.txt` file.\n\nFor several Maven versions, the following graph gives the average of ten heap allocations caused by the application of `mvn validate` on Apache Camel:\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"maven-perf/measures/mvn-validate-on-camel.png\"\u003e\n\u003c/p\u003e\n\nFor this graph, you can consult:\n* [the measures](maven-perf/measures/maven-memory-allocation-2019-09-01-18-48-41.csv)\n* [the execution context](maven-perf/measures/execution-context-2019-09-01-18-48-41.txt)\n\nMeasures took around one hour and a quarter. \n\nFrom Maven versions 3.2.5 to 3.6.2, heap allocation level is the highest with Maven 3.2.5 and the smallest with Maven 3.6.2. *The heap allocation decreases from ~7 Gb with Maven 3.6.1 to ~3 Gb with Maven 3.6.2*.\n\nControl and reduce heap allocation is an important matter for Maven project. Indeed, a part of the heap allocation is going to be garbage collected and the garbage collection activity is succeptible to slow down your build. In addition, less heap allocation means that you may execute Maven with a smaller heap size.\n\nBut where the allocation comes from? In the following part we will see how to spot the Java methods allocating a lot.\n\n# Investigate where heap allocation comes from\n\nYou can use `org.quickperf.maven.bench.MvnValidateProfilingTest` to understand the origin of heap allocation.\nSome of the set up requirements can be found in [General setup](#General-setup) part.\n\nThe Maven version under test can be set with the `MAVEN_3_VERSION` constant:\n``` java\n    public static org.quickperf.maven.bench.projects.Maven3Version MAVEN_3_VERSION = org.quickperf.maven.bench.projects.Maven3Version.V_3_6_2;\n```\n\nA test method is annotated with [@ProfileJvm](https://github.com/quick-perf/doc/wiki/JVM-annotations#Profile-or-check-your-JVM) to profile the test method with Java Flight Recorder (JFR).\n\nThe JFR file location is going to be displayed in the console:\n```\n[QUICK PERF] JVM was profiled with Java File Recorder (JFR).\nThe recording file can be found here: C:\\Users\\JEANBI~1\\AppData\\Local\\Temp\\QuickPerf-46868616\\jvm-profiling.jfr\nYou can open it with Java Mission Control (JMC).\n```\n\nYou can open it with Java Mission Control (JMC) to discover the methods contributing the most to heap allocation. \n\nBelow a JFR file for Maven 3.2.5 and opened with JMC 5.5:\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"maven-perf/measures/Maven3.2.5-JMC.5.5JPG.jpg\"\u003e\n\u003c/p\u003e\n\n\nBy the way, you can also benefit from an automatic performance analysis with [@ExpectNoJvmIssue](https://github.com/quick-perf/doc/wiki/JVM-annotations#ExpectNoJvmIssue).\nFor example, the following warning is reported with Maven 3.2.5:\n```\nRule: Thrown Exceptions\nSeverity: WARNING\nScore: 97\nMessage: The program generated 20 482 exceptions per second during 26,722 s starting at \n03/09/19 17:08:31.\n```\n\n# Perspectives\nWe have developed a test bench that is able to compare the heap allocation level between several Maven versions. We also have given a method to understand the origin of heap allocation.\n\nFeel free to play with this bench and [QuickPerf](https://github.com/quick-perf/doc/wiki/QuickPerf), to perform measures (heap allocation, execution time, ...) with different plugins/goals, use different JDK or garbage collectors, ..., suggest new ideas, create new features or share your measures with PR!\nSome issues are also available [here](https://github.com/quick-perf/maven-test-bench/issues)!\n\nYou also have [QuickPerf issues](https://github.com/quick-perf/quickperf/issues) to build new performance tools!\n\n# Contributors\nMany thanks to all our contributors!\n\n\u003ctable\u003e\n    \u003ctr\u003e\n        \u003ctd align=\"center\"\u003e\n            \u003ca href=\"https://github.com/jeanbisutti\"\u003e\n                \u003cimg src=\"https://avatars1.githubusercontent.com/u/14811066?v=4\" width=\"100px;\" alt=\"Jean Bisutti\"/\u003e\n                \u003cbr/\u003e\n                \u003csub\u003e\u003cb\u003eJean Bisutti\u003c/b\u003e\u003c/sub\u003e\n            \u003c/a\u003e\n            \u003cbr/\u003e\n            \u003ca href=\"https://github.com/quick-perf/maven-test-bench/commits?author=jeanbisutti\" title=\"Ideas\"\u003e🤔\u003c/a\u003e\n            \u003ca href=\"https://github.com/quick-perf/maven-test-bench/commits?author=jeanbisutti\" title=\"Code\"\u003e💻\u003c/a\u003e\n            \u003ca href=\"https://github.com/quick-perf/maven-test-bench/commits?author=jeanbisutti\" title=\"Documentation\"\u003e📖\u003c/a\u003e\n            \u003ca href=\"https://github.com/quick-perf/maven-test-bench/commits?author=jeanbisutti\" title=\"Reviewed Pull Requests\"\u003e👀\u003c/a\u003e\n        \u003c/td\u003e       \n        \u003ctd align=\"center\"\u003e\n            \u003ca href=\"https://github.com/hboutemy\"\u003e\n                \u003cimg src=\"https://avatars1.githubusercontent.com/u/237462?v=4\" width=\"100px;\" alt=\"Hervé Boutemy\"/\u003e\n                \u003cbr/\u003e\n                \u003csub\u003e\u003cb\u003eHervé Boutemy\u003c/b\u003e\u003c/sub\u003e\n            \u003c/a\u003e\n            \u003cbr/\u003e\n            \u003ca href=\"https://github.com/quick-perf/maven-test-bench/commits?author=hboutemy\" title=\"Ideas\"\u003e🤔\u003c/a\u003e\n            \u003ca href=\"https://github.com/quick-perf/maven-test-bench/commits?author=hboutemy\" title=\"Documentation\"\u003e📖\u003c/a\u003e\n        \u003c/td\u003e        \n        \u003ctd align=\"center\"\u003e\n            \u003ca href=\"https://github.com/albertotn\"\u003e\n                \u003cimg src=\"https://avatars1.githubusercontent.com/u/12526457?v=4\" width=\"100px;\" alt=\"Alberto Martinelli\"/\u003e\n                \u003cbr/\u003e\n                \u003csub\u003e\u003cb\u003eAlberto Martinelli\u003c/b\u003e\u003c/sub\u003e\n            \u003c/a\u003e\n            \u003cbr/\u003e\n            \u003ca href=\"https://github.com/quick-perf/maven-test-bench/commits?author=albertotn\" title=\"Code\"\u003e💻\u003c/a\u003e\n        \u003c/td\u003e\n        \u003ctd align=\"center\"\u003e\n            \u003ca href=\"https://github.com/pcavezzan\"\u003e\n                \u003cimg src=\"https://avatars2.githubusercontent.com/u/3405916?v=4\" width=\"100px;\" alt=\"Patrice Cavezzan\"/\u003e\n                \u003cbr/\u003e\n                \u003csub\u003e\u003cb\u003ePatrice Cavezzan\u003c/b\u003e\u003c/sub\u003e\n            \u003c/a\u003e\n            \u003cbr/\u003e\n            \u003ca href=\"https://github.com/quick-perf/maven-test-bench/commits?author=pcavezzan\" title=\"Code\"\u003e💻\u003c/a\u003e\n        \u003c/td\u003e        \n    \u003c/tr\u003e\n\u003c/table\u003e\n\u003ca href = \"https://allcontributors.org/docs/en/emoji-key\"\u003eemoji key\u003c/a\u003e\n\n# License\n[Apache License 2.0](/LICENSE.txt)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fquick-perf%2Fmaven-test-bench","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fquick-perf%2Fmaven-test-bench","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fquick-perf%2Fmaven-test-bench/lists"}