{"id":20249600,"url":"https://github.com/jonico/speedup-builds","last_synced_at":"2025-04-10T22:45:06.941Z","repository":{"id":54459723,"uuid":"68734945","full_name":"jonico/speedup-builds","owner":"jonico","description":"Some ideas on how to speed up builds / tests with Jenkins partitioning example","archived":false,"fork":false,"pushed_at":"2021-02-16T14:53:55.000Z","size":7685,"stargazers_count":43,"open_issues_count":0,"forks_count":17,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-04-10T22:45:01.669Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jonico.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-09-20T17:10:38.000Z","updated_at":"2023-03-03T09:53:19.000Z","dependencies_parsed_at":"2022-08-13T16:20:20.920Z","dependency_job_id":null,"html_url":"https://github.com/jonico/speedup-builds","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonico%2Fspeedup-builds","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonico%2Fspeedup-builds/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonico%2Fspeedup-builds/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonico%2Fspeedup-builds/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jonico","download_url":"https://codeload.github.com/jonico/speedup-builds/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248312190,"owners_count":21082637,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-14T09:54:37.306Z","updated_at":"2025-04-10T22:45:06.922Z","avatar_url":"https://github.com/jonico.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# How to speed up builds and tests in your CI environment \n\n* What is taking most of the time in your builds? Cloning, building, testing?\n* Improve cloning times by\n  - Use a [reference repository](https://git-scm.com/docs/git-clone#git-clone---shared) (under Advanced Clone Behaviors) so that you can point to a local base repo instead of copying the entire history for every project, you may also check out @larsxschneider's [cached clone](https://github.com/larsxschneider/cached-clone) script that also caches submodules and Git LFS objects\n  - [Narrow refspec](https://youtu.be/TsWkZLLU-s4?t=2m18s) by selecting \"Honor refspec on initial clone\" in Advanced Clone Behaviors and specfying an appropriate refspec\n  - Sparse checkout only the directories that are needed for the operation\n  - reduced repository sizes by moving large binary files to Git LFS using [bfg-repo cleaner](https://github.com/rtyley/bfg-repo-cleaner/releases/tag/v1.12.5)\n  - [push based build triggering](https://wiki.jenkins-ci.org/display/JENKINS/GitHub+Plugin#GitHubPlugin-TriggerabuildwhenachangeispushedtoGitHub), + use [ngrok](https://ngrok.com/) or open a port if needed\n  - get version control system server and artifact repository closer to the build server (maybe by hosting it yourself or moving the build server closer to the rest)\n* Improve build times by\n  - [only building components once and transport them to all downstream build jobs and machines](https://wiki.jenkins-ci.org/display/JENKINS/Copy+Artifact+Plugin)\n  - [using an artifact repository for shared components](http://www.slideshare.net/SonatypeCorp/white-paper-concepts-and-benefits-of-repository-management)\n  - SSDs, more cores and more memory\n  - [Optimized compiler switches](https://rogerkeays.com/why-is-maven-so-slow) - mvn client VM, [parallel build options](https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3), memory options\n  - [use a build system that is caching build results globally and does build parallelization automatically](http://prezi.com/20fxomjc_goy/?utm_campaign=share\u0026utm_medium=copy\u0026rc=ex0share)\n* Improve test times by\n  - identifying which tests are still testing used functionality and disable the rest\n  - rewriting integration and system tests to use more mocking (one process without IPC and file system interaction preferred)\n  - analyze bottlenecks and silent exceptions using New Relic, Dynatrace, OverOps or just VisualVM\n  - Break monolythic applications done into multiple components that can be tested in isolation (and only retested if they changed)\n  - dedicate a person per team to improve test execution time\n  - [test partitioning across multiple machines using Jenkins Parallel Test Executor Plugin](https://github.com/jenkinsci/pipeline-plugin/blob/master/TUTORIAL.md) (this repository)\n  - SSDs, more cores and more memory\n  - learn more about [Jenkins Pipelines](https://jenkins.io/solutions/pipeline/), its job resuming functionalities and its [GitHub Integration](https://go.cloudbees.com/docs/cloudbees-documentation/cje-user-guide/chapter-github-branch-source.html#github-branch-source)\n  - if you like to use a pre-setup environment using Docker that integrates with GitHub.com, you can follow [this tutorial](https://hub.docker.com/r/jenkinsci/pipeline-as-code-github-demo/).\n  - pre-heating of integration test systems (e.g. by unfreezing memory snapshots of already fully up and running application server and database VMs)\n  - only do smoke tests in build, all tests in nightly build (CI result is now also shown in code browser out of the pull request)\n  - [optimized compiler switches](https://rogerkeays.com/why-is-maven-so-slow), parallel test runs (Maven 3 now supports this [better](https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3)), compile options like -XX:+TieredCompilation -XX:TieredStopAtLevel=1 for short running mvn invocations\n  - [use different commit status contexts for different tests](https://wiki.jenkins-ci.org/display/JENKINS/GitHub+Plugin), only make smoke test status checks [mandatory](https://help.github.com/articles/enabling-required-status-checks/)\n  - [find the best tradeoff of number of tests/test duration and code coverage](https://www.atlassian.com/software/clover)\n  - [use a build system that is automatically caching test results parallelizing tests](http://prezi.com/20fxomjc_goy/?utm_campaign=share\u0026utm_medium=copy\u0026rc=ex0share) (if code tested and test have not changed)\n\n# Interesting links\n\n* [Tools for Continuous Integration at Google Scale](https://www.youtube.com/watch?v=KH2_sB1A6lA)\n* [Facebook's Build Environment](https://www.youtube.com/watch?v=X0VH78ye4yY)\n* [Speeding up builds with buck](https://buckbuild.com/)\n* [Culture as code](https://www.youtube.com/watch?v=YIpNpptGX6Q)\n* [Inner Source](https://paypal.github.io/InnerSourceCommons/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjonico%2Fspeedup-builds","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjonico%2Fspeedup-builds","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjonico%2Fspeedup-builds/lists"}