https://github.com/jenkins-infra/release

Contains every things needed to release jenkins core from the jenkins infra project
https://github.com/jenkins-infra/release

jenkins release

Last synced: 3 months ago
JSON representation

Contains every things needed to release jenkins core from the jenkins infra project

• Host: GitHub
• URL: https://github.com/jenkins-infra/release
• Owner: jenkins-infra
• License: mit
• Created: 2019-09-20T09:05:30.000Z (over 5 years ago)
• Default Branch: master
• Last Pushed: 2024-10-28T00:04:20.000Z (3 months ago)
• Last Synced: 2024-10-28T04:11:56.740Z (3 months ago)
• Topics: jenkins, release
• Language: Shell
• Size: 990 KB
• Stars: 15
• Watchers: 10
• Forks: 22
• Open Issues: 16
• Metadata Files:
  • Readme: README.adoc
  • License: LICENSE
  • Codeowners: .github/CODEOWNERS

Awesome Lists containing this project

README

        
= Jenkins Release

:toc: right

== Introduction

This repository contains everything needed to release https://github.com/jenkinsci/jenkins[Jenkins core].

This includes:

* Documentation how to release Jenkins LTS releases, available at link:docs/releases.md[doc/releases.md].

* Jenkinsfiles for Releasing and Packaging.

* Kubernetes POD templates definition for releasing and packaging environments on Linux and Windows operating systems.

* Any configuration or utility scripts

The release process implies following Git repository:

* https://github.com/jenkinsci/jenkins[jenkinsci/jenkins] or any other git repository containing the Jenkins code to be released.

* https://github.com/jenkinsci/packaging[jenkinsci/packaging] contains tools and configuration to package Jenkins for OS distribution [Debian, Red Hat, SUSE, MSI, WAR]

* https://github.com/jenkins-infra/chart[jenkins-infra/chart] contains the Helm chart used to deploy link:https://release.ci.jenkins.io/[release.ci.jenkins.io] on top of Kubernetes.

* https://github.com/jenkins-infra/azure[jenkins-infra/azure] contains the terraform code used to provision infrastructure needed by the release environment

== Settings

[cols="1,3a,0,0", options="header"]

|===

| Variable | Description | Release | Packaging

| `GIT_EMAIL`

| Defines git email used in git commits.

| x

| x

| `GIT_NAME`

| Defines git name used in git commits.

| x

| x

| `GIT_SSH_COMMAND`

| Defines options that can be passed to every git commands used in release scripts.

| x

| x

| `GPG_KEYNAME`

| Define the GPG key name that will  be used to sign artifacts.

| x

| x

| `GPG_FILE`

| Define the GPG file name that will  be used to import gpg key into keyring.

| x

| x

| `GPG_VAULT_NAME`

| Define the Azure key vault containing the gpg key.

| x

| x

| `JENKINS_ASC`

| Defines the Jenkins signature file name.

|

| x

| `JENKINS_DOWNLOAD_URL`

| Define the url used to download jenkins.war.

|

| x

| `JENKINS_VERSION`

| Define which version of Jenkins will be downloaded for packaging.

It accepts following values:

- `latest`, fetches the latest version available in a maven repository based on the maven-metadata.xml.

- `version`, search the latest version available based on version.

More information in https://github.com/jenkins-infra/release/blob/master/utils/getJenkinsVersion.py[getJenkinsVersion.py]

[IMPORTANT]

====

 . Versions available depends on the maven repository defined by `MAVEN_REPOSITORY_NAME`

 . `JENKINS_VERSION` is totally independent of `RELEASE_PROFILE` explained below

====

|

| x

| `JENKINS_WAR`

| Defines the jenkins.war file name downloaded from a maven repository using `getJenkinsVersion.py`.

|

| x

| `RELEASE_GIT_REPOSITORY`

| Defines the Jenkins git repository read by the build. It is also the repository where generated commits and new tags will be pushed.

| x

| x

| `RELEASE_GIT_BRANCH`

| Defines the Jenkins branch used to checkout and then pushes commits related to the release.

| x

| x

| `RELEASE_GIT_PRODUCTION_BRANCH`

| Defines the branch used as the production branch during promotion.

|

| x

| `RELEASE_GIT_PRODUCTION_REPOSITORY`

| Defines the git repository used as the production git repository during promotion.

|

| x

| `RELEASE_GIT_STAGING_BRANCH`

| Defines the branch used as the staging git branch used to promote to the production one.

|

| x

| `RELEASE_GIT_STAGING_REPOSITORY`

| Defines the git repository used as the staging git repository during promotion.

|

| x

| `RELEASE_GIT_STAGING_REPOSITORY_PATH`

| Defines path where staging git repository will be be checkout.

|

| x

|`MAVEN_REPOSITORY_URL`

| Defines the maven repository URL where is pushed generated artifacts.

| x

| x

|`MAVEN_REPOSITORY_NAME`

| Defines the maven repository name where is pushed generated artifacts.

| x

| x

|`MAVEN_REPOSITORY_PASSWORD`

| Defines the maven repository password.

| x

| x

|`MAVEN_REPOSITORY_USERNAME`

| Defines the maven repository username.

| x

| x

|`MAVEN_REPOSITORY_PRODCUTION_NAME`

| Defines the production maven repository name used during maven promotion.

|

| x

| `PKGSERVER`

| Defines where the different packages will be published.

|

| x

| `PKGSERVER_SSH_OPTS`

| Defines custom ssh options used to connect to `PKGSERVER`.

|

| x

| `PROMOTE_STAGING_MAVEN_ARTIFACTS_ARGS`

| Defines parameters used by `promoteMavenArtifacts.py`.   

Default value is set to `item --mode copy --source $MAVEN_REPOSITORY_NAME --destination $MAVEN_REPOSITORY_PRODUCTION_NAME --url $MAVEN_REPOSITORY_URL --username $MAVEN_REPOSITORY_USERNAME --password $MAVEN_REPOSITORY_PASSWORD --search '/org/jenkins-ci/main' $(./utils/getJenkinsVersion.py --version)}"`

|

| x

| `RELEASELINE`

| Define the release line used by packaging scripts in https://github.com/jenkinsci/packaging[jenkinsci/packaging].

|

| x

| `RELEASE_PROFILE`

| Define a file containing environment variables specific to a release, located in the profile.d directory.

| x

| x

| `SIGN_ALIAS`

| Define code signing certificate name.

| x

| x

| `SIGN_KEYSTORE_FILENAME`

| Define code signing certificate file name.

| x

| x

| `SIGN_KEYSTORE`

| Define signing keystore.

| x

| x

| `SIGN_CERTIFICATE`

| Define code signing certificate file name.

| x

| x

|===

== Process

=== Introduction

The release process is divided in two categories.

The first part that we mention by using the term the *release*, is when we create a new java code release. It relies on the Maven Release Plugin to perform the release. It involves signing with a GPG key and a code signing certificate. At the end of this operation signed maven artifacts are pushed to a Maven repository.

The second part that we name *packaging* is when we retrieve from a Maven repository, the version we want to package. Then we build distribution packages, publish them, promote artifacts between staging and production environment if needed and finally we ensure that our mirrors are up to date.

==== Required

In order to trigger a new release, you must fulfil following requirements:

- [x] https://github.com/jenkins-infra/docker-openvpn#howto-get-client-access[Jenkins private VPN Access] access

- [x] Being part of LDAP group 'release-core'

- [x] https://release.ci.jenkins.io[release.ci.jenkins.io] access

=== Release

At this stage, we are going to retrieve the Java code, release a new version using the maven release plugin and then publish artifacts on a maven repository.

It's important to notice that we do not use the maven release plugin to checkout git repositories neither to push changes. This allow us to release from a different git repository than the one defined in the pom.xml. We also need to be able to push commits to a different repository than the one defined in the pom.xml.

. link:http://maven.apache.org/maven-release/maven-release-plugin/perform-mojo.html#localCheckout[localCheckout] must be set to true

. link:http://maven.apache.org/maven-release/maven-release-plugin/prepare-mojo.html#pushChanges[pushChanges] must be set to false

==== Steps

Estimated time +- 1h30

. Connect to the Jenkins private VPN (private.vpn.jenkins.io).

. Open your favorite browser to https://release.ci.jenkins.io.

. Trigger the release job on the master branch. https://release.ci.jenkins.io/job/core/job/release/[Link].

. Once triggered, it asks you which release line you want to do. It's important to know that the release line matches one of the profiles file defined https://github.com/jenkins-infra/release/tree/master/profile.d[here], so please carefully review the settings and be sure that it does what you are looking for.

. At the end of the job, git commits and maven artifacts will be pushed to their respective locations.

==== Validate

To validate that the release went well, excepted by having a green build, you can double-check that your artifacts have been correctly pushed to Maven repository located on `$MAVEN_REPOSITORY_URL/$MAVEN_REPOSITORY_NAME/org/jenkins-ci/main/jenkins-war/`. You could also run `jarsigner -verify ` is correct.

==== Stage

In order to have private maven releases, you can modify `RELEASE_GIT_REPOSITORY` or `MAVEN_REPOSITORY_NAME`, to respectively use code from a private git repository and then push artifacts to a private maven repository.

Artifact promotion is done in the next stage 'packaging'.

=== Packaging

The packaging process looks after the latest Jenkins version published on a Maven repository and then build and publish artifacts for Debian, Red Hat, SUSE, Windows. It also republishes the War file on the package server.

If enabled, then it promotes git commits between git repository, promotes maven artifacts between maven repository.

NOTE: Packages are not re-published if they already exist, only package website is overridden so it's safe to re-trigger the job.

==== Steps

Estimated time +- 30min

. Connect to the Jenkins private VPN (private.vpn.jenkins.io)

. Open your favorite browser to https://release.ci.jenkins.io[release.ci.jenkins.io]

. Trigger the packaging job on the master branch. https://release.ci.jenkins.io/job/core/job/package/[Link]

. Once triggered, it asks you which release line you want to package for. The release line matches one of the profile defines in https://github.com/jenkins-infra/release/tree/master/profile.d[profile.d], so please carefully review those settings in order to validate that's what you are looking for.

Once the job is done, every package will be published and then mirror synchronized.

==== Validate

Ensure that packages are correctly published on pkg.jenkins.io and correctly signed.

==== Stage

Staging packages is not yet fully supported, more information on link:https://issues.jenkins-ci.org/browse/INFRA-1363[INFRA-1363] and link:https://issues.jenkins-ci.org/browse/INFRA-2608[INFRA-2608]

== Profile

The release profile is used to identify the kind of release we are going to do. They are mainly influenced by following elements:

. Do we want to releases based on different repository branch?

. Do we want to release based on different git repository?

At the moment we identify three release types:

. Weekly

. Stable

. Security (both weekly and LTS)

[NOTE]

====

link:https://release.ci.jenkins.io[release.ci.jenkins.io] has two generic jobs, one for release and a second one for packaging.

One job per release type triggers the two generic jobs with different parameters.

While it isn't required to trigger a release type job, it increases visibility and reduces the risk of human error.

====

=== Weekly

The weekly release is the default release. It is scheduled every Tuesday as defined by this link:https://github.com/jenkins-infra/release/blob/19685def608c641496e6e2de3d40c275ca5e913d/Jenkinsfile.d/core/weekly#L15[cron].

It uses parameters defined in this link:https://github.com/jenkins-infra/release/blob/master/profile.d/weekly[file]

It releases using the repository https://github.com/jenkinsci/jenkins[jenkinsci/jenkins] from the branch master. Artifacts are pushed to the default maven repository 'Releases'.

If for some reason the release job needs to be re-triggered, you can:

. Connect to the Jenkins private VPN (private.vpn.jenkins.io)

. Open your favorite browser to link:https://release.ci.jenkins.io[release.ci.jenkins.io]

. Review the weekly environment https://github.com/jenkins-infra/release/blob/master/profile.d/weekly[file]:

. Trigger the weekly link:https://release.ci.jenkins.io/job/core/job/weekly/job/release/[job]

[NOTE]

====

You can re-trigger individually the two downstream jobs, release and packaging.

- Re-triggering the release will do a version bump then push new artifacts.

- Re-triggering the packaging job won't published artifacts if they already exist but it will update website html.

====

=== Stable Release Candidate

A stable release-candidate is a manually triggered release that happens around once a month.

It uses parameters defined in this link:https://github.com/jenkins-infra/release/blob/master/profile.d/stable-rc[file].

Before triggering a new stable release candidatae release, some steps are required:

. Prepare `jenkinsci/jenkins` repository -> missing documentation link.

. Create a branch on jenkins-infra/release with a branch name that match the release branch from jenkinsci/jenkins like `rc-stable-`.

. Review and update the stable environment https://github.com/jenkins-infra/release/blob/master/profile.d/stable[file] with:

.. `RELEASE_GIT_BRANCH` set to the `jenkinsci/jenkins` release branch like `stable-2.235`

.. `PACKAGING_GIT_BRANCH` set to the appropriate `jenkinsci/packaging` branch, e.g. `stable-2.235`

. Trigger the stable link:https://release.ci.jenkins.io/job/core/job/stable/job/release/[job]

[NOTE]

====

You can re-trigger individually the two downstream jobs, release and packaging.

- Re-triggering the release will do a version bump then push new artifacts.

- Re-triggering the packaging job won't published artifacts if they already exist but it will update website html.

====

=== Stable

A stable release is a manually triggered release that happens around once a month.

Refer to link:https://www.jenkins.io/download/lts/[LTS Release Line] for more detailed information.

It uses parameters defined in this link:https://github.com/jenkins-infra/release/blob/master/profile.d/stable[file].

Before triggering a new stable release, some steps are required:

. Prepare `jenkinsci/jenkins` repository -> missing documentation link.

. Create a branch on jenkins-infra/release with a branch name that match the release branch from jenkinsci/jenkins like `stable-`.

. Review and update the stable environment https://github.com/jenkins-infra/release/blob/master/profile.d/stable[file] with:

.. `RELEASE_GIT_BRANCH` set to the `jenkinsci/jenkins` release branch like `stable-2.235`

.. `JENKINS_VERSION` set to the final release version that will be packaged. If set to 'stable' then the packaging job will try to guess the version based on what was pushed to the maven repository. cfr settings.

.. `PACKAGING_GIT_BRANCH` set to the appropriated `jenkinsci/packaging` branch

. Trigger the stable link:https://release.ci.jenkins.io/job/core/job/stable/job/release/[job]

[NOTE]

====

You can re-trigger individually the two downstream jobs, release and packaging.

- Re-triggering the release will do a version bump then push new artifacts.

- Re-triggering the packaging job won't published artifacts if they already exist but it will update website html.

====

=== Security

The security release follows the same process as the stable one except that artifacts are published in private. So we need to promote git commits from a private repository to the public one then promote maven artifacts from a private maven repository to the public one.

The following sections assume you have prepared jenkinsci-cert/jenkins with security fixes and created a Maven staging repository as documented by the security team.

All steps need to be done twice: Once for weekly, once for LTS.

==== Preparation

. Create a (origin) branch on jenkins-infra/release with a branch name corresponding to the specific release, e.g. `security-2.287` or `security-stable-2.303.2`. Base them on the `master` (weekly) or `stable-2.303` (LTS) branch, respectively.

. In your fork, update the security environment https://github.com/jenkins-infra/release/blob/master/profile.d/security[file] with the following (new) entries:

.. `RELEASE_GIT_BRANCH` set to the `jenkinsci-cert/jenkins` release branch like `security-stable-2.303`

.. `MAVEN_REPOSITORY_NAME` set to the maven repository name where we are going to publish staging maven artifacts, e.g. `caravelli`. This is also the source location used by the packaging job to build distribution packages.

.. `JENKINS_VERSION` set to the final release version that will be packaged.

.. `RELEASELINE` set to '-stable' for an LTS release, otherwise leave empty or undefined.

.. Open a PR from your fork's branch into the origin repository's branch to allow review.

==== Staging (before release day)

To stage the Maven artifacts, trigger the generic Release link:https://release.ci.jenkins.io/job/core/job/release/[job] from the appropriate branch like `security-stable-2.303.2`.

To do that, follow these steps:

.. Force repository scan

.. Trigger the first build to have access to job parameter and immediately abort it

.. Trigger a job with the correct parameters

... `RELEASE_PROFILE` set to `security`

... `RELEASE_GIT_BRANCH` set to `unused` as we already define it in the release profile file, which overrides the job parameter

... `MAVEN_REPOSITORY_NAME` set to `unused` as we already define it in the release profile file, which overrides the job parameter

... `VALIDATION_ENABLED` set to true if the validation stage should run

==== Publishing (on release day)

. To create and publish packages, trigger the generic Packaging job link:https://release.ci.jenkins.io/job/core/job/package/[job] from the appropriate branch like `security-stable-2.303.2` with correct parameters

.. `RELEASE_PROFILE` set to `security`

.. `RELEASE_GIT_BRANCH`  set to `unused` same reason as before

.. `MAVEN_REPOSITORY_NAME` set to `unused` same reason as before

.. `MAVEN_REPOSITORY_PRODUCTION_NAME` set to `unused`

.. `MAVEN_STAGING_REPOSITORY_PROMOTION_ENABLED` set to false (manually done by publishing-tool in a parallel process)

.. `GIT_STAGING_REPOSITORY_PROMOTION_ENABLED` set to false (manually merged by security team)

.. `VALIDATION_ENABLED` set to true

== Certificate

The Jenkins project uses a Digicert account provided by CDF to request code signing certificate.

The release environment is designed to download a pkcs12 certificate from Azure key vault.

. Request a code signing certificate from Digicert

. Convert the code signing certificate from Digicert from `p7b` format to a `pfx` (with pkcs12) certificate which also includes the private key but not export password

. Upload the `pfx` certificate to Azure Key Vault

. Update the release environment credentials with appropriated password

.Certificate Fields

----

   Country Name: US

   State: DE

   Organization: CDF Binary Project a Series of LF Projects, LLC

   Organization Unit: Jenkins Project

   Common Name: Jenkins

----

Generate a new code signing certificate private key and a certificate signing request:

  openssl req -out jenkins-release.csr -new -newkey rsa:4096 -keyout jenkins-release.key

Show csr information

  openssl req -text -noout -verify -in jenkins-release.csr

Show private key information

  openssl rsa -in jenkins-release.key -check

Show certificate information

  openssl x509 -in jenkins-release.crt -text -noout

Convert p7b to pkcs12

  

  # Based from https://knowledge.digicert.com/solution/SO26449.html and https://github.com/jenkins-infra/release/blob/7a03f98eff839d4fed75ea96cf7bebbc963e3a91/README.adoc#certificate

  # P7B to PFX: 1/2

  openssl pkcs7 -print_certs -in digicert.p7b -out jenkins-release.crt

    ## Asks for the Export password, transmitted by Digicert from another channel

    ## Asks for the `jenkins-release.key` private key passphrase

  

  # Check for the intermediate certificate attributes

  openssl x509 -in jenkins-release.crt -text -noout

  

  # P7B to PFX: 2/2

  openssl pkcs12 -export -in jenkins-release.crt -inkey jenkins-release.key -out jenkins-release.pfx

    ## Asks for an Export password: do not set any (type enter only)

  Enter Export Password: # Empty!!

  Verifying - Enter Export Password: # Empty!!

Check PFX (pkcs12 format) attributes

  openssl pkcs12 -info -in jenkins.pfx

== Core Maintainers

More information about Jenkins Core maintainers and the different roles can be found in https://github.com/jenkinsci/jenkins/blob/master/docs/MAINTAINERS.adoc[MAINTAINERS].

== FAQ

**The stage release failed and we already push commits during the maven release.**

The problem here, is if we re-trigger the release job, we also update the release version again.

So if the process fails on running  `mvn release:stage`, then we can re-trigger it from inside the container as custom parameters are located in `settings-release.xml`.

```

kubectl get pods -n release # Looking for the correct jenkins agent name

kubectl exec -i -t -n release -c maven  bash

cd /home/jenkins/agent/workspace/core_release_master/release

mvn -B -DstagingRepository=releases::https://repo.jenkins-ci.org/releases -s settings-release.xml --no-transfer-progress release:stage

```

== Miscellaneous

* Jenkins docker images are describe from https://github.com/jenkinsci/docker[jenkinsci/docker] repository and deployed to DockerHub as link:https://hub.docker.com/r/jenkins/jenkins[jenkins/jenkins]

* https://pkg.jenkins.io[pkg.jenkins.io], service to download Jenkins packages

* https://repo.jenkins-ci.org[repo.jenkins-ci.org], Jenkins Maven repository

* https://mirrors.jenkins.io[mirrors.jenkins.io]

* https://issues.jenkins-ci.org/browse/INFRA-910[INFRA-910] - EPIC for the new Jenkins Core Release Environment

* https://github.com/jenkins-infra/docker-packaging[docker-packaging], defines the docker image used to execute packaging scripts