Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/threatcode/threat-ci-pipeline


https://github.com/threatcode/threat-ci-pipeline

Last synced: 1 day ago
JSON representation

Awesome Lists containing this project

README 

        
# Debian pipeline for Developers



Build and test on reproducible environments on every push.



TL;DR: Use this CI/CD configuration file setting:



```

recipes/debian.yml@salsa-ci-team/pipeline

```



## Table of contents

* [Introduction](#introduction)

* [What does this pipeline give to my project?](#what-does-this-pipeline-provide-for-my-projectpackage)

* [Basic Use](#basic-use)

* [Advanced Use](#advanced-use)

* [Contributing](#contributing)

* [Support](#support)



## Introduction



The Salsa CI Team's work aims to improve the Debian packaging lifecycle by providing [Continuous Integration](https://about.gitlab.com/product/continuous-integration/) fully compatible with Debian packaging.



Currently, all the building and testing performed by Debian QA is run asynchronously and takes a long time to give feedback because it is only accessible after pushing a release to the archive.



Our [pipeline](https://salsa.debian.org/help/ci/pipelines/index.md) definition is focused on speeding up this process by giving developers faster feedback.



## What does this _pipeline_ provide for my project/package?



The [pipeline](https://salsa.debian.org/help/ci/pipelines/index.md) builds your package(s) and runs multiple checks on them after every push to Salsa.



This provides you with instant feedback about any problems the changes you made may have created or solved, without the need to do a push to the archive, speeding up your development cycle and improving the quality of packages uploaded to Debian.



While the pipeline is a Work-In-Progress project, it will always try to replicate the tests run by Debian QA.

The services we got working are the following:



 * Building the package from the source (only gbp is supported)

 * [Lintian](https://lintian.debian.org)

 * Reproducible build using [Reprotest](https://reproducible-builds.org/tools)

 * [Piuparts](https://piuparts.debian.org)

 * [Autopkgtest](https://salsa.debian.org/ci-team/autopkgtest/raw/master/doc/README.package-tests.rst)

 * [Buildd Log Scanner](https://qa.debian.org/bls/)



Those services are enabled by something we called `salsa-pipeline` and it will be shared for all Salsa projects who adopt it.

Having this on GitLab CI ensures that every package accomplishes the minimum quality to be in the archive and if we improve or add a new service the project will get the benefit instantaneously.



## Basic Use



To use the Salsa Pipeline, the first thing to do is to enable the project's Pipeline. Go to `Settings` (General), expand `Visibility, project features, permissions`, and in `Repository`, enable `CI/CD`. This makes the `CI/CD` settings and menu available.

Then, change the project's setting to make it point to the pipeline's config file.

This can be done on `Settings` -> `CI/CD` (on the expanded menu, don't click on the CI / CD rocket) -> `General Pipelines` -> `CI/CD configuration file`.



If the base pipeline configuration fits your needs without further modifications, the recommended way is to use `recipes/debian.yml@salsa-ci-team/pipeline` as the config path, which refers to a file kept in the salsa-ci-team/pipeline repository.



On the other hand, if you want to use the base configuration and apply customizations on top, the recommended path to create this file is `debian/salsa-ci.yml`.

It should contain at least the following lines:



```yaml

---

include:

  - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/recipes/debian.yml

```



> :warning: **Note:** On Debian projects, you would normally want to put this file under the `debian/` folder.



## Advanced Use



Following the basic instructions will allow you to add all the building and testing stages as provided by the salsa-ci-team.

However, customization of the scripts is possible.



The [`salsa-ci.yml`](https://salsa.debian.org/salsa-ci-team/pipeline/blob/master/salsa-ci.yml) template delivers the jobs definitions.

Including only this file, no job will be added to the pipeline.

On the other hand, [`pipeline-jobs.yml`](https://salsa.debian.org/salsa-ci-team/pipeline/blob/master/pipeline-jobs.yml) includes all the jobs' instances.



### Changing the Debian Release



By default, everything will run on the `'unstable'` suite.

Changing the release is as easy as setting a `RELEASE` variable.



```yaml

---

include:

  - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/salsa-ci.yml

  - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/pipeline-jobs.yml



variables:

  RELEASE: 'buster'

```



The following releases are currently supported:

* stretch

* stretch-backports

* buster

* buster-backports

* bullseye

* bullseye-backports

* stable

* testing

* unstable

* experimental



### Avoid running CI on certain branches



It is possible to configure the pipeline to skip branches you don't want CI to run on.

The `SALSA_CI_IGNORED_BRANCHES` variable can be set to a regex that will be compared against the ref name and will decide if a pipeline is created.



By default, pipelines are only created for branches that contain a `debian/` folder.



```yaml

---

include:

  - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/salsa-ci.yml

  - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/pipeline-jobs.yml



variables:

  SALSA_CI_IGNORED_BRANCHES: 'some-branch|another-ref'

```



### Building with non-free dependencies

By default, only `main` repositories are used.

If your package has dependencies or build-dependencies in the `contrib` or `non-free` components (archive areas), set `SALSA_CI_COMPONENTS` to indicate this:



```yaml

---

include:

  - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/salsa-ci.yml

  - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/pipeline-jobs.yml



variables:

    RELEASE: 'stretch'

    SALSA_CI_COMPONENTS: 'main contrib non-free'

```



This is currently used for `piuparts`, but is likely to be used for

other stages in future.



It is possible to use the `SALSA_CI_EXTRA_REPOSITORY` support to add a

suitable apt source to the build environment and allow builds to access

build-dependencies from contrib and non-free. You will need permission

to modify the Salsa Settings for the project.



The CI/CD settings are at a URL like:



`https://salsa.debian.org///-/settings/ci_cd`

Expand the section on Variables and add a **File** type variable:



> Key: SALSA_CI_EXTRA_REPOSITORY



> Value: deb [signed-by=/usr/share/keyrings/debian-archive-keyring.gpg] https://deb.debian.org/debian/ sid contrib non-free



The apt source should reference `sid` or `unstable`.



Many `contrib` and `non-free` packages only build on `amd64`, so the

32-bit x86 build (`build i386`) should be disabled. (refer to the [Disabling building on i386](#Disabling-building-on-i386) Section).



### Skipping a job

There are many ways to skip a certain job. The recommended way is to set to `1` (or "`yes`" or "`true`") the `SALSA_CI_DISABLE_*` variables that have been created for this purpose.



```yaml

---

include:

  - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/salsa-ci.yml

  - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/pipeline-jobs.yml



# This sample disables all default tests, only disable those that you

# don't want

variables:

  SALSA_CI_DISABLE_APTLY: 1

  SALSA_CI_DISABLE_AUTOPKGTEST: 1

  SALSA_CI_DISABLE_BLHC: 1

  SALSA_CI_DISABLE_LINTIAN: 1

  SALSA_CI_DISABLE_PIUPARTS: 1

  SALSA_CI_DISABLE_REPROTEST: 1

  SALSA_CI_DISABLE_BUILD_PACKAGE_ALL: 1

  SALSA_CI_DISABLE_BUILD_PACKAGE_ANY: 1

  SALSA_CI_DISABLE_BUILD_PACKAGE_I386: 1

  SALSA_CI_DISABLE_CROSSBUILD_ARM64: 1

```



Alternatively, it is possible to disable all tests, and then enable only the

ones you want, which is good when testing a specific test:



```yaml

---

include:

  - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/salsa-ci.yml

  - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/pipeline-jobs.yml



# This sample disables all default tests, and then enables just autopkgtest

variables:

  SALSA_CI_DISABLE_ALL_TESTS: 1

  SALSA_CI_ENABLE_AUTOPKGTEST: 1

```



### Disabling building on i386

The `build i386` job builds packages against the 32-bit x86 architecture. If for any reason you need to skip this job, set the `SALSA_CI_DISABLE_BUILD_PACKAGE_I386` in the variables' block to `1`, '`yes`' or '`true`'.  i.e;



```yaml



variables:

  SALSA_CI_DISABLE_BUILD_PACKAGE_I386: 1

```



### Allowing a job to fail

Without completely disabling a job, you can allow it to fail without failing the whole pipeline. That way, if the job fails, the pipeline will pass and show an orange warning telling you something went wrong.



For example, even though reproducible builds are important, `reprotest`'s behavior can sometimes be a little erratic and fail randomly on packages that aren't totally reproducible (yet!). In such case, you can allow `reprotest` to fail by adding this variable in your salsa-ci.yml manifest:



```

include:

 - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/salsa-ci.yml

 - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/pipeline-jobs.yml



reprotest:

  allow_failure: true

```



### Set build timeout

At times your job may fail because it reached its max duration (either job timeout, or runner timeout).

In that case, the job would stop immediately without entering the `after_script` phase, and without saving the cache and without saving the artifacts.



To prevent this, the build phase of the build job and the build phase of the reprotest job have a timeout of `2.75h` (the runner's timeout is 3h). This permits also saving the cache of ccache. That way, on the next run, there is more chance to finish the job since it can use ccache's cache.



You can set the `SALSA_CI_BUILD_TIMEOUT_ARGS` variable to override this. The arguments can be any valid argument used by the `timeout` command. For example, you may set:



```

variables:

  SALSA_CI_BUILD_TIMEOUT_ARGS: "0.75h"

```



### Enabling the pipeline for tags

By default, the pipeline is run only for commits, tags are ignored. To run the pipeline against tags as well, export the `SALSA_CI_ENABLE_PIPELINE_ON_TAGS` variable and set it to one of "1", "yes" or "true", like in the following example:



```yaml

---

include:

  - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/salsa-ci.yml

  - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/pipeline-jobs.yml



variables:

  SALSA_CI_ENABLE_PIPELINE_ON_TAGS: 1

```



### Skipping the whole pipeline on push



There may be reasons to skip the whole pipeline for a `git push`, for example when you are adding `salsa-ci.yml` to hundreds of repositories or doing other mass changes.



You can achieve this in two ways:



You can insert `[ci skip]` or `[skip ci]`, using any capitalization, in the commit message. With this marker, GitLab will not run the pipeline when the commit is pushed.



Alternatively, one can pass the `ci.skip` [Git push](https://git-scm.com/docs/git-push#Documentation/git-push.txt--oltoptiongt) option if using Git 2.10 or newer:



```

git push --push-option=ci.skip    # using git 2.10+

git push -o ci.skip               # using git 2.18+

```



See also https://salsa.debian.org/help/ci/pipelines/index.md#skip-a-pipeline



### Adding your private repositories to the builds



The variables `SALSA_CI_EXTRA_REPOSITORY` and `SALSA_CI_EXTRA_REPOSITORY_KEY` can be

used to add private apt repositories to the sources.list, to be

used by the build and tests, and (optionally) the signing key for the

repositories in armor format. These variables are of

[type file](https://salsa.debian.org/help/ci/variables/index.md#cicd-variable-types),

which eases the multiline handling, but have the disadvantage that they can't

be set on the salsa-ci.yml file.



### Setting variables on pipeline creation



You can set these and other similar variables when launching a new pipeline in different ways:

 * Using the web interface under CI/CD, Run Pipeline, and setting the

desired variables.

 * Using the [GitLab API](https://salsa.debian.org/help/api/index.md). For example,  check the script

[salsa_drive_build.py](https://salsa.debian.org/maxy/qt-kde-ci/blob/tooling/salsa_drive_build.py),

in particular the function

[launch_pipelines](https://salsa.debian.org/maxy/qt-kde-ci/blob/tooling/salsa_drive_build.py#L568).

 * Setting them as part of a pipeline-triggered build.



### Only running selected jobs



If you want to use the definitions provided by the Salsa CI Team, but want to explicitly define which jobs to run, you might want to declare your YAML as follows:



```yaml

---

include: https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/salsa-ci.yml



extract-source:

    extends: .provisioning-extract-source



variables:

  RELEASE: 'experimental'



build:

  extends: .build-package



test-build-any:

  extends: .test-build-package-any



test-build-all:

  extends: .test-build-package-all



test-crossbuild-arm64:

  extends: .test-crossbuild-package-arm64



reprotest:

  extends: .test-reprotest



lintian:

  extends: .test-lintian



autopkgtest:

  extends: .test-autopkgtest



blhc:

  extends: .test-blhc



piuparts:

  extends: .test-piuparts



aptly:

  extends: .publish-aptly

```



On the previous example, the package is built on Debian experimental and checked through on all tests currently provided.

You can choose to run only some of the jobs by deleting any of the definitions above.



As new changes are expected to happen from time to time, we **firmly recommend NOT to do define all jobs manually**.

Please consider if [skipping jobs](#skipping-a-job) meets your needs instead.



### Testing build of arch=any and arch=all packages



If your package contains binary packages for `all` or `any`, you may want to test if those can be built in isolation from the full build normally done.



This verifies the Debian buildds can build your package correctly when building for other architectures that the one you uploaded in or in case a binNMU is needed or you want to do source-only uploads.



The default `pipeline-jobs.yml` does this automatically based on the contents of `debian/control`, but if you manually define the jobs to run, you also need to include the `test-build-any` and `test-build-all` jobs manually as well:



```yaml

test-build-any:

  extends: .test-build-package-any



test-build-all:

  extends: .test-build-package-all

```



`.test-build-package-any` runs `dpkg-buildpackage` with the option `--build=any` and will only build arch-specific packages.



`.test-build-package-all` does the opposite and runs `dpkg-buildpackage` with the option `--build=all` building only arch-indep packages.



### Enable generation of dbgsym packages



To reduce the size of the artifacts produced by the build jobs, auto-generation

of dbgsym packages is disabled by default. This behaviour can be controlled by

the `SALSA_CI_DISABLE_BUILD_DBGSYM`. Set it to anything different than 1, 'yes'

or 'true', to generate those packages.



```yaml

variables:

  SALSA_CI_DISABLE_BUILD_DBGSYM: 0

```



### Enable building packages twice in a row



The job `test-build-twice` can be used to check whether it is possible to run

`dpkg-buildpackage` twice in a row. To enable this check, either run your

pipeline manually with `SALSA_CI_DISABLE_BUILD_PACKAGE_TWICE` set to anything

different than 1, 'yes' or 'true' or by adding the following to your

`debian/salsaci.yml`:



```yaml

variables:

  SALSA_CI_DISABLE_BUILD_PACKAGE_TWICE: 0

```



### Customizing Lintian



The Lintian job can be customized to ignore certain tags.



To ignore a tag, add it to the setting `SALSA_CI_LINTIAN_SUPPRESS_TAGS`.



By default, the Lintian jobs fail either if a Lintian run-time error occurs or if Lintian finds a tag of the error category.



To also fail the job on findings of the category warning, set `SALSA_CI_LINTIAN_FAIL_WARNING` to 1 (or "yes" or "true").



To make Lintian shows overridden tags, set `SALSA_CI_LINTIAN_SHOW_OVERRIDES` to 1 (or "yes" or "true").



```yaml

---

include:

  - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/salsa-ci.yml

  - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/pipeline-jobs.yml



variables:

  SALSA_CI_LINTIAN_FAIL_WARNING: 1

  SALSA_CI_LINTIAN_SUPPRESS_TAGS: 'orig-tarball-missing-upstream-signature'

```



It is possible to add any other lintian argument using `SALSA_CI_LINTIAN_ARGS`. Those

arguments will be appended after the arguments generated by the pipeline.



### Adding extra arguments to autopkgtest



Sometimes it is desirable to add arguments to autopkgtest.



You can do this by setting the arguments in the `SALSA_CI_AUTOPKGTEST_ARGS` variable.



```yaml

---

include:

  - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/salsa-ci.yml

  - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/pipeline-jobs.yml



variables:

  SALSA_CI_AUTOPKGTEST_ARGS: '--debug'

```



Note that autopkgtest can access the repository in the current directory, making it possible for `--setup-commands` to read commands from a file. For example:



```yaml

----

include

  - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/salsa-ci.yml

  - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/pipeline-jobs.yml



variables:

  SALSA_CI_AUTOPKGTEST_ARGS: '--setup-commands=ci/pin-django-from-backports.sh'

```



### Adding extra arguments to dpkg-buildpackage



Sometimes it is desirable to add direct options to the dpkg-buildpackage that is run for the package building.



You can do this using the `SALSA_CI_DPKG_BUILDPACKAGE_ARGS` variable.



```yaml

---

include:

  - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/salsa-ci.yml

  - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/pipeline-jobs.yml



variables:

  SALSA_CI_DPKG_BUILDPACKAGE_ARGS: --your-option

```



### Adding extra arguments to gbp-buildpackage



Sometimes it is desirable to add direct options to the `gbp buildpackage` command.



You can do this using the `SALSA_CI_GBP_BUILDPACKAGE_ARGS` variable.



```yaml

---

include:

  - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/salsa-ci.yml

  - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/pipeline-jobs.yml



variables:

  SALSA_CI_GBP_BUILDPACKAGE_ARGS: --your-option

```



### Executing a pre-install / post-install script in piuparts



Sometimes it is desirable to execute a pre-install or post-install scripts in piuparts.



You can do this using the `SALSA_CI_PIUPARTS_PRE_INSTALL_SCRIPT` and `SALSA_CI_PIUPARTS_POST_INSTALL_SCRIPT` variables, that may point to the path of scripts in the project repository. This could be used, for instance, to pin dependencies. For example, if you had a `ci/pin-django-from-backports.sh` script:



```yaml

---

include:

  - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/salsa-ci.yml

  - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/pipeline-jobs.yml



variables:

  SALSA_CI_PIUPARTS_PRE_INSTALL_SCRIPT: 'ci/pin-django-from-backports.sh'

```



The `ci/pin-django-from-backports.sh`:

```shell

#!/bin/sh



set -e



# Ensure that python3-django from bullseye-backports will be

# installed as needed (bullseye has too old version)

cat >/etc/apt/preferences.d/99django-backports <