https://github.com/ecmwf-projects/cads-build-farm
Build farm
https://github.com/ecmwf-projects/cads-build-farm
cads
Last synced: 5 months ago
JSON representation
Build farm
- Host: GitHub
- URL: https://github.com/ecmwf-projects/cads-build-farm
- Owner: ecmwf-projects
- Created: 2023-01-24T14:02:04.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2025-11-18T19:17:27.000Z (7 months ago)
- Last Synced: 2025-11-18T21:06:58.099Z (7 months ago)
- Topics: cads
- Language: Python
- Homepage:
- Size: 543 KB
- Stars: 0
- Watchers: 6
- Forks: 1
- Open Issues: 7
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# cads-build-farm
This repository manages the "build and push" GitHub Actions for all the services in CADS project. This GitOps part is an important component of the whole CI/CD pipeline that includes also FluxCD configuration on [cads-deployment](https://github.com/ecmwf-projects/cads-deployment).
When developing please install pre-commit hooks with:
`pre-commit install`
## CI/CD Workflow
The current pipeline is designed in order to have the **Continuous Deployment** part fully automated. At the moment, the **Integration** part remains manual to grant the developers more control on releases.
#### FluxCD automated Deployment
As described [here](https://github.com/ecmwf-projects/cads-deployment) FluxCD automatically reconcile what's present in a source with the status of a Kubernetes cluster. In this pipeline this tool has two scopes: with two dedicated configurations, Flux is able to keep updated each service inside the `cds2-dev` namespace and deploy on `cds2-test` only the tagged images. This setting allow us to have a continuously-deployed dev environment with the last changes on the code of all repositories and have a production environment (`cds2-test`) updated only with images that match with a specific tag (release versioning).
#### Integration tests
To verify the proper functioning of the code deployed on a specific namespace the user(developer) could run manually the integration tests from the [cads-api-client repo](https://github.com/ecmwf-projects/cads-api-client). The integration tests check all the services and api exposed.
#### Build and Push
Once that the developer is satisfied with all the changes the [Build all images](https://github.com/ecmwf-projects/cads-build-farm/actions/workflows/build-images.yml) could be triggered manually. During this step is possible to specify the tag to use for the build. This will automatically trigger the "build and push" workflow of each CADS service that build the image with the GitHub BuildX provided engine and push the builded image on the Harbor private registry with a `tag`. During the manual triggering step is possible to specify the `tag` to use for the build.
#### Tag release commit
Once that all the images are builded and stored on the private registry the next step is to change the Kustomize definition files specifying the `tag` of the version for the images used into `cds2-test` namespace. This will trigger the deploy on that namespace with the new images selected.
### Release pipeline example
At the end of a Work Package cycle we want to release the version `2023.02.0` of the CADS portal.
1. All the developers merges their working branches on main of each service repo. This will trigger Flux.
1. FluxCD configuration updates `cds2-dev` namespace inside the Kubernetes cluster with all the changes.
1. Run the integration tests as described from the cads-api-client repo README.md
```
cd /cads-api-client
conda activate DEVELOP
CADS_API_URL=http://cds2-dev.copernicus-climate.eu/api make integration-tests
```
If somehow anything is broken or some of the tests have **FAILED** status you can safely check it, fix and commit changes on the related repo
1. Now that all the test have **PASSED** status and everything works as expected, we can manually trigger the GitHub Action `Build all images` on this repo from the Action panel. I specify that the tag will be `2023-02.0`
Wait until all the workflows are finished
1. Wait until all the workflows are finished. Now from `cads-deployment` repo edit this file
```
cads-deployment/kustomize/overlays/test/kustomization.yaml
```
and change the `newTag` parameter for all the related services under the `images` list
```
images:
- name: cads-broker
newName: my-private-registry/broker
newTag: '2023-02.0'
- name: catalogue-api
newName: my-private-registry/catalogue-api
newTag: '2023-02.0'
- name: cads-catalogue-manager
[...]
```
Commit to trigger Flux Automated Deployment
1. FluxCD configuration updates `cds2-test` namespace inside the Kubernetes cluster. All the components will be deployed using the selected image. Now you can set the version tag also in Flux configuration file.
To do that edit this file
```
cads-deployment/clusters/ecmwf/test/cads-deployment-test-source.yaml
```
and change the value for the `ref` parameter to `2023-02.0`