https://github.com/seatgeek/buildkit-prestop-script
A simple PreStop script for BuildKit that ensures all ongoing builds complete before Kubernetes stops the pod.
https://github.com/seatgeek/buildkit-prestop-script
buildkit kubernetes prestop
Last synced: 3 months ago
JSON representation
A simple PreStop script for BuildKit that ensures all ongoing builds complete before Kubernetes stops the pod.
- Host: GitHub
- URL: https://github.com/seatgeek/buildkit-prestop-script
- Owner: seatgeek
- License: apache-2.0
- Created: 2024-08-13T11:21:07.000Z (9 months ago)
- Default Branch: main
- Last Pushed: 2024-12-11T20:38:40.000Z (5 months ago)
- Last Synced: 2024-12-11T21:27:24.143Z (5 months ago)
- Topics: buildkit, kubernetes, prestop
- Language: Shell
- Homepage:
- Size: 37.1 KB
- Stars: 10
- Watchers: 7
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Contributing: .github/CONTRIBUTING.md
- License: LICENSE
- Code of conduct: .github/CODE_OF_CONDUCT.md
- Security: .github/SECURITY.md
Awesome Lists containing this project
README
# buildkit-prestop-script
[](LICENSE)
[](https://github.com/seatgeek/buildkit-prestop-script/actions?query=workflow%3ATests+branch%3Amain)
[](https://github.com/seatgeek/buildkit-prestop-script/releases)
[](./buildkit-prestop.sh)
A simple [PreStop] script for [BuildKit] that ensures all ongoing build complete before Kubernetes stops the pod.
## Why?
[BuildKit does not (currently) support graceful shutdowns when it receives a `SIGTERM` signal][buildkit-4090].
This means that if a BuildKit pod is stopped while any builds are in progress, those builds will be terminated immediately,
resulting in failed builds.
We can mitigate this by providing a script that will wait for all ongoing builds to finish before allowing the pod to stop.
## Usage
### Option 1: Bake the script into your BuildKit image
1. Add [the `buildkit-prestop.sh` script](./buildkit-prestop.sh) to your BuildKit pod's container image. For example:
```Dockerfile
FROM moby/buildkit:v0.15.1-rootless
ADD --chmod=755 https://raw.githubusercontent.com/seatgeek/buildkit-prestop-script/main/buildkit-prestop.sh /usr/local/bin/buildkit-prestop.sh
```
2. Add the following to your BuildKit pod's spec:
```yaml
lifecycle:
preStop:
exec:
command: ["/usr/local/bin/buildkit-prestop.sh"]
```
### Option 2: Mount the script with a ConfigMap
Instead of extending the buildkit docker image, it's also possible to mount the preStop script inside the pod via a `ConfigMap`:
1. Download [the `buildkit-prestop.sh` script](./buildkit-prestop.sh) to your local machine.
2. Create a `ConfigMap` from that file:
```bash
kubectl create configmap buildkit-prestop-script --from-file=buildkit-prestop.sh
```
Which should create a `ConfigMap` named `buildkit-prestop-script`:
```
apiVersion: v1
kind: ConfigMap
metadata:
name: buildkit-prestop-script
data:
buildkit-prestop.sh: |
#!/bin/bash
...
```
3. Update your BuildKit `Deployment` manifest to mount the `ConfigMap` as a volume and reference the script:
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: buildkitd
name: buildkitd
spec:
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 0
template:
spec:
terminationGracePeriodSeconds: 1200
containers:
- name: buildkitd
image: moby/buildkit:v0.16.0
lifecycle:
preStop:
exec:
command: ["/bin/sh", "/usr/local/bin/buildkit-prestop.sh"]
volumeMounts:
- name: prestop-script-volume
mountPath: /usr/local/bin/buildkit-prestop.sh
subPath: buildkit-prestop.sh
volumes:
- name: prestop-script-volume
configMap:
name: buildkit-prestop-script
defaultMode: 0777 # Ensure the script is executable
```
## How it works
BuildKit does not provide any built-in API for querying the count and status of ongoing builds, forcing us to rely on
external observations to determine when it is safe to stop the pod.
We found that the most reliable method is to check whether any clients are connected to the BuildKit daemon via `netstat`.
If we see any established connections then we assume that there is an ongoing build process.
## Configuration
The script contains a few configuration options that you can adjust to suit your needs. See the script for details.
## License
This project is licensed under the Apache License, Version 2.0. See [LICENSE](./LICENSE) for the full license text.
[PreStop]: https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks
[BuildKit]: https://github.com/moby/buildkit
[buildkit-4090]: https://github.com/moby/buildkit/issues/4090