Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/billziss-gh/pmci
Poor Man's CI - Hosted CI with shell scripting and duct tape
https://github.com/billziss-gh/pmci
Last synced: about 2 months ago
JSON representation
Poor Man's CI - Hosted CI with shell scripting and duct tape
- Host: GitHub
- URL: https://github.com/billziss-gh/pmci
- Owner: billziss-gh
- License: bsd-2-clause
- Created: 2018-06-16T06:53:01.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2018-08-01T22:52:52.000Z (about 6 years ago)
- Last Synced: 2024-04-17T00:19:40.135Z (5 months ago)
- Language: JavaScript
- Homepage:
- Size: 201 KB
- Stars: 32
- Watchers: 4
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: License.txt
Awesome Lists containing this project
README
# Poor Man's CI
Poor Man's CI (PMCI - Poor Man's Continuous Integration) is a collection of scripts that taken together work as a simple CI solution that runs on Google Cloud. While there are many advanced hosted CI systems today, and many of them are free for open source projects, none of them seem to offer a solution for the BSD operating systems (FreeBSD, NetBSD, OpenBSD, etc.)
The architecture of Poor Man's CI is system agnostic. However in the implementation provided in this repository the only supported systems are FreeBSD and NetBSD. Support for additional systems is possible.
Poor Man's CI runs on the Google Cloud. It is possible to set it up so that the service fits within the Google Cloud ["Always Free"](https://cloud.google.com/free/docs/always-free-usage-limits) limits. In doing so the provided CI is not only hosted, but is also free! (Disclaimer: I am not affiliated with Google and do not otherwise endorse their products.)
## ARCHITECTURE
A CI solution listens for "commit" (or more usually "push") events, builds the associated repository at the appropriate place in its history and reports the results. Poor Man's CI implements this very basic CI scenario using a simple architecture, which we present in this section.
Poor Man's CI consists of the following components and their interactions:
- **`Controller`**: Controls the overall process of accepting GitHub `push` events and starting builds. The `Controller` runs in the Cloud Functions environment and is implemented by the files in the `controller` source directory. It consists of the following components:
- **`Listener`**: Listens for GitHub `push` events and posts them as `work` messages to the `workq` PubSub.
- **`Dispatcher`**: Receives `work` messages from the `workq` PubSub and a free instance `name` from the `Builder Pool`. It instantiates a `builder` instance named `name` in the Compute Engine environment and passes it the link of a repository to build.
- **`Collector`**: Receives `done` messages from the `doneq` PubSub and posts the freed instance `name` back to the `Builder Pool`.
- **`PubSub Topics`**:
- **`workq`**: Transports `work` messages that contain the link of the repository to build.
- **`poolq`**: Implements the `Builder Pool`, which contains the `name`'s of available `builder` instances. To acquire a `builder` name, pull a message from the `poolq`. To release a `builder` name, post it back into the `poolq`.
- **`doneq`**: Transports `done` messages (`builder` instance terminate and delete events). These message contain the `name` of freed `builder` instances.
- **`builder`**: A `builder` is a Compute Engine instance that performs a build of a repository and shuts down when the build is complete. A `builder` is instantiated from a VM `image` and a `startx` (startup-exit) script.
- **`Build Logs`**: A Storage bucket that contains the logs of builds performed by `builder` instances.
- **`Logging Sink`**: A `Logging Sink` captures `builder` instance terminate and delete events and posts them into the `doneq`.
A structural view of the system is presented below:
A behavioral view of the system follows:
## DEPLOYMENT
Prerequisites:
- Empty project in Google Cloud.
- Google Cloud SDK installed.
- `gcloud init` command has been run.
Instructions:
- Obtain a `SECRET` that will guard access to your PMCI deployment.
```
$ openssl rand -hex 16
SECRET
```
- Deploy PMCI to your project:
```
$ ./pmci deploy SECRET
```
- Obtain your personal access `TOKEN` by visiting github.com > Account > Settings > Developer settings > Personal access tokens.
- On every project you want to use PMCI go to github.com > Project > Settings > Webhooks > Add Webhook.
- URL: `https://REGION-PROJECT.cloudfunctions.net/listener?secret=SECRET&image=IMAGE&token=TOKEN`
- Set `REGION` and `PROJECT` accordingly.
- Set `IMAGE` to `freebsd` or `netbsd`.
- Content-type: `application/json`
- "Just the `push` event."
- Add a shell script named `.pmci/IMAGE.sh` (where `IMAGE` is `freebsd` or `netbsd`) to your project. This script will be run by PMCI on every push. For example, here is my [cgofuse](https://github.com/billziss-gh/cgofuse) script for FreeBSD:
```shell
set -ex
# FUSE
kldload fuse
pkg install -y fusefs-libs
# cgofuse: build and test
export GOPATH=/tmp/go
mkdir -p /tmp/go/src/github.com/billziss-gh
cp -R /tmp/repo/cgofuse /tmp/go/src/github.com/billziss-gh
cd /tmp/go/src/github.com/billziss-gh/cgofuse
go build ./examples/memfs
go build ./examples/passthrough
go test -v ./fuse
```
- You should now have working FreeBSD and/or NetBSD builds! Try pushing something into your GitHub project.
- To undeploy PMCI:
```
$ ./pmci undeploy
```
**NOTE**: The default deployment uses a single builder instance of `f1-micro` with a 30GB HDD. This fits within the "Always Free" tier and is therefore free. However it is also extremely slow and can even run out of memory when compiling bigger projects (e.g. it runs out of memory 1 out of 5 times when compiling Go in June 2018). Here are some ways to improve the performance:
- Use a machine type that is faster and has more memory, such as `n1-standard-1`.
- Use a larger HDD or an SSD.
- FreeBSD only: Use a custom image that has already performed `firstboot`. The default FreeBSD image `freebsd-11-1-release-amd64` performs a system update and other expensive work when booted for the first time (i.e. the `/firstboot` file exists). An image that has already done this work boots much faster.
```
$ ./pmci freebsd_builder_create builder0
# wait until builder has fully booted; it will do so twice;
# when the login prompt is presented in the serial console proceed
$ ./pmci builder_stop builder0
$ ./pmci builder_image_create builder0 freebsd-builder
$ ./pmci builder_delete builder0
# now modify your controller/index.js file to point to your custom freebsd-builder image
$ ./pmci deploy SECRET
```
### BADGES
PMCI supports status badges that show the last status of your build. Use them as follows in Markdown:
Badge:
```markdown
```
Badge that links to the build log:
```markdown
[](http://storage.googleapis.com/PROJECT-logs/github.com/USER/REPO/IMAGE/build.html)
```
## BUGS
- The `Builder Pool` is currently implemented as a PubSub; messages in the PubSub contain the names of available `builder` instances. Unfortunately a PubSub retains its messages for a maximum of 7 days. It is therefore possible that messages will be discarded and that your PMCI deployment will suddenly find itself out of builder instances. If this happens you can reseed the `Builder Pool` by running the commands below. However this is a serious BUG that should be fixed. For a related discussion see https://tinyurl.com/ybkycuub.
```
$ ./pmci queue_post poolq builder0
# ./pmci queue_post poolq builder1
# ... repeat for as many builders as you want
```
- The `Dispatcher` is implemented as a Retry Background Cloud Function. It accepts `work` messages from the `workq` and attempts to pull a free `name` from the `poolq`. If that fails it returns an error, which instructs the infrastructure to retry. Because the infrastructure does not provide any retry controls, this currently happens immediately and the `Dispatcher` spins unproductively. This is currently mitigated by a "sleep" (`setTimeout`), but the Cloud Functions system still counts the Function as running and charges it accordingly. While this fits within the "Always Free" limits, it is something that should eventually be fixed (perhaps by the PubSub team). For a related discussion see https://tinyurl.com/yb2vbwfd.