Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/knl/pulley

A service to expose Prometheus metrics of your CI’s validation of Pull Requests, using GitHub webhooks.
https://github.com/knl/pulley

github-webhooks golang prometheus service

Last synced: 13 days ago
JSON representation

A service to expose Prometheus metrics of your CI’s validation of Pull Requests, using GitHub webhooks.

Host: GitHub
URL: https://github.com/knl/pulley
Owner: knl
License: bsd-3-clause
Created: 2020-01-28T09:23:20.000Z (almost 5 years ago)
Default Branch: master
Last Pushed: 2023-02-15T06:13:41.000Z (almost 2 years ago)
Last Synced: 2024-10-12T00:31:12.017Z (about 1 month ago)
Topics: github-webhooks, golang, prometheus, service
Language: Go
Homepage:
Size: 68.4 KB
Stars: 4
Watchers: 3
Forks: 0
Open Issues: 0
Metadata Files:
- Readme: README.adoc
- License: LICENSE

Awesome Lists containing this project

README

= Pulley

A service to expose Prometheus metrics of your CI's validation of Pull Requests
(PRs), using GitHub webhooks.

image:https://github.com/knl/pulley/workflows/CI/badge.svg[CI]
image:https://github.com/knl/pulley/workflows/goreleaser/badge.svg[release]
image:https://goreportcard.com/badge/github.com/knl/pulley[link=https://goreportcard.com/report/github.com/knl/pulley, alt=Go Report Card]
image:https://codebeat.co/badges/56655dd4-44a7-45fd-9d33-0654b5a90452[link=https://codebeat.co/projects/github-com-knl-pulley-master]

== Why

The best way to have service level objectives (SLOs) in place is to measure the
same way the other party is observing. Sometimes your CI is providing you
metrics, but they are not sufficient or do not match the reality. Pulley uses
GitHub's webhooks to produce metrics as observed by your developers. That way,
you can optimize your CI's pipelines in a way it matters.

=== Features

Pulley assumes that a PR is mergeable when a required status check is
successful. Thus, it lets you configure which status checks to monitor, and only
treat the PR as validated (mergeable) when that status check is green. Just
exactly like how developers (or even other automation) observe and act on PRs.

The basic metrics Pulley exposes are (all times are histograms):

- The time it takes for the CI to send the first `pending` status for a PR
- The time it takes for the CI to send the `success`/`failure`/`error` for the
required status check, from the time PR has been open
- The time it takes for a PR to be merged since it got open
- The build duration on the CI, per build
- How many PRs have been open/closed
- How many times branches have been rebased
- The total number of status checks received
- The total number of `success`/`failure`/`error` status checks received,
without a preceding `pending` status check

Pulley can track any repository on GitHub, as long as that repository is
configured to send https://developer.github.com/webhooks/[webhook events] to it.

== Usage

Pulley is a service that should run listening on a public IP (as an endpoint is
needed to be accessible by GitHub's servers). It is completely configurable via
environment variables.

=== Configuration

Pulley understands the following environment variables:

|===
| Environment Variable | Description

| PULLEY_HOST
| The hostname to which Pulley will bind. Defaults to `localhost`.

| PULLEY_PORT
| The port number to which Pulley will bind. Defaults to `1701`.

| PULLEY_WEBHOOK_PATH
| URL path on which Pulley receives webhooks. Defaults to an empty string,
meaning that webhook events are handled on the root path (that is,
`http://$PULLEY_HOST:$PULLEY_PORT/`).

| PULLEY_WEBHOOK_TOKEN
| A **base64** encoded string representing a secret token, used to validate the
events coming from GitHub. Defaults to an empty string. More details on
https://developer.github.com/webhooks/securing/.

| PULLEY_METRICS_PATH
| URL path on which Pulley exposes Prometheus metrics. Defaults to `metrics`.

| PULLEY_PR_TIMING_STRATEGY
| Which strategy Pulley should use to time the PRs. That is, how to detect when
PR building started and ended. Currently, the only available one is `regex`.
If missing or unset, defaults to `regex`.

| PULLEY_STRATEGY_AGGREGATE_REPO_REGEX_
| Set of regular expressions defining contexts to monitor for matching
repository names. Defaults to regex `:all-jobs$` matching all repositories
(`.*`). For more details, consult the next section.

| PULLEY_STRATEGY_AGGREGATE_CONTEXT_REGEX_
| See above.

| PULLEY_TRACK_BUILD_TIMES
| If true, Pulley will track and export build times for each build (that is,
different context it sees). Accepted range of values is `1`, `t`, `T`, `TRUE`,
`true`, `True`, `0`, `f`, `F`, `FALSE`, `false`, `False`.
|===

==== PR Timing Strategies

A PR timing strategy is how Pulley determines when the CI started building the
PR, and when CI finished. It deals with received status checks coming from
GitHub.

At the moment, Pulley supports only a single PR timing strategy, `aggregate`.

===== The aggregate strategy

This strategy assumes that each repository has one _aggregate_ job defined, that
finishes when all other jobs are done and emits a corresponding status check.
This job determines if the PR is to be merged. Another use of this strategy is
for the case when there is a single required status check defined in a repo.

To provide versatility in configuration, while keeping it simple and
configurable via the environment variables, Pulley resorts to using the regular
expressions (regexes) and the following scheme for the environment variables
when encoding that information:

PULLEY_STRATEGY_AGGREGATE_REPO_REGEX_ = $repo_name_regex
PULLEY_STRATEGY_AGGREGATE_CONTEXT_REGEX_ = $status_check_name_regex

That is, we mimic a prioritized list of regexes to match between the repository
name and the status check name.

The list of regexes is processed in order from the smallest number towards the
highest, looking first for the `_REPO` variant. If there is a match on the
repository name, but not on the status, the search will **NOT** continue.

For example, the following configuration:

PULLEY_STRATEGY_AGGREGATE_REPO_REGEX_0=-deployment$
PULLEY_STRATEGY_AGGREGATE_CONTEXT_REGEX_0=^terraform-validate
PULLEY_STRATEGY_AGGREGATE_REPO_REGEX_1=^knl/pulley$
PULLEY_STRATEGY_AGGREGATE_CONTEXT_REGEX_1=build
PULLEY_STRATEGY_AGGREGATE_REPO_REGEX_100=.*
PULLEY_STRATEGY_AGGREGATE_CONTEXT_REGEX_100=:all-jobs$

would check for status checks whose names begin with `terraform-validate` for
all repositories whose names end with `-deployment`. Then it would check for
`build` status check on repositories whose names begin with `knl/pulley`.
Finally, it would default (due to `.*` regex on the repository name) to status
check whose name ends with `:all-jobs` for all other repositories.

Several things are important to note here:

. Repository names are always in the form: `$OWNER/$REPOSITORY`, that is, they
represent the repository's full name.
. The first entry that matches the repository name is considered only.
. When there are multiple matching status check names for a repository, only the
first one that shows up will be considered.

== Run

Set the environment variables and run:

./pulley

The best is to place Pulley behind a reverse proxy (for example, Nginx) that
terminates HTTPS traffic.

== Requirements

Go version: `1.13`

== Development

To build the code, simply run:

make build

Similarly, the tests are executed via:

make test

Prior to committing the code, you could run

make

to properly format and lint the code

=== Managing releases

Releases are managed with https://goreleaser.com/[goreleaser].

To create a new release, push a tag (for example, a version 0.1.0):

git tag -a v0.1.0 -m "First release"
git push origin v0.1.0

To build a test release, without publishing, run:

make test-release