{"id":15443790,"url":"https://github.com/petergtz/overbook","last_synced_at":"2025-03-28T08:12:18.915Z","repository":{"id":153336012,"uuid":"104682144","full_name":"petergtz/overbook","owner":"petergtz","description":"A utility to augment an existing Concourse pipeline with cross-cutting concerned tasks","archived":false,"fork":false,"pushed_at":"2018-11-20T10:34:45.000Z","size":12,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-02T08:42:21.219Z","etag":null,"topics":["concourse"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/petergtz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-09-24T22:58:22.000Z","updated_at":"2023-10-09T16:18:44.000Z","dependencies_parsed_at":null,"dependency_job_id":"8384e9b6-9758-4af5-90b7-e3f6e7af26dc","html_url":"https://github.com/petergtz/overbook","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/petergtz%2Foverbook","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/petergtz%2Foverbook/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/petergtz%2Foverbook/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/petergtz%2Foverbook/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/petergtz","download_url":"https://codeload.github.com/petergtz/overbook/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245991584,"owners_count":20706129,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["concourse"],"created_at":"2024-10-01T19:36:54.778Z","updated_at":"2025-03-28T08:12:18.909Z","avatar_url":"https://github.com/petergtz.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Overbook\n\n## Introduction\n\nHow do you add more activity to a [Concourse](https://concourse.ci)? By overbooking flights.\n\nOverbook is a utility to insert new tasks to an existing Concourse pipeline. The inputs made available to the new tasks are *all* in-resources consumed so far in a job. See \"Getting Started\" for more details.\n\n**Wait, what are you talking about?**\n\nOkay. In simple words, Overbook makes it feasible to systematically notify people on failed jobs in Concourse pipelines. Other approaches require you to copy this notification-on-failure setting all over the place in your pipelines.\n\n## Install\n\nThe following instructions assume that you have [Go](https://golang.org/dl/) and [Glide](https://github.com/Masterminds/glide#install) installed on your system and that you [set up a Go workspace properly](https://golang.org/doc/code.html#Workspaces).\n\n\n    mkdir -p $GOPATH/github.com/petergtz\n    cd $GOPATH/github.com/petergtz\n    git clone https://github.com/petergtz/overbook.git\n    cd overbook\n    glide install\n    go install github.com/petergtz/overbook/cmd/overbook\n    go install github.com/petergtz/overbook/cmd/render-task-template\n\n\n## Getting Started\n\n### Problem Statement\nSometimes it's necessary to run a task in every job of a pipeline, and it takes all resources as input that have been consumed by `get` resources.\n\nWhile it is possible to add this manually to the pipeline yml, it's a tedious task, involving a lot of copy and paste (with slight changes, because every job might consume a different set of `get` resources), and it litters the pipeline yml with a lot of noise. Furthermore, Concourse task ymls have a fixed set of inputs, which means that for every number of inputs we would need to write its own task yml.\n\nOverbook solves this problem by automating it.\n\n### What Overbook Does\nWhen Overbook runs, it inserts Concourse tasks into jobs. It does this precisely before *every existing task*, with the exception defined below. The inserted task takes all resources consumed so far by the job as input.\n\nThe exception to the mechanism explained below, is that Overbook will not insert a task when the set of resources hasn't changed since the last inserted task.\n\n## Example Use Cases\n\nIt's not obvious how Overbook can help to solve real-world problems. Before going into such a real-world problem in [How to Slack a Committer of a Faulty Commit](#how_to_slack_a_committer_of_a_faulty_commit), let's look into a simpler use case to get a feel for how Overbook works.\n\n### Hello World\n\n*TBD*\n\n### How to Slack a Committer of a Faulty Commit\n\n#### Problem\n\nOften, in Concourse pipelines, you can see an `on_failure` block in all jobs with the following content:\n\n```yaml\n  put: slack\n  params:\n    text: |\n      The Concourse pipeline broke. See:\n      $ATC_EXTERNAL_URL/teams/$BUILD_TEAM_NAME/pipelines/$BUILD_PIPELINE_NAME/jobs/$BUILD_JOB_NAME/builds/$BUILD_NAME\n```\n\nThis notifies the owners of the pipeline that something went wrong.\n\nThe problem with this approach is that it doesn't address the actual person who has caused the breakage.\n\nInstead, let's say every time a Concourse build fails, we'd like to notify **exactly** the person responsible for the failure, so that unnecessary communication between members of the team can be avoided, and a fix can be pushed as quickly as possible.\n\nOne way to do that is to use Slack's _name tags_ so that the person gets notified directly.\n\nSo, roughly what we want to do instead, is:\n\n```yaml\n  put: slack\n  params:\n    text: |\n      $TEXT_FILE_CONTENT The Concourse pipeline broke. See:\n      $ATC_EXTERNAL_URL/teams/$BUILD_TEAM_NAME/pipelines/$BUILD_PIPELINE_NAME/jobs/$BUILD_JOB_NAME/builds/$BUILD_NAME\n    text_file: points-of-contact/slack-users-single-line\n```\n\nWhere `points-of-contact/slack-users-single-line` contains the Slack users who have potentially caused the broken build. But where does this `points-of-contact/slack-users-single-line` come from?\n\nThis is where Overbook comes into play.\n\n#### Implementing the Use Case\n\n##### Creating the Committer Aggregation Task\nCreate a file `aggregate-committers-for-notification.yml.overbook-template` (name can be choosen freely, extension is important) with the following content:\n\n```yaml\nplatform: linux\n\nimage_resource: { type: docker-image, source: { repository: my/docker-repo } }\n\ninputs:\n- name: ci\n$INPUTS\n\noutputs:\n  - name: points-of-contact\n\nrun:\n  path: ci/scripts/aggregate-committers-for-notification.sh\n```\n\nThen,\n\n```sh\nrender-task-template tasks/aggregate-committers-for-notification.yml.overbook-template\n```\n\ngenerates a task config in different versions where `$INPUTS` gets rendered into different numbers of inputs, e.g. in `aggregate-committers-for-notification3.yml` into:\n```yaml\ninputs:\n- name: ci\n\n- name: input0\n- name: input1\n- name: input2\n```\n\nWith that, we can now write `ci/scripts/aggregate-committers-for-notification.sh`. E.g.:\n```bash\n#! /bin/bash -ex\n\ntouch points-of-contact/committers\n\n# Copy all committers from all git input resources into one file:\nfor input in input*; do\n    pushd $input\n        if [ ! -e .git ]; then\n            popd\n            continue\n        fi\n        now=$(date +%s)\n        commit_date=$(git show -s --format=%ct)\n        time_since_commit=$((now-commit_date))\n        # This is really just a heuristic:\n        # commits older than 3 days are not responsible for a broken build\n        if [ \"$time_since_commit\" -gt \"259200\" ]; then # 3 days = 3*24*60*60 seconds\n            popd\n            continue\n        fi\n        echo $(git show -s --format=%ce) \u003e\u003e ../points-of-contact/committers\n    popd\ndone\n\n# for all users do something like this:\nsed -e s/my.email@gmail.com/my-slack-user/gI -i points-of-contact/slack-users\n\n# embed slack users in \u003c@...\u003e:\nawk '{print \"\u003c@\" $0 \"\u003e\"}' points-of-contact/slack-users \u003e points-of-contact/slack-users-with-at\nmv points-of-contact/slack-users-with-at points-of-contact/slack-users\n\n# Make slack users available in a single line:\ntr '\\n' ' ' \u003c points-of-contact/slack-users \u003e points-of-contact/slack-users-single-line\ncat points-of-contact/slack-users-single-line\n```\n\nNote that the line `for input in input*; do` now automatically takes all inputs made available through the task config.\n\n##### Using the Committer Aggregation Task\n\nWith the task from above at hand, we can now augment our pipeline, by running:\n\n```bash\noverbook --config pipeline.yml --task-path ci-tasks/tasks/generated/aggregate-committers-for-notification --resource ci=ci-tasks \u003e pipeline-overbooked.yml\n```\nwhere `--resource` simply lets us specify additional input-mappings.\n\n`pipeline-overbooked.yml` now has additional tasks which simply make sure all potential blame candidates are available whenever an `on_failure` gets triggered. You can see those tasks in your next `fly set-pipeline`. The only remaining thing to add to our `pipeline.yml` is:\n\n```yaml\nnotify: \u0026notify\n  put: slack\n  params:\n    text: |\n      $TEXT_FILE_CONTENT The Concourse pipeline broke. See:\n      $ATC_EXTERNAL_URL/teams/$BUILD_TEAM_NAME/pipelines/$BUILD_PIPELINE_NAME/jobs/$BUILD_JOB_NAME/builds/$BUILD_NAME\n    text_file: points-of-contact/slack-users-single-line\n```\n\nAnd on every job a simple:\n\n```yaml\n  on_failure: *notify\n```\n\nThat's it!\n\n## Conclusion\n\nThe setup seems pretty complex at first, but there two things to keep in mind:\n\n* Changes to `pipeline.yml` are minimal, and hence the yaml file is not convoluted with \"error handling\".\n* Concoourse does not currently provide a built-in feature to make committers available to a task step.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpetergtz%2Foverbook","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpetergtz%2Foverbook","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpetergtz%2Foverbook/lists"}