{"id":18045307,"url":"https://github.com/dpb587/bosh-release-blobs-upgrader-pipeline","last_synced_at":"2026-08-03T10:31:05.203Z","repository":{"id":138803729,"uuid":"115455299","full_name":"dpb587/bosh-release-blobs-upgrader-pipeline","owner":"dpb587","description":"For managing a BOSH release pipeline which upgrades blobs from external resources.","archived":false,"fork":false,"pushed_at":"2019-11-26T16:01:36.000Z","size":221,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-04-15T14:20:49.227Z","etag":null,"topics":["bosh","concourse","version-update"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dpb587.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-12-26T21:14:13.000Z","updated_at":"2024-06-19T11:05:36.369Z","dependencies_parsed_at":null,"dependency_job_id":"a54f5cc0-210f-4201-bf19-80b4c527c8ae","html_url":"https://github.com/dpb587/bosh-release-blobs-upgrader-pipeline","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/dpb587/bosh-release-blobs-upgrader-pipeline","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpb587%2Fbosh-release-blobs-upgrader-pipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpb587%2Fbosh-release-blobs-upgrader-pipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpb587%2Fbosh-release-blobs-upgrader-pipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpb587%2Fbosh-release-blobs-upgrader-pipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dpb587","download_url":"https://codeload.github.com/dpb587/bosh-release-blobs-upgrader-pipeline/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpb587%2Fbosh-release-blobs-upgrader-pipeline/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279076614,"owners_count":26098218,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-15T02:00:07.814Z","response_time":56,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bosh","concourse","version-update"],"created_at":"2024-10-30T18:12:51.859Z","updated_at":"2025-10-15T11:22:55.540Z","avatar_url":"https://github.com/dpb587.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# bosh-release-blobs-upgrader-pipeline\n\nFor managing a [BOSH](https://bosh.io) release pipeline which upgrades blobs from external resources.\n\nDeclare your dependencies as Concourse resources to track versions and ensure they can be downloaded; then configure tests to run after upgrading a blob and before pushing or submitting a pull request with the upgrades.\n\n\n## Usage\n\nThe `config/blobs` directory of a release repository is used to record and track sourcing information of dependencies. Each subdirectory defines a dependency for tracking an external, versioned resource through a Concourse [resource](https://concourse-ci.org/resources.html) definition in the `resource.yml` file. For example, a dependency on [PHP 7](https://php.net/) might have a file `config/blobs/php7/resource.yml` which looks like...\n\n```yaml\ntype: dynamic-metalink\nsource:\n  version_check: |\n    curl -s https://golang.org/dl/?mode=json | jq -r '.[].version[2:]'\n  metalink_get: |\n    curl -s https://golang.org/dl/?mode=json | jq '\n      map(select(.version[2:] == env.version)) | map({\n        \"files\": (.files | map({\n          \"name\": .filename,\n          \"size\": .size,\n          \"urls\": [ { \"url\": \"https://dl.google.com/go/\\(.filename)\" } ],\n          \"hashes\": [ { \"type\": \"sha-256\", \"hash\": .sha256 } ] } ) ) } )[]'\n  include_files:\n  - go*.linux-amd64.tar.gz\n```\n\nEach dependency will be placed into its own directory under `blobs` (since a dependency may have multiple files). When a new version is available, all existing files in its blobs directory will be removed and replaced with the new dependency blobs. Additionally, resource metadata can be committed as well to help document the provenance of the blob. In this example, `config/blobs.yml` would eventually have an entry for `php7/php-7.2.8.tar.gz` and `config/blobs/php7/metalink.meta4` would document where exactly it was downloaded from.\n\nA pipeline defines the basic configuration (e.g. `ci/pipelines/upgrader.yml`) which includes, at a minimum, a `repo` resource which references the release repository and some additional upgrade steps in a `bosh_release_blobs_upgrader` section.\n\n```yaml\nbosh_release_blobs_upgrader:\n  # run integration tests with a dev release of the new blob to make sure it works\n  before_upload:\n    do:\n    - task: create-dev-release\n      file: repo/ci/tasks/create-dev-release/task.yml\n    - task: integration-test\n      file: repo/ci/tasks/integration-test/task.yml\n      privileged: true\n  # push the new blobs to the repo\n  after_upload:\n    put: repo\n    params:\n      repository: repo\n  # notify slack/email to ensure someone manually figures out upgrading\n  on_failure:\n    task: notify-upgrade-failure\n    file: repo/ci/tasks/notify-upgrade-failure/task.yml\n  # track details about where the blobs came from\n  track_files:\n  - .resource/metalink.meta4\nresources:\n# the bosh release repository\n- name: repo\n  type: git\n  source:\n    uri: git@github.com:dpb587/ssoca-bosh-release.git\n    branch: master\n    private_key: ((repo_private_key))\nresource_types:\n# used by individual resource.yml definitions\n- name: dynamic-metalink\n  type: docker-image\n  source:\n    repository: dpb587/dynamic-metalink-resource\n```\n\nOnce the blobs and base pipeline have been configured, `bosh-release-blobs-upgrader-pipeline` can be used to generate the pipeline for passing to `fly`. The added blob upgrade jobs will require several variables to be set: `release_private_yml` (for uploading blobs to the blobstore) and `maintainer_email`, `maintainer_name` (for the `git` commit).\n\n```bash\n$ fly set-pipeline -p my-release:upgrader \\\n  -c \u003c( bosh-release-blobs-upgrader-pipeline ci/pipelines/upgrader.yml ) \\\n  -l \u003c( lpass show 'pipeline=my-release:upgrader' )\n```\n\nThe blob jobs automatically trigger whenever a new version is available. When syncing blobs, old blobs in the directory are removed, new blobs are added (not yet uploaded), and a commit message referencing the specific blob and new version are prepared. When uploading blobs, `config/blobs.yml` is updated with the new blobstore references, and any other already-staged files are committed. After uploading blobs, the repository should be pushed.\n\n\n## Examples\n\nA couple repositories using this to manage some upstream dependencies...\n\n* [dpb587/openvpn-bosh-release](https://github.com/dpb587/openvpn-bosh-release/blob/master/ci/pipelines/upgrader.yml)\n* [dpb587/ssoca-bosh-release](https://github.com/dpb587/ssoca-bosh-release)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdpb587%2Fbosh-release-blobs-upgrader-pipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdpb587%2Fbosh-release-blobs-upgrader-pipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdpb587%2Fbosh-release-blobs-upgrader-pipeline/lists"}