{"id":23070100,"url":"https://github.com/finbourne/uav","last_synced_at":"2025-08-15T13:32:39.061Z","repository":{"id":42013585,"uuid":"150438302","full_name":"finbourne/uav","owner":"finbourne","description":"Unmanned Air Vehicle - the self-flying machine.","archived":false,"fork":false,"pushed_at":"2024-06-24T13:20:36.000Z","size":276,"stargazers_count":4,"open_issues_count":9,"forks_count":4,"subscribers_count":5,"default_branch":"master","last_synced_at":"2024-06-24T14:56:23.558Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/finbourne.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-09-26T14:18:58.000Z","updated_at":"2024-06-24T14:56:43.888Z","dependencies_parsed_at":"2024-06-24T14:56:28.713Z","dependency_job_id":null,"html_url":"https://github.com/finbourne/uav","commit_stats":null,"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/finbourne%2Fuav","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/finbourne%2Fuav/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/finbourne%2Fuav/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/finbourne%2Fuav/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/finbourne","download_url":"https://codeload.github.com/finbourne/uav/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":229916223,"owners_count":18144111,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-16T06:19:25.717Z","updated_at":"2024-12-16T06:19:26.261Z","avatar_url":"https://github.com/finbourne.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"![LUSID_by_Finbourne](./resources/Finbourne_Logo_Teal.svg)\n\n# UAV \n\n![release-tag](https://github.com/finbourne/uav/workflows/release-tag/badge.svg)\n![release-build](https://github.com/finbourne/uav/workflows/release-build/badge.svg)\n![Go-build-status](https://github.com/finbourne/uav/workflows/Go-build-status/badge.svg)\n\n## Example usage\nThe following command will take the pipeline defined in `my.pipeline.yaml` and output the result to `stdout`.  If you wanted to output to a file, redirect the output or use the `-o` flag.\n\n`uav merge -p my.pipeline.yaml`\n\n```yaml\njobs:\n- name: deploy-ci\n  plan:\n  - config:\n      image_resource:\n        source:\n          repository: test/docker-container\n        type: docker-image\n      platform: linux\n      run:\n        args:\n        - -cel\n        - |\n          echo Hello ci!\n        path: /bin/bash\n    task: task1\n  serial: true\n- name: deploy-qa\n  plan:\n  - config:\n      image_resource:\n        source:\n          repository: test/docker-container\n        type: docker-image\n      platform: linux\n      run:\n        args:\n        - -cel\n        - |\n          echo Hello qa!\n        path: /bin/bash\n    task: task1\n  serial: true\n\n```\n`my.pipeline.yaml`\n```yaml\nmerge:\n- template: jobs/test.yaml\n  args:\n    envs:\n    -\n      env: ci\n    -\n      env: qa\n```\n`jobs/test.yaml`\n```yaml\njobs:\n{{ range .envs }}\n\n- name: deploy-{{ .env }}\n  serial: true\n  plan:\n  - task: task1\n    config:\n      platform: linux\n\n      image_resource:\n        type: docker-image\n        source:\n          repository: test/docker-container\n      run:\n        path: /bin/bash\n        args:\n        - -cel\n        - |\n          echo Hello {{ .env }}!\n\n{{- end }}\n```\nTo explain all that:\n* One new pipeline construct has been added; `merge`.  \nThis will be evaluated recursively, so you can add a `merge` that references your resources from each of your pipelines.\n* Arguments can be passed to the templates in `merge` via the `args` map.  \n`args` is a yaml map.  In the example above, it has one entry `envs` which is itself an array of maps.\n* All the power of golang text/template is at your fingertips.  \nSo things like loops `{{ range }}` and if's `{{ if }}` are available to use in the pipelines.  \nSee https://golang.org/pkg/text/template/ for further information.\n* It is worth noting that the yaml that goes in may not resemble the yaml that comes out.  This is because yaml maps aren't order specific, so when processed, end up being printed out in alphabetical order.\n* Note that the pipeline is located in the current directory `.` and it references files in a subdirectory.\n\n## Another example\n`my.pipeline.yaml`\n```yaml\nmerge:\n- template: jobs/test.yaml\n  args:\n    env: qa\n    repo_master: github\n```\n`jobs/test.yaml`\n```yaml\njobs:\n- name: deploy-{{ .env }}\n  serial: true\n  plan:\n  - get: repo\n  - task: task1\n    config:\n      platform: linux\n\n      image_resource:\n        type: docker-image\n        source:\n          repository: test/docker-container\n      run:\n        path: /bin/bash\n        args:\n        - -cel\n        - |\n          cd repo\n          echo Hello {{ .env }}!\nmerge:\n- template: resources/repo.yaml\n  args:\n    repo_master: {{ .repo_master }}\n```\n`resources/repo.yaml`\n```yaml\nresources:\n- name: test\n  type: git\n  source:\n    uri: git@{{ .repo_master }}.com:concourse/concourse.git\n    branch: master\n    private_key: ((github.privatekey))\n```\n`uav merge -p my.pipeline.yaml`\n\nAnd the output from that should be:\n```yaml\njobs:\n- name: deploy-qa\n  serial: true\n  plan:\n  - get: repo\n  - task: task1\n    config:\n      platform: linux\n\n      image_resource:\n        type: docker-image\n        source:\n          repository: test/docker-container\n      run:\n        path: /bin/bash\n        args:\n        - -cel\n        - |\n          cd repo\n          echo Hello qa!\nresources:\n- name: test\n  type: git\n  source:\n    uri: git@github.com:concourse/concourse.git\n    branch: master\n    private_key: ((github.privatekey))\n```\n\n* Subsequent merges still use the working directory as the source for templates.  In this case, the template for the resource was not in `jobs/resources/`, but instead was in `resources/`\n\n# Templates\nAs well as the 'top-level' Concourse pipeline objects specified by the `merge` clause, snippets may be provided as Go templates. These are imported into the template using the `include` function:\n\n`include \"\u003ctemplate name\u003e\" [\u003carg1\u003e...]`\n\nPer standard Go templating rules, the `template name` may either be a named template (if multiple templates reside in one file) or the basename of a file containing a single unnamed template. As the basename of the file is used, it is therefore not possible to have multiple template files with the same name, even if they reside in different directories.\n\nUAV provides two mechanisms for making these templates available for inclusion into other templates:\n* Directories containing templates or nested subdirectories containing templates may be provided using the `--directory` or `-d` flag:\n`--directory \u003cdir1\u003e [\u003cdir2\u003e...]`\n* Individual template file(s) may be provided as arguments.\n\n# Template Functions\nIn addition to the standard functions from the Go text/template package, the functions from the Sprig library (http://masterminds.github.io/sprig/) are available.\n\nUAV also provides a few functions of its own:\n* `indentSub n \"text\"` - where `text` is some text to be indented (often piped from an `include`) and `n` is the number of spaces all lines but the first in the `text` will be indented by. This is useful for including snippets at the correct level of indentation to ensure the result is valid YAML.\n* `toYaml object` - marshall an arbitrary object (Go `struct`, `map` or `slice`) into YAML.\n* `fromYaml` - unmarshall YAML into a Go `map[string]interface{}` (a map of string keys to arbitrary objects).\n* `toJson` - marshall an arbitrary object (Go `struct`, `map` or `slice`) into JSON.\n* `fromJson` - unmarshall JSON into a Go `map[string]interface{}` (a map of string keys to arbitrary objects).\n* `skipLines n \"text\"` - where `text` is some text (often piped from another function) and `n` is the number of lines from the input to skip in the output.\n\n# Example Project Layout\n\nA typical project layout showing how UAV is used at [Finbourne](https://www.finbourne.com):\n\n```\n├── pipelines\n│   ├── build\n│   │   └── components.tpl\n│   ├── deploy\n│   │   └── components.tpl\n│   └── verify\n│       └── verify.tpl\n├── resource_types\n│   ├── semver.tpl\n│   └── slack-notification.tpl\n├── resources\n│   ├── git\n│   │   └── pipelines.tpl\n│   ├── semver\n│   │   └── versions.tpl\n│   └── slack\n│       └── slack-alert.tpl\n├── pipeline.tpl\n└── templates\n    └── on_failure.tpl\n```\n\nThis UAV project is \"built\" using the following invocation:\n\n`uav merge --pipeline telemetry.pipeline.tpl --directory templates \u003e .pipeline.yml`\n\n* The parent directory contains the file `pipeline.ypl` which contains a single `merge` containing an array of `template`s:\n\n```YAML\n{{$COMPONENTS := `[\"prometheus\", \"grafana\"]`}}\nmerge:\n  - template: pipelines/build/components.tpl\n    args:\n      components: {{$COMPONENTS}}\n  - template: pipelines/verify/verify.tpl\n    args:\n      envs:\n        - name: CI\n          dependencies:\n            - build-telemetry-deployables\n        - name: QA\n          dependent_on_env: CI\n        - name: PROD\n          dependent_on_env: QA        \n      components: {{$COMPONENTS}}\n  - template: pipelines/deploy/components.tpl\n    args:\n      envs:\n        - name: CI\n        - name: QA\n        - name: PROD\n      components: {{$COMPONENTS}}  \n```\n\n* The `pipelines/build/components.tpl` file contains the Concourse `job` which builds the component deployables and uploads to cloud storage:\n\n```YAML\nmerge:\n  - template: resources/semver/versions.tpl\n    args:\n      semvers:\n        - name: telemetry-version\n          file: telemetry.version  \n  - template: resources/git/pipelines.tpl\n\n  # Used by the \"on_failure.tpl\" include\n  - template: resources/slack/slack-alert.tpl\njobs:\n  - name: build-telemetry-deployables\n    plan:\n    - aggregate:\n      - get: pipelines\n      - get: telemetry-version\n        trigger: true\n    - aggregate:\n      {{range .components}}\n      - task: Create {{. | title}} deploy zip\n        config:\n          platform: linux\n          image_resource:\n            type: docker-image\n            source:\n              repository: ...          \n          inputs:\n            - name: pipelines\n            - name: telemetry-version\n          params:\n            component: {{.}}\n          run:\n            ...\n      {{end}}\n\n    {{include \"on_failure.tpl\" | indentSub 4}}\n```\n\n* The `resources/semver/versions.tpl` file contains the Concourse `resource` definition for a `semver` resource:\n\n```YAML\nmerge:\n  - template: resource_types/semver.tpl\n\nresources:\n  {{range .semvers}}\n  - name: {{.name}}\n    type: semver    \n    source:\n      driver: git\n      branch: master\n      uri: ...\n      file: {{.file}}\n      private_key: ((gitlab-private-key))\n  {{end}}\n```\n\n* The `resource_types/semver.tpl` file contains the Concourse `resource_type` definition for a custom `semver` resource (this would not be required when using the built-in resource of the same name):\n\n```YAML\nresource_types:\n- name: semver  \n  type: docker-image\n  source:\n    repository: ...\n```\n\n* The `resources/git/pipelines.tpl` file contains the Concourse `resource` definition for a `git` resource:\n\n```YAML\nresources:\n- name: pipelines\n  type: git  \n  source:\n    uri: ...\n    branch: master\n    private_key: ((gitlab-private-key))\n```\n\n* The `resources/slack/slack-alert.tpl` file contains the Concourse `resource` definition for a `slack-notification` resource:\n\n```YAML\nmerge:\n  - template: resource_types/slack-notification.tpl\n\nresources:\n  - name: slack-alert    \n    type: slack-notification\n    source:\n      url: ((slack.url))\n```\n\n* The `resource_types/slack-notification.tpl` file contains the Concourse `resource_type` definition for a third-party `slack-notification` resource:\n\n```yaml\nresource_types:\n- name: slack-notification\n  type: docker-image\n  source:\n    repository: cfcommunity/slack-notification-resource\n    platform: latest\n```\n\n* The `on_failure.tpl` template is contained in file `templates/on_failure.tpl` (see discussion of Go template naming conventions above) and contains a snippet of Concourse `job` configuration - specifically, the `on_failure` clause. The same boilerplate is used for every job in the pipeline, hence it is moved into its own file:\n\n```YAML\non_failure:\n  put: slack-alert\n  params:\n    channel: '#build'\n    icon_emoji: ':warning:'\n    text: |\n      HEY \u003c!channel\u003e! Something went wrong with the build ($BUILD_PIPELINE_NAME/$BUILD_JOB_NAME).      \n```\nThe key thing to notice here is that a \"snippet\" (i.e. some re-usable boilerplate which is not a \"top-level\" Concourse type) is `include`d into a larger template rather than `merge`d. This necessitates the \"forward-declaring\" of the `resource` named `slack-alert` in the template files which `include` the `on_failure.tpl` template.\n\n* The `pipelines/verify/verify.tpl` file contains the Concourse `job` which builds the verifies the environments are ready for the installation of the deployables:\n\n```YAML\nmerge:  \n  - template: resources/semver/versions.tpl\n    args:\n      semvers:\n        - name: telemetry-version\n          file: telemetry.version\n  - template: resources/git/pipelines.tpl\n\n  # Used by the \"on_failure.tpl\" include\n  - template: resources/slack/slack-alert.tpl  \njobs:\n{{range .envs}}\n{{$env := .}}\n  - name: verify environment [{{$env.name}}]\n    plan:\n      - aggregate:\n        - get: pipelines    \n        - get: telemetry-version\n          passed:\n          {{if $env.dependencies}}\n          {{range $env.dependencies}}          \n          - {{.}}\n          {{end}}\n          {{else}}          \n          {{range $.components}}\n          - {{printf \"telemetry-%s-deploy [%s]\" . $env.dependent_on_env}}\n          {{end}}\n          {{end}}\n          trigger: true\n      - aggregate:\n        - task: Run verifications\n          config:\n            platform: linux\n            image_resource:\n              type: docker-image\n              source:\n                repository: ...                \n                username: ((docker-username))\n                password: ((docker-password))\n            run:\n              path: /var/app/start.sh\n              dir: /var/app\n\n    {{include \"on_failure.tpl\" | indentSub 4}}\n{{end}}\n```\n\nNotice that all `resource`s are declared again, even though they were declared previously in the `pipelines/build/components.tpl` file. This does not cause an issue in the final YAML which is created as UAV will merge repeated identical definitions. The advantage of this approach is that it declares the resources needed by a job \"locally\", allowing larger pipelines to be composed from individual job templates - each job will \"bundle\" its dependencies with it.\n\n* The `pipelines/deploy/components.tpl` file contains the Concourse `job` which installs the deployables onto the environments:\n\n```YAML\nmerge:  \n  - template: resources/semver/versions.tpl\n    args:\n      semvers:\n        - name: telemetry-version\n          file: telemetry.version  \n  - template: resources/git/pipelines.tpl\n\n  # Used by the \"on_failure.tpl\" include\n  - template: resources/slack/slack-alert.tpl\njobs:\n  {{range .envs}}\n  {{$env := .}}\n  {{range $.components}}\n  - name: \"telemetry-{{.}}-deploy [{{$env.name}}]\"\n    plan:\n      - aggregate:\n        - get: pipelines\n        - get: telemetry-version\n          trigger: true\n          passed:\n            - \"verify environment [{{$env.name}}]\"      \n      - task: Deploy    \n        config:\n          platform: linux\n          image_resource:\n            type: docker-image\n            source:\n              repository: ...\n          inputs:\n            - name: telemetry-version\n            - name: pipelines\n          params:            \n            component: {{.}}\n          run:\n            path: /bin/bash\n            args: ...\n      - task: Check application health\n        file: pipelines/tasks/...\n        params:        \n          envName: \"telemetry-{{.}}\"        \n\n    {{include \"on_failure.tpl\" | indentSub 4}}\n{{end}}\n{{end}}\n```\n\n# Restrictions \u0026 Known Bugs\n* There is currently no way of using the templating engine's `template` function.  Arguments would need to be added to pull in additional named template in order to fulfil this condition were it to become a requirement.\n* I'm sure there are quirks.  Find them, and we can fix them.\n* There are occasions that the ranging doesn't work.  Haven't figured out why.  It's deterministic, it's just my lack of understanding around something there.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffinbourne%2Fuav","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffinbourne%2Fuav","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffinbourne%2Fuav/lists"}