{"id":17204191,"url":"https://github.com/akshaymankar/dhall-concourse","last_synced_at":"2026-01-06T07:03:15.635Z","repository":{"id":52972829,"uuid":"159702076","full_name":"akshaymankar/dhall-concourse","owner":"akshaymankar","description":"Library to type check concourse configurations in dhall","archived":false,"fork":false,"pushed_at":"2021-07-08T14:41:37.000Z","size":214,"stargazers_count":20,"open_issues_count":1,"forks_count":2,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-01-30T09:12:13.331Z","etag":null,"topics":["concourse","dhall","dhall-lang"],"latest_commit_sha":null,"homepage":"","language":"Dhall","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/akshaymankar.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-11-29T17:17:24.000Z","updated_at":"2024-07-31T09:25:54.000Z","dependencies_parsed_at":"2022-09-04T07:12:55.645Z","dependency_job_id":null,"html_url":"https://github.com/akshaymankar/dhall-concourse","commit_stats":null,"previous_names":[],"tags_count":35,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/akshaymankar%2Fdhall-concourse","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/akshaymankar%2Fdhall-concourse/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/akshaymankar%2Fdhall-concourse/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/akshaymankar%2Fdhall-concourse/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/akshaymankar","download_url":"https://codeload.github.com/akshaymankar/dhall-concourse/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245437181,"owners_count":20615230,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["concourse","dhall","dhall-lang"],"created_at":"2024-10-15T02:20:54.475Z","updated_at":"2026-01-06T07:03:10.606Z","avatar_url":"https://github.com/akshaymankar.png","language":"Dhall","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Dhall Concourse\nConcourse types and helpers in dhall.\n\nDhall Concourse provides Dhall bindings for Concourse, so you can generate concourse pipelines from Dhall expressions. This lets the pipelines be easily typechecked, templated and modularized.\n\n## Why do I need this?\n\nThere are a lot of issues one could face while building any non-trivial pipeline. Few of them could be:\n1. Pipeline yaml becomes very big and unmanageable\n2. Same set of jobs are required to be run in different environments\n3. Same set of hooks but with slight changes in all jobs. E.g. slack notifications, releasing resources on failure, etc.\n\nMost common ways to deal with these have been to use a templating language like erb or tools like spruce. But this gets very messy very fast. We can do a lot better with a typed total language. I'll let [dhall speak for itself](https://dhall-lang.org/).\n\n## Usage\n\n### Using dhall-fly\n\nOne way to translate pipelines written in dhall-concourse into yaml is using [dhall-fly](https://github.com/akshaymankar/dhall-fly#installation).\n\n### Using dhall-to-json and jq\n\n#### Jobs without Groups\n\nTo use native rendering to render a list of jobs in a file called `jobs.dhall`, you'd have to write a dhall expression like this:\n\n```dhall\nlet Concourse = \n      https://raw.githubusercontent.com/akshaymankar/dhall-concourse/0.6.0/package.dhall\n\nlet jobs = ./jobs.dhall\n\nin Concourse.render.pipeline jobs\n```\n\nNow you can render this using dhall-to-json and jq like this:\n\n```bash\ndhall-to-json \u003c\u003c\u003c './pipeline.dhall' \\\n  | jq '.resources = (.resources|unique)' \\\n  | jq '.resource_types = (.resource_types|unique)'\n```\n\n#### Jobs with groups\n\nSimilarly, to render a list of `GroupedJob`s in a filed called `grouped-jobs.dhall`, this would be the expression to render:\n\n```dhall\nlet Concourse = \n      https://raw.githubusercontent.com/akshaymankar/dhall-concourse/0.6.0/package.dhall\n\nlet groupedJobs = ./grouped-jobs.dhall\n\nin Concourse.render.groupedJobs groupedJobs\n```\n\nNow you can render this using dhall-to-json and jq like this:\n\n```bash\ndhall-to-json \u003c\u003c\u003c './pipeline.dhall' \\\n  | jq '.resources = (.resources|unique)' \\\n  | jq '.resource_types = (.resource_types|unique)' \\\n  | jq '.groups = (.groups | group_by(.name) | map({name: .[0].name, jobs: (map(.jobs) | flatten) }))'\n```\n\n## Defining a pipeline\n\n### Example 1: Hello World\n\nThis dhall expression will create a pipeline with one job, which would have one task. The task would run in a busybox container and echo \"Hello Dhall\".\n\n```dhall\nlet Concourse =\n      https://raw.githubusercontent.com/akshaymankar/dhall-concourse/0.5.0/package.dhall\n\nlet Prelude =\n      https://prelude.dhall-lang.org/v11.1.0/package.dhall sha256:99462c205117931c0919f155a6046aec140c70fb8876d208c7c77027ab19c2fa\n\nlet busyboxImage =\n      Concourse.schemas.ImageResource::{\n      , type = \"docker-image\"\n      , source = Some (toMap { repository = Prelude.JSON.string \"busybox\" })\n      }\n\nlet job =\n      Concourse.schemas.Job::{\n      , name = \"hello\"\n      , plan =\n          [ Concourse.helpers.taskStep\n              Concourse.schemas.TaskStep::{\n              , task = \"hello\"\n              , config =\n                  Concourse.Types.TaskSpec.Config\n                    Concourse.schemas.TaskConfig::{\n                    , image_resource = Some busyboxImage\n                    , run =\n                        Concourse.schemas.TaskRunConfig::{\n                        , path = \"bash\"\n                        , args = Some [ \"-c\", \"echo Hello Dhall\" ]\n                        }\n                    }\n              }\n          ]\n      }\n\nin  [ job ]\n```\n\nTo set the pipeline, run this command:\n\n```\nfly -t \u003cTARGET\u003e set-pipeline -p hello-dhall -c \u003c(dhall-fly \u003cexample1.dhall)\n```\n\n### Example 2 (Hello World with groups)\n\n```dhall\nlet Concourse =\n      https://raw.githubusercontent.com/akshaymankar/dhall-concourse/0.5.0/package.dhall\n\nlet Prelude =\n      https://prelude.dhall-lang.org/v11.1.0/package.dhall sha256:99462c205117931c0919f155a6046aec140c70fb8876d208c7c77027ab19c2fa\n\n-- Assuming above file is here\nlet jobs = ./example1.dhall\n\nin  Prelude.List.map\n      Concourse.Types.Job\n      Concourse.Types.GroupedJob\n      (λ(j : Concourse.Types.Job) → { job = j, groups = [ \"hello-world\" ] })\n      jobs\n```\n\nTo notify `dhall-fly` that we'd be passing it a `List Concourse.Types.GroupedJob` instead of `List Concourse.Types.Job`, we have to call it with `--pipeline-type grouped-jobs` like this:\n```\nfly -t \u003cTARGET\u003e set-pipeline -p hello-dhall -c \u003c(dhall-fly --pipeline-type grouped-jobs \u003cexample1.dhall)\n```\n\n\n### Example 3: Pipeline for this repository\n\nThe pipeline is defined in [./ci/pipeline.dhall](./ci/pipeline.dhall). \n\nThe script [./ci/set-pipeline.sh](./ci/set-pipeline.sh) is used to set the pipeline. \n\nThe pipeline can be viewed [here](https://concourse.gdn/teams/main/pipelines/dhall-concourse).\n\n## Why are Steps so complicated?\n\nIn concourse, a step can be one of `get`, `put`, `task`, `in_parallel`, `aggregate`, `do` or `try`. The `in_parallel`, `do` and `aggregate` are list of steps, `try` represents one step. This makes definition of the `Step` type recursive, as in to define a `Step` we'd already need definition of a Step. In total languages like dhall, this is a little tricky to do. There is an explanation in [dhall docs about how to do this](https://docs.dhall-lang.org/howtos/How-to-translate-recursive-code-to-Dhall.html). An example of this can be found in dhall prelude's definition of the [JSON Type](https://github.com/dhall-lang/dhall-lang/blob/master/Prelude/JSON/Type)\n\n**But there is more!**: Each step can also have 4 types of `hooks`: `on_failure`, `on_success`, `on_abort` and `ensure`. Each of these is optional and you guessed it, is of type Step.\n\nSo, I decided the type definition for `Step` would look like this:\n\n```dhall\nlet StepHooks =\n        λ(Step : Type)\n      → { on_failure : Optional Step\n        , on_success : Optional Step\n        , on_abort : Optional Step\n        , ensure : Optional Step\n        }\n\nlet StepConstructors =\n        λ(Step : Type)\n      → { get : ./GetStep.dhall → StepHooks Step → Step\n        , put : ./PutStep.dhall → StepHooks Step → Step\n        , task : ./TaskStep.dhall → StepHooks Step → Step\n        , aggregate : List Step → StepHooks Step → Step\n        , in_parallel : ./InParallelStep.dhall → StepHooks Step → Step\n        , do : Step → StepHooks Step → Step\n        , try : Step → StepHooks Step → Step\n        }\n\nin  ∀(Step : Type) → StepConstructors Step → Step\n\n```\n\nThe way I read it is, a Step is defined for any type for which one can provide constructors for get, put, task, aggregate, etc.\n\nThis is the reason this repository includes helpers to construct steps with or without hooks.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fakshaymankar%2Fdhall-concourse","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fakshaymankar%2Fdhall-concourse","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fakshaymankar%2Fdhall-concourse/lists"}