{"id":15663018,"url":"https://github.com/joscha/buildkite-graph","last_synced_at":"2025-04-28T15:57:20.381Z","repository":{"id":35026966,"uuid":"193539337","full_name":"joscha/buildkite-graph","owner":"joscha","description":"A graph-based generator for Buildkite pipelines","archived":false,"fork":false,"pushed_at":"2024-11-05T15:54:26.000Z","size":1687,"stargazers_count":18,"open_issues_count":18,"forks_count":6,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-28T15:56:45.725Z","etag":null,"topics":["buildkite","buildkite-pipelines","dag","dependencies","dynamic-pipelines","graph","serialization","yaml"],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/joscha.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-06-24T16:16:28.000Z","updated_at":"2024-12-30T22:24:19.000Z","dependencies_parsed_at":"2024-06-21T15:35:02.792Z","dependency_job_id":"99e38da9-fe38-4160-9811-4ceae3086ad8","html_url":"https://github.com/joscha/buildkite-graph","commit_stats":{"total_commits":215,"total_committers":7,"mean_commits":"30.714285714285715","dds":0.5813953488372092,"last_synced_commit":"d404a97740356a4d0b34d89d13816b652b648def"},"previous_names":[],"tags_count":49,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joscha%2Fbuildkite-graph","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joscha%2Fbuildkite-graph/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joscha%2Fbuildkite-graph/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joscha%2Fbuildkite-graph/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/joscha","download_url":"https://codeload.github.com/joscha/buildkite-graph/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251342720,"owners_count":21574244,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["buildkite","buildkite-pipelines","dag","dependencies","dynamic-pipelines","graph","serialization","yaml"],"created_at":"2024-10-03T13:35:14.793Z","updated_at":"2025-04-28T15:57:20.335Z","avatar_url":"https://github.com/joscha.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# buildkite-graph\n\nA graph-based generator for Buildkite pipelines\n\nThis module allows you to generate Buildkite pipelines by defining their dependencies via a graph. This graph gets then serialized into the Buildkite-specific YAML format.\nAll standard Buildkite features are supported.\n\nThe main advantage of using this module is:\n\n- Easy reuse and recombination of steps\n- Defining dependencies between steps explicitly\n- Wait steps are not defined explicitly and manually managed but derived from the graph, always providing the most optimized graph\n- Steps can be defined conditionally via an acceptor function, allowing for completely dynamic pipelines\n- The graph can be serialzed into [dot](https://www.graphviz.org/) format, allowing you to see the whole of the pipeline in one glance. Clusters denote which parts of the graph are dependendent.\n- Timeouts can be defined on a per-command basis, the step will then accumulate the timeouts accordingly\n\n## Example in a nutshell\n\n```ts\nconst install = new Command('yarn', 2);\n\nconst lint = new CommandStep([install, new Command('yarn lint', 1)]).withKey(\n  'lint',\n);\nconst test = new CommandStep([install, new Command('yarn test', 2)])\n  .withKey('test')\n  .dependsOn(lint);\nconst build = new CommandStep([install, new Command('yarn build', 5)])\n  .withKey('build')\n  .dependsOn(lint);\nconst integration = new CommandStep([\n  install,\n  new Command('yarn integration', 10),\n])\n  .withKey('integration-test')\n  .dependsOn(build);\n\nconst pipeline = new Pipeline('My pipeline').add(test).add(integration);\n```\n\n\u003e Do you see how we don't have to add the `lint` or `build`step? Because other steps depend on them, they will become part of the graph automatically in the right place. This allows you to define graphs with complex dependencies and only add the steps which have an important signal - no more manually adding auxiliary steps.\n\nwill serialize to:\n\n```yaml\nsteps:\n  - command:\n      - yarn\n      - yarn lint\n    timeout_in_minutes: 3\n  - wait: ~\n  - command:\n      - yarn\n      - yarn build\n    timeout_in_minutes: 7\n  - command:\n      - yarn\n      - yarn test\n    timeout_in_minutes: 4\n  - wait: ~\n  - command:\n      - yarn\n      - yarn integration\n    timeout_in_minutes: 12\n```\n\n\u003e Did you see how the `wait` step got added for you? How cool is that, hey :)\n\n\u003e And did you also see how the timeouts for the steps are derived from the commands?\n\nSince version 5 we also support the [new `depends_on` syntax](https://buildkite.com/changelog/84-introducing-pipeline-step-dependencies):\n\n```yaml\nsteps:\n  - key: lint\n    command:\n      - yarn\n      - yarn lint\n    timeout_in_minutes: 3\n  - key: build\n    depends_on:\n      - step: lint\n    command:\n      - yarn\n      - yarn build\n    timeout_in_minutes: 7\n  - key: test\n    depends_on:\n      - step: lint\n    command:\n      - yarn\n      - yarn test\n    timeout_in_minutes: 4\n  - key: integration-test\n    depends_on:\n      - step: build\n    command:\n      - yarn\n      - yarn integration\n    timeout_in_minutes: 12\n```\n\nyou can get this format by using a flag on the yaml serializer:\n\n```ts\nconsole.log(\n  await new YamlSerializer({ explicitDependencies: true }).serialize(pipeline),\n);\n```\n\nThe same with graphviz:\n\n```dot\ndigraph \"My pipeline\" {\n  graph [ compound =true ];\nsubgraph cluster_0 {\n  graph [ color = \"black\" ];\n  \"\u003cyarn \u0026\u0026 yarn lint\u003e\" [ color = \"grey\" ];\n}\n\nsubgraph cluster_1 {\n  graph [ color = \"black\" ];\n  \"\u003cyarn \u0026\u0026 yarn build\u003e\" [ color = \"grey\" ];\n  \"\u003cyarn \u0026\u0026 yarn test\u003e\" [ color = \"grey\" ];\n}\n\nsubgraph cluster_2 {\n  graph [ color = \"black\" ];\n  \"\u003cyarn \u0026\u0026 yarn integration\u003e\" [ color = \"grey\" ];\n}\n\n  \"\u003cyarn \u0026\u0026 yarn lint\u003e\";\n  \"\u003cyarn \u0026\u0026 yarn build\u003e\";\n  \"\u003cyarn \u0026\u0026 yarn test\u003e\";\n  \"\u003cyarn \u0026\u0026 yarn integration\u003e\";\n  \"\u003cyarn \u0026\u0026 yarn lint\u003e\" -\u003e \"\u003cyarn \u0026\u0026 yarn build\u003e\" [ ltail = \"cluster_0\", lhead = \"cluster_1\" ];\n  \"\u003cyarn \u0026\u0026 yarn lint\u003e\" -\u003e \"\u003cyarn \u0026\u0026 yarn test\u003e\" [ ltail = \"cluster_0\", lhead = \"cluster_1\" ];\n  \"\u003cyarn \u0026\u0026 yarn build\u003e\" -\u003e \"\u003cyarn \u0026\u0026 yarn integration\u003e\" [ ltail = \"cluster_1\", lhead = \"cluster_2\" ];\n}\n```\n\nwhich will visualize to:\n\n\u003cimg src=\"https://user-images.githubusercontent.com/188038/61578524-b6cfc280-ab3b-11e9-87ab-28fa6be480ff.png\" width=\"500\"\u003e\n\n\u003e See the clusters (the square boxes)? The are legs of multiple steps in your pipeline, each separated by a wait step.\n\n...but wait there's more; you can use the structural serializer to produce a textual output of the pipeline:\n\n```\n* Build Editor\n* Test Editor\n* [wait; continues on failure]\n* Annotate failures\n* Upload coverage\n* [wait]\n* Integration tests [x 8]\n* [wait]\n* :saucelabs: Integration tests [x 8]\n* Visreg baseline update\n* [wait; continues on failure]\n* Annotate cucumber failures\n* [wait]\n* Copy to deploy bucket\n* [wait]\n* Update checkpoint\n* Deploy to tech\n* Deploy to usertesting\n* [wait]\n* [block for 'Release editor']\n```\n\njust by calling:\n\n```ts\nconsole.log(await new StructuralSerializer().serialize(pipeline));\n```\n\nThis will allow you to create snapshot tests for your dynamic pipelines which represent the step composition and order.\n\n## Side-effects (Weak dependencies)\n\nSometimes a dependency is not a strong one, e.g. if you have a unit test you probably want to deploy the coverage, but adding the deploy coverage step should not trigger the unit tests unless they need to. In that case you can use `.isEffectOf()` instead of `.dependsOn()`:\n\n```ts\ndeploCoverageStep.isEffectOf(conditionalUnitTestStep);\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoscha%2Fbuildkite-graph","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjoscha%2Fbuildkite-graph","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoscha%2Fbuildkite-graph/lists"}