{"id":21519423,"url":"https://github.com/cermakm/argo-python-dsl","last_synced_at":"2026-06-12T09:01:55.144Z","repository":{"id":57411439,"uuid":"227359950","full_name":"CermakM/argo-python-dsl","owner":"CermakM","description":"Python DSL for Argo Workflows | Mirrored to https://github.com/argoproj-labs/argo-python-dsl","archived":false,"fork":false,"pushed_at":"2020-03-19T14:10:33.000Z","size":214,"stargazers_count":55,"open_issues_count":11,"forks_count":6,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-04-09T22:09:20.259Z","etag":null,"topics":["argo","argo-workflows","kubernetes","python"],"latest_commit_sha":null,"homepage":"https://github.com/argoproj/argo","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CermakM.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.rst","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null}},"created_at":"2019-12-11T12:23:26.000Z","updated_at":"2023-08-24T14:59:51.000Z","dependencies_parsed_at":"2022-09-09T22:23:19.370Z","dependency_job_id":null,"html_url":"https://github.com/CermakM/argo-python-dsl","commit_stats":null,"previous_names":["cermakm/argo-python-sdk"],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CermakM%2Fargo-python-dsl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CermakM%2Fargo-python-dsl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CermakM%2Fargo-python-dsl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CermakM%2Fargo-python-dsl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CermakM","download_url":"https://codeload.github.com/CermakM/argo-python-dsl/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248119294,"owners_count":21050755,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["argo","argo-workflows","kubernetes","python"],"created_at":"2024-11-24T00:57:43.031Z","updated_at":"2025-12-13T21:32:12.563Z","avatar_url":"https://github.com/CermakM.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# argo-python-dsl \u0026nbsp; [![Release](https://img.shields.io/github/v/tag/argoproj-labs/argo-python-dsl.svg?sort=semver\u0026label=Release)](https://github.com/argoproj-labs/argo-python-dsl/releases/latest)\n\n[![License](https://img.shields.io/github/license/argoproj-labs/argo-python-dsl)](https://github.com/argoproj-labs/argo-python-dsl/blob/master/LICENSE) \u0026nbsp; [![CI](https://github.com/argoproj-labs/argo-python-dsl/workflows/CI/badge.svg)](https://github.com/argoproj-labs/argo-python-dsl/actions) \u0026nbsp;\n\n### Python DSL for [Argo Workflows](https://github.com/argoproj/argo)\n\n\u003cdiv style=\"text-align: justify\"\u003e\n\nIf you're new to Argo, we recommend checking out the examples in pure YAML. The language is descriptive and the Argo [examples](https://github.com/argoproj/argo/tree/master/examples) provide an exhaustive explanation.\n\nFor a more experienced audience, this DSL grants you the ability to programatically define Argo Workflows in Python which is then translated to the Argo YAML specification.\n\nThe DSL makes use of the Argo models defined in the [Argo Python client](https://github.com/argoproj-labs/argo-client-python) repository. Combining the two approaches we are given the whole low-level control over Argo Workflows.\n\n\u003c/div\u003e\n\n## Getting started\n\n#### Hello World\n\n\u003cdiv style=\"text-align: justify\"\u003e\n\nThis example demonstrates the simplest functionality. Defining a `Workflow` by subclassing the `Workflow` class and a single template with the `@template` decorator.\n\nThe entrypoint to the workflow is defined as an `entrypoint` class property.\n\n\u003c/div\u003e\n\n\u003ctable\u003e\n\u003ctr\u003e\u003cth\u003eArgo YAML\u003c/th\u003e\u003cth\u003eArgo Python\u003c/th\u003e\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd valign=\"top\"\u003e\u003cp\u003e\n\n```yaml\n# @file: hello-world.yaml\napiVersion: argoproj.io/v1alpha1\nkind: Workflow\nmetadata:\n  name: hello-world\n  generateName: hello-world-\nspec:\n  entrypoint: whalesay\n  templates:\n  - name: whalesay\n    container:\n      name: whalesay\n      image: docker/whalesay:latest\n      command: [cowsay]\n      args: [\"hello world\"]\n```\n\n\u003c/p\u003e\u003c/td\u003e\n\u003ctd valign=\"top\"\u003e\u003cp\u003e\n\n```python\nfrom argo.workflows.dsl import Workflow\nfrom argo.workflows.dsl import template\n\nfrom argo.workflows.dsl.templates import V1Container\n\n\nclass HelloWorld(Workflow):\n\n    entrypoint = \"whalesay\"\n\n    @template\n    def whalesay(self) -\u003e V1Container:\n        container = V1Container(\n            image=\"docker/whalesay:latest\",\n            name=\"whalesay\",\n            command=[\"cowsay\"],\n            args=[\"hello world\"]\n        )\n\n        return container\n```\n\n\u003c/p\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n#### DAG: Tasks\n\n\u003cdiv style=\"text-align: justify\"\u003e\n\nThis example demonstrates tasks defined via dependencies forming a *diamond* structure. Tasks are defined using the `@task` decorator and they **must return a valid template**.\n\nThe entrypoint is automatically created as `main` for the top-level tasks of the `Workflow`.\n\n\u003c/div\u003e\n\n\u003ctable\u003e\n\u003ctr\u003e\u003cth\u003eArgo YAML\u003c/th\u003e\u003cth\u003eArgo Python\u003c/th\u003e\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd valign=\"top\"\u003e\u003cp\u003e\n\n```yaml\n# @file: dag-diamond.yaml\n# The following workflow executes a diamond workflow\n#\n#   A\n#  / \\\n# B   C\n#  \\ /\n#   D\napiVersion: argoproj.io/v1alpha1\nkind: Workflow\nmetadata:\n  name: dag-diamond\n  generateName: dag-diamond-\nspec:\n  entrypoint: main\n  templates:\n  - name: main\n    dag:\n      tasks:\n      - name: A\n        template: echo\n        arguments:\n          parameters: [{name: message, value: A}]\n      - name: B\n        dependencies: [A]\n        template: echo\n        arguments:\n          parameters: [{name: message, value: B}]\n      - name: C\n        dependencies: [A]\n        template: echo\n        arguments:\n          parameters: [{name: message, value: C}]\n      - name: D\n        dependencies: [B, C]\n        template: echo\n        arguments:\n          parameters: [{name: message, value: D}]\n\n  # @task: [A, B, C, D]\n  - name: echo\n    inputs:\n      parameters:\n      - name: message\n    container:\n      name: echo\n      image: alpine:3.7\n      command: [echo, \"{{inputs.parameters.message}}\"]\n```\n\n\u003c/p\u003e\u003c/td\u003e\n\u003ctd valign=\"top\"\u003e\u003cp\u003e\n\n```python\nfrom argo.workflows.dsl import Workflow\n\nfrom argo.workflows.dsl.tasks import *\nfrom argo.workflows.dsl.templates import *\n\n\nclass DagDiamond(Workflow):\n\n    @task\n    @parameter(name=\"message\", value=\"A\")\n    def A(self, message: V1alpha1Parameter) -\u003e V1alpha1Template:\n        return self.echo(message=message)\n\n    @task\n    @parameter(name=\"message\", value=\"B\")\n    @dependencies([\"A\"])\n    def B(self, message: V1alpha1Parameter) -\u003e V1alpha1Template:\n        return self.echo(message=message)\n\n    @task\n    @parameter(name=\"message\", value=\"C\")\n    @dependencies([\"A\"])\n    def C(self, message: V1alpha1Parameter) -\u003e V1alpha1Template:\n        return self.echo(message=message)\n\n    @task\n    @parameter(name=\"message\", value=\"D\")\n    @dependencies([\"B\", \"C\"])\n    def D(self, message: V1alpha1Parameter) -\u003e V1alpha1Template:\n        return self.echo(message=message)\n\n    @template\n    @inputs.parameter(name=\"message\")\n    def echo(self, message: V1alpha1Parameter) -\u003e V1Container:\n        container = V1Container(\n            image=\"alpine:3.7\",\n            name=\"echo\",\n            command=[\"echo\", \"{{inputs.parameters.message}}\"],\n        )\n\n        return container\n\n```\n\n\u003c/p\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n#### Artifacts\n\n\u003cdiv style=\"text-align: justify\"\u003e\n\n`Artifacts` can be passed similarly to `parameters` in three forms: `arguments`, `inputs` and `outputs`, where `arguments` is the default one (simply `@artifact` or `@parameter`).\n\nI.e.: `inputs.artifact(...)`\n\nBoth artifacts and parameters are passed **one by one**, which means that for multiple artifacts (parameters), one should call:\n\n\n```python\n@inputs.artifact(name=\"artifact\", ...)\n@inputs.parameter(name=\"parameter_a\", ...)\n@inputs.parameter(...)\ndef foo(self, artifact: V1alpha1Artifact, prameter_b: V1alpha1Parameter, ...): pass\n```\n\nA complete example:\n\n\u003c/div\u003e\n\n\u003ctable\u003e\n\u003ctr\u003e\u003cth\u003eArgo YAML\u003c/th\u003e\u003cth\u003eArgo Python\u003c/th\u003e\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd valign=\"top\"\u003e\u003cp\u003e\n\n```yaml\n# @file: artifacts.yaml\napiVersion: argoproj.io/v1alpha1\nkind: Workflow\nmetadata:\n  name: artifact-passing\n  generateName: artifact-passing-\nspec:\n  entrypoint: main\n  templates:\n  - name: main\n    dag:\n      tasks:\n      - name: generate-artifact\n        template: whalesay\n      - name: consume-artifact\n        template: print-message\n        arguments:\n          artifacts:\n          # bind message to the hello-art artifact\n          # generated by the generate-artifact step\n          - name: message\n            from: \"{{tasks.generate-artifact.outputs.artifacts.hello-art}}\"\n\n  - name: whalesay\n    container:\n      name: \"whalesay\"\n      image: docker/whalesay:latest\n      command: [sh, -c]\n      args: [\"cowsay hello world | tee /tmp/hello_world.txt\"]\n    outputs:\n      artifacts:\n      # generate hello-art artifact from /tmp/hello_world.txt\n      # artifacts can be directories as well as files\n      - name: hello-art\n        path: /tmp/hello_world.txt\n\n  - name: print-message\n    inputs:\n      artifacts:\n      # unpack the message input artifact\n      # and put it at /tmp/message\n      - name: message\n        path: /tmp/message\n    container:\n      name: \"print-message\"\n      image: alpine:latest\n      command: [sh, -c]\n      args: [\"cat\", \"/tmp/message\"]\n```\n\n\u003c/p\u003e\u003c/td\u003e\n\u003ctd valign=\"top\"\u003e\u003cp\u003e\n\n```python\nfrom argo.workflows.dsl import Workflow\n\nfrom argo.workflows.dsl.tasks import *\nfrom argo.workflows.dsl.templates import *\n\nclass ArtifactPassing(Workflow):\n\n    @task\n    def generate_artifact(self) -\u003e V1alpha1Template:\n        return self.whalesay()\n\n    @task\n    @artifact(\n        name=\"message\",\n        _from=\"{{tasks.generate-artifact.outputs.artifacts.hello-art}}\"\n    )\n    def consume_artifact(self, message: V1alpha1Artifact) -\u003e V1alpha1Template:\n        return self.print_message(message=message)\n\n    @template\n    @outputs.artifact(name=\"hello-art\", path=\"/tmp/hello_world.txt\")\n    def whalesay(self) -\u003e V1Container:\n        container = V1Container(\n            name=\"whalesay\",\n            image=\"docker/whalesay:latest\",\n            command=[\"sh\", \"-c\"],\n            args=[\"cowsay hello world | tee /tmp/hello_world.txt\"]\n        )\n\n        return container\n\n    @template\n    @inputs.artifact(name=\"message\", path=\"/tmp/message\")\n    def print_message(self, message: V1alpha1Artifact) -\u003e V1Container:\n        container = V1Container(\n            name=\"print-message\",\n            image=\"alpine:latest\",\n            command=[\"sh\", \"-c\"],\n            args=[\"cat\", \"/tmp/message\"],\n        )\n\n        return container\n```\n\n\u003c/p\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n\u003cbr\u003e\n\n## Going further: `closure` and `scope`\n\n\u003cdiv style=\"text-align: justify\"\u003e\n\nThis is where it gets quite interesting. So far, we've only scratched the benefits that the Python implementation provides.\n\nWhat if we want to use native Python code and execute it as a step in the Workflow. What are our options?\n\n**Option A)** is to reuse the existing mindset, dump the code in a string, pass it as the source to the `V1ScriptTemplate` model and wrap it with the `template` decorator.\nThis is illustrated in the following code block:\n\n\u003c/div\u003e\n\n```python\nimport textwrap\n\nclass ScriptsPython(Workflow):\n\n    ...\n\n    @template\n    def gen_random_int(self) -\u003e V1alpha1ScriptTemplate:\n        source = textwrap.dedent(\"\"\"\\\n          import random\n          i = random.randint(1, 100)\n          print(i)\n        \"\"\")\n\n        template = V1alpha1ScriptTemplate(\n            image=\"python:alpine3.6\",\n            name=\"gen-random-int\",\n            command=[\"python\"],\n            source=source\n        )\n\n        return template\n```\n\nWhich results in:\n\n```yaml\napi_version: argoproj.io/v1alpha1\nkind: Workflow\nmetadata:\n  generate_name: scripts-python-\n  name: scripts-python\nspec:\n  entrypoint: main\n\n  ...\n\n  templates:\n  - name: gen-random-int\n    script:\n      command:\n      - python\n      image: python:alpine3.6\n      name: gen-random-int\n      source: 'import random\\ni = random.randint(1, 100)\\nprint(i)\\n'\n```\n\n\u003cdiv style=\"text-align: justify\"\u003e\n\nNot bad, but also not living up to the full potential. Since we're already writing Python, why would we wrap the code in a string? This is where we introduce `closure`s.\n\n#### `closure`s\n\nThe logic of `closure`s is quite simple. Just wrap the function you want to execute in a container in the `@closure` decorator. The `closure` then takes care of the rest and returns a `template` (just as the `@template` decorator).\n\nThe only thing we need to take care of is to provide it an image which has the necessary Python dependencies installed and is present in the cluster.\n\n\u003e There is a plan to eliminate even this step in the future, but currently it is inavoidable.\n\nFollowing the previous example:\n\n\u003c/div\u003e\n\n```python\nclass ScriptsPython(Workflow):\n\n    ...\n\n    @closure(\n      image=\"python:alpine3.6\"\n    )\n    def gen_random_int() -\u003e V1alpha1ScriptTemplate:\n          import random\n\n          i = random.randint(1, 100)\n          print(i)\n```\n\n\u003cdiv style=\"text-align: justify\"\u003e\n\nThe closure implements the `V1alpha1ScriptTemplate`, which means that you can pass in things like `resources`, `env`, etc...\n\nAlso, make sure that you `import` whatever library you are using, the context is not preserved --- `closure` behaves as a staticmethod and is *sandboxed* from the module scope.\n\n#### `scope`s\n\nNow, what if we had a function (or a whole script) which is quite big. Wrapping it in a single Python function is not very Pythonic and it gets tedious. This is where we can make use of `scope`s.\n\nSay that we, for example, wanted to initialize logging before running our `gen_random_int` function.\n\n\u003c/div\u003e\n\n```python\n    ...\n\n    @closure(\n      scope=\"main\",\n      image=\"python:alpine3.6\"\n    )\n    def gen_random_int(main) -\u003e V1alpha1ScriptTemplate:\n          import random\n\n          main.init_logging()\n\n          i = random.randint(1, 100)\n          print(i)\n\n    @scope(name=\"main\")\n    def init_logging(level=\"DEBUG\"):\n        import logging\n\n        logging_level = getattr(logging, level, \"INFO\")\n        logging.getLogger(\"__main__\").setLevel(logging_level)\n```\n\nNotice the 3 changes that we've made:\u003c/div\u003e\n\n```python\n    @closure(\n      scope=\"main\",  # \u003c--- provide the closure a scope\n      image=\"python:alpine3.6\"\n    )\n    def gen_random_int(main):  # \u003c--- use the scope name\n```\n\n```python\n    @scope(name=\"main\")  # \u003c--- add function to a scope\n    def init_logging(level=\"DEBUG\"):\n```\n\n\u003cdiv style=\"text-align: justify\"\u003e\n\nEach function in the given scope is then namespaced by the scope name and injected to the closure.\n\nI.e. the resulting YAML looks like this:\u003c/div\u003e\n\n```yaml\n...\nspec:\n  ...\n  templates:\n    - name: gen-random-int\n      script:\n        command:\n        - python\n        image: python:alpine3.6\n        name: gen-random-int\n        source: |-\n          import logging\n          import random\n\n          class main:\n            \"\"\"Scoped objects injected from scope 'main'.\"\"\"\n\n            @staticmethod\n            def init_logging(level=\"DEBUG\"):\n              logging_level = getattr(logging, level, \"INFO\")\n              logging.getLogger(\"__main__\").setLevel(logging_level)\n\n\n          main.init_logging()\n\n          i = random.randint(1, 100)\n          print(i)\n```\n\nThe compilation also takes all imports to the front and remove duplicates for convenience and more natural look so that you don't feel like poking your eyes when you look at the resulting YAML.\n\n\u003cbr\u003e\n\nFor more examples see the [examples](https://github.com/argoproj-labs/argo-python-dsl/tree/master/examples) folder.\n\n\u003cbr\u003e\n\n---\n\nAuthors:\n- [ Maintainer ] Marek Cermak \u003cmacermak@redhat.com\u003e, \u003cprace.mcermak@gmail.com\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcermakm%2Fargo-python-dsl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcermakm%2Fargo-python-dsl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcermakm%2Fargo-python-dsl/lists"}