{"id":22131125,"url":"https://github.com/slamdev/argo-workflows-showcase","last_synced_at":"2026-03-19T21:53:22.547Z","repository":{"id":43334887,"uuid":"431579104","full_name":"slamdev/argo-workflows-showcase","owner":"slamdev","description":null,"archived":false,"fork":false,"pushed_at":"2023-12-15T14:30:40.000Z","size":151,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-26T11:56:16.774Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Makefile","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/slamdev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-11-24T17:51:05.000Z","updated_at":"2021-12-14T09:56:07.000Z","dependencies_parsed_at":"2024-12-01T18:41:52.961Z","dependency_job_id":null,"html_url":"https://github.com/slamdev/argo-workflows-showcase","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/slamdev/argo-workflows-showcase","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/slamdev%2Fargo-workflows-showcase","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/slamdev%2Fargo-workflows-showcase/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/slamdev%2Fargo-workflows-showcase/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/slamdev%2Fargo-workflows-showcase/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/slamdev","download_url":"https://codeload.github.com/slamdev/argo-workflows-showcase/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/slamdev%2Fargo-workflows-showcase/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29567591,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-18T00:47:08.760Z","status":"online","status_checked_at":"2026-02-18T02:00:09.468Z","response_time":162,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-01T18:30:09.366Z","updated_at":"2026-02-18T03:36:59.158Z","avatar_url":"https://github.com/slamdev.png","language":"Makefile","funding_links":[],"categories":[],"sub_categories":[],"readme":"# argo-workflows-showcase\n\nProject explores functionality of [Argo Workflows](https://argoproj.github.io/workflows/).\n\nCovered features:\n\n- [x] Directed acyclic graph\n- [x] Reusable templates\n\n---\n**NOTE**\n\nThis is not a production ready project and should be used only for learning purposes. Refer\nto [Production-ready checklist](#production-ready-checklist) section for more details.\n\n---\n\nImplemented workflow:\n\n1. Clone csv, avro, parquet files from AWS S3 to internal S3-compatible storage\n2. Link Hive tables to files from the first step\n3. Query tables\n\nWorkflow is described in [workflow.yaml](showcase/workflow.yaml) file.\n\n![workflow](workflow.png)\n\n## Findings\n\n### Verbosity\n\nWorkflow syntax is very verbose, especially in yaml. Extracting as much as possible to external templates helps but not\nmuch. A simple DSL with compile-time validation can reduce the complexity.\n\n### Dev experience\n\n#### Workflows\n\nDuring the workflow implementation the development cycle looks like:\n\n```text\nchange spec -\u003e run -\u003e wait until the change is executed -\u003e fail -\u003e repeat\n```\n\nThe process is pretty time consuming.\n\nArgo is lacking of admission controller to do a deep validation before applying manifests. That could help with syntax\nerrors.\n\nThere is no simple way to run a part of the workflow locally with mocked inputs and outputs. Therefore no unit testing\nas well.\n\n#### Templates\n\nIssues above are applicable for external templates development as well. But the impact is worse: multiple workflows can\nbreak by introducing a bug in a template.\n\n### Documentation\n\nThere is no build-in tool to generate docs for external templates so users of the templates have to check the template\nimplementation to understand how to use it in their workflows.\n\nAlthough templates are defined in yaml, so creation of a docs generator is an easy task.\n\n### Versioning\n\nExternal templates have to be versioned and maintain the backwards compatibility with the corresponding workflows.\n\nArgo doesn't support this concept out of the box, so some standards between template authors and template users need to\nbe defined.\n\n## Local development\n\nList of required tools:\n\n1. [make](https://command-not-found.com/make)\n2. [skaffold](https://skaffold.dev/docs/install/#standalone-binary)\n3. [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl)\n4. [helm](https://helm.sh/docs/intro/install/)\n5. [kind](https://kind.sigs.k8s.io/docs/user/quick-start#installation)\n6. [docker](https://docs.docker.com/get-docker/)\n\n**make** is used to provide a convenient way of deploying all the components to a cluster.\n\n`make apply` deploys the whole project to the currently active cluster, so make sure you have a correct kubeconfig file\nselected\n\n`make apply/{app}` (e.g. `make apply/showcase/rides-workflow`) deploys a single app\n\n**skaffold** provides a unified way of working with plain kube manifests and helm charts\n\n**kind** allows to run a local kube cluster via a single binary\n\nThere are scripts in [.github/kind](.github/kind) folder that can create\\destroy a kind cluster for this project.\n\nRun to expose Argo Server UI on http://localhost:2746 :\n\n```shell\nkubectl -nargo port-forward svc/argo-workflows-server 2746:2746\n```\n\nRun to query trino:\n\n```shell\nkubectl -nshowcase port-forward svc/rides-trino 8080:8080\ndocker run --rm trivadis/trino-cli:353 trino --server=host.docker.internal:8080 --execute=\"SELECT 1\"\n```\n\n## CI\n\nThere is a single [Github Action](https://github.com/slamdev/argo-workflows-showcase/actions/) that starts a **kind**\ncluster, deploys all the components there and verifies that workflow is completed successfully.\n\n## Production-ready checklist\n\n- [ ] secret management - secrets are hardocded and stored in plain text in the code; this should be changed to managing\n  secrets in [Vault](https://www.vaultproject.io/) and linking it to kube cluster\n  via [external-secrets-operator](https://github.com/slamdev/external-secrets-operator)\n- [ ] resource requests\\limits - apps should be analyzed for the resource usage and corresponding values should be set\n- [ ] observability - all the components should expose metrics and write logs in json format, some dashboards have to be\n  created to visualize the state of the components, as well as alerts should be send in case there are issues in\n  runtime, the observability stack to go with can be the following:\n    * [Prometheus](https://prometheus.io/) to scrape the metrics from components\n    * [Thanos](https://thanos.io/) as a long term stateless metrics storage\n    * [Loki](https://grafana.com/oss/loki/) as a long term stateless logs storage\n    * [Grafana](https://grafana.com/oss/grafana/) to visualize logs and metrics\n- [ ] build infra - even though make+skaffold provide a convenient way of deploying apps, the solution doesn't work in a\n  long run:\n    * no caching -\u003e builds are slow\n    * no manifests validation, e.g. via [kubeconform](https://github.com/yannh/kubeconform)\n    * no way to track if an app was rolled out successfully after the deploy\n\n  as an option, it can be replaced with [bazel](https://bazel.build/) that fits the requirements\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fslamdev%2Fargo-workflows-showcase","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fslamdev%2Fargo-workflows-showcase","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fslamdev%2Fargo-workflows-showcase/lists"}