{"id":17343766,"url":"https://github.com/zbrookle/goflow","last_synced_at":"2025-09-09T21:37:51.906Z","repository":{"id":47302172,"uuid":"310500632","full_name":"zbrookle/goflow","owner":"zbrookle","description":"A Kubernetes native task manager that functions similarly to Airflow","archived":false,"fork":false,"pushed_at":"2021-09-03T21:09:06.000Z","size":35863,"stargazers_count":6,"open_issues_count":10,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-14T20:13:37.525Z","etag":null,"topics":["apache-airflow","go","golang","kubernetes","scheduler"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zbrookle.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-11-06T05:26:57.000Z","updated_at":"2023-09-09T07:40:16.000Z","dependencies_parsed_at":"2022-07-22T01:02:16.417Z","dependency_job_id":null,"html_url":"https://github.com/zbrookle/goflow","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zbrookle%2Fgoflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zbrookle%2Fgoflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zbrookle%2Fgoflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zbrookle%2Fgoflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zbrookle","download_url":"https://codeload.github.com/zbrookle/goflow/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248952352,"owners_count":21188426,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-airflow","go","golang","kubernetes","scheduler"],"created_at":"2024-10-15T16:10:17.775Z","updated_at":"2025-04-14T20:13:43.450Z","avatar_url":"https://github.com/zbrookle.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# GoFlow\n\n[![CI](https://github.com/zbrookle/goflow/workflows/CI/badge.svg?branch=master)](https://github.com/zbrookle/goflow/actions)\n[![go report](https://goreportcard.com/badge/github.com/zbrookle/goflow)](https://goreportcard.com/report/github.com/zbrookle/goflow)\n\nGoFlow is a job scheduler inspired by Apache Airflow and is designed to be a more cohesive package,\ndirectly integrated with Kubernetes.\n\nIn GoFlow, DAGS are supported in the following formats:\n\n- Tradiitonal Airflow DAG (Coming soon...)\n- Golang DAG (Coming soon...)\n\n## Why GoFlow\n\nGoFlow has the potential to surpass Apache Airflow in terms of design and scalability for a few key reasons:\n\n1. GoFlow is written in Golang, a language that is compiled and has native support for multithreading. This\n   removes the dependency that Airflow has traditionally had on Redis, for scalability and drastically removes\n   the CPU footprint of running separate Python processes.\n\n1. GoFlow is designed to be cohesive and less flexible than Airflow. Airflow has grown in complexity over the\n   years and suits a great number of use cases. However, it is very difficult to set up because of this flexibility.\n   GoFlow only will run on Kubernetes, meaning that it will require less configuration and gives developers back the\n   time they need to focus on code rather than infrastructure.\n\n1. Because Golang is statically typed, it is much less prone to typing errors than Apache Airflow's language Python\n   which is dynamically typed.\n\n1. GoFlow will have built in support for ElasticSearch logging. (Coming soon...)\n\n## Design\n\nGoFlow relies on a central orchestrator to maintain and monitor the CronJobs that it has placed into the Kubernetes\nenvironment. These jobs are declared using DAGs as is traditional in Apache Airflow. GoFlow keeps track of the days\nthat jobs have run and can alert users if jobs fail or have not been run yet. GoFlow also collects performance\nmetrics from these CronJobs.\n\nGoFlow also features a comprehensive UI, that allows users to easily create new jobs, view performance of running\njobs, and track the health of the server itself\n\n### Job Information\n\nGoFlow collects all DAG and DAG run information in a database for convenience and backup purposes. This information may\nbe useful for tracking tasks externally.\n\n#### DAGs\n\nTable includes:\n\n- id\n- name\n- namespace\n- version\n- file_path\n- file_format\n- created_date\n- last_updated_date\n\n#### DAGRuns\n\nTable includes:\n\n- dag_id\n- status\n- execution_date\n- start_date\n- end_date\n- last_updated_date\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzbrookle%2Fgoflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzbrookle%2Fgoflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzbrookle%2Fgoflow/lists"}