{"id":13413238,"url":"https://github.com/fieldryand/goflow","last_synced_at":"2025-03-14T19:31:39.851Z","repository":{"id":38327676,"uuid":"249262734","full_name":"fieldryand/goflow","owner":"fieldryand","description":"Simple but powerful DAG scheduler and dashboard","archived":false,"fork":false,"pushed_at":"2024-12-08T12:28:30.000Z","size":969,"stargazers_count":395,"open_issues_count":10,"forks_count":31,"subscribers_count":7,"default_branch":"main","last_synced_at":"2024-12-08T13:26:23.418Z","etag":null,"topics":["airflow","dashboard","directed-acyclic-graph","go","job-scheduler","schedule","workflow-engine"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fieldryand.png","metadata":{"files":{"readme":"Readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-03-22T20:03:31.000Z","updated_at":"2024-12-04T03:02:07.000Z","dependencies_parsed_at":"2023-02-17T22:46:20.708Z","dependency_job_id":"4469b6fc-0c5c-4971-89e1-a55cf2f56498","html_url":"https://github.com/fieldryand/goflow","commit_stats":null,"previous_names":[],"tags_count":43,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fieldryand%2Fgoflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fieldryand%2Fgoflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fieldryand%2Fgoflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fieldryand%2Fgoflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fieldryand","download_url":"https://codeload.github.com/fieldryand/goflow/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243635276,"owners_count":20322909,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","dashboard","directed-acyclic-graph","go","job-scheduler","schedule","workflow-engine"],"created_at":"2024-07-30T20:01:35.835Z","updated_at":"2025-03-14T19:31:39.446Z","avatar_url":"https://github.com/fieldryand.png","language":"Go","readme":"![Build Status](https://github.com/fieldryand/goflow/actions/workflows/go.yml/badge.svg)\n[![codecov](https://codecov.io/gh/fieldryand/goflow/branch/master/graph/badge.svg)](https://codecov.io/gh/fieldryand/goflow)\n[![Go Report Card](https://goreportcard.com/badge/github.com/fieldryand/goflow)](https://goreportcard.com/report/github.com/fieldryand/goflow)\n[![GoDoc](https://pkg.go.dev/badge/github.com/fieldryand/goflow/v2?status.svg)](https://pkg.go.dev/github.com/fieldryand/goflow/v2?tab=doc)\n[![Release](https://img.shields.io/github/v/release/fieldryand/goflow)](https://github.com/fieldryand/goflow/releases)\n\n# Goflow\n\nA simple but powerful DAG scheduler and dashboard, written in Go.\n\n![goflow-demo](https://user-images.githubusercontent.com/3333324/147818084-ade84547-4404-4d58-a697-c18ecb06fd30.gif)\n\n------\n\n**Use it if:**\n- you need a directed acyclic graph (DAG) scheduler like Apache Airflow, but without the complexity.\n- you have a variety of clusters or services performing heavy computations and you want something small and light to orchestrate them.\n- you want a monitoring dashboard.\n- you want the easiest possible deployment with a single binary or container, saving you time. Volume mounts etc are too much headache.\n- you want it to run on a single tiny VM, saving on cloud costs.\n- you want to choose your storage technology--embedded, Postgres, Redis, S3, DynamoDB or something else.\n- you prefer to define your DAGs with code rather than configuration files. This approach can make it easier to manage complex DAGs.\n\n**Don't use it if:**\n- you need to queue a huge number of tasks. Goflow is not tested at massive scale and does not support horizontal scaling.\n\n## Contents\n\n- [Quick start](#quick-start)\n   - [With Docker](#with-docker)\n   - [Without Docker](#without-docker)\n- [Development overview](#development-overview)\n   - [Jobs and tasks](#jobs-and-tasks)\n   - [Custom Operators](#custom-operators)\n   - [Retries](#retries)\n   - [Task dependencies](#task-dependencies)\n   - [Trigger rules](#trigger-rules)\n   - [The Goflow engine](#the-goflow-engine)\n   - [Available operators](#available-operators)\n- [Storage](#storage)\n- [API and integration](#api-and-integration)\n\n## Quick start\n\n### With Docker\n\n```shell\ndocker run -p 8181:8181 ghcr.io/fieldryand/goflow-example:latest\n```\n\nCheck out the dashboard at `localhost:8181`.\n\n### Without Docker\n\nIn a fresh project directory:\n\n```shell\ngo mod init # create a new module\ngo get github.com/fieldryand/goflow/v2 # install dependencies\n```\n\nCreate a file `main.go` with contents:\n```go\npackage main\n\nimport \"github.com/fieldryand/goflow/v2\"\n\nfunc main() {\n        options := goflow.Options{\n                UIPath: \"ui/\",\n                ShowExamples:  true,\n                WithSeconds:  true,\n        }\n        gf := goflow.New(options)\n        gf.Use(goflow.DefaultLogger())\n        gf.Run(\":8181\")\n}\n```\n\nDownload and untar the dashboard:\n\n```shell\nwget https://github.com/fieldryand/goflow/releases/latest/download/goflow-ui.tar.gz\ntar -xvzf goflow-ui.tar.gz\nrm goflow-ui.tar.gz\n```\n\nNow run the application with `go run main.go` and see it in the browser at localhost:8181.\n\n## Development overview\n\nFirst a few definitions.\n\n- `Job`: A Goflow workflow is called a `Job`. Jobs can be scheduled using cron syntax.\n- `Task`: Each job consists of one or more tasks organized into a dependency graph. A task can be run under certain conditions; by default, a task runs when all of its dependencies finish successfully.\n- Concurrency: Jobs and tasks execute concurrently.\n- `Operator`: An `Operator` defines the work done by a `Task`. Goflow comes with a handful of basic operators, and implementing your own `Operator` is straightforward.\n- Retries: You can allow a `Task` a given number of retry attempts. Goflow comes with two retry strategies, `ConstantDelay` and `ExponentialBackoff`.\n- Streaming: Goflow uses server-sent events to stream the status of jobs and tasks to the dashboard in real time.\n\n### Jobs and tasks\n\nLet's start by creating a function that returns a job called `my-job`. There is a single task in this job that sleeps for one second.\n\n```go\npackage main\n\nimport (\n\t\"errors\"\n\n\t\"github.com/fieldryand/goflow/v2\"\n)\n\nfunc myJob() *goflow.Job {\n\tj := \u0026goflow.Job{Name: \"my-job\", Schedule: \"* * * * *\", Active: true}\n\tj.Add(\u0026goflow.Task{\n\t\tName:     \"sleep-for-one-second\",\n\t\tOperator: goflow.Command{Cmd: \"sleep\", Args: []string{\"1\"}},\n\t})\n\treturn j\n}\n```\n\nBy setting `Active: true`, we are telling Goflow to apply the provided cron schedule for this job when the application starts.\nJob scheduling can be activated and deactivated from the dashboard.\n\n### Custom operators\n\nA custom `Operator` needs to implement the `Run` method. Here's an example of an operator that adds two positive numbers.\n\n```go\ntype PositiveAddition struct{ a, b int }\n\nfunc (o PositiveAddition) Run() (interface{}, error) {\n\tif o.a \u003c 0 || o.b \u003c 0 {\n\t\treturn 0, errors.New(\"Can't add negative numbers\")\n\t}\n\tresult := o.a + o.b\n\treturn result, nil\n}\n```\n\n### Retries\n\nLet's add a retry strategy to the `sleep-for-one-second` task:\n\n```go\nfunc myJob() *goflow.Job {\n\tj := \u0026goflow.Job{Name: \"my-job\", Schedule: \"* * * * *\"}\n\tj.Add(\u0026goflow.Task{\n\t\tName:       \"sleep-for-one-second\",\n\t\tOperator:   goflow.Command{Cmd: \"sleep\", Args: []string{\"1\"}},\n\t\tRetries:    5,\n\t\tRetryDelay: goflow.ConstantDelay{Period: 1},\n\t})\n\treturn j\n}\n```\n\nInstead of `ConstantDelay`, we could also use `ExponentialBackoff` (see https://en.wikipedia.org/wiki/Exponential_backoff).\n\n### Task dependencies\n\nA job can define a directed acyclic graph (DAG) of independent and dependent tasks. Let's use the `SetDownstream` method to\ndefine two tasks that are dependent on `sleep-for-one-second`. The tasks will use the `PositiveAddition` operator we defined earlier,\nas well as a new operator provided by Goflow, `Get`.\n\n```go\nfunc myJob() *goflow.Job {\n\tj := \u0026goflow.Job{Name: \"my-job\", Schedule: \"* * * * *\"}\n\tj.Add(\u0026goflow.Task{\n\t\tName:       \"sleep-for-one-second\",\n\t\tOperator:   goflow.Command{Cmd: \"sleep\", Args: []string{\"1\"}},\n\t\tRetries:    5,\n\t\tRetryDelay: goflow.ConstantDelay{Period: 1},\n\t})\n\tj.Add(\u0026goflow.Task{\n\t\tName:       \"get-google\",\n\t\tOperator:   goflow.Get{Client: \u0026http.Client{}, URL: \"https://www.google.com\"},\n\t})\n\tj.Add(\u0026goflow.Task{\n\t\tName:       \"add-two-plus-three\",\n\t\tOperator:   PositiveAddition{a: 2, b: 3},\n\t})\n\tj.SetDownstream(j.Task(\"sleep-for-one-second\"), j.Task(\"get-google\"))\n\tj.SetDownstream(j.Task(\"sleep-for-one-second\"), j.Task(\"add-two-plus-three\"))\n\treturn j\n}\n```\n\n### Trigger rules\n\nBy default, a task has the trigger rule `allSuccessful`, meaning the task starts executing when all the tasks directly\nupstream exit successfully. If any dependency exits with an error, all downstream tasks are skipped, and the job exits with an error.\n\nSometimes you want a downstream task to execute even if there are upstream failures. Often these are situations where you want\nto perform some cleanup task, such as shutting down a server. In such cases, you can give a task the trigger rule `allDone`.\n\nLet's modify `sleep-for-one-second` to have the trigger rule `allDone`.\n\n\n```go\nfunc myJob() *goflow.Job {\n\t// other stuff\n\tj.Add(\u0026goflow.Task{\n\t\tName:        \"sleep-for-one-second\",\n\t\tOperator:    goflow.Command{Cmd: \"sleep\", Args: []string{\"1\"}},\n\t\tRetries:     5,\n\t\tRetryDelay:  goflow.ConstantDelay{Period: 1},\n\t\tTriggerRule: \"allDone\",\n\t})\n\t// other stuff\n}\n```\n\n### The Goflow Engine\n\nFinally, let's create a Goflow engine, register our job, attach a logger, and run the application.\n\n```go\nfunc main() {\n\tgf := goflow.New(goflow.Options{Streaming: true})\n\tgf.AddJob(myJob)\n\tgf.Use(goflow.DefaultLogger())\n\tgf.Run(\":8181\")\n}\n```\n\nYou can pass different options to the engine. Options currently supported:\n- `Store`: This is [described in more detail below.](#storage)\n- `UIPath`: The path to the dashboard code. The default value is an empty string, meaning Goflow serves only the API and not the dashboard. Suggested value if you want the dashboard: `ui/`\n- `ShowExamples`: Whether to show the example jobs. Default value: `false`\n- `WithSeconds`: Whether to include the seconds field in the cron spec. See the [cron package documentation](https://github.com/robfig/cron) for details. Default value: `false`\n\nGoflow is built on the [Gin framework](https://github.com/gin-gonic/gin), so you can pass any Gin handler to `Use`.\n\n### Available operators\n\nGoflow provides several operators for common tasks. [See the package documentation](https://pkg.go.dev/github.com/fieldryand/goflow) for details on each.\n\n- `Command` executes a shell command.\n- `Get` makes a GET request.\n- `Post` makes a POST request.\n\n## Storage\n\nFor persisting your job execution history, Goflow allows you to plug in many different key-value stores thanks to the [excellent gokv package](https://github.com/philippgille/gokv/). This way you can recover from a crash or deploy a new version of your app without losing your data.\n\n\u003e Note: the gokv API is not yet stable. Goflow has been tested against v0.6.0.\n\nBy default, Goflow uses an in-memory database, but you can easily replace it with Postgres, Redis, S3 or any other `gokv.Store`. Here is an example:\n\n```go\npackage main\n\nimport \"github.com/fieldryand/goflow/v2\"\nimport \"github.com/philippgille/gokv/redis\"\n\nfunc main() {\n        // create a storage client\n        client, err := redis.NewClient(redis.DefaultOptions)\n        if err != nil {\n                panic(err)\n        }\n        defer client.Close()\n\n        // pass the client as a Goflow option\n        options := goflow.Options{\n                Store: client,\n                UIPath: \"ui/\",\n                Streaming: true,\n                ShowExamples:  true,\n        }\n        gf := goflow.New(options)\n        gf.Use(goflow.DefaultLogger())\n        gf.Run(\":8181\")\n}\n```\n\n\n## API and integration\n\nYou can use the API to integrate Goflow with other applications, such as an existing dashboard. Here is an overview of available endpoints:\n- `GET /api/health`: Check health of the service\n- `GET /api/jobs`: List registered jobs\n- `GET /api/jobs/{jobname}`: Get the details for a given job\n- `GET /api/executions`: Query and list job executions\n- `POST /api/jobs/{jobname}/submit`: Submit a job for execution\n- `POST /api/jobs/{jobname}/toggle`: Toggle a job schedule on or off\n- `/stream`: This endpoint returns Server-Sent Events with a `data` payload matching the one returned by `/api/executions`. The dashboard that ships with Goflow uses this endpoint.\n\nCheck out the OpenAPI spec for more details. Easiest way is to clone the repo, then within the repo use Swagger as in the following:\n\n```shell\ndocker run -p 8080:8080 -e SWAGGER_JSON=/app/swagger.json -v $(pwd):/app swaggerapi/swagger-ui\n```\n","funding_links":[],"categories":["Job Scheduler","Go","Full fledged product","作业调度器"],"sub_categories":["Search and Analytic Databases","检索及分析资料库","Advanced Console UIs"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffieldryand%2Fgoflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffieldryand%2Fgoflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffieldryand%2Fgoflow/lists"}