Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/runabol/tork
A distributed workflow engine
https://github.com/runabol/tork
background-jobs distributed docker go golang job-queue task-queue work-queue workflow-engine
Last synced: 4 days ago
JSON representation
A distributed workflow engine
- Host: GitHub
- URL: https://github.com/runabol/tork
- Owner: runabol
- License: mit
- Created: 2023-05-24T01:17:07.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-05-10T12:57:07.000Z (6 months ago)
- Last Synced: 2024-05-10T13:27:50.855Z (6 months ago)
- Topics: background-jobs, distributed, docker, go, golang, job-queue, task-queue, work-queue, workflow-engine
- Language: Go
- Homepage: https://tork.run
- Size: 7.47 MB
- Stars: 480
- Watchers: 7
- Forks: 22
- Open Issues: 18
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-ccamel - runabol/tork - A distributed workflow engine (Go)
README
Features •
Installation •
Documentation •
Quick Start •
REST API •
Web UI
Tork is a highly-scalable, general-purpose workflow engine.
## Features:
- [REST API](https://www.tork.run/rest)
- [Highly extensible](https://www.tork.run/extend)
- Horizontally scalable
- Task isolation - tasks are executed within a container to provide isolation, idempotency, and in order to enforce resource [limits](https://www.tork.run/tasks#limits)
- Automatic recovery of tasks in the event of a worker crash
- Supports both stand-alone and [distributed](https://www.tork.run/installation#running-in-a-distributed-mode) setup
- Retry failed tasks
- [Middleware](https://www.tork.run/customize#middleware)
- [Pre/Post tasks](https://www.tork.run/tasks#pre-post-tasks)
- No single point of failure
- Task timeout
- [Full-text search](https://www.tork.run/rest#list-jobs)
- [Expression Language](https://www.tork.run/tasks#expressions)
- [Conditional Tasks](https://www.tork.run/tasks#expressions)
- [Parallel Tasks](https://www.tork.run/tasks#parallel-task)
- [For-Each Task](https://www.tork.run/tasks#each-task)
- [Subjob Task](https://www.tork.run/tasks#sub-job-task)
- [Task Priority](https://www.tork.run/tasks#priority)
- [Sandbox Mode](https://www.tork.run/runtime#sandbox-mode-experimental)
- [Secrets](https://www.tork.run/tasks#secrets)
- [Web UI](https://www.tork.run/web-ui)
## Documentation
See [tork.run](https://tork.run) for the full documentation.
## Quick Start
1. Ensure you have [Docker](https://docs.docker.com/get-docker/) with API Version >= 1.42 (use `docker version | grep API` to check).
2. Download the binary for your system from the [releases](https://github.com/runabol/tork/releases/latest) page.
### Hello World
Start in `standalone` mode:
```
./tork run standalone
```
Submit a job in another terminal:
```yaml
# hello.yaml
---
name: hello job
tasks:
- name: say hello
image: ubuntu:mantic #docker image
run: |
echo -n hello world
- name: say goodbye
image: ubuntu:mantic
run: |
echo -n bye world
```
```bash
JOB_ID=$(curl \
-s \
-X POST \
--data-binary @hello.yaml \
-H "Content-type: text/yaml" \
http://localhost:8000/jobs | jq -r .id)
```
Query for the status of the job:
```bash
curl -s http://localhost:8000/jobs/$JOB_ID | jq .
{
"id": "ed0dba93d262492b8cf26e6c1c4f1c98",
"state": "COMPLETED",
...
"execution": [
{
...
"state": "COMPLETED",
}
],
}
```
### A slightly more interesting example
The following job:
1. Downloads a remote video file using a `pre` task to a shared `/tmp` volume.
2. Converts the first 5 seconds of the downloaded video using `ffmpeg`.
3. Uploads the converted video to a destination using a `post` task.
```yaml
# convert.yaml
---
name: convert a video
inputs:
source: https://upload.wikimedia.org/wikipedia/commons/1/18/Big_Buck_Bunny_Trailer_1080p.ogv
tasks:
- name: convert the first 5 seconds of a video
image: jrottenberg/ffmpeg:3.4-alpine
run: |
ffmpeg -i /tmp/input.ogv -t 5 /tmp/output.mp4
mounts:
- type: volume
target: /tmp
pre:
- name: download the remote file
image: alpine:3.18.3
env:
SOURCE_URL: "{{ inputs.source }}"
run: |
wget \
$SOURCE_URL \
-O /tmp/input.ogv
post:
- name: upload the converted file
image: alpine:3.18.3
run: |
wget \
--post-file=/tmp/output.mp4 \
https://devnull-as-a-service.com/dev/null
```
Submit the job in another terminal:
```
JOB_ID=$(curl \
-s \
-X POST \
--data-binary @convert.yaml \
-H "Content-type: text/yaml" \
http://localhost:8000/jobs | jq -r .id)
```
### More examples
Check out the [examples](examples/) folder.
## REST API
See the [REST API](https://www.tork.run/rest) documentation.
### Swagger Docs
Make sure you have CORS configured in your [config file](https://www.tork.run/config):
```toml
[middleware.web.cors]
enabled = true
```
Start Tork in `standalone` or `coordinator` mode.
```shell
go run cmd/main.go run standalone
```
Serve the Swagger Docs
```shell
docker compose up -d swagger
```
Visit [http://localhost:9000](http://localhost:9000)
## Web UI
[Tork Web](https://www.tork.run/web-ui) is a web based tool for interacting with Tork.
## License
Copyright (c) 2023-present Arik Cohen. Tork is free and open-source software licensed under the MIT License.