Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/sbstnerhrdt/go-job-dispatcher
A golang job queue / dispatcher using sql and redis
https://github.com/sbstnerhrdt/go-job-dispatcher
go golang job-dispatcher job-queue job-scheduler mysql postgresql redis
Last synced: about 1 month ago
JSON representation
A golang job queue / dispatcher using sql and redis
- Host: GitHub
- URL: https://github.com/sbstnerhrdt/go-job-dispatcher
- Owner: SbstnErhrdt
- License: apache-2.0
- Created: 2021-09-24T13:29:44.000Z (about 3 years ago)
- Default Branch: master
- Last Pushed: 2023-10-14T04:29:22.000Z (about 1 year ago)
- Last Synced: 2024-06-20T16:33:19.861Z (6 months ago)
- Topics: go, golang, job-dispatcher, job-queue, job-scheduler, mysql, postgresql, redis
- Language: Go
- Homepage:
- Size: 140 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Job-Dispatcher
This repository contains the code of an `atomic like` jobs dispatcher. It is backed with sql database or with a redis
key-value store.
## Status
Work in progress
## Features
* Counts the attempts of jobs
* Heartbeat to store the current status of the job
* Priority
* Multi-Instance Jobs
## Dependencies
* SQL
* Redis
# Environment Variables
```
# SQL Database
SQL_TYPE=MYSQL
SQL_HOST=localhost
SQL_USER=root
SQL_PASSWORD=test
SQL_PORT=3306
SQL_DATABASE=test
# Clean jobs after n seconds / minutes ... SQL INTERVAL
CLEAN_STALLED_JOBS_INTERVAL=20 SECOND
# Application port
PORT=18989
# Job dispatcher prefix for redis
JOB_DISPATCHER_REDIS_PREFIX=job_dispatcher
# Redis Key Value Store
REDIS_HOST=192.168.157.33
REDIS_PORT=6379
REDIS_PASSWORD=
REDIS_DATABASE=0
```
# Tests
Run the tests via the script
```
sh tests.sh
```
or via the golang command
```
go test ./...
```
To test the redis version just add the following environment variable
```
export TEST_ENV=REDIS
```
# Client
Create a new job
```go
// create uuid
uid := uuid.New()
// init job dispatcher client
client := job_dispatcher.NewClient(
https: //job-dispatcher.endpoint.net,
"web-crawler",
uid,
)
// create new job
newJob := job_dispatcher.NewJobDTO{
Name: "Search",
Priority: 10,
WorkerInstance: "web-crawler",
Tasks: []job_dispatcher.JobTask{
{
Version: "0.1",
Name: "daily search",
Type: "search",
Execute: map[string]Interface{}{
"Name": "Company Name",
},
},
},
}
// send job to dispatcher
res, err := client.CreateJob(newJob)
```
Receive the latest job in the queue
```go
// create uuid
uid := uuid.New()
// init job dispatcher client
client := job_dispatcher.NewClient(
https: //job-dispatcher.endpoint.net,
"web-crawler",
uid,
)
// get the latest job from the queue
err := client.GetJob("job-type")
if err != nil {
return
}
// start the job
err := client.StartCurrentJob()
// send a heartbeat with metrics of the job to the queue
err := client.HeartBeat(...)
// release the current job if something fails
_ = client.ReleaseCurrentJob()
// complete the jobs
_ := client.MarkCurrentJobAsCompleted()
```
# Deployment
Via docker
````shell
docker run -p 8080:8080 ese7en/go-job-dispatcher
````
* [Kubernetes](deployments/kubernetes)
* [Docker-Compose](deployments/kubernetes)