{"id":36908312,"url":"https://github.com/svaloumas/valet","last_synced_at":"2026-01-12T15:55:38.012Z","repository":{"id":41811728,"uuid":"471752989","full_name":"svaloumas/valet","owner":"svaloumas","description":"Job queuing service and async task runner.","archived":false,"fork":false,"pushed_at":"2023-05-04T17:01:23.000Z","size":1883,"stargazers_count":59,"open_issues_count":2,"forks_count":4,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-11-22T17:20:47.316Z","etag":null,"topics":["async","go","golang","gomodule","hexagonal-architecture","job-queue","job-scheduler","task-runner"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/svaloumas.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-03-19T16:36:59.000Z","updated_at":"2025-10-26T18:02:42.000Z","dependencies_parsed_at":"2024-06-19T02:16:59.925Z","dependency_job_id":"38f561cd-7248-451b-b97d-00d81690c60c","html_url":"https://github.com/svaloumas/valet","commit_stats":null,"previous_names":[],"tags_count":11,"template":false,"template_full_name":null,"purl":"pkg:github/svaloumas/valet","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/svaloumas%2Fvalet","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/svaloumas%2Fvalet/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/svaloumas%2Fvalet/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/svaloumas%2Fvalet/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/svaloumas","download_url":"https://codeload.github.com/svaloumas/valet/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/svaloumas%2Fvalet/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28341887,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-12T15:50:39.657Z","status":"ssl_error","status_checked_at":"2026-01-12T15:49:49.297Z","response_time":98,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["async","go","golang","gomodule","hexagonal-architecture","job-queue","job-scheduler","task-runner"],"created_at":"2026-01-12T15:55:37.896Z","updated_at":"2026-01-12T15:55:37.987Z","avatar_url":"https://github.com/svaloumas.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Valet\n[![CI](https://github.com/svaloumas/valet/actions/workflows/ci.yml/badge.svg)](https://github.com/svaloumas/valet/actions/workflows/ci.yml)\n[![codecov](https://codecov.io/gh/svaloumas/valet/branch/main/graph/badge.svg?token=9CI4Q74JJK)](https://codecov.io/gh/svaloumas/valet)\n[![Go Report Card](https://goreportcard.com/badge/github.com/svaloumas/valet)](https://goreportcard.com/report/github.com/svaloumas/valet)\n[![Go Reference](https://pkg.go.dev/badge/github.com/svaloumas/valet.svg)](https://pkg.go.dev/github.com/svaloumas/valet)\n[![License](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/svaloumas/valet/blob/develop/LICENSE)\n\nStateless Go server responsible for running tasks asynchronously and concurrently.\n\n* [Overview](#overview)\n  * [Job](#job)\n  * [Pipeline](#pipeline)\n* [Architecture](#architecture)\n* [Installation](#installation)\n* [Configuration](#configuration)\n* [Secrets](#secrets)\n* [Usage](#usage)\n* [Tests](#tests)\n\n\u003ca name=\"overview\"/\u003e\n\n## Overview\n\nAt its core, `valet` is a simple job queuing system and an asynchronous task runner. A task is a user-defined `func` that is run as a callback by the service.\n\n\u003ca name=\"job\"/\u003e\n\n### Job\n\nThe implementation uses the notion of `job`, which describes the work that needs to be done and carries information about the task that will run for the\nspecific job. User-defined tasks are assigned to jobs. Every job can be assigned with a different task, a JSON payload with the data required for the task\nto be executed, and an optional timeout interval. Jobs can be scheduled to run at a specified time or instantly.\n\nAfter the tasks have been executed, their results along with the errors (if any) are stored into a storage system.\n\n\u003ca name=\"pipeline\"/\u003e\n\n### Pipeline\n\nA `pipeline` is a sequence of jobs that need to be executed in a specified order, one by one. Every job in the pipeline can be assigned with a different task\nand parameters, and each task callback can optionally use the results of the previous task in the job sequence. A pipeline can also be scheduled to run sometime in the future, or immediately.\n\n\u003ca name=\"architecture\"/\u003e\n\n## Architecture\n\nYour callback functions can live in any repo and should be registered to your own build of `valet`. To unlock this level of flexibility,\nthe service is provided as a Go `pkg` rather than a `cmd`, enabling the task registration before building the executable.\n\nInternally, the service consists of the following components.\n\n* Server - Exposes a RESTful API to enable communication with external services.\n* Job queue - A FIFO queue that supports the job queuing mechanism of the service.\n* Scheduler - Responsible for dispatching the jobs from the job queue to the worker pool and of course for scheduling the tasks for future execution.\n* Worker pool - A number of available go-routines, responsible for the concurrent execution of the jobs.\n* Storage - The storage system where jobs and their results persist.\n\nThe design intends to follow the hexagonal architecture pattern and to support modularity and extendability. \n\nSo far, `valet` provides the following interfaces and can be configured accordingly to function with any of the listed technologies.\n\n#### API\n\n* HTTP - Powered by [gin](https://github.com/gin-gonic/gin). Find the swagger files under `doc/swagger/`.\n* gRPC\n\n#### Storage\n\n* In-memory key-value storage.\n* MySQL\n* PostgreSQL\n* Redis\n\n#### Job queue\n\n* In-memory job queue.\n* RabbitMQ\n* Redis\n\n\u003ca name=\"installation\"/\u003e\n\n## Installation\n\n1. Download the `pkg`.\n\n```bash\ngo get github.com/svaloumas/valet\n```\n\n2. Define your own task functions in your repo by implementing the type `func(...interface{}) (interface{}, error)`.\n\n```go\npackage somepkg\n\nimport (\n\t\"github.com/svaloumas/valet\"\n)\n\n// DummyParams is an example of a task params structure.\ntype DummyParams struct {\n\tURL string `json:\"url,omitempty\"`\n}\n\n// DummyTask is a dummy task callback.\nfunc DummyTask(args ...interface{}) (interface{}, error) {\n\tdummyParams := \u0026DummyParams{}\n\tvar previousResultsMetadata string\n\tvalet.DecodeTaskParams(args, dummyParams)\n\tvalet.DecodePreviousJobResults(args, \u0026resultsMetadata) // Applies to pipelines.\n\n\tmetadata, err := downloadContent(params.URL)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\treturn metadata, nil\n}\n\nfunc downloadContent(URL string) (string, error) {\n\treturn \"some metadata\", nil\n}\n```\n\n\u003e `args[0]` holds the task parameters as they were given through the API call for the job/pipeline creation. `args[1]` holds the results of the \n\u003e previous task only in case of a pipeline execution. Prefer to safely access the arguments by using `valet.DecodeTaskParams` and `valet.DecodePreviousJobResults`\n\u003e to decode them into your custom task param structs.\n\n3. Copy `config.yaml` from the repo and set a configuration according to your needs.\n\n4. Initialize `valet` in a `main` function under your repo, register your tasks to the service and run it.\n\n```bash\nmkdir -p cmd/valetd/\ntouch cmd/valetd/main.go\n```\n\n```go\n// cmd/valetd/main.go\npackage main\n\nimport (\n\t\"github.com/svaloumas/valet\"\n\t\"path/to/somepkg\"\n)\n\nfunc main() {\n\tv := valet.New(\"/path/to/config.yaml\")\n\tv.RegisterTask(\"mytask\", somepkg.DummyTask)\n\tv.Run()\n}\n```\n\n5. Build and run the service.\n\n* To run the service and its dependencies as Docker containers, use the `Dockerfile`, `docker-compose` and `Makefile` files provided.\n\n    ```bash\n    docker-compose up -d --build\n    ```\n\n* Build and run the service as a standalone binary.\n\n    \u003e Optionally set the corresponding environment variables depending on your configuration options.\n\n    ```bash\n    export POSTGRES_DSN=\n    export RABBITMQ_URI=\n    export MYSQL_DSN=\n    export REDIS_URL=\n    ```\n\n    ```bash\n    go build -o valetd cmd/valted/*.go\n    ./valetd\n\n    ```\n\n\u003ca name=\"configuration\"/\u003e\n\n## Configuration\n\nAll configuration is set through `config.yaml`, which lives under the project's root directory.\n\n```yaml\n# Server config section\nserver:\n  protocol: http                    # string - options: http, grpc\n  http:\n    port: 8080                      # int\n  grpc:\n    port: 50051                     # int\n# Job queue config section\njob_queue:\n  option: rabbitmq                  # string - options: memory, rabbitmq, redis\n  memory_job_queue:\n    capacity: 100                   # int\n  rabbitmq:\n    queue_params:\n      queue_name: job               # string\n      durable: false                # boolean\n      deleted_when_unused: false    # boolean\n      exclusive: false              # boolean\n      no_wait: false                # boolean\n    consume_params:\n      name: rabbitmq-consumer       # string\n      auto_ack: true                # boolean\n      exclusive: false              # boolean\n      no_local: false               # boolean\n      no_wait: false                # boolean\n    publish_params:\n      exchange:                     # string\n      routing_key: job              # string\n      mandatory: false              # boolean\n      immediate: false              # boolean\n  redis:\n    key_prefix: valet               # string\n    min_idle_conns: 10              # int\n    pool_size: 10                   # int\n# Workerpool config section\nworker_pool:\n  workers: 4                        # int\n  queue_capacity: 4                 # int\n# Scheduler config section\nscheduler:\n  job_queue_polling_interval: 5     # int\n  storage_polling_interval: 60      # int\n# Storage config section\nstorage:\n  option: memory                    # string - options:  memory, mysql, postgres, redis\n  mysql:\n    connection_max_lifetime:        # int\n    max_idle_connections:           # int\n    max_open_connections:           # int\n  postgres:\n    connection_max_lifetime:        # int\n    max_idle_connections:           # int\n    max_open_connections:           # int\n  redis:\n    key_prefix: valet               # string\n    min_idle_conns: 10              # int\n    pool_size: 10                   # int\n# Global config section\ntimeout_unit: second                # string - options: millisecond, second\nlogging_format: text                # string - options: text, json\n```\n\n\u003ca name=\"secrets\"/\u003e\n\n## Secrets\n\nCurrently, the secrets depending on the configuration are the following: MySQL DSN, PostgreSQL DSN, Redis URL and the RabbitMQ URI.\nEach can be provided as an environment variable. Alternatively, if you choose to use the provided Docker compose files, you can create the\ncorresponding Docker secrets.\n\n| Env variable   | Docker secret        |\n| -------------- | -------------------- |\n| `MYSQL_DSN`    | `valet-mysql-dsn`    |\n| `POSTGRES_DSN` | `valet-postgres-dsn` |\n| `REDIS_URL`    | `valet-redis-url`    |\n| `RABBITMQ_URI` | `valet-rabbitmq-uri` |\n\n\u003ca name=\"usage\"/\u003e\n\n## Usage\n\nCreate a new job by making a POST HTTP call to `/jobs` or via gRPC to `job.Job.Create` service method. You can inject arbitrary parameters for your task to run\nby including them in the request body.\n\n```json\n{\n    \"name\": \"a job\",\n    \"description\": \"what this job is all about, but briefly\",\n    \"task_name\": \"dummytask\",\n    \"task_params\": {\n        \"url\": \"www.some-fake-url.com\"\n    },\n    \"timeout\": 10\n}\n```\n\nTo schedule a new job to run at a specific time, add `run_at` field to the request body.\n\n```json\n{\n    \"name\": \"a scheduled job\",\n    \"description\": \"what this scheduled job is all about, but briefly\",\n    \"task_name\": \"dummytask\",\n    \"run_at\": \"2022-06-06T15:04:05.999\",\n    \"task_params\": {\n        \"url\": \"www.some-fake-url.com\"\n    },\n    \"timeout\": 10\n}\n```\n\nCreate a new pipeline by making a POST HTTP call to `/pipelines` or via gRPC to `pipeline.Pipeline.Create` service method. You can inject arbitrary parameters\nfor your tasks to run by including them in the request body. Optionally, you can tune your tasks to use any results of the previous task in the pipeline, creating\na `bash`-like command pipeline. Pipelines can also be scheduled for execution at a specific time, by adding `run_at` field to the request payload\njust like it's done with the jobs.\n\n```json\n{\n    \"name\": \"a scheduled pipeline\",\n    \"description\": \"what this pipeline is all about\",\n    \"run_at\": \"2022-06-06T15:04:05.999\",\n    \"jobs\": [\n        {\n            \"name\": \"the first job\",\n            \"description\": \"some job description\",\n            \"task_name\": \"dummytask\",\n            \"task_params\": {\n                \"url\": \"www.some-fake-url.com\"\n            }\n        },\n        {\n            \"name\": \"a second job\",\n            \"description\": \"some job description\",\n            \"task_name\": \"anothertask\",\n            \"task_params\": {\n                \"url\": \"www.some-fake-url.com\"\n            },\n            \"use_previous_results\": true,\n            \"timeout\": 10\n        },\n        {\n            \"name\": \"the last job\",\n            \"description\": \"some job description\",\n            \"task_name\": \"dummytask\",\n            \"task_params\": {\n                \"url\": \"www.some-fake-url.com\"\n            },\n            \"use_previous_results\": true\n        }\n    ]\n}\n```\n\n\u003ca name=\"tests\"/\u003e\n\n## Tests\n\nRun the complete test suite.\n\n```bash\ndocker-compose up -d\nmake test\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsvaloumas%2Fvalet","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsvaloumas%2Fvalet","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsvaloumas%2Fvalet/lists"}