{"id":48940504,"url":"https://github.com/eproxus/gaffer","last_synced_at":"2026-04-27T15:00:41.355Z","repository":{"id":346828174,"uuid":"1191786807","full_name":"eproxus/gaffer","owner":"eproxus","description":"A reliable job queue implemented in Erlang","archived":false,"fork":false,"pushed_at":"2026-04-17T12:23:57.000Z","size":297,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-17T13:39:12.818Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Erlang","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/eproxus.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.md","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2026-03-25T15:30:09.000Z","updated_at":"2026-04-17T12:24:01.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/eproxus/gaffer","commit_stats":null,"previous_names":["eproxus/gaffer"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/eproxus/gaffer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eproxus%2Fgaffer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eproxus%2Fgaffer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eproxus%2Fgaffer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eproxus%2Fgaffer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/eproxus","download_url":"https://codeload.github.com/eproxus/gaffer/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eproxus%2Fgaffer/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32341455,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-26T23:26:28.701Z","status":"online","status_checked_at":"2026-04-27T02:00:06.769Z","response_time":128,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-04-17T13:30:31.996Z","updated_at":"2026-04-27T15:00:41.349Z","avatar_url":"https://github.com/eproxus.png","language":"Erlang","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003c!-- markdownlint-disable-line MD013 --\u003e\n# gaffer [![CI Status][ci-img]][ci] [![Hex.pm Version][hex-img]][hex] [![Docs][docs-img]][docs] [![Minimum Erlang Version][erlang-img]][erlang] [![License][license-img]][license]\n\n[ci]:          https://github.com/eproxus/gaffer/actions/workflows/ci.yml?query=branch%3Amain\n[ci-img]:      https://img.shields.io/github/actions/workflow/status/eproxus/gaffer/ci.yml?label=ci\n[hex]:         https://hex.pm/packages/gaffer\n[hex-img]:     https://img.shields.io/hexpm/v/gaffer\n[docs]:        https://hexdocs.pm/gaffer\n[docs-img]:    https://img.shields.io/badge/docs-hexdocs-blue\n[erlang]:      https://github.com/eproxus/gaffer/blob/main/mise.toml\n[erlang-img]:  https://img.shields.io/badge/erlang-28+-blue.svg\n[license]:     LICENSE.md\n[license-img]: https://img.shields.io/badge/license-MIT-blue.svg\n\nA reliable job queue implemented in Erlang.\n\n## Features\n\n- [x] Priority-based execution\n- [x] Per-queue concurrency limits (local and global)\n- [x] Pluggable storage drivers (ETS for dev/test, Postgres for production)\n- [x] Hooks for queue and job events\n- [x] Dead-letter queues (`on_discard`)\n- [x] Queue introspection and automatic/manual job pruning\n- [x] Delayed job scheduling\n- [x] Automatic retries with backoff\n- [ ] Drain and flush (graceful shutdown)\n- [x] Job execution timeouts\n- [ ] Worker shutdown timeouts\n\n## Usage\n\n### Shell\n\nFor simple jobs, pass an anonymous function as a worker:\n\n```erlang\n1\u003e ok = gaffer:ensure_queue(#{\n       name =\u003e greetings,\n       driver =\u003e ets,\n       worker =\u003e fun(#{payload := #{~\"name\" := Name}}) -\u003e\n           io:format(~\"Hello, ~s!~n\", [Name]),\n           complete\n       end\n   }).\nok\n2\u003e gaffer:insert(greetings, #{~\"name\" =\u003e ~\"world\"}).\n#{id =\u003e \u003c\u003c...\u003e\u003e, queue =\u003e greetings, state =\u003e available, ...}\nHello, world!\n```\n\n### Application\n\n#### Define a worker\n\nImplement the `gaffer_worker` behaviour:\n\n```erlang\n-module(email_sender).\n-behaviour(gaffer_worker).\n-export([perform/1]).\n\nperform(#{payload := #{~\"to\" := To, ~\"body\" := Body}}) -\u003e\n    logger:info(~\"Sending email to ~s: ~s\", [To, Body]),\n    complete.\n```\n\nThe `perform/1` callback can return:\n\n- `complete` - mark the job as completed\n- `{complete, Result}` - complete with a result\n- `{fail, Reason}` - fail and retry (up to `max_attempts`)\n- `{cancel, Reason}` - cancel the job permanently\n- `{schedule, Timestamp}` - reschedule the job for later\n\nCrashes are treated as failures and their reason recorded.\n\n#### Create a queue\n\n```erlang\nDriver = gaffer_driver_pgo:start(#{\n    pool =\u003e my_pool,\n    start =\u003e #{host =\u003e ~\"localhost\", database =\u003e ~\"my_app\", pool_size =\u003e 5}\n}),\ngaffer:ensure_queue(#{\n    name =\u003e emails,\n    driver =\u003e {gaffer_driver_pgo, Driver},\n    worker =\u003e email_sender\n}).\n```\n\n#### Insert a job\n\n```erlang\nJob = gaffer:insert(emails, #{~\"to\" =\u003e ~\"user@example.com\", ~\"body\" =\u003e ~\"Welcome!\"}).\n```\n\n## Configuration\n\nQueues are configured via `gaffer:queue_conf()` maps:\n\n- `name` (`atom()`, **required**)\n\n  Queue identifier.\n\n- `worker` (`module() | fun/1`, **required**)\n\n  Worker callback module or function.\n\n- `driver` (`{module(), state()}`)\n\n  Storage driver.\n\n- `max_workers` (`pos_integer()`, default = `1`)\n\n  Max concurrent workers per node.\n\n- `global_max_workers` (`pos_integer() | infinity`, default = `infinity`)\n\n  Max concurrent workers across all nodes.\n\n- `poll_interval` (`pos_integer() | infinity`, default = `1000`).\n\n  Polling interval in ms.\n\n- `max_attempts` (`pos_integer()`, default = `3`).\n\n  Max execution attempts.\n\n- `timeout` (`pos_integer()`, default = `30000`).\n\n  Execution timeout in ms.\n\n- `backoff` (`[non_neg_integer()]`, default = `[1000]`).\n\n  Retry backoff schedule in ms.\n\n- `priority` (`integer()`, default = `0`).\n\n  Default job priority. Can be negative. Jobs with higher values are claimed\n  first.\n\n- `shutdown_timeout` (`pos_integer()`, default = `5000`).\n\n  Worker shutdown grace period in ms.\n\n- `on_discard` (`atom()`).\n\n  Dead-letter queue name.\n\n- `hooks` (`[hook()]`, default = `[]`).\n\n  Hook modules or funs called after queue and job events. See [Hooks](#hooks).\n\n- `prune` (`prune_conf()`)\n\n  Pruning configuration. A per-queue pruner periodically deletes jobs in\n  terminal states older than the configured max age.\n\n    - `interval` (`pos_integer() | infinity`)\n\n      Prune interval in ms.\n\n    - `max_age` (`#{job_state() | '_' =\u003e age()}`)\n\n      Per-state max age in milliseconds. `infinity` means never prune. `'_'` sets\n      a default for all states.\n\n      Default: `completed`, `failed`, and `cancelled` jobs are pruned\n      immediately, others are kept indefinitely.\n\n## Hooks\n\nGaffer notifies registered hooks after queue and job events. Each hook\nreceives an event path (a list of atoms) and a payload map carrying an `actor`\nfield that identifies which Gaffer process or public API call caused the\nevent.\n\nHooks can be registered per queue via the `hooks` configuration option or\nglobally via the `gaffer` application's `hooks` environment variable.\n\nSee [`gaffer_hooks`](https://hexdocs.pm/gaffer/gaffer_hooks.html) for the full\nlist of events and their payload shapes.\n\n## Changelog\n\nSee the [Releases](https://github.com/eproxus/gaffer/releases) page.\n\n## Code of Conduct\n\nFind this project's code of conduct in [Contributor Covenant Code of Conduct](CODE_OF_CONDUCT.md).\n\n## Contributing\n\nFirst of all, thank you for contributing with your time and energy.\n\nIf you want to request a new feature make sure to [open an issue](https://github.com/eproxus/gaffer/issues/new?template=feature_request.md)\nso we can discuss it first.\n\nBug reports and questions are also welcome, but do check you're using the latest version of the\napplication - if you found a bug - and/or search the issue database - if you have a question,\nsince it might have already been answered before.\n\nContributions will be subject to the MIT License. You will retain the copyright.\n\nFor more information check out [CONTRIBUTING.md](CONTRIBUTING.md).\n\n## Security\n\nThis project's security policy is made explicit in [SECURITY.md](SECURITY.md).\n\n## Conventions\n\n### Versions\n\nThis project adheres to\n[Semantic Versioning](https://semver.org/spec/v2.0.0.html).\n\n### License\n\nThis project uses the [MIT License](LICENSE.md).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feproxus%2Fgaffer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Feproxus%2Fgaffer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feproxus%2Fgaffer/lists"}