{"id":16445304,"url":"https://github.com/lloydmeta/tasques","last_synced_at":"2025-03-23T08:32:14.300Z","repository":{"id":37178334,"uuid":"240384309","full_name":"lloydmeta/tasques","owner":"lloydmeta","description":"Background Tasks system backed by Elasticsearch with support for language agnostic Workers.","archived":false,"fork":false,"pushed_at":"2023-01-20T22:27:34.000Z","size":1029,"stargazers_count":9,"open_issues_count":15,"forks_count":3,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-18T19:29:27.854Z","etag":null,"topics":["api","apm","background-jobs","background-tasks","background-worker","distributed-systems","docker","elasticsearch","go","golang","job-queue","job-scheduler","k8s","kibana","kubernetes","language-agnostic","rest","restful-api","stateless","swagger"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lloydmeta.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-02-13T22:58:27.000Z","updated_at":"2023-03-13T12:24:04.000Z","dependencies_parsed_at":"2023-02-12T06:15:19.007Z","dependency_job_id":null,"html_url":"https://github.com/lloydmeta/tasques","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lloydmeta%2Ftasques","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lloydmeta%2Ftasques/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lloydmeta%2Ftasques/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lloydmeta%2Ftasques/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lloydmeta","download_url":"https://codeload.github.com/lloydmeta/tasques/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245078067,"owners_count":20557274,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api","apm","background-jobs","background-tasks","background-worker","distributed-systems","docker","elasticsearch","go","golang","job-queue","job-scheduler","k8s","kibana","kubernetes","language-agnostic","rest","restful-api","stateless","swagger"],"created_at":"2024-10-11T09:43:57.878Z","updated_at":"2025-03-23T08:32:13.523Z","avatar_url":"https://github.com/lloydmeta.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Tasques [![Build Status](https://travis-ci.org/lloydmeta/tasques.svg?branch=master)](https://travis-ci.org/lloydmeta/tasques) [![codecov](https://codecov.io/gh/lloydmeta/tasques/branch/master/graph/badge.svg)](https://codecov.io/gh/lloydmeta/tasques) [![Go Report Card](https://goreportcard.com/badge/github.com/lloydmeta/tasques)](https://goreportcard.com/report/github.com/lloydmeta/tasques) [![](https://images.microbadger.com/badges/commit/lloydmeta/tasques.svg)](https://microbadger.com/images/lloydmeta/tasques \"tasques docker image details\")\n\n\u003e Task queues backed by Elasticsearch (ES): Tasques \n\u003e\n\u003e Pronounced: /tɑːsks/, like \"tasks\"\n\u003e \n\u003e You know, for Background Tasks !\n\nWhy use ES as a Tasks data store? It's horizontally scalable, highly-available, and offers a lot of built-in ways to manage your data\n(lifecycle management, snapshots, etc).\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"dashboard.png\" width=\"500\"  alt=\"dashboard\"/\u003e\n\u003c/p\u003e\n\n### Features:\n\n- Easily scalable:\n  - Servers are stateless; just spin more up as needed\n  - The storage engine is Elasticsearch, nuff' said.\n- Tasks are configurable:\n  - Priority\n  - Schedule to run later\n  - Retries with exponential increase in retry delays.\n- Idempotency\n- Recurring Tasks that are repeatedly enqueued at configurable intervals (cron format with basic macro support à la \n  `@every 1m`)\n  - Also supports skipping if outstanding Tasks exist for a given Recurring Task\n- Timeouts for Tasks that are picked up by Workers but either don't report in or finish on time.\n- Archiving of completed Tasks (DONE or DEAD), also configurable\n  - If Index Lifecycle Management (ILM) is enabled (default), the archive index is set to roll over automatically for \n    easy management of old data. \n- Unclaiming allows Tasks that get picked up but can't be handled to be requeued without consequence.\n- API is exposed as Swagger; easily generate clients in any language:\n  - Use the client to enqueue Tasks from your application\n  - Workers are just a loop around the client, then your own business logic to do the actual work.\n- Pre-seeded Kibana Index Patterns and Dashboards for monitoring tasks.\n- Simple configuration: use a config file, optionally override with environment variables (12 factor-ready).\n- Application Performance monitoring: metrics are exported to [APM](https://www.elastic.co/apm) and available again from\n  Kibana (more below)\n\n### Usage\n\n#### Running\n\nTasques is available as a [small Docker image](https://hub.docker.com/repository/docker/lloydmeta/tasques), with images\npublished automatically by CI upon pushes/merges to `master` and tag pushes.\n\nTo get a quick idea of what is included and how to get up and running with Kubernetes:\n\n1. Go to `docker/k8s` and run `make install-eck deploy`, and wait until the pods are all ready (`kubectl get pods`)\n2. For Swagger, go to [localhost:8080/swagger/index.html](http://localhost:8080/swagger/index.html)\n    ![Swagger](swagger.png)\n3. To log into Kibana for dashboards and APM stats, run `make show-credentials` to get the `elastic` user password, and\n   go to [localhost:5601](http://localhost:5601) to log in.\n\nThere is also an example project that demonstrates the application-tasques-worker relationship more thoroughly; please\nsee `example/ciphers` for more details.\n\n### Monitoring\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"apm.png\" width=\"500\"  alt=\"APM\"/\u003e\n\u003c/p\u003e\n\nThe server supports APM, as configured according to the [official docs](https://www.elastic.co/guide/en/apm/agent/go/current/getting-started.html#configure-setup).\n\n### High availability\n\nThe Tasque server is in principle stateless; but there are internal recurring jobs that need to be taken care off, like\nmonitoring claimed Tasks and timing them out as needed, and scheduling recurring Tasks.\n\nThese internal jobs only occur on the Leader server, as determined by a leader lock. By spinning up more than one Tasque server,\nyou not only gain the benefits of being able to handle more load, but also shield yourself from potential disruptions in the running\nof these internal Tasks, as a new Leader will be elected and take over if the current Leader loses connectivity or is terminated.\n\nRunning multiple servers also allows for zero-downtime rollouts of new versions of Tasques server.\n\n### Delivery\n\nAssuming there is no data loss at the ES level, Tasques provides at-least-once delivery, ensuring that it only allows a\nsingle Worker (identified by Id) to have claim on a Task at any given time.\n\nIf there is data loss/recovery (snapshot recovery, ES node loss), jobs might be handed out twice, so it's a good idea to \nmake job handling idempotent.\n\n### Idempotency\n\nWhen submitting/creating a Task, you can optionally specify an \"id\" field, which acts a Queue-specific idempotency key\n(a UUID is generated and used if not specified). If there is already a Task in the Queue you specified with that key, \nthe submission will fail.\n\nNote that idempotency is only for un-archived Tasks: it's possible for a Task to be created with the same Id as another \nalready archived Task. The `archive_older_than` period config can be tweaked if this is an issue.\n\n### Recurring Tasks\n\nTasques comes with support for scheduling Tasks enqueued at a given cron expression (delegating to [robconfig/cron](https://github.com/robfig/cron),\nso check there for supported expressions).\n\nThere is also support to skip enqueueing Tasks for a Recurring Task if there are existing outstanding (not dead or done)\nthat belong to it. For high-frequency Recurring Tasks (higher than once every ~2 seconds) this is subject to limitations\nin ES itself, since it refreshes Indices on a [configurable interval](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-refresh.html).\nTo address this, you can configure the ES index refresh interval to be more frequent, or you can configure\ntasques itself to be more aggressive about refreshing indices (see `config/tasques.example.yaml` for details).  \n\n### Dev\n\nRequires [Go](https://golang.org) 1.13+.\n\n1. [Install `Go`](https://golang.org/doc/install)\n2. Use your favourite editor/IDE\n3. For updating Swagger docs:\n    1. Install [Swaggo](https://github.com/swaggo/swag#getting-started)\n    2. Run `swag init -g app/main.go` from the root project dir\n        * Check that there are no `time.Time` fields... there's a race condition in there somewhere\n    3. Commit the generated files.\n4. For updating the Go Client:\n    1. Install [go-swagger](https://goswagger.io/generate/client.html)\n    2. Run `swagger generate client -f docs/swagger.yaml`\n    3. Commit the generated files.\n    \n### Running tests\n\n**Unit tests**: `go test ./...`\n\n**Integration tests**: `go test -tags=integration ./...` (finds `// +build integration` at the top o IT files)\n\n#### Code guidelines\n\nThe code emphasises the following:\n\n1. **Safety**: the code needs to do the right thing. Use built-in features (locks, typed ids) and tests to help maximise safety.\n2. **Efficiency**: the server and its code should be reasonably efficient so that it can handle high loads.\n3. **Observability**: where reasonable, pass a `Context` around so we can export performance metrics. This is a must when making any kind of IO call.\n4. **High availability**: the server needs to be able to run in a highly available way to maximise robusness.\n5. **Simplicity**: the API needs to be simple and easy to use\n6. **Maintainability**: the internals need to be as simple as possible and invite contributions. Use the rule of 3 to know when something should be generalised. If in doubt, repeat yourself.\n\n### Credit\n\nThis project was inspired by the following, in no particular order\n\n* [Celery](http://www.celeryproject.org)\n* [Resque](https://github.com/resque/resque)\n* [Sidekiq](http://sidekiq.org)\n* [Faktory](https://contribsys.com/faktory/)\n* [esqueue](https://github.com/elastic/esqueue)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flloydmeta%2Ftasques","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flloydmeta%2Ftasques","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flloydmeta%2Ftasques/lists"}