{"id":13413383,"url":"https://github.com/hibiken/asynq","last_synced_at":"2025-05-12T05:19:12.515Z","repository":{"id":37382529,"uuid":"221849341","full_name":"hibiken/asynq","owner":"hibiken","description":"Simple, reliable, and efficient distributed task queue in Go","archived":false,"fork":false,"pushed_at":"2025-04-01T06:06:12.000Z","size":8034,"stargazers_count":11133,"open_issues_count":239,"forks_count":787,"subscribers_count":83,"default_branch":"master","last_synced_at":"2025-05-12T02:43:21.723Z","etag":null,"topics":["asynchronous-tasks","background-jobs","go","golang","redis","task-queue","worker-pool"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hibiken.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null},"funding":{"github":["hibiken"],"open_collective":"ken-hibino"}},"created_at":"2019-11-15T05:17:55.000Z","updated_at":"2025-05-12T00:51:31.000Z","dependencies_parsed_at":"2024-01-29T07:53:55.918Z","dependency_job_id":"6446efae-08f7-4bf1-aa9e-3876bd5f4047","html_url":"https://github.com/hibiken/asynq","commit_stats":{"total_commits":879,"total_committers":52,"mean_commits":"16.903846153846153","dds":"0.12059158134243464","last_synced_commit":"3dbda603333da7c47449e3c1fc14f3c681ac58a3"},"previous_names":[],"tags_count":53,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hibiken%2Fasynq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hibiken%2Fasynq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hibiken%2Fasynq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hibiken%2Fasynq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hibiken","download_url":"https://codeload.github.com/hibiken/asynq/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253672696,"owners_count":21945480,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["asynchronous-tasks","background-jobs","go","golang","redis","task-queue","worker-pool"],"created_at":"2024-07-30T20:01:39.081Z","updated_at":"2025-05-12T05:19:12.484Z","avatar_url":"https://github.com/hibiken.png","language":"Go","funding_links":["https://github.com/sponsors/hibiken","https://opencollective.com/ken-hibino"],"categories":["Popular","HarmonyOS","Go","开源类库","Uncategorized","消息","Messaging","Open source library","Relational Databases","语言资源库","消息系统","\u003ca name=\"Go\"\u003e\u003c/a\u003eGo","机器学习","Repositories"],"sub_categories":["Windows Manager","任务/定时器","Uncategorized","检索及分析资料库","Search and Analytic Databases","Task/Timer","go","SQL 查询语句构建库","Advanced Console UIs"],"readme":"\u003cimg src=\"https://user-images.githubusercontent.com/11155743/114697792-ffbfa580-9d26-11eb-8e5b-33bef69476dc.png\" alt=\"Asynq logo\" width=\"360px\" /\u003e\n\n# Simple, reliable \u0026 efficient distributed task queue in Go\n\n[![GoDoc](https://godoc.org/github.com/hibiken/asynq?status.svg)](https://godoc.org/github.com/hibiken/asynq)\n[![Go Report Card](https://goreportcard.com/badge/github.com/hibiken/asynq)](https://goreportcard.com/report/github.com/hibiken/asynq)\n![Build Status](https://github.com/hibiken/asynq/workflows/build/badge.svg)\n[![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](https://opensource.org/licenses/MIT)\n[![Gitter chat](https://badges.gitter.im/go-asynq/gitter.svg)](https://gitter.im/go-asynq/community)\n\nAsynq is a Go library for queueing tasks and processing them asynchronously with workers. It's backed by [Redis](https://redis.io/) and is designed to be scalable yet easy to get started.\n\nHighlevel overview of how Asynq works:\n\n- Client puts tasks on a queue\n- Server pulls tasks off queues and starts a worker goroutine for each task\n- Tasks are processed concurrently by multiple workers\n\nTask queues are used as a mechanism to distribute work across multiple machines. A system can consist of multiple worker servers and brokers, giving way to high availability and horizontal scaling.\n\n**Example use case**\n\n![Task Queue Diagram](https://user-images.githubusercontent.com/11155743/116358505-656f5f80-a806-11eb-9c16-94e49dab0f99.jpg)\n\n## Features\n\n- Guaranteed [at least one execution](https://www.cloudcomputingpatterns.org/at_least_once_delivery/) of a task\n- Scheduling of tasks\n- [Retries](https://github.com/hibiken/asynq/wiki/Task-Retry) of failed tasks\n- Automatic recovery of tasks in the event of a worker crash\n- [Weighted priority queues](https://github.com/hibiken/asynq/wiki/Queue-Priority#weighted-priority)\n- [Strict priority queues](https://github.com/hibiken/asynq/wiki/Queue-Priority#strict-priority)\n- Low latency to add a task since writes are fast in Redis\n- De-duplication of tasks using [unique option](https://github.com/hibiken/asynq/wiki/Unique-Tasks)\n- Allow [timeout and deadline per task](https://github.com/hibiken/asynq/wiki/Task-Timeout-and-Cancelation)\n- Allow [aggregating group of tasks](https://github.com/hibiken/asynq/wiki/Task-aggregation) to batch multiple successive operations\n- [Flexible handler interface with support for middlewares](https://github.com/hibiken/asynq/wiki/Handler-Deep-Dive)\n- [Ability to pause queue](/tools/asynq/README.md#pause) to stop processing tasks from the queue\n- [Periodic Tasks](https://github.com/hibiken/asynq/wiki/Periodic-Tasks)\n- [Support Redis Sentinels](https://github.com/hibiken/asynq/wiki/Automatic-Failover) for high availability\n- Integration with [Prometheus](https://prometheus.io/) to collect and visualize queue metrics\n- [Web UI](#web-ui) to inspect and remote-control queues and tasks\n- [CLI](#command-line-tool) to inspect and remote-control queues and tasks\n\n## Stability and Compatibility\n\n**Status**: The library relatively stable and is currently undergoing **moderate development** with less frequent breaking API changes.\n\n\u003e ☝️ **Important Note**: Current major version is zero (`v0.x.x`) to accommodate rapid development and fast iteration while getting early feedback from users (_feedback on APIs are appreciated!_). The public API could change without a major version update before `v1.0.0` release.\n\n### Redis Cluster Compatibility\n\nSome of the lua scripts in this library may not be compatible with Redis Cluster.\n\n## Sponsoring\nIf you are using this package in production, **please consider sponsoring the project to show your support!**\n\n## Quickstart\nMake sure you have Go installed ([download](https://golang.org/dl/)). The **last two** Go versions are supported (See https://go.dev/dl).\n\nInitialize your project by creating a folder and then running `go mod init github.com/your/repo` ([learn more](https://blog.golang.org/using-go-modules)) inside the folder. Then install Asynq library with the [`go get`](https://golang.org/cmd/go/#hdr-Add_dependencies_to_current_module_and_install_them) command:\n\n```sh\ngo get -u github.com/hibiken/asynq\n```\n\nMake sure you're running a Redis server locally or from a [Docker](https://hub.docker.com/_/redis) container. Version `4.0` or higher is required.\n\nNext, write a package that encapsulates task creation and task handling.\n\n```go\npackage tasks\n\nimport (\n    \"context\"\n    \"encoding/json\"\n    \"fmt\"\n    \"log\"\n    \"time\"\n    \"github.com/hibiken/asynq\"\n)\n\n// A list of task types.\nconst (\n    TypeEmailDelivery   = \"email:deliver\"\n    TypeImageResize     = \"image:resize\"\n)\n\ntype EmailDeliveryPayload struct {\n    UserID     int\n    TemplateID string\n}\n\ntype ImageResizePayload struct {\n    SourceURL string\n}\n\n//----------------------------------------------\n// Write a function NewXXXTask to create a task.\n// A task consists of a type and a payload.\n//----------------------------------------------\n\nfunc NewEmailDeliveryTask(userID int, tmplID string) (*asynq.Task, error) {\n    payload, err := json.Marshal(EmailDeliveryPayload{UserID: userID, TemplateID: tmplID})\n    if err != nil {\n        return nil, err\n    }\n    return asynq.NewTask(TypeEmailDelivery, payload), nil\n}\n\nfunc NewImageResizeTask(src string) (*asynq.Task, error) {\n    payload, err := json.Marshal(ImageResizePayload{SourceURL: src})\n    if err != nil {\n        return nil, err\n    }\n    // task options can be passed to NewTask, which can be overridden at enqueue time.\n    return asynq.NewTask(TypeImageResize, payload, asynq.MaxRetry(5), asynq.Timeout(20 * time.Minute)), nil\n}\n\n//---------------------------------------------------------------\n// Write a function HandleXXXTask to handle the input task.\n// Note that it satisfies the asynq.HandlerFunc interface.\n//\n// Handler doesn't need to be a function. You can define a type\n// that satisfies asynq.Handler interface. See examples below.\n//---------------------------------------------------------------\n\nfunc HandleEmailDeliveryTask(ctx context.Context, t *asynq.Task) error {\n    var p EmailDeliveryPayload\n    if err := json.Unmarshal(t.Payload(), \u0026p); err != nil {\n        return fmt.Errorf(\"json.Unmarshal failed: %v: %w\", err, asynq.SkipRetry)\n    }\n    log.Printf(\"Sending Email to User: user_id=%d, template_id=%s\", p.UserID, p.TemplateID)\n    // Email delivery code ...\n    return nil\n}\n\n// ImageProcessor implements asynq.Handler interface.\ntype ImageProcessor struct {\n    // ... fields for struct\n}\n\nfunc (processor *ImageProcessor) ProcessTask(ctx context.Context, t *asynq.Task) error {\n    var p ImageResizePayload\n    if err := json.Unmarshal(t.Payload(), \u0026p); err != nil {\n        return fmt.Errorf(\"json.Unmarshal failed: %v: %w\", err, asynq.SkipRetry)\n    }\n    log.Printf(\"Resizing image: src=%s\", p.SourceURL)\n    // Image resizing code ...\n    return nil\n}\n\nfunc NewImageProcessor() *ImageProcessor {\n\treturn \u0026ImageProcessor{}\n}\n```\n\nIn your application code, import the above package and use [`Client`](https://pkg.go.dev/github.com/hibiken/asynq?tab=doc#Client) to put tasks on queues.\n\n```go\npackage main\n\nimport (\n    \"log\"\n    \"time\"\n\n    \"github.com/hibiken/asynq\"\n    \"your/app/package/tasks\"\n)\n\nconst redisAddr = \"127.0.0.1:6379\"\n\nfunc main() {\n    client := asynq.NewClient(asynq.RedisClientOpt{Addr: redisAddr})\n    defer client.Close()\n\n    // ------------------------------------------------------\n    // Example 1: Enqueue task to be processed immediately.\n    //            Use (*Client).Enqueue method.\n    // ------------------------------------------------------\n\n    task, err := tasks.NewEmailDeliveryTask(42, \"some:template:id\")\n    if err != nil {\n        log.Fatalf(\"could not create task: %v\", err)\n    }\n    info, err := client.Enqueue(task)\n    if err != nil {\n        log.Fatalf(\"could not enqueue task: %v\", err)\n    }\n    log.Printf(\"enqueued task: id=%s queue=%s\", info.ID, info.Queue)\n\n\n    // ------------------------------------------------------------\n    // Example 2: Schedule task to be processed in the future.\n    //            Use ProcessIn or ProcessAt option.\n    // ------------------------------------------------------------\n\n    info, err = client.Enqueue(task, asynq.ProcessIn(24*time.Hour))\n    if err != nil {\n        log.Fatalf(\"could not schedule task: %v\", err)\n    }\n    log.Printf(\"enqueued task: id=%s queue=%s\", info.ID, info.Queue)\n\n\n    // ----------------------------------------------------------------------------\n    // Example 3: Set other options to tune task processing behavior.\n    //            Options include MaxRetry, Queue, Timeout, Deadline, Unique etc.\n    // ----------------------------------------------------------------------------\n\n    task, err = tasks.NewImageResizeTask(\"https://example.com/myassets/image.jpg\")\n    if err != nil {\n        log.Fatalf(\"could not create task: %v\", err)\n    }\n    info, err = client.Enqueue(task, asynq.MaxRetry(10), asynq.Timeout(3 * time.Minute))\n    if err != nil {\n        log.Fatalf(\"could not enqueue task: %v\", err)\n    }\n    log.Printf(\"enqueued task: id=%s queue=%s\", info.ID, info.Queue)\n}\n```\n\nNext, start a worker server to process these tasks in the background. To start the background workers, use [`Server`](https://pkg.go.dev/github.com/hibiken/asynq?tab=doc#Server) and provide your [`Handler`](https://pkg.go.dev/github.com/hibiken/asynq?tab=doc#Handler) to process the tasks.\n\nYou can optionally use [`ServeMux`](https://pkg.go.dev/github.com/hibiken/asynq?tab=doc#ServeMux) to create a handler, just as you would with [`net/http`](https://golang.org/pkg/net/http/) Handler.\n\n```go\npackage main\n\nimport (\n    \"log\"\n\n    \"github.com/hibiken/asynq\"\n    \"your/app/package/tasks\"\n)\n\nconst redisAddr = \"127.0.0.1:6379\"\n\nfunc main() {\n    srv := asynq.NewServer(\n        asynq.RedisClientOpt{Addr: redisAddr},\n        asynq.Config{\n            // Specify how many concurrent workers to use\n            Concurrency: 10,\n            // Optionally specify multiple queues with different priority.\n            Queues: map[string]int{\n                \"critical\": 6,\n                \"default\":  3,\n                \"low\":      1,\n            },\n            // See the godoc for other configuration options\n        },\n    )\n\n    // mux maps a type to a handler\n    mux := asynq.NewServeMux()\n    mux.HandleFunc(tasks.TypeEmailDelivery, tasks.HandleEmailDeliveryTask)\n    mux.Handle(tasks.TypeImageResize, tasks.NewImageProcessor())\n    // ...register other handlers...\n\n    if err := srv.Run(mux); err != nil {\n        log.Fatalf(\"could not run server: %v\", err)\n    }\n}\n```\n\nFor a more detailed walk-through of the library, see our [Getting Started](https://github.com/hibiken/asynq/wiki/Getting-Started) guide.\n\nTo learn more about `asynq` features and APIs, see the package [godoc](https://godoc.org/github.com/hibiken/asynq).\n\n## Web UI\n\n[Asynqmon](https://github.com/hibiken/asynqmon) is a web based tool for monitoring and administrating Asynq queues and tasks.\n\nHere's a few screenshots of the Web UI:\n\n**Queues view**\n\n![Web UI Queues View](https://user-images.githubusercontent.com/11155743/114697016-07327f00-9d26-11eb-808c-0ac841dc888e.png)\n\n**Tasks view**\n\n![Web UI TasksView](https://user-images.githubusercontent.com/11155743/114697070-1f0a0300-9d26-11eb-855c-d3ec263865b7.png)\n\n**Metrics view**\n\u003cimg width=\"1532\" alt=\"Screen Shot 2021-12-19 at 4 37 19 PM\" src=\"https://user-images.githubusercontent.com/10953044/146777420-cae6c476-bac6-469c-acce-b2f6584e8707.png\"\u003e\n\n**Settings and adaptive dark mode**\n\n![Web UI Settings and adaptive dark mode](https://user-images.githubusercontent.com/11155743/114697149-3517c380-9d26-11eb-9f7a-ae2dd00aad5b.png)\n\nFor details on how to use the tool, refer to the tool's [README](https://github.com/hibiken/asynqmon#readme).\n\n## Command Line Tool\n\nAsynq ships with a command line tool to inspect the state of queues and tasks.\n\nTo install the CLI tool, run the following command:\n\n```sh\ngo install github.com/hibiken/asynq/tools/asynq@latest\n```\n\nHere's an example of running the `asynq dash` command:\n\n![Gif](/docs/assets/dash.gif)\n\nFor details on how to use the tool, refer to the tool's [README](/tools/asynq/README.md).\n\n## Contributing\n\nWe are open to, and grateful for, any contributions (GitHub issues/PRs, feedback on [Gitter channel](https://gitter.im/go-asynq/community), etc) made by the community.\n\nPlease see the [Contribution Guide](/CONTRIBUTING.md) before contributing.\n\n## License\n\nCopyright (c) 2019-present [Ken Hibino](https://github.com/hibiken) and [Contributors](https://github.com/hibiken/asynq/graphs/contributors). `Asynq` is free and open-source software licensed under the [MIT License](https://github.com/hibiken/asynq/blob/master/LICENSE). Official logo was created by [Vic Shóstak](https://github.com/koddr) and distributed under [Creative Commons](https://creativecommons.org/publicdomain/zero/1.0/) license (CC0 1.0 Universal).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhibiken%2Fasynq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhibiken%2Fasynq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhibiken%2Fasynq/lists"}