{"id":27057783,"url":"https://github.com/dumbmachine/nq","last_synced_at":"2025-04-05T11:33:16.419Z","repository":{"id":53919617,"uuid":"521896002","full_name":"DumbMachine/nq","owner":"DumbMachine","description":"Cancellable, Efficient and Reliable  Distributed Task Queue in Go","archived":false,"fork":false,"pushed_at":"2022-10-03T12:58:36.000Z","size":3819,"stargazers_count":93,"open_issues_count":0,"forks_count":4,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-06-21T03:17:41.807Z","etag":null,"topics":["asynchronous-tasks","background-jobs","go","golang","nats","task-queue","worker-queue"],"latest_commit_sha":null,"homepage":"https://pkg.go.dev/github.com/dumbmachine/nq","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DumbMachine.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-08-06T08:57:49.000Z","updated_at":"2024-05-27T14:39:56.000Z","dependencies_parsed_at":"2022-08-13T04:20:19.417Z","dependency_job_id":null,"html_url":"https://github.com/DumbMachine/nq","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DumbMachine%2Fnq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DumbMachine%2Fnq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DumbMachine%2Fnq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DumbMachine%2Fnq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DumbMachine","download_url":"https://codeload.github.com/DumbMachine/nq/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247331991,"owners_count":20921847,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["asynchronous-tasks","background-jobs","go","golang","nats","task-queue","worker-queue"],"created_at":"2025-04-05T11:33:12.063Z","updated_at":"2025-04-05T11:33:16.410Z","avatar_url":"https://github.com/DumbMachine.png","language":"Go","readme":"# Reliable, Efficient and Cancellable Distributed Task Queue in Go\n\n[![Go Report Card](https://goreportcard.com/badge/github.com/dumbmachine/nq)](https://goreportcard.com/report/github.com/dumbmachine/nq)\n[![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](https://opensource.org/licenses/MIT)\n[![GoDoc](https://godoc.org/github.com/dumbmachine/nq?status.svg)](https://godoc.org/github.com/dumbmachine/nq)\n\nNQ ( Nats Queue ) is Go package for queuing and processing jobs in background with workers. Based on [nats](https://nats.io/) with a focus on cancel-ability of enqueued jobs.\n\nNQ requires nats-server version that supports both jetstream support and key-value store\n\n**How does it work?:**\n\n|                                                           ![Task Queue Figure](/docs//assets/1.svg) ![Task Queue Figure](/docs//assets/2.svg)                                                           |\n| :-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |\n| _This package was designed such that a task should always be cancellable by client. Workers can be configured to cancel and quit instantly upon network partision ( eg. disconnect from nats-server )._ |\n\n\u003c!-- NQ Client submits `task` to queues which are _load-balanced_ across servers ( subscribed to said queues ). When a task is to be `cancelled`, client issues `cancel` request to all servers subsribed to the `queue`, think multicast, and responsible server `cancels` executing task via go-context. A `task` in `state ∈ {Completed, Failed, Cancelled, Deleted}` cannot be cancelled. While a `task` still in queue, pending for it's execution, will be removed from the queue ( marked as `deleted` ) and a `task` in execution will marked be `cancelled` by calling `cancel` method on it's context.\nFor successful cancellations it is important that `ProcessingFunc`, the function executing said task, should respect `context` provided to it. --\u003e\n\n## Features\n\n\u003c!-- - Retries of failed tasks // todo --\u003e\n\n- Multiple task queues\n- [Deadline and Timeout for tasks](#task-options-walkthrough)\n- [Tasks can be cancelled via context](#task-cancellations)\n- Horizontally scalable workers\n- [Automatic failover](#automatic-failover)\n- [Reconnection to nats-server for automatic failover](#reconnection)\n- [Monitoring and Alerting](#monitoring-and-alerting)\n- [CLI](#cli-usage)\n\u003c!-- - TODO: Nats cluster --\u003e\n\n# Task Options Walkthrough\n\n## Watch for updates\n\n( Introduced in v0.3 )\n\nListen for updates to task metadata\n\n```go\nfunc main() {\n\tclient := nq.NewPublishClient(nq.NatsClientOpt{\n\t\tAddr: \"nats://127.0.0.1:4222\",\n\t}, nq.NoAuthentcation(),\n\t)\n\n\tdefer client.Close()\n\n\tbytesPayload1, err := json.Marshal(UrlPayload{Url: \"https://httpstat.us/200?sleep=10000\"})\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\n\tvar wg sync.WaitGroup\n\ttask1 := nq.NewTask(QueueDev, bytesPayload1)\n\tif ack, err := client.Enqueue(task1); err == nil {\n\t\tlog.Printf(\"Watching updates queue=%s taskID=%s payload=%s\", ack.Queue, ack.ID, ack.Payload)\n\t\twg.Add(1)\n\t\tupdates, err := client.GetUpdates(ack.ID)\n\t\tif err != nil {\n\t\t\tpanic(err)\n\t\t}\n\t\t// listening for updates\n\t\tgo func() {\n\t\t\tdefer wg.Done()\n\n\t\t\tfor {\n\t\t\t\tmsg, ok := \u003c-updates\n\t\t\t\tif !ok {\n\t\t\t\t\t// channel closed\n\t\t\t\t\treturn\n\t\t\t\t}\n\t\t\t\tlog.Printf(\"Change detected, status=%s\", msg.GetStatus())\n\t\t\t}\n\t\t}()\n\t} else {\n\t\tlog.Printf(\"err=%s\", err)\n\t}\n\twg.Wait()\n}\n\n```\n\n```\n2022/08/29 22:17:15 Watching updates queue=scrap-url-dev taskID=yzaKwBIcbGEt8sMGgMJcZ0 payload={\"url\":\"https://httpstat.us/200?sleep=10000\"}\n2022/08/29 22:17:15 Change detected, status=pending\n2022/08/29 22:17:16 Change detected, status=processing\n2022/08/29 22:17:28 Change detected, status=completed\n```\n\n## Retrying\n\nBy default `task` is submitted for retry, if it returns non-nil error.\n\n```go\n// a task that will be retried 2 before being marked as `failed`\ntaskWithRetry := nq.NewTask(\"my-queue\", bytesPayload, nq.Retry(2))\n```\n\nCustom filtering function for error, to mark task as failed only on specific error.\nHere if a task fails due to `ErrFailedDueToInvalidApiKeys`, it will be consider as `failure` and will be retried\n\n```go\nvar ErrFailedDueToInvalidApiKeys = errors.New(\"failed to perform task, invalid api keys\")\n\nsrv := nq.NewServer(nq.NatsClientOpt{Addr: nats.DefaultURL}, nq.Config{\n\tIsFailureFn: func(err error) bool {\n\t\treturn errors.Is(err, ErrFailedDueToInvalidApiKeys)\n\t},\n\tServerName:  nq.GenerateServerName(),\n})\n\n```\n\n## Deadline / Timeout for tasks\n\n```go\n// a task that executes till time.Now() + 1 hour\ntaskWithDeadline := nq.NewTask(\"my-queue\", bytesPayload, nq.Deadline(time.Now().Add(time.Hour)), nq.TaskID(\"deadlineTaskID\"))\n\n// a task that executes for 10 minutes\ntaskWithTimeout := nq.NewTask(\"my-queue\", bytesPayload, nq.Timeout(time.Minute * 10), nq.TaskID(\"timeoutTaskID\"))\n```\n\n## Task cancellations\n\nTasks that are either waiting for execution or being executed on any worker, can be `cancelled`. Cancellation of a task requires it's `taskID`.\n\n```go\n// Cancel a task by ID\ntaskSignature := nq.NewTask(\"my-queue\", []byte())\nack, err := client.Enqueue(taskSignature);\nclient.Cancel(ack.ID)\n```\n\nA **Task** can handle cancel like so:\n\n```go\nfunc longRunningOperation(ctx context.Context, task *nq.TaskPayload) error {\n\tif ctx.Err() != nil {\n\t\treturn ctx.Err()\n\t}\n\tfor i := 0; i \u003c 1000; i++ {\n\t\ttimeout := time.Millisecond * 20\n\t\tprintln(\"sleeping for: \",timeout)\n\t\ttime.Sleep(timeout)\n\t\tif ctx.Err() != nil {\n\t\t\treturn ctx.Err()\n\t\t}\n\t}\n\treturn nil\n}\n```\n\nNOTE: Successful cancellation depends on task function respecting `context.Done()`.\n\n# Automatic Failover\n\n`ShutdownOnNatsDisconnect` option will shutdown workers and server is connection to nats-server is broken. Useful when tasks being `cancellable` at all times is required.\n\nNote: When disconnect is observed, workers would stop processing new messages. The workers would be cancelled in `shutdownTimeout` duration. If any tasks is/are not completed after this, they will be cancelled and still be available in task queue for future / other workers to process.\n\nAuto-shutdown of worker server if at any time server is incapable of respecting a cancel request. Eg. losing connection to nats-server\n\n```go\nsrv := nq.NewServer(nq.NatsClientOpt{\n\tAddr: \"nats://127.0.0.1:4222\",\n}, nq.Config{\n\tServerName:  nq.GenerateServerName(),\n\tConcurrency: 2,\n\tLogLevel:    nq.InfoLevel,\n}, nq.ShutdownOnNatsDisconnect(),\n)\n```\n\n```go\n$ go run examples/simple.go sub\nnq: pid=24914 2022/08/21 15:43:45.650999 INFO: Registered queue=scrap-url-dev\nnq: pid=24914 2022/08/21 15:43:45.652720 INFO: Started Server@DumbmachinePro-local/24914\nnq: pid=24914 2022/08/21 15:43:45.652739 INFO: [*] Listening for messages\nnq: pid=24914 2022/08/21 15:43:45.652742 INFO: cmd/ctrl + c to terminate the process\nnq: pid=24914 2022/08/21 15:43:45.652744 INFO: cmd/ctrl + z to stop processing new tasks\nnq: pid=24914 2022/08/21 15:43:48.363110 ERROR: Disconnected from nats\nnq: pid=24914 2022/08/21 15:43:48.363173 INFO: Starting graceful shutdown\nnq: pid=24914 2022/08/21 15:43:53.363535 INFO: Waiting for all workers to finish...\nnq: pid=24914 2022/08/21 15:43:53.363550 INFO: All workers have finished\nnq: pid=24914 2022/08/21 15:43:53.363570 INFO: Exiting\n```\n\n## Reconnection\n\nServer can configured to not shutdown and instead try to reconnect to nats, when disconnected.\n\n```go\nsrv := nq.NewServer(nq.NatsClientOpt{\n\t\tAddr:          \"nats://127.0.0.1:4222\",\n\t\tReconnectWait: time.Second * 5, // controls timeout between reconnects\n\t\tMaxReconnects: 100, // controls total number of reconnects before giving up\n\t}, nq.Config{ServerName:  \"local-serv-1\"})\n```\n\nIf nats-server is up again:\n\n1. With previous state ( i.e with expected queue data )\n\n   ```bash\n   nq: pid=7988 2022/08/22 17:24:44.349815 INFO: Registered queue=scrap-url-dev\n   nq: pid=7988 2022/08/22 17:24:44.356378 INFO: Registered queue=another-one\n   nq: pid=7988 2022/08/22 17:24:44.356393 INFO: Started Server@DumbmachinePro-local/7988\n   nq: pid=7988 2022/08/22 17:24:44.356444 INFO: [*] Listening for messages\n   nq: pid=7988 2022/08/22 17:24:44.356455 INFO: cmd/ctrl + c to terminate the process\n   nq: pid=7988 2022/08/22 17:24:44.356459 INFO: cmd/ctrl + z to stop processing new tasks\n   disconnected from nats\n   2022/08/22 22:55:02 reconnection found nats://127.0.0.1:4222\n   nq: pid=7988 2022/08/22 17:25:02.860051 INFO: Re-registering subscriptions to nats-server\n   nq: pid=7988 2022/08/22 17:25:02.864988 INFO: Registration successful[nats://127.0.0.1:4222]\n   disconnected from nats\n   ```\n\n2. Without previous state\n   If registered queues are not found in nats-server, they will be created\n   ```bash\n   nq: pid=7998 2022/08/22 17:26:44.349815 INFO: Registered queue=scrap-url-dev\n   nq: pid=7998 2022/08/22 17:26:44.356378 INFO: Registered queue=another-one\n   nq: pid=7998 2022/08/22 17:26:44.356393 INFO: Started Server@DumbmachinePro-local/7998\n   nq: pid=7998 2022/08/22 17:26:44.356444 INFO: [*] Listening for messages\n   nq: pid=7998 2022/08/22 17:26:44.356455 INFO: cmd/ctrl + c to terminate the process\n   nq: pid=7998 2022/08/22 17:26:44.356459 INFO: cmd/ctrl + z to stop processing new tasks\n   disconnected from nats\n   2022/08/22 22:57:25 reconnection found nats://127.0.0.1:4222\n   nq: pid=7998 2022/08/22 17:27:25.518079 INFO: Re-registering subscriptions to nats-server\n   nq: pid=7998 2022/08/22 17:27:25.524895 WARN: stream=scrap-url-dev re-registering\n   nq: pid=7998 2022/08/22 17:27:25.542725 INFO: Registered queue=scrap-url-dev\n   nq: pid=7998 2022/08/22 17:27:25.543668 WARN: stream=another-one re-registering\n   nq: pid=7998 2022/08/22 17:27:25.554961 INFO: Registered queue=another-one\n   nq: pid=7998 2022/08/22 17:27:25.555002 INFO: Registration successful[nats://127.0.0.1:4222]\n   ```\n\n## Monitoring and Alerting\n\nRefer [nats monitoring section](https://docs.nats.io/running-a-nats-service/configuration/monitoring) and [monitoring tool by nats-io](https://github.com/nats-io/nats-surveyor)\n\n# CLI Usage\n\nInstall CLI\n\n```go\ngo install github.com/dumbmachine/nq/tools/nq@latest\n```\n\n- Cancel task\n\n```bash\n$ nq -u nats://127.0.0.1:4222 task cancel --id customID\nCancel message sent task=customID\n```\n\n- Status of task\n\n```bash\n$ nq -u nats://127.0.0.1:4222 task status --id customID\ntaskID=customID status=Cancelled\n```\n\n- Queue stats\n\n```bash\n$ nq -u nats://127.0.0.1:4222 queue stats --name scrap-url-dev\nqueue: scrap-url-dev | MessagesPending: 11 | Size: 3025 Bytes\n```\n\n## Quickstart\n\nInstall NQ library\n\n```bash\ngo get -u github.com/dumbmachine/nq\n```\n\nMake sure you have nats-server running locally or in a container. Example:\n\n```bash\ndocker run --rm -p 4222:4222 --name nats-server -ti nats:latest -js\n```\n\nNow create a client to publish jobs.\n\n```go\n// Creating publish client\npackage main\n\nimport (\n\t\"encoding/json\"\n\t\"log\"\n\n\t\"github.com/dumbmachine/nq\"\n)\n\ntype Payload struct {\n\tUrl string `json:\"url\"`\n}\n\nfunc main() {\n\tclient := nq.NewPublishClient(nq.NatsClientOpt{\n\t\tAddr: \"nats://127.0.0.1:4222\",\n\t}, nq.NoAuthentcation(),\n\t// see godoc for more options\n\t)\n\tdefer client.Close()\n\n\tbPayload, err := json.Marshal(Payload{Url: \"https://httpstat.us/200?sleep=10000\"})\n\tif err != nil {\n\t\tlog.Println(err)\n\t}\n\n\ttaskSig := nq.NewTask(\"scrap-url-dev\", bPayload)\n\tif ack, err := client.Enqueue(taskSig); err == nil {\n\t\tlog.Printf(\"Submitted queue=%s taskID=%s payload=%s\", ack.Queue, ack.ID, ack.Payload)\n\t} else {\n\t\tlog.Printf(\"err=%s\", err)\n\t}\n}\n\n\n```\n\n```go\n// creating worker server\npackage main\n\nimport (\n\t\"context\"\n\t\"encoding/json\"\n\t\"errors\"\n\t\"net/http\"\n\t\"time\"\n\n\t\"github.com/dumbmachine/nq\"\n)\n\ntype Payload struct {\n\tUrl string `json:\"url\"`\n}\n\n// Processing function\nfunc fetchHTML(ctx context.Context, task *nq.TaskPayload) error {\n\tvar payload Payload\n\tif err := json.Unmarshal(task.Payload, \u0026payload); err != nil {\n\t\treturn errors.New(\"invalid payload\")\n\t}\n\tclient := \u0026http.Client{}\n\treq, _ := http.NewRequest(\"GET\", payload.Url, nil)\n\treq = req.WithContext(ctx)\n\tif _, err := client.Do(req); err != nil {\n\t\treturn err\n\t}\n\treturn nil\n}\n\nfunc main() {\n\n\tsrv := nq.NewServer(nq.NatsClientOpt{\n\t\tAddr:          \"nats://127.0.0.1:4222\",\n\t\tReconnectWait: time.Second * 2,\n\t\tMaxReconnects: 100,\n\t}, nq.Config{\n\t\tServerName:  nq.GenerateServerName(),\n\t\tConcurrency: 1,\n\t\tLogLevel:    nq.InfoLevel,\n\t},\n\t)\n\n\tsrv.Register(\"scrap-url-dev\", fetchHTML)\n\n\tif err := srv.Run(); err != nil {\n\t\tpanic(err)\n\t}\n}\n```\n\nNote: New messages are fetched from queue in sequencial order of their registration. NQ does not implement any custom priority order for registered queue yet.\n\n\u003c!-- For more checkout [Getting Started](/link/to/wiki) --\u003e\n\nTo learn more about `nq` APIs, see [godoc](https://pkg.go.dev/github.com/dumbmachine/nq)\n\n## Acknowledgements\n\n[Async](https://github.com/hibiken/asynq) : Many of the design ideas are taken from async\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdumbmachine%2Fnq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdumbmachine%2Fnq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdumbmachine%2Fnq/lists"}