{"id":13471203,"url":"https://github.com/kalbhor/Tasqueue","last_synced_at":"2025-03-26T13:30:54.025Z","repository":{"id":37861788,"uuid":"489248786","full_name":"kalbhor/Tasqueue","owner":"kalbhor","description":"A background jobs library for Go that allows pluggable brokers/store for distribution.","archived":false,"fork":false,"pushed_at":"2025-02-24T05:31:55.000Z","size":179,"stargazers_count":408,"open_issues_count":4,"forks_count":22,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-21T06:01:33.754Z","etag":null,"topics":["background-jobs","distributed-systems","go","golang","jobqueue","queue","tasqueue"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kalbhor.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-05-06T06:54:29.000Z","updated_at":"2025-03-15T23:21:25.000Z","dependencies_parsed_at":"2023-11-30T09:46:43.377Z","dependency_job_id":"e3de2206-71f6-471e-9d1a-da4bac5ce30c","html_url":"https://github.com/kalbhor/Tasqueue","commit_stats":null,"previous_names":[],"tags_count":15,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kalbhor%2FTasqueue","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kalbhor%2FTasqueue/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kalbhor%2FTasqueue/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kalbhor%2FTasqueue/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kalbhor","download_url":"https://codeload.github.com/kalbhor/Tasqueue/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245662770,"owners_count":20652078,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["background-jobs","distributed-systems","go","golang","jobqueue","queue","tasqueue"],"created_at":"2024-07-31T16:00:41.539Z","updated_at":"2025-03-26T13:30:53.691Z","avatar_url":"https://github.com/kalbhor.png","language":"Go","readme":"\u003ca href=\"https://zerodha.tech\"\u003e\u003cimg src=\"https://zerodha.tech/static/images/github-badge.svg\" align=\"right\" /\u003e\u003c/a\u003e\n\n![taskqueue](https://user-images.githubusercontent.com/14031096/170992942-3b62e055-6d9e-4c08-a277-ed6d6e9a4c2a.png)\n\n[![Run Tests](https://github.com/kalbhor/tasqueue/v2/actions/workflows/test.yml/badge.svg)](https://github.com/kalbhor/tasqueue/v2/actions/workflows/test.yml) [![Go Report Card](https://goreportcard.com/badge/github.com/kalbhor/tasqueue/v2)](https://goreportcard.com/report/github.com/kalbhor/tasqueue/v2)\n\n**Tasqueue** is a simple, lightweight distributed job/worker implementation in Go\n\n### Installation\n\n`go get -u github.com/kalbhor/tasqueue/v2`\n\n- [Concepts](#concepts)\n- [Server](#server)\n  - [Options](#server-options)\n  - [Usage](#usage)\n  - [Task Options](#task-options)\n  - [Registering Tasks](#registering-tasks)\n  - [Starting Server](#start-server)\n- [Job](#job)\n  - [Options](#job-options)\n  - [Creating a job](#creating-a-job)\n  - [Enqueuing a job](#enqueuing-a-job)\n  - [Getting job message](#getting-a-job-message)\n  - [JobCtx](#jobctx)\n- [Group](#group)\n  - [Creating a group](#creating-a-group)\n  - [Enqueuing a group](#enqueuing-a-group)\n  - [Getting group message](#getting-a-group-message)\n- [Chain](#chain)\n  - [Creating a chain](#creating-a-chain)\n  - [Enqueuing a chain](#enqueuing-a-chain)\n  - [Getting chain message](#getting-a-group-chain)\n- [Result](#result)\n  - [Get Result](#get-result)\n\n## Concepts\n\n- `tasqueue.Broker` is a generic interface to enqueue and consume messages from a single queue. Currently supported brokers are\n  [redis](./brokers/redis/) and [nats-jetstream](./brokers/nats-js/). Note: It is important for the broker (or your enqueue, consume implementation) to guarantee atomicity. ie : Tasqueue does not provide locking capabilities to ensure unique job consumption.\n- `tasqueue.Results` is a generic interface to store the status and results of jobs. Currently supported result stores are\n  [redis](./results/redis/) and [nats-jetstream](./results/nats-js/).\n- `tasqueue.Task` is a pre-registered job handler. It stores a handler functions which is called to process a job. It also stores callbacks (if set through options), executed during different states of a job.\n- `tasqueue.Job` represents a unit of work pushed to a queue for consumption. It holds:\n  - `[]byte` payload (encoded in any manner, if required)\n  - task name used to identify the pre-registed task which will processes the job.\n\n### Server\n\nA tasqueue server is the main store that holds the broker and the results interfaces. It also acts as a hub to register tasks.\n\n#### Server Options\n\nServer options are used to configure the server. Broker \u0026 Results are mandatory, while logger and open telemetry provider are optional. Refer to the [in-memory](./examples/in-memory/main.go) example for an open telemetry implementation.\n\n```go\ntype ServerOpts struct {\n\t// Mandatory results \u0026 broker implementations.\n\tBroker        Broker\n\tResults       Results\n\n\t// Optional logger and telemetry provider.\n\tLogger        logf.Logger\n\tTraceProvider *trace.TracerProvider\n}\n```\n\n#### Usage\n\n```go\npackage main\n\nimport (\n\t\"log\"\n\n\t\"github.com/kalbhor/tasqueue/v2\"\n\trb \"github.com/kalbhor/tasqueue/v2/brokers/redis\"\n\trr \"github.com/kalbhor/tasqueue/v2/results/redis\"\n\t\"github.com/zerodha/logf\"\n)\n\nfunc main() {\n\tlo := logf.New(logf.Opts{})\n\n\tbroker := rb.New(rb.Options{\n\t\tAddrs:    []string{\"127.0.0.1:6379\"},\n\t\tPassword: \"\",\n\t\tDB:       0,\n\t}, lo)\n\tresults := rr.New(rr.Options{\n\t\tAddrs:    []string{\"127.0.0.1:6379\"},\n\t\tPassword: \"\",\n\t\tDB:       0,\n\t}, lo)\n\n\tsrv, err := tasqueue.NewServer(tasqueue.ServerOpts{\n\t\tBroker:        broker,\n\t\tResults:       results,\n\t\tLogger:        lo,\n\t})\n\tif err != nil {\n\t\tlog.Fatal(err)\n\t}\n}\n```\n\n#### Task Options\nQueue is the name of the queue assigned to the task. By default the value is \"tasqueue:tasks\". Queues can be \nshared between tasks.\n\nConcurrency defines the number of processor go-routines running on the queue. By default this number is equal\nto `runtime.GOMAXPROCS(0)` (number of CPUs on the system). Ideally, it is recommended that the client tweak this number according\nto their tasks.\n\n```go\ntype TaskOpts struct {\n\tConcurrency  uint32\n\tQueue        string\n\tSuccessCB    func(JobCtx)\n\tProcessingCB func(JobCtx)\n\tRetryingCB   func(JobCtx)\n\tFailedCB     func(JobCtx)\n}\n```\n\n#### Registering tasks\n\nA task can be registered by supplying a name, handler and options.\nJobs can be processed using a task registered by a particular name.\nA handler is a function with the signature `func([]byte, JobCtx) error`. It is the responsibility of the handler to deal with the `[]byte` payload in whatever manner (decode, if required).\n\n```go\npackage tasks\n\nimport (\n\t\"encoding/json\"\n\n\t\"github.com/kalbhor/tasqueue/v2\"\n)\n\ntype SumPayload struct {\n\tArg1 int `json:\"arg1\"`\n\tArg2 int `json:\"arg2\"`\n}\n\ntype SumResult struct {\n\tResult int `json:\"result\"`\n}\n\n// SumProcessor prints the sum of two integer arguements.\nfunc SumProcessor(b []byte, m tasqueue.JobCtx) error {\n\tvar pl SumPayload\n\tif err := json.Unmarshal(b, \u0026pl); err != nil {\n\t\treturn err\n\t}\n\n\trs, err := json.Marshal(SumResult{Result: pl.Arg1 + pl.Arg2})\n\tif err != nil {\n\t\treturn err\n\t}\n\n\tm.Save(rs)\n\n\treturn nil\n}\n```\n\nOnce a queue is created if the client creates a task with an existing queue but supplies a different concurrency \nin the `TaskOpts`, then `RegisterTask` will return an error.\n\n```go\n// This creates the q1 queue if it doesn't exist and assigns 5 concurrency to it\nerr := srv.RegisterTask(\"add\", tasks.SumProcessor, TaskOpts{Queue:\"q1\", Concurrency: 5})\nif err != nil {\n\tlog.Fatal(err)\n}\n\n// No error\nerr := srv.RegisterTask(\"div\", tasks.DivProcessor, TaskOpts{Queue:\"q1\"})\nif err != nil {\n\tlog.Fatal(err)\n}\n\n// No error\nerr := srv.RegisterTask(\"sub\", tasks.SubProcessor, TaskOpts{Queue:\"q1\", Concurrency: 5})\nif err != nil {\n\tlog.Fatal(err)\n}\n\n// This will return an error since q1 is already created and its concurrency cannot be modified\nerr := srv.RegisterTask(\"multiplication\", tasks.MulProcessor, TaskOpts{Queue:\"q1\", Concurrency: 10})\nif err != nil {\n\tlog.Fatal(err)\n}\n```\n\n#### Start server\n\n`Start()` starts the job consumer and processor. It is a blocking function. It listens for jobs on the queue and spawns processor go routines.\n\n```go\nsrv.Start(ctx)\n```\n\n### Job\n\nA tasqueue job represents a unit of work pushed onto the queue, that requires processing using a registered Task. It holds a `[]byte` payload, a task name (which will process the payload) and various options.\n\n#### Job Options\n\n```go\n// JobOpts holds the various options available to configure a job.\ntype JobOpts struct {\n\t// Optional ID passed by client. If empty, Tasqueue generates it.\n\tID string\n\n\tQueue      string\n\tMaxRetries uint32\n\tSchedule   string\n\tTimeout    time.Duration\n}\n```\n\n#### Creating a job\n\n`NewJob` returns a job with the supplied payload. It accepts the name of the task, the payload and a list of options.\n\n```go\nb, _ := json.Marshal(tasks.SumPayload{Arg1: 5, Arg2: 4})\njob, err := tasqueue.NewJob(\"add\", b, tasqueue.JobOpts{})\nif err != nil {\n\tlog.Fatal(err)\n}\n```\n\n#### Enqueuing a job\n\nOnce a job is created, it can be enqueued via the server for processing. Calling `srv.Enqueue` returns a job id which can be used to query the status of the job.\n\n```go\nid, err := srv.Enqueue(ctx, job)\nif err != nil {\n\tlog.Fatal(err)\n}\n```\n\n#### Getting a job message\n\nTo query the details of a job that was enqueued, we can use `srv.GetJob`. It returns a `JobMessage` which contains details related to a job.\n\n```go\njobMsg, err := srv.GetJob(ctx, id)\nif err != nil {\n\tlog.Fatal(err)\n}\n```\n\nFields available in a `JobMessage` (embeds `Meta`):\n\n```go\n// Meta contains fields related to a job. These are updated when a task is consumed.\ntype Meta struct {\n\tID          string\n\tOnSuccessID string\n\tStatus        string\n\tQueue         string\n\tSchedule      string\n\tMaxRetry      uint32\n\tRetried       uint32\n\tPrevErr       string\n\tProcessedAt   time.Time\n\n\t// PrevJobResults contains any job result set by the previous job in a chain.\n\t// This will be nil if the previous job doesn't set the results on JobCtx.\n\tPrevJobResult []byte\n}\n```\n\n#### JobCtx\n\n`JobCtx` is passed to handler functions and callbacks. It can be used to view the job's meta information (`JobCtx` embeds `Meta`) and also to save arbitrary results for a job using `func (c *JobCtx) Save(b []byte) error`\n\n### Group\n\nA tasqueue group holds multiple jobs and pushes them all simultaneously onto the queue, the Group is considered successful only if all the jobs finish successfully.\n\n#### Creating a group\n\n`NewGroup` returns a Group holding the jobs passed.\n\n```go\nvar group []tasqueue.Job\n\nfor i := 0; i \u003c 3; i++ {\n\tb, _ := json.Marshal(tasks.SumPayload{Arg1: i, Arg2: 4})\n\tjob, err := tasqueue.NewJob(\"add\", b)\n\tif err != nil {\n\t\t\tlog.Fatal(err)\n\t}\n\tgroup = append(group, job)\n}\n\ngrp, err := tasqueue.NewGroup(group, tasqueue.GroupOpts{})\nif err != nil {\n\tlog.Fatal(err)\n}\n```\n\n#### Enqueuing a group\n\nOnce a group is created, it can be enqueued via the server for processing. Calling `srv.EnqueueGroup` returns a group id which can be used to query the status of the group.\n\n```go\ngroupID, err := srv.EnqueueGroup(ctx, grp)\nif err != nil {\n\tlog.Fatal(err)\n}\n```\n\n#### Getting a group message\n\nTo query the details of a group that was enqueued, we can use `srv.GetGroup`. It returns a `GroupMessage` which contains details related to a group.\n\n```go\ngroupMsg, err := srv.GetGroup(ctx, groupID)\nif err != nil {\n\tlog.Fatal(err)\n}\n```\n\nFields available in a `GroupMessage` (embeds `GroupMeta`):\n\n```go\n// GroupMeta contains fields related to a group job. These are updated when a task is consumed.\ntype GroupMeta struct {\n\tID   string\n\tStatus string\n\t// JobStatus is a map of individual job id -\u003e status\n\tJobStatus map[string]string\n}\n```\n\n### Chain\n\nA tasqueue chain holds multiple jobs and pushes them one after the other (after a job succeeds), the Chain is considered successful only if the final job completes successfuly.\n\n#### Creating a chain\n\n`NewChain` returns a chain holding the jobs passed in the order.\n\n```go\nvar chain []tasqueue.Job\n\nfor i := 0; i \u003c 3; i++ {\n\tb, _ := json.Marshal(tasks.SumPayload{Arg1: i, Arg2: 4})\n\ttask, err := tasqueue.NewJob(\"add\", b)\n\tif err != nil {\n\t\tlog.Fatal(err)\n\t}\n\tchain = append(chain, task)\n}\n\nchn, err := tasqueue.NewChain(chain, tasqueue.ChainOpts{})\nif err != nil {\n\tlog.Fatal(err)\n}\n```\n\n#### Enqueuing a chain\n\nOnce a chain is created, it can be enqueued via the server for processing. Calling `srv.EnqueueChain` returns a chain id which can be used to query the status of the chain.\n\n```go\nchainID, err := srv.EnqueueChain(ctx, chn)\nif err != nil {\n\tlog.Fatal(err)\n}\n```\n\n#### Getting results of previous job in a chain\n\nA job in the chain can access the results of the previous job in the chain by getting `JobCtx.Meta.PrevJobResults`. This will contain any job result saved by the previous job by `JobCtx.Save()`.\n\n#### Getting a chain message\n\nTo query the details of a chain that was enqueued, we can use `srv.GetChain`. It returns a `ChainMessage` which contains details related to a chian.\n\n```go\nchainMsg, err := srv.GetChain(ctx, chainID)\nif err != nil {\n\tlog.Fatal(err)\n}\n```\n\nFields available in a `ChainMessage` (embeds `ChainMeta`):\n\n```go\n// ChainMeta contains fields related to a chain job.\ntype ChainMeta struct {\n\tID string\n\t// Status of the overall chain\n\tStatus string\n\t// ID of the current job part of chain\n\tJobID string\n\t// List of IDs of completed jobs\n\tPrevJobs []string\n}\n```\n\n### Result\n\nA result is arbitrary `[]byte` data saved by a handler or callback via `JobCtx.Save()`.\n\n#### Get Result\n\nIf the `jobID` does not exist, `ErrNotFound` will be returned\n\n```go \nb, err := srv.GetResult(ctx, jobID)\nif err != nil {\n\tlog.Fatal(err)\n}\n```\n\n#### Delete Result\n\nDeleteJob removes the job's saved metadata from the store\n\n```go\nerr := srv.DeleteResult(ctx, jobID)\nif err != nil {\n\tlog.Fatal(err)\n}\n```\n\n## Credits\n\n- [@knadh](github.com/knadh) for the logo \u0026 feature suggestions\n\n## License\n\nBSD-2-Clause-FreeBSD\n","funding_links":[],"categories":["Uncategorized","Go"],"sub_categories":["Uncategorized"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkalbhor%2FTasqueue","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkalbhor%2FTasqueue","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkalbhor%2FTasqueue/lists"}