{"id":40900637,"url":"https://github.com/btubbs/pgq","last_synced_at":"2026-01-22T02:31:35.811Z","repository":{"id":34915140,"uuid":"163380519","full_name":"btubbs/pgq","owner":"btubbs","description":"An easy-to-use Postgres job queue library, for Go programs.  Supports retries and exponential backoff.","archived":false,"fork":false,"pushed_at":"2022-03-11T15:37:30.000Z","size":65,"stargazers_count":47,"open_issues_count":1,"forks_count":9,"subscribers_count":5,"default_branch":"master","last_synced_at":"2024-06-18T20:25:28.436Z","etag":null,"topics":["golang","postgresql","queue","worker"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/btubbs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-12-28T07:28:09.000Z","updated_at":"2024-06-10T23:15:14.000Z","dependencies_parsed_at":"2022-08-08T02:16:16.596Z","dependency_job_id":null,"html_url":"https://github.com/btubbs/pgq","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/btubbs/pgq","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/btubbs%2Fpgq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/btubbs%2Fpgq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/btubbs%2Fpgq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/btubbs%2Fpgq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/btubbs","download_url":"https://codeload.github.com/btubbs/pgq/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/btubbs%2Fpgq/sbom","scorecard":{"id":256915,"data":{"date":"2025-08-11","repo":{"name":"github.com/btubbs/pgq","commit":"0a3335913e86a402013ee81a9e45ffbe502bbffe"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":3,"checks":[{"name":"Code-Review","score":0,"reason":"Found 0/26 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"SAST","score":0,"reason":"no SAST tool detected","details":["Warn: no pull requests merged into dev branch"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}}]},"last_synced_at":"2025-08-17T09:54:56.134Z","repository_id":34915140,"created_at":"2025-08-17T09:54:56.135Z","updated_at":"2025-08-17T09:54:56.135Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28651783,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-22T01:17:37.254Z","status":"online","status_checked_at":"2026-01-22T02:00:07.137Z","response_time":144,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["golang","postgresql","queue","worker"],"created_at":"2026-01-22T02:31:33.960Z","updated_at":"2026-01-22T02:31:35.805Z","avatar_url":"https://github.com/btubbs.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pgq [![Build Status](https://travis-ci.org/btubbs/pgq.svg?branch=master)](https://travis-ci.org/btubbs/pgq) [![Coverage Status](https://coveralls.io/repos/github/btubbs/pgq/badge.svg?branch=master)](https://coveralls.io/github/btubbs/pgq?branch=master)\n\npgq is a Go library for job queues that use Postgres for persistence.  It builds on the [SKIP\nLOCKED](https://blog.2ndquadrant.com/what-is-select-skip-locked-for-in-postgresql-9-5/)\nfunctionality added in Postgres 9.5, which provides safe locking of in-progress jobs while keeping \n queries very simple and readable.\n\n## Installation and Setup\n\nYou can install pgq with `go get`:\n\n    go get github.com/btubbs/pgq\n\nBefore you can enqueue or process jobs with pgq, your Postgres database will need the table and\nindex defined in `sql/create_table.sql`.  You can paste this right into `psql` command line while\nconnected to your database, or paste it into the migration tool of your choice.  (I'm partial to\n[Pomegranate](https://github.com/btubbs/pomegranate).)\n\n## Usage\n\nThere are a few steps to using pgq:\n\n1. Define job handler functions.\n2. Instantiate a new Worker.\n3. Register your handler functions with the Worker.\n4. Start the Worker.\n5. In another process, enqueue jobs with a Worker.\n\nEach of these steps will be explained in more detail below.\n\n### Defining Job Handlers\n\nWhile pgq will take care of pulling jobs off the queue, it's up to you to tell it exactly what each\njob does.  This is done by defining and registering handler functions.  A job handler func takes one\n`[]byte` argument, and returns either an error or `nil`.  Because job payloads are sent as bytes,\nyou can put whatever data you need in there.  It may be plain text, a URL, JSON, or a binary\nencoding like protobuf.\n\nHere's a simple example:\n\n```go\nfunc SayHello(data []byte) error {\n  name := string(data)\n  fmt.Println(\"Hello \" + name)\n}\n```\n\nThat example might be too simple for a real app though.  You'll probably want your jobs to have\naccess to some resources, like config or a database.  In that case you can use either struct methods or\nclosures to inject those resources.  Here's an example of a struct method job handler:\n\n```go\ntype MyApp struct {\n  db *sql.DB\n}\n\nfunc (app *MyApp) SayGoodbye(data []byte) error {\n  // this is a silly example of a DB query, but illustrates the point.\n  var message string\n  err := app.db.QueryRow(`SELECT 'Goodbye ' || $1`, string(data)).Scan(\u0026message)\n  if err != nil {\n    return err\n  }\n  fmt.Println(message)\n}\n```\n\nHere's the equivalent using a closure:\n\n```go\npackage main\n\nfunc main() {\n  db, _ := sql.Open(\"postgres\", \"postgres://postgres@/my_database?sslmode=disable\")\n\n  myJobHandler := buildGoodbyeHandler(db)\n}\n\nfunc buildGoodbyeHandler(db *sql.DB) func([]byte) error {\n\treturn func(data []byte) error {\n\t\t// we have access to db here because this function is declared in the same scope that db lives in\n\t\tvar message string\n\t\terr := db.QueryRow(`SELECT 'Goodbye ' || $1`, string(data)).Scan(\u0026message)\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\t\tfmt.Println(message)\n\t}\n}\n```\n\n(That example, like several below omits some error handling code.  Don't do that in real life.  It's\nonly done here so you can see details about using pgq without getting lost in error handling noise.)\n\n### Processing Jobs with the Worker \n\nOnce you've defined a handler function, you need to register it with an instance of `pgq.Worker`.\nYou can get one of those by passing your Postgres `*sql.DB` instance into `pgq.NewWorker`.\n\n```go\ndb, _ := sql.Open(\"postgres\", \"postgres://postgres@/my_database?sslmode=disable\")\nworker := pgq.NewWorker(db)\n```\n\nNow you can register your job handler functions with this worker by calling its `RegisterQueue`\nmethod and passing two arguments: the queue name (which can be any string you like), and your\nhandler function.\n\n```go\nerr := worker.RegisterQueue(\"hello\", SayHello)\n```\n\nIf you're using struct methods as job handler functions, you need to instantiate your struct and\nthen register its methods.  This example uses the `MyApp` struct shown above.\n\n```go\ndb, _ := sql.Open(\"postgres\", \"postgres://postgres@/my_database?sslmode=disable\")\nmyApp := \u0026MyApp{db: db}\nerr := worker.RegisterQueue(\"goodbye\", myApp.SayGoodbye)\n```\n\nIf you attempt to register the same queue name more than once on the same worker, you'll get an\nerror.\n\nWith your handler functions now registered, you start the job worker by calling its `Run`\nmethod.  This call will query for uncompleted jobs and call the appropriate handler function for each\nof them.\n\n```go\nerr := worker.Run()\n```\n\nThis will loop and perform jobs forever.  It will only stop if it hits an unrecoverable error, or\nthe program is terminated, or your program stops the loop by sending a value on the worker's stop\nchannel, like this:\n\n```go\nworker.StopChan \u003c- true\n```\n\nThe worker will only query for jobs that have a queue name that matches one of its registered\nhandlers.  If there's a job in the queue with an unregistered queue name, it will be ignored.  You\ncan use this feature to start separate processes for handling different job types.  This is useful\nif some queues need to be scaled up to more worker processes than others.\n\n### Putting Jobs on the Queue\n\nTo enqueue jobs you need to have a Worker instance and then call its `EnqueueJob` method, which\ntakes a queue name, a `[]byte` payload, and some optional arguments we'll cover later.  `EnqueueJob`\nwill return an integer job ID (only useful for logging/debugging), and possibly an error.\n\n```go\ndb, _ := sql.Open(\"postgres\", \"postgres://postgres@/my_database?sslmode=disable\")\nworker := pgq.NewWorker(db)\njobID, err := worker.EnqueueJob(\"hello\", []byte(\"Brent\"))\n```\n\nIf you're using the same Postgres database for both pgq and your own application tables, you might\nwant to have the enqueueing of jobs happen in the same database transaction where you're doing other\ndatabase calls.  In that case, you can use the worker's `EnqueueJobInTx` method, which lets you pass\nin your own transaction object.  It's up to you to ensure that the transaction is properly committed\nor rolled back. Example:\n\n```go\ndb, _ := sql.Open(\"postgres\", \"postgres://postgres@/my_database?sslmode=disable\")\nworker := pgq.NewWorker(db)\n\n// later on in some other part of your app...\ntx, _ := db.Begin()\n\n// we use the same transaction for creating the user and enqueueing the welcome email\n// so both will succeed or fail together, and we avoid sending a welcome email to a user \n// whose account never actually got created because the tx got rolled back.\ncreateUser(tx, someUserData)\njobID, err := worker.EnqueueJobInTx(tx, \"sendWelcomeEmail\", someWelcomeEmailData)\n\nif thereWasAnErrorSomewhereElse {\n  tx.Rollback()\n} else {\n  tx.Commit()\n}\n\n```\n\n### Additional Features \n\nThere are some additional things you can configure, both at the per-worker level and the per-job\nlevel.\n\n#### Runner Options\n\nThese options are passed as additional arguments when calling `pgq.NewWorker`, using the [functional\noptions](https://dave.cheney.net/2014/10/17/functional-options-for-friendly-apis) pattern.\n\n##### JobPollingInterval\n\nAfter a worker completes a job, it immediately queries for another one.  If there are no jobs\nwaiting in a queue, it will sleep for a few seconds before querying again.  By default that's 10\nseconds.  You can change that default by passing in the `JobPollingInterval` option with a\ntime.Duration to `NewWorker`.  In this example we increase it to 30 seconds.\n\n```go\nworker, err := pgq.NewWorker(\n  db,\n  pgq.JobPollingInterval(time.Second * 30),\n)\n```\n\n##### PreserveCompletedJobs\n\nBy default, jobs are deleted from the `pgq_jobs` table after being performed.  If you would prefer\nto leave them in the table for analytics or debugging purposes, you can pass the\n`PreserveCompletedJobs` option to `pgq.NewWorker`:\n\n```go\nworker, err := pgq.NewWorker(\n  db,\n  pgq.PreserveCompletedJobs,\n)\n```\n\n#### Job Options\n\n##### After\n\nIf you want a job to be processed at some time in the future instead of immediately, you can pass\nthe `pgq.After` option to the worker's `EnqueueJob` method.  This option takes a `time.Time`.  This\nexample sets a job to be run 24 hours in the future.\n\n```go\njobID, err := worker.EnqueueJob(\n  \"sendWelcomeEmail\",\n  someWelcomeData,\n  pgq.After(time.Now().Add(time.Minute * 60 * 24)),\n)\n```\n\n##### RetryWaits\n\nSometimes jobs fail through no fault of their own.  It could be because of downtime in a database,\nor a third party API, or a flaky network.  It can be useful to retry jobs automatically when such\nthings happen.\n\nBut you don't want to retry immediately.  A problem that's happening now will probably still be\nhappening one second from now.\n\nFor this reason pgq supports setting a per-job option for the number of retries to perform and how\nlong to wait before each.  If a job returns an error, and it's configured to allow one or more\nretries, then it will be run again.\n\nBy default pgq does three retries, with one minute, ten minute, and 30 minute waits. You can\noverride this by passing the `RetryWaits` option to the worker's `EnqueueJob` method.  This option\ntakes in a slice of durations specifying how long each wait should be. In this example we set two\nretries, one after an hour and another after 6 hours:\n\n```go\njobID, err := worker.EnqueueJob(\n  \"sendWelcomeEmail\",\n  someWelcomeData,\n  pgq.RetryWaits([]time.Duration{\n    time.Minute * 60,\n    time.Minute * 60 * 6,\n  }),\n)\n```\n\n#### Exponential Backoffs\n\nWhile retries may help a single job be resilient to intermittent failures, they won't slow down a\nbig queue of jobs that are all failing because some backend system is down.  For that you need to\ntell the worker to slow down its processing of jobs on the whole queue.  You can do that by having\nyour job return an error that implements this interface:\n\n```go\n// Backoffer is a type of error that can also indicate whether a queue should slow down.\ntype Backoffer interface {\n\terror\n\tBackoff() bool\n}\n```\n\nIf the error returned by your job implements that interface, and its `Backoff()` method returns\n`true`, then the worker will pause for a short time (100 milliseconds) before pulling jobs off that\nqueue again.  The next time a job returns a backoff, that time will be doubled, and so on until\nreaching the max backoff time (60 seconds).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbtubbs%2Fpgq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbtubbs%2Fpgq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbtubbs%2Fpgq/lists"}