{"id":13413242,"url":"https://github.com/albrow/jobs","last_synced_at":"2026-01-11T02:57:14.227Z","repository":{"id":27094307,"uuid":"30561475","full_name":"albrow/jobs","owner":"albrow","description":"A persistent and flexible background jobs library for go.","archived":true,"fork":false,"pushed_at":"2018-06-16T21:00:16.000Z","size":217,"stargazers_count":499,"open_issues_count":17,"forks_count":47,"subscribers_count":20,"default_branch":"master","last_synced_at":"2024-07-30T20:43:28.796Z","etag":null,"topics":["go","golang","redis","worker-pool"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/albrow.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-02-09T22:13:29.000Z","updated_at":"2024-06-27T06:40:47.000Z","dependencies_parsed_at":"2022-07-14T01:20:32.326Z","dependency_job_id":null,"html_url":"https://github.com/albrow/jobs","commit_stats":null,"previous_names":["albrow/zazu"],"tags_count":12,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/albrow%2Fjobs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/albrow%2Fjobs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/albrow%2Fjobs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/albrow%2Fjobs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/albrow","download_url":"https://codeload.github.com/albrow/jobs/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243635293,"owners_count":20322912,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["go","golang","redis","worker-pool"],"created_at":"2024-07-30T20:01:35.917Z","updated_at":"2026-01-11T02:57:14.187Z","avatar_url":"https://github.com/albrow.png","language":"Go","funding_links":[],"categories":["Job Scheduler","Miscellaneous","作业调度","Relational Databases","杂项","作业调度器","\u003cspan id=\"其他-miscellaneous\"\u003e其他 Miscellaneous\u003c/span\u003e","雜項","其他"],"sub_categories":["Search and Analytic Databases","Advanced Console UIs","SQL 查询语句构建库","高级控制台界面","检索及分析资料库","\u003cspan id=\"高级控制台用户界面-advanced-console-uis\"\u003e高级控制台用户界面 Advanced Console UIs\u003c/span\u003e","高級控制台界面","交流"],"readme":"Jobs\n====\n\nDevelopment Status\n------------------\n\nJobs is ***no longer*** being actively developed. I will still try my best to\nrespond to issues and pull requests, but in general you should not expect much\nsupport. No new features will be added. Still, Jobs is reasonably\nwell-tested, and it is probably fine to use it for low-traffic hobby sites. If\nyou are looking for something for more serious, production use-cases, consider\nalternatives such as [RabbitMQ](https://github.com/streadway/amqp).\n\nJobs follows semantic versioning but offers no guarantees of backwards compatibility until version\n1.0.\n\nAbout\n-----\n\nJobs is a persistent and flexible background jobs library for go.\n\n[![Version](https://img.shields.io/badge/version-0.4.2-5272B4.svg)](https://github.com/albrow/jobs/releases)\n[![Circle CI](https://img.shields.io/circleci/project/albrow/jobs.svg)](https://circleci.com/gh/albrow/jobs)\n[![GoDoc](https://godoc.org/github.com/albrow/jobs?status.svg)](https://godoc.org/github.com/albrow/jobs)\n\nJobs is powered by Redis and supports the following features:\n\n - A job can encapsulate any arbitrary functionality. A job can do anything\n   which can be done in a go function.\n - A job can be one-off (only executed once) or recurring (scheduled to\n   execute at a specific interval).\n - A job can be retried a specified number of times if it fails.\n - A job is persistent, with protections against power loss and other worst\n   case scenarios. (See the [Guarantees](#guarantees) section below)\n - Work on jobs can be spread amongst any number of concurrent workers across any\n   number of machines.\n - Provided it is persisted to disk, every job will be executed *at least* once,\n \tand in ideal network conditions will be executed *exactly* once. (See the\n \t[Guarantees](#guarantees) section below)\n - You can query the database to find out e.g. the number of jobs that are\n   currently executing or how long a particular job took to execute.\n - Any job that permanently fails will have its error captured and stored.\n\n\nWhy is it Useful?\n-----------------\n\nJobs is intended to be used in web applications. It is useful for cases where you need\nto execute some long-running code, but you don't want your users to wait for the code to\nexecute before rendering a response. A good example is sending a welcome email to your users\nafter they sign up. You can use Jobs to schedule the email to be sent asynchronously, and\nrender a response to your user without waiting for the email to be sent. You could use a\ngoroutine to accomplish the same thing, but in the event of a server restart or power loss,\nthe email might never be sent. Jobs guarantees that the email will be sent at some time\nand allows you to spread the work between different machines.\n\n\nInstallation\n------------\n\nJobs requires Go version \u003e= 1.2. If you do not already have it, follow these instructions:\n\n- [Install Go](http://golang.org/doc/install)\n- Follow the instructions for [setting up your go workspace](https://golang.org/doc/code.html)\n\nJobs requires access to a Redis database. If you plan to have multiple worker pools spread\nout across different machines, they should all connect to the same Redis database. If you\nonly want to run one worker pool, it is safe to install Redis locally and run it on the same\nmachine. In either case, if you need to install Redis, follow these instructions:\n\n- [Install Redis](http://redis.io/download).\n- Follow the instructions in the section called\n  [Installing Redis more properly](http://redis.io/topics/quickstart#installing-redis-more-properly).\n- Make sure you understand how [Redis Persistence](http://redis.io/topics/persistence) works and have\n  edited your config file to get your desired persistence. We recommend using both RDB and AOF and setting\n  fsync to either \"always\" or \"everysec\".\n\nAfter that, you can install Jobs like you would any other go package: `go get github.com/albrow/jobs`.\nIf you want to update the package later, use `go get -u github.com/albrow/jobs`. Then you can import\nJobs like you would any other go package by adding `import github.com/albrow/jobs` to your go source\nfile.\n\n\nQuickstart Guide\n----------------\n\n### Connecting to Redis\n\nYou can configure the connection to Redis by editing Config.Db. Here are the options:\n\n- Address is the address of the redis database to connect to. Default is\n  \"localhost:6379\".\n- Network is the type of network to use to connect to the redis database\nDefault is \"tcp\".\n- Database is the redis database number to use for storing all data. Default\n  is 0.\n- Password is a password to use for connecting to a redis database via the\n  AUTH command. If empty, Jobs will not attempt to authenticate. Default is\n  \"\" (an empty string).\n\nYou should edit Config.Db during program initialization, before running Pool.Start\nor scheduling any jobs. Here's an example of how to configure Jobs to use databse #10\nand authenticate with the password \"foobar\":\n\n``` go\nfunc main() {\n\t// Configure database options at the start of your application\n\tjobs.Config.Db.Database = 10\n\tjobs.Config.Db.Password = \"foobar\"\n}\n```\n\n### Registering Job Types\n\nJobs must be organized into discrete types. Here's an example of how to register a job\nwhich sends a welcome email to users:\n\n``` go\n// We'll specify that we want the job to be retried 3 times before finally failing\nwelcomeEmailJobs, err := jobs.RegisterType(\"welcomeEmail\", 3, func(user *User) error {\n\tmsg := fmt.Sprintf(\"Hello, %s! Thanks for signing up for foo.com.\", user.Name)\n\tif err := emails.Send(user.EmailAddress, msg); err != nil {\n\t\t// The returned error will be captured by a worker, which will then log the error\n\t\t// in the database and trigger up to 3 retries.\n\t\treturn err\n\t}\n})\n```\n\nThe final argument to the RegisterType function is a [HandlerFunc](http://godoc.org/github.com/albrow/jobs#HandlerFunc)\nwhich will be executed when the job runs. HandlerFunc must be a function which accepts either\nzero or one arguments and returns an error.\n\n### Scheduling a Job\n\nAfter registering a job type, you can schedule a job using the Schedule or ScheduleRecurring\nmethods like so:\n\n``` go\n// The priority argument lets you choose how important the job is. Higher\n// priority jobs will be executed first.\njob, err := welcomeEmailJobs.Schedule(100, time.Now(), \u0026User{EmailAddress: \"foo@example.com\"})\nif err != nil {\n\t// Handle err\n}\n```\n\nYou can use the [Job object](http://godoc.org/github.com/albrow/jobs#Job) returned by Schedule\nor ScheduleRecurring to check on the status of the job or cancel it manually.\n\n### Starting and Configuring Worker Pools\n\nYou can schedule any number of worker pools across any number of machines, provided every machine\nagrees on the definition of the job types. If you want, you can start a worker pool on the same\nmachines that are scheduling jobs, or you can have each worker pool running on a designated machine.\nSince each pool is assigned an id based on a unique hardware identifier, you must only run one\nworker pool per machine.\n\nTo create a new pool with the [default configuration](http://godoc.org/github.com/albrow/jobs#pkg-variables),\njust pass in nil:\n\n``` go\npool, err := jobs.NewPool(nil)\nif err != nil {\n\t// Handle err\n}\n```\n\nYou can also specify a different configuration by passing in\n[*PoolConfig](http://godoc.org/github.com/albrow/jobs#PoolConfig). Any zero values in the config you pass\nin will fallback to the default values. So here's how you could start a pool with 10 workers and a batch\nsize of 10, while letting the other options remain the default.\n\n``` go\npool, err := jobs.NewPool(\u0026jobs.PoolConfig{\n\tNumWorkers: 10,\n\tBatchSize: 10,\n})\nif err != nil {\n\t// Handle err\n}\n```\n\nAfter you have created a pool, you can start it with the Start method. Once started, the pool will\ncontinuously query the database for new jobs and delegate those jobs to workers. Any program that calls\nPool.Start() should also wait for the workers to finish before exiting. You can do so by wrapping Close and\nWait in a defer statement. Typical usage looks something like this:\n\n``` go\nfunc main() {\n\tpool, err := jobs.NewPool(nil)\n\tif err != nil {\n\t\t// Handle err\n\t}\n\tdefer func() {\n\t\tpool.Close()\n\t\tif err := pool.Wait(); err != nil {\n\t\t\t// Handle err\n\t\t}\n\t}()\n\tif err := pool.Start(); err != nil {\n\t\t// Handle err\n\t}\n}\n```\n\nYou can also call Close and Wait at any time to manually stop the pool from executing new jobs. In this\ncase, any jobs that are currently being executed will still finish.\n\n\nTesting\n-------\n\nTo run the tests, make sure you have Redis running and accepting unix socket connections on the address\n/tmp/redis.sock. The tests will use database #14. **WARNING:** After each test is run, database #14 will be completely\nerased, so make sure you do not have any important data stored there.\n\nTo run the tests just run `go test .` If anything fails, please report an issue and describe what happened.\n\n\nContributing\n------------\n\nSee [Contributing.md](https://github.com/albrow/jobs/blob/master/CONTRIBUTING.md)\n\n\nGuarantees\n-----------\n\n### Persistence\n\nSince jobs is powered by Redis, there is a chance that you can lose data with the default Redis configuration.\nTo get the best persistence guarantees, you should set Redis to use both AOF and RDB persistence modes and set\nfsync to \"always\". With these settings, Redis is more or less\n[as persistent as a database like postgres](http://redis.io/topics/persistence#ok-so-what-should-i-use). If want\nbetter performance and are okay with a slightly greater chance of losing data (i.e. jobs not executing), you can\nset fsync to \"everysec\".\n\n[Read more about Redis persistence](http://redis.io/topics/persistence).\n\n### Atomicity\n\nJobs is carefully written using Redis transactions and lua scripting so that all database changes are atomic.\nIf Redis crashes in the middle of a transaction or script execution, it is possible that your AOF file can become\ncorrupted. If this happens, Redis will refuse to start until the AOF file is fixed. It is relatively easy to fix\nthe problem with the redis-check-aof tool, which will remove the partial transaction from the AOF file. In effect,\nthis guarantees that modifications of the database are atomic, even in the event of a power loss or hard reset,\nwith the caveat that you may need to use the redis-check-aof tool in the worst case scenario.\n\nRead more about [Redis transactions](http://redis.io/topics/transactions) and\n[scripts](http://redis.io/commands#scripting).\n\n### Job Execution\n\nJobs guarantees that a job will be executed *at least* once, provided it has been persisted on disk. (See the section\non [Persistence](#persistence) directly above). A job can only picked up by one pool at a time because a pool\natomically pops (gets and immediately moves) the next available jobs from the database. A job can only be executed\nby one worker at a time because the jobs are delegated to workers via a shared channel. Each worker pool checks on\nthe health of all the other pools when it starts. If a pool crashes or is otherwise disconnected, any jobs it had\ngrabbed from the database that did not yet finish will be re-queued and picked up by a different pool.\n\nThis is in no way an exhaustive list, but here are some known examples of scenarios that may cause a job to be\nexecuted more than once:\n\n1. If there is a power failure or hard reset while a worker is in the middle of executing a job, the job may be\n   stuck in a half-executed state. Since there is no way to know how much of the job was successfully completed,\n   the job will be re-queued and picked up by a different pool, where it may be partially or fully executed\n   more than once.\n2. If a pool becomes disconnected, it will be considered stale and its jobs will be re-queued and reclaimed\n   by a different pool. However, if the stale pool is able to partly or fully execute jobs without a reliable\n   internet connection, any jobs belonging to the stale pool might be executed more than once. You can increase\n   the [StaleTimeout](https://godoc.org/github.com/albrow/jobs#PoolConfig) parameter for a pool to make this\n   scenario less likely.\n\n\nLicense\n-------\n\nJobs is licensed under the MIT License. See the LICENSE file for more information.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falbrow%2Fjobs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falbrow%2Fjobs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falbrow%2Fjobs/lists"}