{"id":37154321,"url":"https://github.com/kevinburke/rickover","last_synced_at":"2026-01-14T18:12:10.707Z","repository":{"id":45922719,"uuid":"58006693","full_name":"kevinburke/rickover","owner":"kevinburke","description":"A job queue and scheduler written in Go, backed by Postgres, and available over HTTP","archived":false,"fork":true,"pushed_at":"2023-11-07T18:29:49.000Z","size":5343,"stargazers_count":25,"open_issues_count":5,"forks_count":2,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-12-22T18:03:00.598Z","etag":null,"topics":["golang","job-queue","postgresql"],"latest_commit_sha":null,"homepage":"https://godoc.org/github.com/kevinburke/rickover","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":"Shyp/rickover","license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kevinburke.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-05-03T22:55:28.000Z","updated_at":"2024-01-10T02:37:51.000Z","dependencies_parsed_at":"2023-02-08T19:46:29.911Z","dependency_job_id":null,"html_url":"https://github.com/kevinburke/rickover","commit_stats":null,"previous_names":[],"tags_count":24,"template":false,"template_full_name":null,"purl":"pkg:github/kevinburke/rickover","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kevinburke%2Frickover","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kevinburke%2Frickover/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kevinburke%2Frickover/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kevinburke%2Frickover/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kevinburke","download_url":"https://codeload.github.com/kevinburke/rickover/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kevinburke%2Frickover/sbom","scorecard":{"id":556949,"data":{"date":"2025-08-11","repo":{"name":"github.com/kevinburke/rickover","commit":"7ed8402e7e087c5e610b2ac38ace874c2bb0eb02"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":3.4,"checks":[{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Code-Review","score":0,"reason":"Found 0/30 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Token-Permissions","score":0,"reason":"detected GitHub workflow tokens with excessive permissions","details":["Warn: no topLevel permission defined: .github/workflows/ci.yml:1","Info: no jobLevel write permissions found"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:10: update your workflow using https://app.stepsecurity.io/secureworkflow/kevinburke/rickover/ci.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:14: update your workflow using https://app.stepsecurity.io/secureworkflow/kevinburke/rickover/ci.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:22: update your workflow using https://app.stepsecurity.io/secureworkflow/kevinburke/rickover/ci.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:28: update your workflow using https://app.stepsecurity.io/secureworkflow/kevinburke/rickover/ci.yml/master?enable=pin","Warn: goCommand not pinned by hash: .github/workflows/ci.yml:37","Info:   0 out of   4 GitHub-owned GitHubAction dependencies pinned","Info:   0 out of   1 goCommand dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 5 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-20T12:37:40.816Z","repository_id":45922719,"created_at":"2025-08-20T12:37:40.816Z","updated_at":"2025-08-20T12:37:40.816Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28429929,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-14T16:38:47.836Z","status":"ssl_error","status_checked_at":"2026-01-14T16:34:59.695Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["golang","job-queue","postgresql"],"created_at":"2026-01-14T18:12:09.924Z","updated_at":"2026-01-14T18:12:10.702Z","avatar_url":"https://github.com/kevinburke.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Rickover\n\nThis holds the code for a scheduler and a job queue written in Go and\nbacked by Postgres.\n\nThe goals/features of this project are:\n\n- Visibility into the system using a tool our team is familiar with (Postgres)\n- Correctness - jobs shouldn't get stuck, or dequeued twice, unless that's\n  desirable\n- Good memory performance - with 300 dequeuers, the server and worker take\n  about 30MB in total.\n- No long-running database queries, or open transactions\n- All queue actions have an HTTP API.\n- All enqueue/dequeue actions are idempotent - it's OK if any part of the\n  system gets restarted\n\nIt might not be the most performant, but it should be easy to use and deploy!\n\n## Server Endpoints\n\nThe only supported content type for uploads and responses is JSON.\n\n#### Create a job type\n\nTo start enqueueing and dequeueing work, you need to create a job type. Define\na job type with a name, a delivery strategy (idempotent == `\"at_least_once\"`,\nnot idempotent == `\"at_most_once\"`), and a concurrency - the maximum number\nof jobs that can be in flight at once. If the job is idempotent, you can add\n\"attempts\" - the number of times to try to send the job to the downstream\nserver before giving up.\n\n```\nPOST /v1/jobs\n{\n    \"id\": \"invoice-shipments\",\n    \"delivery_strategy\": \"at_least_once\",\n    \"attempts\": 3,\n    \"concurrency\": 5\n}\n```\n\nThis returns a [models.Job][job-type] on success.\n\nCreating a new job will also send a signal to the dequeuer to ask it to restart\nitself (so it can create workers to process jobs using the new job type). If you\ndo not want to enable this behavior, set `DisableMetaShutdown: true` in the\n`Config` for the dequeuer or for the server.\n\n[job-type]: https://godoc.org/github.com/kevinburke/rickover/models#Job\n\n#### Enqueue a new job\n\nOnce you have a job type, you can enqueue new jobs. Note the client is\nresponsible for generating a UUID.\n\n```\nPUT /v1/jobs/invoice-shipments/job_282227eb-3c76-4ef7-af7e-25dff933077f\n{\n    \"data\": {\n        \"shipmentId\": \"shp_123\",\n    }\n    \"id\": \"job_282227eb-3c76-4ef7-af7e-25dff933077f\",\n    \"run_after\": \"2016-01-11T18:26:26.000Z\",\n    \"expires_at\": \"2016-01-11T20:26:26.000Z\"\n}\n```\n\nThis inserts a record into the `queued_jobs` table and returns a\n[models.QueuedJob][queued-job]. The client can and should retry in the event of\nfailure.\n\nYou can put any valid JSON in the `data` field; we'll send this to the\ndownstream worker.\n\nThere are two special fields - `run_after` indicates the earliest possible\ntime this job can run (or `null` to indicate it can run now), and `expires_at`\nindicates the latest possible time this job can run. If a job is dequeued after\nthe `expires_at` date, we don't send it to the downstream worker, and insert it\nimmediately into the `archived_jobs` table with status `expired`.\n\n[queued-job]: https://godoc.org/github.com/kevinburke/rickover/models#QueuedJob\n\n#### Record a job's success or failure\n\nOnce the downstream worker has completed work, record the status of the job by\nmaking a POST request to the same URI.\n\n```\nPOST /v1/jobs/invoice-shipments/job_123 HTTP/1.1\n{\n    \"status\": \"succeeded\"\n    \"attempt\": 3,\n}\n```\n\nNote you must include the attempt number in your callback; we use this\nfor idempotency, and to avoid stale writes. Valid values for `status` are\n\"succeeded\" or \"failed\". If a failed job is retryable, we'll insert the job\nback into the `queued_jobs` table with a `run_after` date set a small amount\nof time in the future. If a failed job is retryable but should not be retried,\ninclude `\"retryable\": false` in the body of the POST request, which will\nimmediately archive the job.\n\n#### Replay a job\n\nThis is handy if the initial job failed, the downstream server had an outage,\nor you want to re-run something on an adhoc basis.\n\n```\nPOST /v1/jobs/invoice-shipments/job_123/replay HTTP/1.1\n```\n\nWill create a new UUID and enqueue the job to be run immediately. If you\nattempt to replay an expired job, the new job will be immediately archived with\na status of \"expired\".\n\n#### Get information about a job\n\n```\nGET /v1/jobs/invoice-shipments/job_123 HTTP/1.1\n```\n\nThis looks in the queued_jobs table first, then the archived_jobs table, and\nreturns whatever it finds. Note the fields in these tables don't match up 100%.\n\n#### List completed jobs\n\n```\nGET /v1/archived-jobs HTTP/1.1\n```\n\nReturns a list of archived jobs. Pass `name=x` to filter by jobs named `x`. Pass\n`limit=x` to limit results.\n\n### Server Authentication\n\nBy default, the server uses an in-memory secret for authentication. Call\n[`server.AddUser`][add-user] to add an authenticated user and password for the\n[DefaultServer][default-server].\n\nYou can use your own authentication scheme with any code that satisifies the\n[server.Authorizer][authorizer] interface:\n\n```go\n// Authorizer can authorize the given user and token to access the API.\ntype Authorizer interface {\n\tAuthorize(user string, token string) error\n}\n```\n\nThen, get a http.Handler with your authorizer by calling\n\n```go\nimport \"github.com/kevinburke/rickover/server\"\n\nhandler := server.Get(authorizer)\nhttp.ListenAndServe(\":9090\", handler)\n```\n\n[add-user]: https://godoc.org/github.com/kevinburke/rickover/server#AddUser\n[default-server]: https://godoc.org/github.com/kevinburke/rickover/server#pkg-variables\n[authorizer]: https://godoc.org/github.com/kevinburke/rickover/server#Authorizer\n\n## Processing jobs\n\nWhen you get a job from the database, you can do whatever you want with it -\nyour dequeuer just needs to satisfy the [Worker][worker] interface.\n\n```go\n// A Worker does some work with a QueuedJob.\ntype Worker interface {\n\tDoWork(*models.QueuedJob) error\n}\n```\n\n[worker]: https://godoc.org/github.com/kevinburke/rickover/dequeuer#Worker\n\nA default Worker is provided as [services.JobProcessor][job-processor],\nwhich makes an API request to a downstream service. The default client is\n[downstream.Client][downstream-client]. You'll need to set the URL\nand password for the downstream service:\n\n```go\nimport \"github.com/kevinburke/rickover/dequeuer\"\nimport \"github.com/kevinburke/rickover/services\"\n\nfunc main() {\n\tpassword := \"hymanrickover\"\n\t// Basic auth - username \"jobs\", password password\n\tjp := services.NewJobProcessor(\"http://downstream-service.example.com\", password)\n\n\tpools, err := dequeuer.CreatePools(jp)\n\tfmt.Println(err)\n}\n```\n\n[job-processor]: https://godoc.org/github.com/kevinburke/rickover/services#JobProcessor\n[downstream-client]: https://godoc.org/github.com/kevinburke/rickover/downstream#Client\n\nThe [downstream.Client][downstream-client] will make a POST request to\n`/v1/jobs/:job-name/:job-id`:\n\n```\nPOST /v1/jobs/invoice-shipment/job_123 HTTP/1.1\nHost: downstream.shyp.com\nContent-Type: application/json\nAccept: application/json\n{\n    \"data\": {\n        \"shipmentId\": \"shp_123\"\n    },\n    \"id\": \"job_123\",\n    \"attempts\": 3\n}\n```\n\n## Callbacks\n\nAll actions in the system are designed to be short-lived. When the downstream\nserver has finished processing the job, it should make a callback to the\nRickover server, reporting on the status of the job, with `status` set to\n`succeeded` or `failed`.\n\n```\nPOST /v1/jobs/invoice-shipments/job_123 HTTP/1.1\nHost: rickover.shyp.com\nContent-Type: application/json\n{\n    \"status\": \"succeeded\"\n    \"attempt\": 3,\n}\n```\n\nIf this request times out or errors, you can try it again; the `attempt` number\nis used to avoid making a stale update.\n\nYou can also report status of a job by calling\n[services.HandleStatusCallback][status-callback] directly, with success or\nfailure.\n\n[status-callback]: https://godoc.org/github.com/kevinburke/rickover/services#HandleStatusCallback\n\n## Failure Handling\n\nIf the downstream worker never hits the callback, the JobProcessor will time\nout after 5 minutes and mark the job as failed.\n\nIf the dequeuer gets killed while waiting for a response, we'll time out the\njob after 7 minutes, and mark it as failed. (This means the maximum allowable\ntime for a job is 7 minutes.)\n\n## Dashboard\n\nThe homepage can embed an iframe of your choice, configurable via the\n`HOMEPAGE_IFRAME_URL` environment variable. We set up a Librato space with the\nmetrics we send from this service, and embed that in the homepage:\n\n\u003cimg src=\"https://monosnap.com/file/nr399cxUwpBkEk5gAjJ5hTG4c3aR9J.png\"\u003e\n\n#### Stuck jobs\n\nIf a dequeuer gets restarted after it's Acquire()d a job but before it can send\nit downstream, or if the downstream worker gets restarted before it can hit the\ncallback, the job can get stuck in-progress indefinitely. Run WatchStuckJobs in\na goroutine to periodically check for in-progress jobs and mark them as failed:\n\n```go\n// This should be longer than the timeout in the JobProcessor\nstuckJobTimeout := 7 * time.Minute\ngo services.WatchStuckJobs(1*time.Minute, stuckJobTimeout)\n```\n\n## Database Table Layout\n\nThere are three tables, plus one for keeping track of ran migrations.\n\n- `jobs` - Contains information about a job's name, retry strategy, desired\n  concurrency.\n\n```\n                          Table \"public.jobs\"\n      Column       |           Type           |       Modifiers\n-------------------+--------------------------+------------------------\n name              | text                     | not null\n delivery_strategy | delivery_strategy        | not null\n attempts          | smallint                 | not null\n concurrency       | smallint                 | not null\n created_at        | timestamp with time zone | not null default now()\nIndexes:\n    \"jobs_pkey\" PRIMARY KEY, btree (name)\nCheck constraints:\n    \"jobs_attempts_check\" CHECK (attempts \u003e 0)\n    \"jobs_concurrency_check\" CHECK (concurrency \u003e= 0)\nReferenced by:\n    TABLE \"archived_jobs\" CONSTRAINT \"archived_jobs_name_fkey\" FOREIGN KEY (name) REFERENCES jobs(name)\n    TABLE \"queued_jobs\" CONSTRAINT \"queued_jobs_name_fkey\" FOREIGN KEY (name) REFERENCES jobs(name)\n```\n\n- `queued_jobs` - The \"hot\" table, this contains rows that are scheduled to be\ndequeued. Should be small, so queries are fast.\n\n```\n                   Table \"public.queued_jobs\"\n   Column   |           Type           |       Modifiers\n------------+--------------------------+------------------------\n id         | uuid                     | not null\n name       | text                     | not null\n attempts   | smallint                 | not null\n run_after  | timestamp with time zone | not null\n expires_at | timestamp with time zone |\n created_at | timestamp with time zone | not null default now()\n updated_at | timestamp with time zone | not null default now()\n status     | job_status               | not null\n data       | jsonb                    | not null\nIndexes:\n    \"queued_jobs_pkey\" PRIMARY KEY, btree (id)\n    \"find_queued_job\" btree (name, run_after) WHERE status = 'queued'::job_status\n    \"queued_jobs_created_at\" btree (created_at)\nCheck constraints:\n    \"queued_jobs_attempts_check\" CHECK (attempts \u003e= 0)\nForeign-key constraints:\n    \"queued_jobs_name_fkey\" FOREIGN KEY (name) REFERENCES jobs(name)\n```\n\n- `archived_jobs` - Insert-only table containing historical records of all\njobs. May grow very large.\n\n```\n            Table \"public.archived_jobs\"\n   Column   |           Type           |       Modifiers\n------------+--------------------------+------------------------\n id         | uuid                     | not null\n name       | text                     | not null\n attempts   | smallint                 | not null\n status     | archived_job_status      | not null\n created_at | timestamp with time zone | not null default now()\n data       | jsonb                    | not null\nIndexes:\n    \"archived_jobs_pkey\" PRIMARY KEY, btree (id)\nCheck constraints:\n    \"archived_jobs_attempts_check\" CHECK (attempts \u003e= 0)\nForeign-key constraints:\n    \"archived_jobs_name_fkey\" FOREIGN KEY (name) REFERENCES jobs(name)\n```\n\n## Example servers and dequeuers\n\nExample server and dequeuer instances are stored in commands/server and\ncommands/dequeuer. You will probably want to modify these to provide your own\nauthentication scheme.\n\n## Configure the server\n\nYou can use the following variables to tune the server:\n\n- `PG_SERVER_POOL_SIZE` - Maximum number of database connections from an\nindividual instance. Across every database connection in the cluster, you want\nto have the number of active Postgres connections equal to 2 * (num CPUs on the\nPostgres machine). Currently set to 15.\n\n- `PORT` - which port to listen on.\n\n- `LIBRATO_TOKEN` - This library uses Librato for metrics. This environment\nvariable sets the Librato token for publishing.\n\n- `DATABASE_URL` - Postgres database URL. Currently only connections to the\n  primary are allowed, there are not a lot of reads in the system, and all\n  queries are designed to be short.\n\n## Configure the dequeuer\n\nThe number of dequeuers is determined by the number of entries in the `jobs`\ntable. There is currently no way to adjust the number of dequeuers on the fly,\nyou must update the database and then restart the worker process.\n\n- `PG_WORKER_POOL_SIZE` - How many workers to use. Workers hit Postgres in a\nbusy loop asking for work with a `SELECT ... FOR UPDATE`, which skips rows\nif they are active, so queries from the worker tend to cause more active\nconnections than those from the server.\n\n- `DATABASE_URL` - Postgres database URL. Currently only connections to the\n  primary are allowed, there are not a lot of reads in the system, and all\n  queries are designed to be short.\n\n- `DOWNSTREAM_URL` - When you dequeue a job, hit this URL to tell something to\n  do some work.\n\n- `DOWNSTREAM_WORKER_AUTH` - Basic auth password for the downstream service\n  (user is \"jobs\").\n\n## Configure metrics\n\nThe default metrics client uses Librato - see examples of how to configure it in\n`example_server_test.go` or `example_dequeuer_test.go`. However, you can\noverride the default metrics client with any client that implements the\n`metrics.Metrics` interface, something like this:\n\n```go\nfunc (m myCustomClient) Increment(metric string) { ... }\n// implement the other interfaces\n\nfunc main() {\n    metrics.Client = myCustomClient{}\n}\n```\n\nYou can choose which metrics to send by overriding `metrics.Exclude` to\nselectively exclude some metrics. By default, no metrics are excluded.\n\n```go\nmetrics.Exclude = func(metric string) bool {\n    return !strings.HasSuffix(\"latency\") // or whatever\n}\n```\n\n## Local development\n\nWe use [goose][goose] for database migrations. The test database is\n`rickover_test` and the development database is `rickover`. The authenticating\nuser is `rickover`.\n\nTo run all migrations, run:\n\n```\ngoose --env=test up\n```\n\nTo get the database status:\n\n```\ngoose --env=test status\n```\n\nYou should also be able to use goose to run migrations in your production\nenvironment. Set the DATABASE_URL environment variable to a Postgres string,\nthen use the `cluster` environment, for example\n\n```\nDATABASE_URL=$(heroku config:get DATABASE_URL --app myapp) goose --env=cluster up\n```\n\n[goose]: https://bitbucket.org/liamstask/goose\n\n### Start the server\n\n```\nmake serve\n```\n\nWill start the example server on port 8080.\n\n### Start the dequeuer\n\n```\nmake dequeue\n```\n\nWill try to pull jobs out of the database and send them to the downstream\nworker. Note you will need to set `DOWNSTREAM_WORKER_AUTH` as the basic auth\npassword for the downstream service (the user is hardcoded to \"jobs\"), and\n`DOWNSTREAM_URL` as the URL to hit when you have a job to dequeue.\n\n## Debugging variables\n\n- `DEBUG_HTTP_TRAFFIC` - Dump all incoming and outgoing http traffic to stdout\n\n## Run the tests\n\nFirst create the test database:\n\n```\nmake test-install\n```\n\nThen run all migrations:\n\n```\nmake migrate\n```\n\nFinally, you can run the tests:\n\n```\nmake test\n```\n\nThe race detector takes longer to run, so we only enable it in CI, but\nyou can run tests with the race detector enabled:\n\n```\nmake race-test\n```\n\n## View the docs\n\nRun `make docs`, which will start a docs server on port 6060 and open it to the\nright documentation page. All public API's will be present, and most function\ncalls should be documented - some with examples.\n\n## Working with Dep\n\nDep (github.com/golang/dep) is the tool for bringing all dependencies into the project.\n\n## Benchmarks\n\nThe `Dequeue` benchmark measures dequeue performance at various concurrency\nlevels. One `kB` is one run, essentially. It does not use the server.\n\nSome benchmark numbers are here: https://docs.google.com/a/shyp.co/spreadsheets/d/1KF3pqCczDMRXZcq-ZqpQeGKo4sPclThltWhsxHdUPTc/edit?usp=sharing\n\nThe second bottleneck was the database. Note the database performed best when\nthe numbers of connection counts and dequeuers were low. In the cluster we will\nwant to have a higher number of dequeuers, simply because we aren't enqueueing\nas many jobs, it's more important to be fast when we need speed than to worry\nabout the optimal number for peak performance.\n\nIn the cluster I was able to dequeue 7,000 jobs per minute with a single $25\nweb node, a $25 dequeuer node, a $50 database and a $25 Node.js worker. The\nfirst place I would look to improve this would be to increase the number of\nNode (downstream) dynos.\n\nI used [`boom`][boom] for enqueuing jobs; I turned off the dequeuer, enqueued\n30000 jobs, then started the dequeuer and measured the timestamp difference\nbetween the first enqueued job and the last.\n\nThere's a builtin `random_id` endpoint which will generate a UUID for you, for\ndoing load testing.\n\n```\nboom -n 30000 -c 100 -d '{\"data\": {\"user-agent\": \"boom\"}}' -m PUT http://localhost:9090/v1/jobs/echo/random_id\n```\n\n[boom]: https://github.com/rakyll/boom\n\n## Testing\n\nTests hit the database, and should either be able to run in parallel with other\ntests or clean up after themselves.\n\nNote you must run tests with `-p=1`, so packages are tested in turn. Otherwise\n`t.Parallel()` will run parallel tests from different suites at the same time as\neach other, which we currently don't support.\n\n## Supported versions\n\nThe database uses `jsonb`, which is only available in Postgres 9.4 and beyond.\nThe Go server exposes `http/pprof/trace`, which is only available in Go 1.5 and\nbeyond.\n\nYou can probably fork the project to remove the http/pprof handlers and replace\njsonb with json and it should compile/run fine.\n\n## Single points of failure\n\nThis project has two points of failure:\n\n- If the database goes down, you can't enqueue any new jobs, or process\nany work. This is acceptable for most companies and projects. If this is\nunacceptable you may want to check out a distributed scheduler like Chronos, or\nuse something else for the job queue, like SQS.\n\n- If the dequeuer goes down, you can't process any work. You can run multiple\ndequeuers, or manually split the single concurrency value across multiple\nmachines, or just wait until the machine comes back up again. Note if the\ndequeuer goes down you can still enqueue jobs, the database queue will continue\nto grow.\n\n## Suggestions for scaling the project\n\n- Pull jobs out of the database in batches, and send them to the dequeuers over\nchannels.\n\n- Use it only as a scheduler, and move the job queue to SQS or something else.\n\n- Run the server on multiple machines. The worker can't run on multiple\n  machines without violating the concurrency guarantees.\n\n- Run the worker on multiple machines, and ignore or update the concurrency\n  guarantees.\n\n- Run the downstream worker on a larger number of machines.\n\n- Shard the Postgres database so jobs A-M are in one database, and N-Z are in\nanother. Would need to update `db.Conn` to be an interface, or wrap it behind a\nfunction.\n\n- Delete/archive all rows from `archived_jobs` that are older than 180 days, on\na rolling basis. I doubt this will help much, but it might.\n\n- Get a bigger Postgres database.\n\n- Upgrade to Postgres 9.5 and update the Acquire() strategy to use SKIP LOCKED.\n\n## Roadmap\n\n- Allow UPDATEs to jobs once they've been created\n\n- API for retrieving recent jobs/paging through archived jobs, by name\n\n- Dequeuer listens on a local socket/port, so you can send commands to it to\n  add/remove dequeuers on the fly.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkevinburke%2Frickover","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkevinburke%2Frickover","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkevinburke%2Frickover/lists"}