{"id":19912383,"url":"https://github.com/nesterow/queue","last_synced_at":"2025-05-03T03:31:32.521Z","repository":{"id":211200236,"uuid":"728451828","full_name":"nesterow/queue","owner":"nesterow","description":"A fault-tolerant cron scheduler and message queue for Deno backed by PostgreSQL.","archived":true,"fork":false,"pushed_at":"2023-12-18T22:20:30.000Z","size":30,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-01T08:45:15.815Z","etag":null,"topics":["celery","deno","javascript","postgresql","queue","typescript"],"latest_commit_sha":null,"homepage":"https://deno.land/x/que","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nesterow.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-07T01:07:50.000Z","updated_at":"2024-09-09T23:54:39.000Z","dependencies_parsed_at":"2024-11-12T21:31:44.523Z","dependency_job_id":"ac3cadff-d978-4263-880e-ccc8a706d392","html_url":"https://github.com/nesterow/queue","commit_stats":null,"previous_names":["nesterow/queue"],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nesterow%2Fqueue","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nesterow%2Fqueue/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nesterow%2Fqueue/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nesterow%2Fqueue/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nesterow","download_url":"https://codeload.github.com/nesterow/queue/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251225475,"owners_count":21555282,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["celery","deno","javascript","postgresql","queue","typescript"],"created_at":"2024-11-12T21:29:11.315Z","updated_at":"2025-05-03T03:31:32.282Z","avatar_url":"https://github.com/nesterow.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Que\n\nA fault-tolerant cron scheduler and message queue for Deno backed by PostgreSQL.\n\n- No dependencies (pg only)\n- Cron pattern support + expire \u0026 once\n- Scales horisontaly\n- Scales vertically\n\n**Motivation:**\n\nI needed a queue that only uses postgres as a backend and works with Deno so I\nmade one.\n\n## Settings\n\nRequired environment variables:\n\n```bash\nDATABASE_URL=postgresql://postgres:postgres@127.0.0.1:54322/queues\n```\n\n## Scheduler basic usage\n\nThe scheduler executes functions for the task `type` on cron schedule.\n\n```typescript\nimport { scheduler } from \"https://deno.land/x/que/mod.ts\";\n\nawait scheduler.start();\n\nscheduler.register(\"fetch\", async (ctx, onError) =\u003e {\n  onError((e) =\u003e {\n    console.error(e);\n  });\n  await fetch(ctx.url);\n});\n\n// fetch some resource every second\nconst fetchEverySec = {\n  type: \"fetch\",\n  pattern: \"* * * * * *\",\n  description: \"fetch url\",\n  context: { url: \"http://google.com\" },\n};\n\nawait scheduler.schedule(fetchEverySec);\n// the next line doesn't have effect,\n// because the task is the same\nawait scheduler.schedule(fetchEverySec);\n```\n\nExecute once at midnight, and deactivate the task:\n\n```typescript\nconst executeOnce = {\n  type: \"fetch\",\n  pattern: \"0 0 * * *\",\n  description: \"fetch url\",\n  context: { url: \"http://google.com\" },\n  once: true,\n};\nawait scheduler.schedule(executeOnce);\n```\n\nExecute every second for one hour:\n\n```typescript\nconst expireInHour = {\n  type: \"fetch\",\n  pattern: \"* * * * * *\",\n  description: \"fetch url\",\n  context: { url: \"http://google.com\" },\n  expire_at: new Date(new Date().getTime() + 1000 * 60 * 60),\n};\nawait scheduler.schedule(expireInHour);\n```\n\nUnschedule the tasks:\n\n```typescript\nawait scheduler.unschedule(fetchEverySec);\nawait scheduler.unschedule(executeOnce);\nawait scheduler.unschedule(expireInHour);\n```\n\n## Queue basic usage\n\nThe queue handles messages in recieving order as soon as possible.\n\n```typescript\nimport { Queue } from \"https://deno.land/x/que/mod.ts\";\nimport type { TimeoutError } from \"https://deno.land/x/que/mod.ts\";\n\nconst que = new Queue({\n  name: \"que-uniq-name\",\n});\n\nawait que.listen(async (msg, onError) =\u003e {\n  const some = new Some();\n  onError((e) =\u003e {\n    // we can abort\n    if (e instanceof TimeoutError) {\n      some.abort();\n    }\n  });\n  // handle msg\n  await some.send(\n    msg.type,\n    msg.uuid,\n    msg.payload,\n  );\n});\n\nawait queue.enqueue({\n  type: \"email\",\n  payload: {\n    to: \"to@test.moc\",\n    from: \"from@test.moc\",\n    body: \"Message\",\n  },\n});\n```\n\n### Queue methods\n\nThe Queue methods are self-descriptive. The constructor options requires\nproviding the `name`\n\n```typescript\nimport { Queue } from 'https://deno.land/x/que/mod.ts'\n\nconst que = new Queue({\n  name: \"name\",\n})\n\nque.listen(callback: ListenCallback)\nque.enqueue(..msgs: { type: string, payload: JsonValue, timeout_ms?: number }) // uuid[]\nque.statusOf(uuid: string)\nque.pause(state: boolean)\nque.close()\n```\n\n### Constructor options\n\n```typescript\ntype QueueOptions = {\n  name: string;\n  max_attempts?: number;\n  batch_size?: number;\n  throttle?: number;\n  storage?: Storage;\n};\n```\n\n_name_ - Required. Unique queue name.\n\n_max_attempts_ - Maximum attempts made after failing\n\n_batch_size_ - Number of the items one instance takes to process at a tick, the\ndefault is `2`. You can use it in pair with `throttle` to control RPS.\n\n_throttle_ - Default is 0. This option controls queque tick timeout. Normally\nused on a master queue when you need to control RPS.\n\n_storage_ - A custom storage implementation.\n\n## Vertical scaling\n\nWhen running on single machine sometimes it makes sence to increase number of\nworkers to speed up the queue processing. This solution provides an `Executor`\nclass which is responsible for worker sync and managemet. The workers spawned\nthrough the `Executor` will sync `pause` and `close` states.\n\n**Environment**\n\nFollowing environment variables control the executor limits:\n\n```bash\nEXECUTOR_WORKER_NUMBER=2 # spawn workers\nEXECUTOR_WORKER_LIMIT=4  # max workers\n```\n\n**Example**\n\n```typescript\nconst exec = new Executor({\n  url: new URL(\"./worker.ts\", import.meta.url),\n  name: \"queue\",\n});\n// exec.increase()\n```\n\n## Horizontal scaling\n\nThe queue workers can work on any machines as long as utilizing one database.\nThe only thing to keep in mind is that the instances are not going to syncronize\n`pause` and `close` states.\n\n## Execution order\n\nThe queue handles messages in recieving order, however the _execution order_ is\nonly guarantied when utilizing a single instance. If you need the _execution\norder_ in tact when scaling the workers - it is possible, relate to the known\npatterns and techniques.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnesterow%2Fqueue","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnesterow%2Fqueue","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnesterow%2Fqueue/lists"}