{"id":51051293,"url":"https://github.com/hexsprite/durabl","last_synced_at":"2026-06-22T17:02:06.884Z","repository":{"id":363436881,"uuid":"1263416938","full_name":"hexsprite/durabl","owner":"hexsprite","description":"A small, durable, Mongo-backed job queue — atomic claim, retries, visibility-timeout leases, dedupe keys, and change-stream push.","archived":false,"fork":false,"pushed_at":"2026-06-09T00:02:43.000Z","size":0,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-09T00:05:57.965Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hexsprite.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2026-06-09T00:01:17.000Z","updated_at":"2026-06-09T00:02:47.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/hexsprite/durabl","commit_stats":null,"previous_names":["hexsprite/durabl"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/hexsprite/durabl","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hexsprite%2Fdurabl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hexsprite%2Fdurabl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hexsprite%2Fdurabl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hexsprite%2Fdurabl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hexsprite","download_url":"https://codeload.github.com/hexsprite/durabl/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hexsprite%2Fdurabl/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34657902,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-22T02:00:06.391Z","response_time":106,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-22T17:02:04.742Z","updated_at":"2026-06-22T17:02:06.879Z","avatar_url":"https://github.com/hexsprite.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# durabl\n\nA small durable job queue backed by MongoDB. Atomic claiming, retries, visibility-timeout leases, dedupe keys, and optional change-stream push — no Redis, no separate worker service, no orchestrator.\n\n\u003e **Status: work in progress.** This is the production job queue I've run inside [Focuster](https://focuster.com) since 2016, just lifted out of the app and decoupled from Meteor. It works and it's tested, but the packaging is young: the API may still shift and the docs are thin in places. Treat `0.x` as \"useful, not yet stable.\"\n\n## Why this exists\n\nFocuster needed a durable queue for calendar sync jobs when Meteor 3 landed and the old `vsivsi:job-collection` package stopped working. I looked at the obvious options first:\n\n- **Temporal.** Full workflow orchestration with deterministic replay. Powerful, and far more machinery than four job types need. It also wants a server to run.\n- **DBOS.** Lovely API, durable workflows checkpointed to Postgres. But it's Postgres, and Focuster's system of record is MongoDB. Adding a second datastore to run background jobs is a tax I didn't want to pay.\n- **BullMQ.** The default answer in Node land, but it needs Redis. Same objection: new infrastructure for a problem the existing database already solves.\n- **Agenda, Keuss, Pulse.** The MongoDB-native options were either stale, archived, or missing features I relied on like priorities and atomic dedupe.\n\nThe actual workload is modest: a handful of job types at concurrency 2–16, polling every few seconds, with one hard requirement — **don't run the same user's sync twice at once**, even across a rolling deploy. MongoDB's `findOneAndUpdate` is exactly the primitive that solves atomic claiming, and a unique partial index solves dedupe. So the queue is ~900 lines of TypeScript over the `mongodb` driver instead of a dependency on Redis or a workflow engine.\n\nI stole the good ideas (pluggable backends from Django's task framework, the dedupe-key concept from BullMQ/SQS) and skipped the heavy ones (step-level replay from Inngest/DBOS — job-level durability is enough for now).\n\n## Features\n\n- **Atomic claim.** `findOneAndUpdate` on pending, due jobs sorted by priority. The MongoDB equivalent of `SELECT ... FOR UPDATE SKIP LOCKED`: two workers never claim the same job.\n- **Visibility-timeout leases.** A claimed job is leased, not removed. Handlers heartbeat to extend the lease, and a reaper returns jobs from dead workers to `pending`.\n- **Retries with attempt caps and backoff.** Failed jobs go back to `pending` with a jittered backoff delay until `maxAttempts`, then land in a terminal `failed` state. `failFatal()` skips retries for unrecoverable errors.\n- **Delayed and prioritized scheduling.** `runAt` delays a job; lower `priority` numbers run first.\n- **Dedupe keys, two scopes.** `pending+active` blocks any duplicate. `pending` allows one pending behind one active, which gives you single-flight coalescing: run now, queue at most one more.\n- **Push/poll hybrid.** Rides MongoDB change streams for sub-100ms pickup, with a reconnect catch-up sentinel so jobs that land during a stream blip aren't missed. Degrades cleanly to polling when change streams are off or unavailable.\n- **Pluggable backends.** One interface, three implementations: `MongoJobQueue` for production, plus `DummyBackend` (records calls) and `ImmediateBackend` (runs inline) for tests. Swap the backend and test your job logic without mocking Mongo.\n\n## Install\n\n```bash\nnpm install durabl mongodb\n```\n\n`mongodb` is a peer dependency — durabl uses your driver instance and version.\n\n## Quickstart\n\n```typescript\nimport { MongoClient } from 'mongodb'\nimport { JobQueue, MongoJobQueue } from 'durabl'\n\nconst client = await MongoClient.connect(process.env.MONGO_URL!)\nconst db = client.db('app')\n\n// 1. Create and start the backend (creates indexes).\nconst backend = new MongoJobQueue({ db })\nawait backend.startup()\n\n// 2. Wrap it in a queue.\nconst queue = new JobQueue(backend)\n\n// 3. Register a processor.\nqueue.process\u003c{ userId: string }\u003e(\n  'welcome-email',\n  async (job, ctx) =\u003e {\n    await sendWelcomeEmail(job.data.userId)\n    await ctx.complete()\n  },\n  { concurrency: 4, pollInterval: 5000 },\n)\n\n// 4. Enqueue. The dedupeKey makes this idempotent: a second enqueue while\n//    the first is still pending/active returns null instead of duplicating.\nconst jobId = await queue.enqueue(\n  'welcome-email',\n  { userId: 'u_123' },\n  { dedupeKey: 'welcome-email:u_123' },\n)\nif (jobId === null) {\n  // A job for this user is already queued — nothing to do.\n}\n```\n\n### Change streams (push pickup)\n\nPass `useChangeStreams: true` to get near-instant pickup instead of waiting for the next poll. This requires a replica set (MongoDB Atlas provides one; a single-node `rs` works for local dev).\n\n```typescript\nconst backend = new MongoJobQueue({ db, useChangeStreams: true })\nawait backend.startup() // throws if the server isn't a replica set\n```\n\nWhen push is active, `JobQueue` bumps its default poll interval to 60s and leans on the stream for latency, keeping the poll loop only as a safety net for dropped events and crash recovery.\n\n### Inline execution with coalescing\n\nFor the \"run it now, but never run two at once, and coalesce a burst into at most one follow-up\" pattern (this replaced a 300-line distributed lock in Focuster), use `claimOrEnqueue` with `dedupeScope: 'pending'`:\n\n```typescript\nconst handle = await queue.claimOrEnqueue(\n  'reschedule',\n  { userId },\n  { dedupeKey: `reschedule:${userId}`, dedupeScope: 'pending' },\n)\n\nif (handle) {\n  // We won the slot — run inline, no poll delay.\n  try {\n    await reschedule(userId)\n    await handle.complete()\n  } catch (err) {\n    await handle.fail(String(err)) // poll loop will retry\n  }\n}\n// else: someone is already running and one run is queued behind them.\n```\n\n## Testing your jobs\n\nThe backend is an interface, so your job logic never has to touch Mongo in a unit test.\n\n```typescript\nimport { DummyBackend, JobQueue } from 'durabl'\n\nconst backend = new DummyBackend() // records, doesn't execute\nconst queue = new JobQueue(backend)\n\nawait myService.doThing() // calls queue.enqueue under the hood\n\nexpect(backend.jobs).toHaveLength(1)\nexpect(backend.jobs[0].dedupeKey).toBe('thing:42')\n```\n\n`ImmediateBackend` runs handlers synchronously on enqueue, which is handy for integration tests where you want side effects without a poll loop.\n\n## API sketch\n\n```typescript\nclass JobQueue {\n  enqueue\u003cT\u003e(type, data, options?): Promise\u003cstring | null\u003e\n  claimOrEnqueue\u003cT\u003e(type, data, options?): Promise\u003cJobHandle\u003cT\u003e | null\u003e\n  process\u003cT\u003e(type, handler, config?): void\n  getStats(type?): Promise\u003cQueueStats\u003e\n  startup(): Promise\u003cvoid\u003e\n  shutdown(timeoutMs?): Promise\u003cvoid\u003e\n}\n\ninterface EnqueueOptions {\n  priority?: number       // lower = higher priority. default 0\n  delay?: number          // ms before claimable. default 0\n  maxAttempts?: number    // default 3\n  dedupeKey?: string\n  dedupeScope?: 'pending' | 'pending+active' // default 'pending+active'\n  // Retry backoff — spaces failed attempts so a fast-failing handler can't\n  // burn every attempt in milliseconds, and an outage doesn't become an\n  // instant-retry storm.\n  backoff?: 'exponential' | 'fixed' // default 'exponential' (full jitter)\n  backoffDelay?: number   // base/floor ms. default 1000\n  backoffMaxDelay?: number // cap ms. default 60000\n}\n\ninterface ProcessorConfig {\n  concurrency?: number    // default 1\n  pollInterval?: number   // default 5000 (60000 when change streams are on)\n}\n```\n\nThe handler receives a `JobContext` with `complete()`, `fail(reason)`, `failFatal(reason)`, `log(message)`, and `heartbeat()`.\n\n## Running the tests\n\n```bash\nnpm install\nnpm test\n```\n\nThe Mongo-backed suites spin up an in-memory single-node replica set via [`mongodb-memory-server`](https://github.com/typegoose/mongodb-memory-server) (the first run downloads a `mongod` binary). To test against a real cluster instead, point it at one:\n\n```bash\nMONGO_URL=\"mongodb://localhost:27017/?replicaSet=rs0\" npm test\n```\n\nThe change-stream suite self-skips if `MONGO_URL` points at a standalone (non-replica-set) server.\n\n## What this is not\n\n- Not a workflow engine. No step-level checkpointing or replay. If a handler crashes halfway, the whole job retries from the top.\n- Not multi-datastore. MongoDB only, for now. The backend interface would accommodate a Postgres implementation (`FOR UPDATE SKIP LOCKED` maps cleanly), and that may land later.\n- Not battle-tested as a standalone package. The *queue* has years of production behind it; the *npm package* does not. File issues.\n\n## License\n\nMIT © Jordan Baker\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhexsprite%2Fdurabl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhexsprite%2Fdurabl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhexsprite%2Fdurabl/lists"}