{"id":47602870,"url":"https://github.com/eyolas/conveyor","last_synced_at":"2026-04-01T18:57:42.147Z","repository":{"id":342000340,"uuid":"1166409907","full_name":"eyolas/conveyor","owner":"eyolas","description":"🚚 A multi-backend job queue for Node.js and Deno. BullMQ-like API with PostgreSQL, SQLite, and in-memory support.","archived":false,"fork":false,"pushed_at":"2026-03-20T10:02:42.000Z","size":625,"stargazers_count":1,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-03-20T14:32:35.240Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/eyolas.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-02-25T07:35:20.000Z","updated_at":"2026-03-20T10:02:47.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/eyolas/conveyor","commit_stats":null,"previous_names":["eyolas/conveyor"],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/eyolas/conveyor","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eyolas%2Fconveyor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eyolas%2Fconveyor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eyolas%2Fconveyor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eyolas%2Fconveyor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/eyolas","download_url":"https://codeload.github.com/eyolas/conveyor/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eyolas%2Fconveyor/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31290983,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-01T13:12:26.723Z","status":"ssl_error","status_checked_at":"2026-04-01T13:12:25.102Z","response_time":53,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-04-01T18:57:42.009Z","updated_at":"2026-04-01T18:57:42.131Z","avatar_url":"https://github.com/eyolas.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/logo.jpeg\" alt=\"Conveyor\" width=\"200\" /\u003e\n\u003c/p\u003e\n\n\u003ch1 align=\"center\"\u003eConveyor\u003c/h1\u003e\n\n\u003cp align=\"center\"\u003e\n  A multi-backend job queue for Deno, Node.js, and Bun.\u003cbr/\u003e\n  BullMQ-like API with PostgreSQL, SQLite, and in-memory support.\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://github.com/eyolas/conveyor/actions/workflows/ci.yml\"\u003e\u003cimg src=\"https://github.com/eyolas/conveyor/actions/workflows/ci.yml/badge.svg\" alt=\"CI\" /\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/eyolas/conveyor/actions/workflows/deploy-docs.yml\"\u003e\u003cimg src=\"https://github.com/eyolas/conveyor/actions/workflows/deploy-docs.yml/badge.svg\" alt=\"Deploy Docs\" /\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n## Why Conveyor?\n\n- **No Redis required** -- use PostgreSQL, SQLite, or in-memory instead\n- **Runtime-agnostic** -- works on Deno 2, Node.js 18+, and Bun 1.1+\n- **BullMQ-compatible API** -- familiar interface, minimal migration effort\n- **Type-safe** -- full TypeScript with generics on job payloads\n- **Adapter pattern** -- implement `StoreInterface` to support any backend\n\n## Features\n\n- FIFO and LIFO processing order\n- Human-readable scheduling (`queue.schedule(\"in 5 minutes\", ...)`, `queue.every(\"2 hours\", ...)`)\n- Job deduplication (payload hash or custom key, with TTL)\n- Retry with backoff (fixed, exponential, custom)\n- Priority queues (lower number = higher priority)\n- Per-worker and global concurrency control\n- Pause/Resume by queue or by job name\n- Recurring jobs with `queue.every()`\n- Cron scheduling with `queue.cron()` (5/6/7-field, timezone support)\n- Rate limiting (sliding window per worker)\n- Real-time job lifecycle events\n- Graceful shutdown with timeout\n- Job timeout support\n\n## Quick Start\n\n```typescript\nimport { Queue, Worker } from '@conveyor/core';\nimport { MemoryStore } from '@conveyor/store-memory';\n\nconst store = new MemoryStore();\nawait store.connect();\n\nconst queue = new Queue('emails', { store });\n\nawait queue.add('send-welcome', { to: 'user@example.com' });\n\nconst worker = new Worker('emails', async (job) =\u003e {\n  console.log(`Sending email to ${job.data.to}`);\n  return { sent: true };\n}, { store, concurrency: 5 });\n\nawait worker.close();\nawait queue.close();\n```\n\n## Packages\n\n| Package                       | Description                | Status |\n| ----------------------------- | -------------------------- | ------ |\n| `@conveyor/core`              | Queue, Worker, Job, Events | Alpha  |\n| `@conveyor/shared`            | Types \u0026 utilities          | Alpha  |\n| `@conveyor/store-memory`      | In-memory store            | Alpha  |\n| `@conveyor/store-pg`          | PostgreSQL store           | Alpha  |\n| `@conveyor/store-sqlite-node` | SQLite store (Node.js)     | Alpha  |\n| `@conveyor/store-sqlite-bun`  | SQLite store (Bun)         | Alpha  |\n| `@conveyor/store-sqlite-deno` | SQLite store (Deno)        | Alpha  |\n| `@conveyor/store-sqlite-core` | SQLite shared base         | Alpha  |\n\n## API\n\n### Queue\n\n```typescript\nconst queue = new Queue\u003cPayloadType\u003e('queue-name', {\n  store: new MemoryStore(),\n  defaultJobOptions: { attempts: 3 },\n});\n```\n\n#### Adding Jobs\n\n```typescript\n// Basic add\nconst job = await queue.add('job-name', { key: 'value' });\n\n// With options\nawait queue.add('job-name', payload, {\n  attempts: 5,\n  backoff: { type: 'exponential', delay: 1000 },\n  priority: 1,\n  delay: 5000,\n  timeout: 30_000,\n  removeOnComplete: true,\n  jobId: 'custom-id',\n});\n\n// Bulk add\nawait queue.addBulk([\n  { name: 'job-1', data: { i: 1 } },\n  { name: 'job-2', data: { i: 2 } },\n]);\n```\n\n#### Scheduling\n\n```typescript\n// Delayed execution with human-readable strings\nawait queue.schedule('5s', 'quick-task', payload);\nawait queue.schedule('in 10 minutes', 'send-reminder', payload);\n\n// Immediate execution\nawait queue.now('urgent-task', payload);\n\n// Recurring jobs\nawait queue.every('2 hours', 'cleanup', payload);\nawait queue.every('30s', 'health-check', payload, { repeat: { limit: 100 } });\n\n// Cron scheduling (5/6/7-field expressions)\nawait queue.cron('0 9 * * *', 'daily-report', payload);\nawait queue.cron('*/30 * * * *', 'health-check', payload);\n\n// Cron with timezone\nawait queue.add('task', payload, {\n  repeat: { cron: '0 9 * * *', tz: 'Europe/Paris' },\n});\n```\n\n#### Deduplication\n\n```typescript\n// By custom key\nawait queue.add('notify', payload, {\n  deduplication: { key: `user-${userId}` },\n});\n\n// By payload hash\nawait queue.add('process', payload, {\n  deduplication: { hash: true },\n});\n\n// With TTL (dedup entry expires after 60s)\nawait queue.add('task', payload, {\n  deduplication: { key: 'my-key', ttl: 60_000 },\n});\n```\n\n#### Pause / Resume\n\n```typescript\n// Pause entire queue\nawait queue.pause();\nawait queue.resume();\n\n// Pause by job name\nawait queue.pause({ jobName: 'send-email' });\nawait queue.resume({ jobName: 'send-email' });\n```\n\n#### Maintenance\n\n```typescript\n// Remove all waiting and delayed jobs\nawait queue.drain();\n\n// Clean old completed jobs (grace period in ms)\nconst removed = await queue.clean('completed', 60_000);\n\n// Count jobs by state\nconst waiting = await queue.count('waiting');\n```\n\n### Worker\n\n```typescript\nconst worker = new Worker('queue-name', async (job) =\u003e {\n  await job.updateProgress(50);\n  await job.log('Processing...');\n  return result;\n}, {\n  store,\n  concurrency: 5,\n  lockDuration: 30_000,\n  stalledInterval: 30_000,\n  limiter: { max: 10, duration: 1000 }, // 10 jobs per second\n});\n```\n\n#### Events\n\n```typescript\nworker.on('active', (job) =\u003e console.log('Started:', job.id));\nworker.on('completed', ({ job, result }) =\u003e console.log('Done:', result));\nworker.on('failed', ({ job, error }) =\u003e console.log('Failed:', error.message));\nworker.on('stalled', (jobId) =\u003e console.log('Stalled:', jobId));\nworker.on('error', (err) =\u003e console.error(err));\n```\n\n#### Lifecycle\n\n```typescript\n// Pause/resume processing\nworker.pause();\nworker.resume();\n\n// Graceful shutdown (waits up to 30s for active jobs)\nawait worker.close(30_000);\n```\n\n### Rate Limiting\n\nLimit the number of jobs a worker processes within a sliding time window:\n\n```typescript\nconst worker = new Worker('api-calls', handler, {\n  store,\n  limiter: { max: 10, duration: 1000 }, // 10 jobs per second\n});\n\n// Or more conservative\nconst worker2 = new Worker('emails', handler, {\n  store,\n  limiter: { max: 100, duration: 60_000 }, // 100 per minute\n});\n```\n\nRate limiting is per-worker (local sliding window). Each worker tracks its own window independently.\n\n### Job\n\n```typescript\n// Inside a worker processor:\nconst worker = new Worker('queue', async (job) =\u003e {\n  console.log(job.id, job.name, job.data);\n\n  await job.updateProgress(50);\n  await job.log('Half done');\n\n  return 'result';\n});\n\n// Outside the processor:\nconst job = await queue.getJob('job-id');\nawait job.moveToFailed(new Error('manual failure'));\nawait job.retry();\nawait job.remove();\n\nconsole.log(await job.isCompleted());\nconsole.log(await job.isFailed());\nconsole.log(await job.isActive());\n```\n\n### JobOptions\n\n| Option             | Type                   | Description                               |\n| ------------------ | ---------------------- | ----------------------------------------- |\n| `attempts`         | `number`               | Max attempts (default: 1)                 |\n| `backoff`          | `BackoffOptions`       | Retry strategy (fixed/exponential/custom) |\n| `delay`            | `number \\| string`     | Delay before execution                    |\n| `repeat`           | `RepeatOptions`        | Recurring job configuration               |\n| `priority`         | `number`               | Lower = higher priority (default: 0)      |\n| `lifo`             | `boolean`              | LIFO mode (default: false)                |\n| `deduplication`    | `DeduplicationOptions` | Dedup by hash or key                      |\n| `removeOnComplete` | `boolean \\| number`    | Auto-remove on completion                 |\n| `removeOnFail`     | `boolean \\| number`    | Auto-remove on failure                    |\n| `timeout`          | `number`               | Job timeout in ms                         |\n| `jobId`            | `string`               | Custom job ID                             |\n\n### Retry / Backoff\n\n```typescript\n// Fixed delay between retries\n{ attempts: 5, backoff: { type: 'fixed', delay: 2000 } }\n\n// Exponential backoff with jitter\n{ attempts: 5, backoff: { type: 'exponential', delay: 1000 } }\n\n// Custom strategy\n{ attempts: 5, backoff: {\n  type: 'custom',\n  delay: 1000,\n  customStrategy: (attempt) =\u003e attempt * 2000,\n}}\n```\n\n### LIFO Mode\n\n```typescript\n// Per-job LIFO\nawait queue.add('task', payload, { lifo: true });\n\n// Or use the store's fetchNextJob with lifo option\n```\n\n## Store Interface\n\nAll storage backends implement `StoreInterface`. To create a custom backend:\n\n```typescript\nimport type { StoreInterface } from '@conveyor/shared';\n\nclass MyStore implements StoreInterface {\n  // Lifecycle\n  async connect(): Promise\u003cvoid\u003e {/* ... */}\n  async disconnect(): Promise\u003cvoid\u003e {/* ... */}\n\n  // CRUD\n  async saveJob(queueName, job): Promise\u003cstring\u003e {/* ... */}\n  async getJob(queueName, jobId): Promise\u003cJobData | null\u003e {/* ... */}\n  // ... implement all methods from StoreInterface\n}\n```\n\n### Store Setup\n\n#### PostgreSQL\n\n```typescript\nimport { PgStore } from '@conveyor/store-pg';\n\nconst store = new PgStore({ connection: 'postgres://user:pass@localhost/mydb' });\nawait store.connect(); // auto-runs migrations\n// ... use with Queue/Worker\nawait store.disconnect();\n```\n\n#### SQLite\n\nChoose the package matching your runtime:\n\n```typescript\n// Node.js\nimport { SqliteStore } from '@conveyor/store-sqlite-node';\n\n// Bun\nimport { SqliteStore } from '@conveyor/store-sqlite-bun';\n\n// Deno\nimport { SqliteStore } from '@conveyor/store-sqlite-deno';\n```\n\n```typescript\nconst store = new SqliteStore({ filename: './data/queue.db' });\nawait store.connect(); // auto-runs migrations, enables WAL\n// ... use with Queue/Worker\nawait store.disconnect();\n\n// Or in-memory for testing\nconst memStore = new SqliteStore({ filename: ':memory:' });\n```\n\nRun the conformance test suite against your store:\n\n```typescript\nimport { runConformanceTests } from './tests/conformance/store.test.ts';\nimport { MyStore } from './my-store.ts';\n\nrunConformanceTests('MyStore', () =\u003e new MyStore());\n```\n\n## Development\n\n```bash\n# Run all tests\ndeno task test\n\n# Run specific tests\ndeno task test:core\ndeno task test:memory\n\n# Lint \u0026 format\ndeno task lint\ndeno task fmt\n\n# Type check\ndeno task check\n```\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feyolas%2Fconveyor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Feyolas%2Fconveyor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feyolas%2Fconveyor/lists"}