{"id":51543941,"url":"https://github.com/izadoesdev/protoqueue","last_synced_at":"2026-07-09T16:01:21.399Z","repository":{"id":289461548,"uuid":"971285454","full_name":"izadoesdev/Protoqueue","owner":"izadoesdev","description":null,"archived":false,"fork":false,"pushed_at":"2025-04-24T19:27:46.000Z","size":73,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-07-07T13:29:17.336Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/izadoesdev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":"ROADMAP.md","authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-04-23T09:38:52.000Z","updated_at":"2026-06-23T10:08:26.000Z","dependencies_parsed_at":"2025-04-23T12:35:01.238Z","dependency_job_id":null,"html_url":"https://github.com/izadoesdev/Protoqueue","commit_stats":null,"previous_names":["izadoesdev/protoqueue"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/izadoesdev/Protoqueue","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/izadoesdev%2FProtoqueue","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/izadoesdev%2FProtoqueue/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/izadoesdev%2FProtoqueue/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/izadoesdev%2FProtoqueue/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/izadoesdev","download_url":"https://codeload.github.com/izadoesdev/Protoqueue/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/izadoesdev%2FProtoqueue/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35304875,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-09T02:00:07.329Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-07-09T16:01:20.185Z","updated_at":"2026-07-09T16:01:21.383Z","avatar_url":"https://github.com/izadoesdev.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Protoqueue\n\nA high-performance, reliable, and scalable task queue system built on [NATS JetStream](https://docs.nats.io/nats-concepts/jetstream) and [Protocol Buffers](https://developers.google.com/protocol-buffers).\n\n![Version](https://img.shields.io/badge/version-1.0.0-blue.svg)\n![License](https://img.shields.io/badge/license-MIT-green.svg)\n\n---\n\n## Table of Contents\n- [Overview](#overview)\n- [Core Concepts](#core-concepts)\n- [Quick Start](#quick-start)\n- [API Reference](#api-reference)\n- [Configuration](#configuration)\n- [Error Handling](#error-handling)\n- [Monitoring](#monitoring)\n- [Performance Tips](#performance-tips)\n- [Roadmap](#roadmap)\n- [Example Applications](#example-applications)\n- [Contributing](#contributing)\n- [License](#license)\n- [Acknowledgments](#acknowledgments)\n\n---\n\n## Overview\n\n**Protoqueue** is a robust, type-safe, and efficient task queue for Node.js and Bun, designed for modern distributed systems. It leverages NATS JetStream for persistence and delivery guarantees, and Protocol Buffers for fast, compact serialization.\n\n**Key Benefits:**\n- At-least-once delivery with automatic retries\n- Dead-letter queue (DLQ) for failed tasks\n- Horizontal scalability (multiple consumers)\n- Real-time queue stats\n- Fully typed TypeScript API\n- Simple, explicit connection and processing model\n\n---\n\n## Core Concepts\n\n**Protoqueue** builds on NATS JetStream. Here are the main concepts:\n\n- **Stream:** A persistent log of messages (tasks) in NATS JetStream.\n- **Subject:** A topic string for publishing and subscribing to tasks (e.g., `tasks.email`).\n- **Consumer:** A worker that pulls and processes tasks from a stream.\n- **ACK/NAK/TERM:**\n  - **ACK:** Acknowledge successful processing (removes task).\n  - **NAK:** Negative acknowledgment (task will be retried).\n  - **TERM:** Terminal failure (task is moved to DLQ or dropped).\n- **DLQ (Dead Letter Queue):** Where tasks go after exceeding max retries.\n- **Protobuf:** Used for efficient, type-safe serialization of task data.\n\n---\n\n## Quick Start\n\n### 1. Install\n\n```bash\nbun add protoqueue\n# or\nnpm install protoqueue\n```\n\n### 2. Prerequisites\n- [NATS Server](https://nats.io/download/nats-io/nats-server/) running with JetStream enabled\n- Node.js 16+ or Bun\n\n### 3. Minimal Example\n\n```typescript\nimport { Protoqueue } from 'protoqueue';\n\n// 1. Create the queue instance\nconst queue = new Protoqueue({\n  streamName: 'my-stream',\n  subject: 'tasks.email',\n});\n\n// 2. Connect to NATS\nawait queue.connect('nats://localhost:4222');\n\n// 3. Enqueue a task\nawait queue.enqueue({\n  data: { to: 'user@example.com', subject: 'Welcome!' },\n  metadata: { priority: 3 }\n});\n\n// 4. Process tasks\nqueue.process(async (task) =\u003e {\n  console.log('Processing:', task.data);\n  // ...do work...\n  return { success: true };\n});\n```\n\n---\n\n## API Reference\n\n### Protoqueue\n\n#### Constructor\n```typescript\nnew Protoqueue(config: ProtoqueueConfig)\n```\n- `config.streamName`: Name of the JetStream stream\n- `config.subject`: Subject to publish/subscribe\n- `config.options`: (Optional) QueueOptions\n- `config.url`: (Optional) NATS server URL\n- `config.verbose`: (Optional) Enable verbose logging\n\n#### Methods\n- `connect(url?: string): Promise\u003cthis\u003e` — Connect to NATS\n- `disconnect(): Promise\u003cvoid\u003e` — Gracefully disconnect\n- `enqueue(task: { data: T, metadata?: Record\u003cstring, any\u003e }): Promise\u003cstring\u003e` — Enqueue a task\n- `enqueueBatch(tasks: Array\u003c{ data: T, metadata?: Record\u003cstring, any\u003e }\u003e): Promise\u003cstring[]\u003e` — Enqueue multiple tasks\n- `process(handler: (task: TaskData) =\u003e Promise\u003cTaskResult\u003e): Promise\u003cthis\u003e` — Start processing tasks\n- `getStats(): Promise\u003cQueueStats\u003e` — Get queue statistics\n\n#### Types\n```typescript\ninterface TaskData {\n  id: string;\n  data: unknown;\n  metadata?: {\n    timestamp?: number;\n    [key: string]: unknown;\n  };\n}\n\ninterface TaskResult {\n  success: boolean;\n  error?: string;\n  details?: Record\u003cstring, any\u003e;\n}\n\ninterface QueueOptions {\n  maxRetries?: number;\n  ackWait?: number;\n  batchSize?: number;\n  retryDelay?: number;\n}\n\ninterface QueueStats {\n  messages: number;\n  bytes: number;\n  firstSequence: number;\n  lastSequence: number;\n  consumer_count: number;\n}\n```\n\n---\n\n## Configuration\n\n### ProtoqueueConfig\n```typescript\ninterface ProtoqueueConfig {\n  url?: string; // NATS server URL\n  streamName: string; // JetStream stream name\n  subject: string; // Subject to publish/subscribe\n  options?: QueueOptions; // Queue options\n  verbose?: boolean; // Enable verbose logging\n}\n```\n\n### QueueOptions\n- `maxRetries` (default: 3): Max retry attempts before moving to DLQ\n- `ackWait` (default: 30000): Time (ms) to wait for task acknowledgment\n- `batchSize` (default: 10): Number of tasks to pull at once\n- `retryDelay` (default: 1000): Delay (ms) before retrying a failed task\n\n---\n\n## Error Handling\n\n- If your handler returns `{ success: false, error: '...' }`, the task will be retried (up to `maxRetries`).\n- If your handler throws, the error is logged and the task is retried.\n- After exceeding `maxRetries`, the task is moved to the DLQ (if configured) or dropped.\n- The `TaskResult.details` field can be used to pass extra error info (e.g., stack trace).\n\n**Example:**\n```typescript\nqueue.process(async (task) =\u003e {\n  try {\n    // ...process...\n    return { success: true };\n  } catch (err) {\n    return { success: false, error: err.message, details: { stack: err.stack } };\n  }\n});\n```\n\n---\n\n## Monitoring\n\nGet real-time stats:\n```typescript\nconst stats = await queue.getStats();\nconsole.log(`Messages: ${stats.messages}, Consumers: ${stats.consumer_count}`);\n```\n\n---\n\n## Performance Tips\n- **Batch Size:** Increase `batchSize` for higher throughput, but ensure your handler can keep up.\n- **ackWait:** Set to the max time your handler might need.\n- **Retry Delay:** Tune for your workload; higher values reduce NATS load.\n- **Horizontal Scaling:** Run multiple consumers for parallel processing.\n- **Explicit Connection:** Always call `connect()` before enqueueing or processing.\n\n---\n\n## Roadmap\n\n- [ ] **DLQ Processing:** Expose `processDLQ` for handling dead-lettered tasks\n- [ ] **Priority Queues:** Support for task prioritization\n- [ ] **Web Dashboard:** Real-time monitoring and management UI\n- [ ] **Pluggable Serializers:** Support for JSON, Avro, etc.\n- [ ] **Advanced Metrics:** Prometheus/Grafana integration\n- [ ] **Multi-Stream Support:** Route tasks to different streams/subjects\n- [ ] **Better Type Inference:** Stronger typing for task data\n- [ ] **Graceful Shutdown:** Improved cancellation and draining\n- [ ] **Cloud Native Examples:** Recipes for Kubernetes, Docker Compose\n\n---\n\n## Example Applications\n\n### Simple Worker\n```typescript\nimport { Protoqueue } from 'protoqueue';\n\nasync function main() {\n  const queue = new Protoqueue({ streamName: 'tasks', subject: 'tasks.email' });\n  await queue.connect();\n\n  queue.process(async (task) =\u003e {\n    // Simulate work\n    await new Promise(r =\u003e setTimeout(r, 100));\n    return { success: true };\n  });\n}\n\nmain();\n```\n\n### Task Producer\n```typescript\nimport { Protoqueue } from 'protoqueue';\n\nasync function sendTasks() {\n  const queue = new Protoqueue({ streamName: 'tasks', subject: 'tasks.email' });\n  await queue.connect();\n\n  for (let i = 0; i \u003c 10; i++) {\n    await queue.enqueue({\n      data: { email: `user${i}@example.com` },\n      metadata: { priority: 1 }\n    });\n  }\n\n  await queue.disconnect();\n}\n\nsendTasks();\n```\n\n---\n\n## Contributing\n\nContributions are welcome! Please:\n1. Fork the repo\n2. Create a feature branch\n3. Commit and push your changes\n4. Open a Pull Request\n\n---\n\n## License\n\nMIT\n\n---\n\n## Acknowledgments\n- [NATS.io](https://nats.io/) for the messaging platform\n- [Protocol Buffers](https://developers.google.com/protocol-buffers) for serialization\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fizadoesdev%2Fprotoqueue","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fizadoesdev%2Fprotoqueue","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fizadoesdev%2Fprotoqueue/lists"}