{"id":22602291,"url":"https://github.com/kingrayhan/poc.nestjs-bullmq-queue","last_synced_at":"2025-03-28T20:45:47.368Z","repository":{"id":264831890,"uuid":"861183376","full_name":"kingRayhan/poc.nestjs-bullmq-queue","owner":"kingRayhan","description":null,"archived":false,"fork":false,"pushed_at":"2024-11-26T10:26:39.000Z","size":93,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-03T05:47:44.907Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kingRayhan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-22T08:19:10.000Z","updated_at":"2024-11-26T10:26:43.000Z","dependencies_parsed_at":"2024-11-26T15:00:39.015Z","dependency_job_id":null,"html_url":"https://github.com/kingRayhan/poc.nestjs-bullmq-queue","commit_stats":null,"previous_names":["kingrayhan/poc.nestjs-bullmq-queue"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kingRayhan%2Fpoc.nestjs-bullmq-queue","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kingRayhan%2Fpoc.nestjs-bullmq-queue/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kingRayhan%2Fpoc.nestjs-bullmq-queue/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kingRayhan%2Fpoc.nestjs-bullmq-queue/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kingRayhan","download_url":"https://codeload.github.com/kingRayhan/poc.nestjs-bullmq-queue/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246100466,"owners_count":20723469,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-08T12:19:45.836Z","updated_at":"2025-03-28T20:45:47.334Z","avatar_url":"https://github.com/kingRayhan.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# NestJS BullMQ Integration Guide\n\nThis guide covers setting up and configuring BullMQ in a NestJS application with production-ready settings.\n\n## Table of Contents\n- [Installation](#installation)\n- [Basic Setup](#basic-setup)\n- [Production Configuration](#production-configuration)\n- [Queue Processor Implementation](#queue-processor-implementation)\n- [Common Patterns](#common-patterns)\n- [Best Practices](#best-practices)\n\n## Installation\n\n```bash\n# Install required packages\nnpm install @nestjs/bullmq bullmq\n\n# Install Redis if you haven't already\n# Using Docker\ndocker run --name redis -p 6379:6379 -d redis\n```\n\n## Basic Setup\n\n### 1. App Module Configuration\n\n```typescript\n// app.module.ts\nimport { Module } from '@nestjs/common';\nimport { BullModule } from '@nestjs/bullmq';\n\n@Module({\n  imports: [\n    BullModule.forRoot({\n      defaultConnection: {\n        host: 'localhost',\n        port: 6379,\n      },\n    }),\n    // Your other modules...\n  ],\n})\nexport class AppModule {}\n```\n\n### 2. Feature Module Setup\n\n```typescript\n// email/email.module.ts\nimport { Module } from '@nestjs/common';\nimport { BullModule } from '@nestjs/bullmq';\n\n@Module({\n  imports: [\n    BullModule.registerQueue({\n      name: 'email-queue',\n    }),\n  ],\n  providers: [EmailProcessor, EmailService],\n  controllers: [EmailController],\n})\nexport class EmailModule {}\n```\n\n### 3. Basic Queue Processor\n\n```typescript\n// email/email.processor.ts\nimport { Processor, WorkerHost } from '@nestjs/bullmq';\nimport { Job } from 'bullmq';\n\n@Processor('email-queue')\nexport class EmailProcessor extends WorkerHost {\n  async process(job: Job\u003cany, any, string\u003e): Promise\u003cany\u003e {\n    // Process your job here\n    return { success: true };\n  }\n}\n```\n\n## Production Configuration\n\n### Recommended Production Settings\n\n```typescript\n// app.module.ts\nBullModule.forRoot({\n  defaultConnection: {\n    host: process.env.REDIS_HOST || 'localhost',\n    port: parseInt(process.env.REDIS_PORT) || 6379,\n    password: process.env.REDIS_PASSWORD,\n    tls: process.env.REDIS_TLS === 'true' ? {} : undefined,\n    maxRetriesPerRequest: 3,\n    enableReadyCheck: true,\n    connectTimeout: 10000,\n  },\n  defaultJobOptions: {\n    // Cleanup settings\n    removeOnComplete: {\n      age: 24 * 3600,    // 24 hours\n      count: 1000,       // Keep max 1000 jobs\n    },\n    removeOnFail: {\n      age: 7 * 24 * 3600,  // 7 days\n      count: 5000,         // Keep max 5000 jobs\n    },\n    // Retry settings\n    attempts: 3,\n    backoff: {\n      type: 'exponential',\n      delay: 1000,\n    },\n    timeout: 5 * 60 * 1000,  // 5 minutes\n  },\n  settings: {\n    maxStalledCount: 2,\n    lockDuration: 30000,\n    stalledInterval: 30000,\n  },\n})\n```\n\n### Queue-Specific Settings\n\n```typescript\n// email/email.module.ts\nBullModule.registerQueue({\n  name: 'email-queue',\n  defaultJobOptions: {\n    attempts: 5,\n    timeout: 30000,\n    removeOnComplete: {\n      age: 3600,\n      count: 500,\n    },\n  },\n  processors: [{\n    concurrency: 5,\n    maxStalledCount: 3,\n  }],\n})\n```\n\n## Queue Processor Implementation\n\n### Production-Ready Processor\n\n```typescript\n@Processor('email-queue')\nexport class EmailProcessor extends WorkerHost {\n  private readonly logger = new Logger(EmailProcessor.name);\n\n  async process(job: Job\u003cany, any, string\u003e): Promise\u003cany\u003e {\n    try {\n      await job.updateProgress(10);\n      const result = await this.sendEmail(job.data);\n      \n      return {\n        success: true,\n        messageId: result.messageId,\n        timestamp: new Date().toISOString(),\n      };\n\n    } catch (error) {\n      this.logger.error(\n        `Failed to process job ${job.id}: ${error.message}`,\n        error.stack\n      );\n      \n      if (this.isRetryableError(error)) {\n        throw error; // Will retry\n      }\n      \n      await job.moveToFailed({\n        message: error.message,\n        code: error.code,\n      }, false);\n    }\n  }\n}\n```\n\n## Common Patterns\n\n### Job Cleanup Options\n\n```typescript\n// Option 1: Boolean\nremoveOnComplete: true    // Remove immediately\nremoveOnFail: true       // Remove immediately\n\n// Option 2: Number\nremoveOnComplete: 1000   // Keep last 1000 jobs\nremoveOnFail: 1000      // Keep last 1000 failed jobs\n\n// Option 3: Age-based\nremoveOnComplete: { \n  age: 3600            // Remove after 1 hour\n}\n\n// Option 4: Count and age combined\nremoveOnComplete: {\n  count: 1000,         // Keep max 1000 jobs\n  age: 3600           // That are no older than 1 hour\n}\n```\n\n### Adding Jobs\n\n```typescript\n// email.service.ts\n@Injectable()\nexport class EmailService {\n  constructor(\n    @InjectQueue('email-queue') private emailQueue: Queue,\n  ) {}\n\n  async sendEmail(data: EmailData) {\n    const job = await this.emailQueue.add('send-email', data, {\n      priority: 1,\n      attempts: 3,\n      backoff: {\n        type: 'exponential',\n        delay: 1000,\n      },\n    });\n    return job;\n  }\n}\n```\n\n## Best Practices\n\n1. **Error Handling**\n    - Implement proper error logging\n    - Distinguish between retryable and non-retryable errors\n    - Set appropriate retry attempts and backoff strategies\n\n2. **Job Cleanup**\n    - Keep completed jobs for a reasonable time (e.g., 24 hours)\n    - Keep failed jobs longer for debugging (e.g., 7 days)\n    - Set maximum counts to prevent Redis memory issues\n\n3. **Monitoring**\n    - Monitor Redis memory usage\n    - Set up alerts for failed jobs\n    - Track job processing times\n    - Monitor queue lengths\n\n4. **Security**\n    - Use environment variables for sensitive data\n    - Enable Redis authentication\n    - Use TLS in production\n    - Set appropriate timeouts\n\n5. **Performance**\n    - Configure appropriate concurrency levels\n    - Set reasonable job timeouts\n    - Use job priorities when needed\n    - Implement rate limiting if required\n\n## Environment Variables\n\n```env\nREDIS_HOST=localhost\nREDIS_PORT=6379\nREDIS_PASSWORD=your-password\nREDIS_TLS=true\n```\n\nRemember to adjust these settings based on your specific requirements and workload patterns.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkingrayhan%2Fpoc.nestjs-bullmq-queue","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkingrayhan%2Fpoc.nestjs-bullmq-queue","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkingrayhan%2Fpoc.nestjs-bullmq-queue/lists"}