{"id":30615618,"url":"https://github.com/swe-himelrana/hs-mail-queue","last_synced_at":"2025-08-30T08:06:10.977Z","repository":{"id":310421416,"uuid":"1039776195","full_name":"Swe-HimelRana/hs-mail-queue","owner":"Swe-HimelRana","description":"A lightweight transactional email API built with Node.js, BullMQ, and Nodemailer. This application provides a robust email queuing system with retry mechanisms, template support, and Docker containerization.","archived":false,"fork":false,"pushed_at":"2025-08-18T00:42:34.000Z","size":90,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-08-18T02:35:26.875Z","etag":null,"topics":["email","email-queue","queue","smtp"],"latest_commit_sha":null,"homepage":"https://hub.docker.com/r/himelranaswe/hs-mail-queue","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Swe-HimelRana.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-08-18T00:39:04.000Z","updated_at":"2025-08-18T00:42:37.000Z","dependencies_parsed_at":"2025-08-18T02:35:29.190Z","dependency_job_id":"19aba55a-0052-4a36-aa0d-5d1a06495b8a","html_url":"https://github.com/Swe-HimelRana/hs-mail-queue","commit_stats":null,"previous_names":["swe-himelrana/hs-mail-queue"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/Swe-HimelRana/hs-mail-queue","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Swe-HimelRana%2Fhs-mail-queue","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Swe-HimelRana%2Fhs-mail-queue/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Swe-HimelRana%2Fhs-mail-queue/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Swe-HimelRana%2Fhs-mail-queue/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Swe-HimelRana","download_url":"https://codeload.github.com/Swe-HimelRana/hs-mail-queue/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Swe-HimelRana%2Fhs-mail-queue/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272821200,"owners_count":24998599,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-30T02:00:09.474Z","response_time":77,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["email","email-queue","queue","smtp"],"created_at":"2025-08-30T08:05:58.100Z","updated_at":"2025-08-30T08:06:10.960Z","avatar_url":"https://github.com/Swe-HimelRana.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# HS Mail Queue\n\nA lightweight transactional email API built with Node.js, BullMQ, and Nodemailer. This application provides a robust email queuing system with retry mechanisms, template support, and Docker containerization.\n\n# Easily deploy or test using docker image from [DockerHub](https://hub.docker.com/r/himelranaswe/hs-mail-queue)\n\n## Features\n\n- 🚀 **Fast \u0026 Lightweight**: Built with Node.js and Express\n- 📧 **Email Queue**: BullMQ + Redis for reliable email processing\n- 🔄 **Retry Mechanism**: Automatic retry with exponential backoff\n- 📝 **Template Support**: Handlebars templates with variable substitution\n- 🐳 **Docker Ready**: Full Docker and Docker Compose support\n- 🔒 **Security**: Helmet.js for security headers\n- ✅ **Validation**: Joi schema validation for all requests\n- 💥 **Detect Disposable Email** Built-in protection of Sending email to disposable email\n- 🔍 **MX Record Verification** - Validate email domains have proper MX records\n- 📊 **Monitoring**: Queue status and health check endpoints\n\n## Quick Start\n\n### Prerequisites\n\n- Node.js 18+ or Docker\n- Redis server\n- SMTP credentials\n- API key (for authentication)\n\n### Using Docker (Recommended)\n\n1. **Clone and setup environment:**\n\n   ```bash\n   git clone \u003crepository-url\u003e\n   cd hs-mail-queue\n   cp env.example .env\n   ```\n2. **Configure your environment variables in `.env`:**\n\n   ```env\n   SMTP_HOST=smtp.yourdomain.com\n   SMTP_PORT=587\n   SMTP_SECURE=false\n   SMTP_USER=your-smtp-username\n   SMTP_PASS=your-smtp-password\n   FROM_EMAIL=noreplay@yourdomain.com\n   FROM_NAME=Your Service Name\n   API_KEYS={\"himosoft\":\"himosoft-strong-api-key\",\"admin\":\"admin-secret-key\",\"client\":\"client-api-key\"}\n   ```\n3. **Start the application:**\n\n   ```bash\n   docker-compose up -d\n   ```\n\n### Using Node.js\n\n1. **Install dependencies:**\n\n   ```bash\n   npm install\n   ```\n2. **Setup environment:**\n\n   ```bash\n   cp env.example .env\n   # Edit .env with your SMTP settings\n   ```\n3. **Start Redis:**\n\n   ```bash\n   # Using Docker\n   docker run -d -p 6379:6379 redis:7-alpine\n\n   # Or install Redis locally\n   ```\n4. **Start the application:**\n\n   ```bash\n   # Terminal 1: Start the API server\n   npm start\n\n   # Terminal 2: Start the worker (optional, API includes worker)\n   npm run worker\n   ```\n\n### Using Docker\n\n1. **Create template volume:**\n\n   ```bash\n   docker volume create hs-mail-queue_email_templates\n   ```\n2. **Create templates directory and add your templates:**\n\n   ```bash\n   mkdir -p templates\n   # Add your HTML templates to templates/ directory\n   ```\n3. **Run the API with templates:**\n\n   ```bash\n   docker run -d \\\n     --name hs-mail-queue-api \\\n     -p 3000:3000 \\\n     -v ./templates:/app/templates:ro \\\n     -v hs-mail-queue_email_templates:/app/templates:ro \\\n     --env-file .env \\\n     himelranaswe/hs-mail-queue:latest\n   ```\n4. **Run the worker with templates:**\n\n   ```bash\n   docker run -d \\\n     --name hs-mail-queue-worker \\\n     -v ./templates:/app/templates:ro \\\n     -v hs-mail-queue_email_templates:/app/templates:ro \\\n     --env-file .env \\\n     himelranaswe/hs-mail-queue:latest \\\n     npm run worker\n   ```\n5. **Copy templates to volume (after editing):**\n\n   ```bash\n   docker run --rm -v \"$(pwd)/templates:/source\" -v hs-mail-queue_email_templates:/dest \\\n     alpine cp -r /source/* /dest/\n   ```\n\n### Using Docker Compose\n\n1. **Create docker-compose.yml:**\n\n   ```yaml\n   version: '3.8'\n   services:\n     api:\n       image: himelranaswe/hs-mail-queue:latest\n       container_name: hs-mail-queue-api\n       ports:\n         - \"3005:3000\"\n       volumes:\n         - ./templates:/app/templates:ro\n       env_file:\n         - .env\n       environment:\n         - NODE_ENV=${NODE_ENV:-production}\n       restart: unless-stopped\n\n     worker:\n       image: himelranaswe/hs-mail-queue:latest\n       container_name: hs-mail-queue-worker\n       command: npm run worker\n       volumes:\n         - ./templates:/app/templates:ro\n       env_file:\n         - .env\n       environment:\n         - NODE_ENV=${NODE_ENV:-production}\n       restart: unless-stopped\n   ```\n2. **Create templates directory and add your templates:**\n\n   ```bash\n   mkdir -p templates\n   # Add your HTML templates to templates/ directory\n   ```\n3. **Start the stack:**\n\n   ```bash\n   docker-compose up -d\n   ```\n\n## API Endpoints\n\n### Send Email\n\n```http\nPOST /api/send-email\nContent-Type: application/json\nX-API-Key: your-secret-api-key-here\n\n{\n  \"to\": \"recipient@example.com\",\n  \"subject\": \"Welcome to our platform!\",\n  \"html\": \"\u003ch1\u003eWelcome {{name}}!\u003c/h1\u003e\u003cp\u003eThank you for joining us.\u003c/p\u003e\",\n  \"text\": \"Welcome {{name}}! Thank you for joining us.\",\n  \"variables\": {\n    \"name\": \"John Doe\"\n  },\n  \"priority\": 5,\n  \"delay\": 0\n}\n```\n\n### Send Email with HTML File Template\n\n```http\nPOST /api/send-email\nContent-Type: application/json\nX-API-Key: your-secret-api-key-here\n\n{\n  \"to\": \"recipient@example.com\",\n  \"subject\": \"Welcome to HIMOSOFT Platform!\",\n  \"templateName\": \"welcome\",\n  \"variables\": {\n    \"name\": \"John Doe\",\n    \"platform\": \"HIMOSOFT Platform\",\n    \"email\": \"recipient@example.com\",\n    \"currentYear\": 2024,\n    \"activationUrl\": \"https://himosoft.com.bd/activate?token=abc123\"\n  }\n}\n```\n\n### Queue Status\n\n```http\nGET /api/queue-status\nX-API-Key: your-secret-api-key-here\n```\n\n### Get Available Templates\n\n```http\nGET /api/templates\nX-API-Key: your-secret-api-key-here\n```\n\n### Health Check\n\n```http\nGET /api/health\n```\n\n## Email Templates\n\nThe API supports both inline Handlebars templates and HTML file templates for dynamic email content:\n\n### HTML File Templates\n\nHTML file templates are stored in the `templates/` directory and can be referenced by name. The system includes several pre-built templates:\n\n- **welcome.html** - Welcome email template\n- **password-reset.html** - Password reset email template\n- **notification.html** - General notification template\n\n#### Using HTML File Templates\n\n```json\n{\n  \"to\": \"user@example.com\",\n  \"subject\": \"Welcome to HIMOSOFT Platform!\",\n  \"templateName\": \"welcome\",\n  \"variables\": {\n    \"name\": \"John Doe\",\n    \"platform\": \"HIMOSOFT Platform\",\n    \"email\": \"user@example.com\",\n    \"currentYear\": 2024,\n    \"activationUrl\": \"https://himosoft.com.bd/activate?token=abc123\"\n  }\n}\n```\n\n#### Password Reset Template\n\n```json\n{\n  \"to\": \"user@example.com\",\n  \"subject\": \"Password Reset Request\",\n  \"templateName\": \"password-reset\",\n  \"variables\": {\n    \"name\": \"Jane Smith\",\n    \"platform\": \"HIMOSOFT Platform\",\n    \"email\": \"user@example.com\",\n    \"currentYear\": 2024,\n    \"resetUrl\": \"https://himosoft.com.bd/reset?token=xyz789\",\n    \"expiryTime\": \"24 hours\"\n  }\n}\n```\n\n### Inline Templates\n\n#### Simple HTML Email\n\n```json\n{\n  \"to\": \"user@example.com\",\n  \"subject\": \"Welcome!\",\n  \"html\": \"\u003ch1\u003eHello {{name}}!\u003c/h1\u003e\u003cp\u003eWelcome to {{platform}}.\u003c/p\u003e\",\n  \"variables\": {\n    \"name\": \"John\",\n    \"platform\": \"Our Platform\"\n  }\n}\n```\n\n#### Template with Both HTML and Text\n\n```json\n{\n  \"to\": \"user@example.com\",\n  \"subject\": \"Password Reset\",\n  \"template\": {\n    \"html\": \"\u003ch1\u003eReset Your Password\u003c/h1\u003e\u003cp\u003eClick \u003ca href='{{resetUrl}}'\u003ehere\u003c/a\u003e to reset.\u003c/p\u003e\",\n    \"text\": \"Reset Your Password\\n\\nClick here to reset: {{resetUrl}}\"\n  },\n  \"variables\": {\n    \"resetUrl\": \"https://example.com/reset?token=abc123\"\n  }\n}\n```\n\n## Configuration\n\n### Environment Variables\n\n| Variable                   | Description                                | Default                    |\n| -------------------------- | ------------------------------------------ | -------------------------- |\n| `PORT`                   | Server port                                | `3000`                   |\n| `NODE_ENV`               | Node.js environment                        | `development`            |\n| `REDIS_URL`              | Redis connection URL                       | `redis://localhost:6379` |\n| `SMTP_HOST`              | SMTP server host                           | Required                   |\n| `SMTP_PORT`              | SMTP server port                           | `587`                    |\n| `SMTP_SECURE`            | Use SSL/TLS                                | `false`                  |\n| `SMTP_USER`              | SMTP username                              | Required                   |\n| `SMTP_PASS`              | SMTP password                              | Required                   |\n| `FROM_EMAIL`             | Default from email                         | Required                   |\n| `FROM_NAME`              | Default from name                          | Required                   |\n| `API_KEYS`               | Multiple API keys with names (JSON format) | Required                   |\n| `QUEUE_NAME`             | BullMQ queue name                          | `email-queue`            |\n| `MAX_RETRIES`            | Maximum retry attempts                     | `3`                      |\n| `RETRY_DELAY`            | Retry delay in ms                          | `5000`                   |\n| `FAKE_EMAIL_FILTER`      | Enable disposable email filtering          | `true`                   |\n| `ENABLE_MX_VERIFICATION` | Enable MX record verification              | `false`                  |\n\n### Example .env file\n\n```env\n# Server Configuration\nPORT=3000\nNODE_ENV=development\n\n# Redis Configuration\nREDIS_URL=your redis URL here\n\n# SMTP Configuration\nSMTP_HOST=smtp.yourdomain.com\nSMTP_PORT=587\nSMTP_SECURE=false\nSMTP_USER=your-smtp-username\nSMTP_PASS=your-smtp-password\n\n# Email Configuration\nFROM_EMAIL=noreplay@yourdomain.com\nFROM_NAME=Your Service Name\n\n# API Security\nAPI_KEYS={\"himosoft\":\"himosoft-strong-api-key\",\"admin\":\"admin-secret-key\",\"client\":\"client-api-key\"}\n\n# Queue Configuration\nQUEUE_NAME=email-queue\nMAX_RETRIES=3\nRETRY_DELAY=5000\n\n# Temp Email Detection\nFAKE_EMAIL_FILTER=false\n\n# MX Record Verification\nENABLE_MX_VERIFICATION=false\n```\n\n### SMTP Providers\n\n#### Gmail\n\n```env\nSMTP_HOST=smtp.gmail.com\nSMTP_PORT=587\nSMTP_SECURE=false\nSMTP_USER=your-email@gmail.com\nSMTP_PASS=your-app-password\n```\n\n#### SendGrid\n\n```env\nSMTP_HOST=smtp.sendgrid.net\nSMTP_PORT=587\nSMTP_SECURE=false\nSMTP_USER=apikey\nSMTP_PASS=your-sendgrid-api-key\n```\n\n#### Amazon SES\n\n```env\nSMTP_HOST=email-smtp.us-east-1.amazonaws.com\nSMTP_PORT=587\nSMTP_SECURE=false\nSMTP_USER=your-ses-smtp-username\nSMTP_PASS=your-ses-smtp-password\n```\n\n## Development\n\n### Scripts\n\n```bash\nnpm start          # Start the API server\nnpm run dev        # Start with nodemon (development)\nnpm run worker     # Start only the worker process\nnpm test           # Run tests\n```\n\n### Project Structure\n\n```\nsrc/\n├── config/\n│   └── database.js      # Redis configuration\n├── middleware/\n│   └── validation.js    # Request validation\n├── routes/\n│   └── emailRoutes.js   # API routes\n├── services/\n│   ├── emailService.js  # Email sending logic\n│   └── queueService.js  # Queue management\n├── server.js            # Express server\n└── worker.js            # Standalone worker\n```\n\n## Monitoring\n\n### Queue Status\n\nMonitor your email queue with the `/api/queue-status` endpoint:\n\n```bash\ncurl http://localhost:3000/api/queue-status\n```\n\nResponse:\n\n```json\n{\n  \"success\": true,\n  \"data\": {\n    \"waiting\": 5,\n    \"active\": 2,\n    \"completed\": 150,\n    \"failed\": 3,\n    \"total\": 160\n  }\n}\n```\n\n### Health Check\n\n```bash\ncurl http://localhost:3000/api/health\n```\n\n## Error Handling\n\nThe application includes comprehensive error handling:\n\n- **Validation Errors**: 400 Bad Request with detailed error messages\n- **Queue Errors**: 500 Internal Server Error with retry mechanism\n- **SMTP Errors**: Automatic retry with exponential backoff\n- **Redis Errors**: Connection retry and graceful degradation\n\n## Security\n\n- **API Key Authentication**: All endpoints (except health check) require API key\n- **Helmet.js**: Security headers\n- **CORS**: Configurable cross-origin requests\n- **Input Validation**: Joi schema validation\n- **Rate Limiting**: Can be added via middleware\n- **Environment Variables**: Secure configuration management\n\n### API Key Authentication\n\nAll API endpoints (except `/api/health`) require authentication using an API key. You can provide the API key in two ways:\n\n1. **X-API-Key header** (recommended):\n\n   ```http\n   X-API-Key: your-secret-api-key-here\n   ```\n2. **Authorization header**:\n\n   ```http\n   Authorization: Bearer your-secret-api-key-here\n   ```\n\n#### Multiple API Keys with Names\n\nSet multiple API keys with names for reference in the `.env` file:\n\n```env\nAPI_KEYS={\"himosoft\":\"himosoft-strong-api-key\",\"admin\":\"admin-secret-key\",\"client\":\"client-api-key\"}\n```\n\n**Benefits of Multiple API Keys:**\n\n- Different keys for different clients/services\n- Easy to revoke individual keys\n- Better security and access control\n- Key names for reference and logging\n\n## Scaling\n\n### Horizontal Scaling\n\nRun multiple worker instances:\n\n```bash\n# Start multiple workers\nnpm run worker  # Terminal 1\nnpm run worker  # Terminal 2\nnpm run worker  # Terminal 3\n```\n\n### Docker Scaling\n\n```bash\n# Scale workers\ndocker-compose up --scale worker=3\n```\n\n## Troubleshooting\n\n### Common Issues\n\n1. **Redis Connection Failed**\n\n   - Ensure Redis is running: `redis-cli ping`\n   - Check `REDIS_URL` in environment\n2. **SMTP Authentication Failed**\n\n   - Verify SMTP credentials\n   - Check if 2FA is enabled (use app password for Gmail)\n   - Ensure SMTP port is correct\n3. **Emails Not Sending**\n\n   - Check worker logs: `docker-compose logs worker`\n   - Verify queue status: `GET /api/queue-status`\n   - Check SMTP configuration\n\n### Logs\n\n```bash\n# Docker logs\ndocker-compose logs -f api\ndocker-compose logs -f worker\n\n# Node.js logs\nnpm start  # Check console output\n```\n\n## License\n\nMIT License - see LICENSE file for details.\n\n## Email Validation Features\n\n### Disposable Email Detection\n\nThe API includes built-in protection against disposable/temporary email addresses using the FakeFilter API.\n\n**Configuration:**\n\n```env\nFAKE_EMAIL_FILTER=true  # Enabled by default\n```\n\n**How it works:**\n\n- Checks email addresses against the FakeFilter API before sending\n- Blocks emails to known disposable email providers\n- Graceful degradation - allows emails if API is unavailable\n- Detailed logging in Docker logs\n\n### MX Record Verification\n\nThe worker validates that email domains have proper MX records before sending emails.\n\n**Configuration:**\n\n```env\nENABLE_MX_VERIFICATION=true  # Disabled by default\n```\n\n**How it works:**\n\n- DNS lookup to check if recipient domain has MX records\n- Blocks emails to domains without valid MX records\n- Prevents sending to non-existent or misconfigured domains\n- Detailed logging showing MX record details\n\n**Example logs:**\n\n```\n🔍 MX verification: ENABLED\n🔍 Checking MX records for domain: gmail.com\n✅ MX records found for gmail.com: gmail-smtp-in.l.google.com (priority: 5), alt1.gmail-smtp-in.l.google.com (priority: 10)\n✅ MX verification: All domains have valid MX records\n```\n\n**When MX verification blocks an email:**\n\n```\n❌ Job 5 BLOCKED: No MX records for 'test@invalid-domain.xyz'\n📧 Job details: {\n  jobId: '5',\n  to: 'test@invalid-domain.xyz',\n  subject: 'Test Email',\n  blockedAddress: 'test@invalid-domain.xyz'\n}\n❌ Job 5 failed: Blocked email 'test@invalid-domain.xyz' - no MX records found\n```\n\n**Benefits:**\n\n- Reduces bounce rates\n- Improves email deliverability\n- Prevents sending to non-existent domains\n- Better sender reputation\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fswe-himelrana%2Fhs-mail-queue","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fswe-himelrana%2Fhs-mail-queue","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fswe-himelrana%2Fhs-mail-queue/lists"}