Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/swe-himelrana/hs-mail-queue

A lightweight transactional email API built with Node.js, BullMQ, and Nodemailer. This application provides a robust email queuing system with retry mechanisms, template support, and Docker containerization.
https://github.com/swe-himelrana/hs-mail-queue

email email-queue queue smtp

Last synced: 10 months ago
JSON representation

A lightweight transactional email API built with Node.js, BullMQ, and Nodemailer. This application provides a robust email queuing system with retry mechanisms, template support, and Docker containerization.

Awesome Lists containing this project

README 

          
# HS Mail Queue



A lightweight transactional email API built with Node.js, BullMQ, and Nodemailer. This application provides a robust email queuing system with retry mechanisms, template support, and Docker containerization.



# Easily deploy or test using docker image from [DockerHub](https://hub.docker.com/r/himelranaswe/hs-mail-queue)



## Features



- 🚀 **Fast & Lightweight**: Built with Node.js and Express

- 📧 **Email Queue**: BullMQ + Redis for reliable email processing

- 🔄 **Retry Mechanism**: Automatic retry with exponential backoff

- 📝 **Template Support**: Handlebars templates with variable substitution

- 🐳 **Docker Ready**: Full Docker and Docker Compose support

- 🔒 **Security**: Helmet.js for security headers

- ✅ **Validation**: Joi schema validation for all requests

- 💥 **Detect Disposable Email** Built-in protection of Sending email to disposable email

- 🔍 **MX Record Verification** - Validate email domains have proper MX records

- 📊 **Monitoring**: Queue status and health check endpoints



## Quick Start



### Prerequisites



- Node.js 18+ or Docker

- Redis server

- SMTP credentials

- API key (for authentication)



### Using Docker (Recommended)



1. **Clone and setup environment:**



   ```bash

   git clone 

   cd hs-mail-queue

   cp env.example .env

   ```

2. **Configure your environment variables in `.env`:**



   ```env

   SMTP_HOST=smtp.yourdomain.com

   SMTP_PORT=587

   SMTP_SECURE=false

   SMTP_USER=your-smtp-username

   SMTP_PASS=your-smtp-password

   FROM_EMAIL=noreplay@yourdomain.com

   FROM_NAME=Your Service Name

   API_KEYS={"himosoft":"himosoft-strong-api-key","admin":"admin-secret-key","client":"client-api-key"}

   ```

3. **Start the application:**



   ```bash

   docker-compose up -d

   ```



### Using Node.js



1. **Install dependencies:**



   ```bash

   npm install

   ```

2. **Setup environment:**



   ```bash

   cp env.example .env

   # Edit .env with your SMTP settings

   ```

3. **Start Redis:**



   ```bash

   # Using Docker

   docker run -d -p 6379:6379 redis:7-alpine



   # Or install Redis locally

   ```

4. **Start the application:**



   ```bash

   # Terminal 1: Start the API server

   npm start



   # Terminal 2: Start the worker (optional, API includes worker)

   npm run worker

   ```



### Using Docker



1. **Create template volume:**



   ```bash

   docker volume create hs-mail-queue_email_templates

   ```

2. **Create templates directory and add your templates:**



   ```bash

   mkdir -p templates

   # Add your HTML templates to templates/ directory

   ```

3. **Run the API with templates:**



   ```bash

   docker run -d \

     --name hs-mail-queue-api \

     -p 3000:3000 \

     -v ./templates:/app/templates:ro \

     -v hs-mail-queue_email_templates:/app/templates:ro \

     --env-file .env \

     himelranaswe/hs-mail-queue:latest

   ```

4. **Run the worker with templates:**



   ```bash

   docker run -d \

     --name hs-mail-queue-worker \

     -v ./templates:/app/templates:ro \

     -v hs-mail-queue_email_templates:/app/templates:ro \

     --env-file .env \

     himelranaswe/hs-mail-queue:latest \

     npm run worker

   ```

5. **Copy templates to volume (after editing):**



   ```bash

   docker run --rm -v "$(pwd)/templates:/source" -v hs-mail-queue_email_templates:/dest \

     alpine cp -r /source/* /dest/

   ```



### Using Docker Compose



1. **Create docker-compose.yml:**



   ```yaml

   version: '3.8'

   services:

     api:

       image: himelranaswe/hs-mail-queue:latest

       container_name: hs-mail-queue-api

       ports:

         - "3005:3000"

       volumes:

         - ./templates:/app/templates:ro

       env_file:

         - .env

       environment:

         - NODE_ENV=${NODE_ENV:-production}

       restart: unless-stopped



     worker:

       image: himelranaswe/hs-mail-queue:latest

       container_name: hs-mail-queue-worker

       command: npm run worker

       volumes:

         - ./templates:/app/templates:ro

       env_file:

         - .env

       environment:

         - NODE_ENV=${NODE_ENV:-production}

       restart: unless-stopped

   ```

2. **Create templates directory and add your templates:**



   ```bash

   mkdir -p templates

   # Add your HTML templates to templates/ directory

   ```

3. **Start the stack:**



   ```bash

   docker-compose up -d

   ```



## API Endpoints



### Send Email



```http

POST /api/send-email

Content-Type: application/json

X-API-Key: your-secret-api-key-here



{

  "to": "recipient@example.com",

  "subject": "Welcome to our platform!",

  "html": "
Welcome {{name}}!
Thank you for joining us.
",

  "text": "Welcome {{name}}! Thank you for joining us.",

  "variables": {

    "name": "John Doe"

  },

  "priority": 5,

  "delay": 0

}

```



### Send Email with HTML File Template



```http

POST /api/send-email

Content-Type: application/json

X-API-Key: your-secret-api-key-here



{

  "to": "recipient@example.com",

  "subject": "Welcome to HIMOSOFT Platform!",

  "templateName": "welcome",

  "variables": {

    "name": "John Doe",

    "platform": "HIMOSOFT Platform",

    "email": "recipient@example.com",

    "currentYear": 2024,

    "activationUrl": "https://himosoft.com.bd/activate?token=abc123"

  }

}

```



### Queue Status



```http

GET /api/queue-status

X-API-Key: your-secret-api-key-here

```



### Get Available Templates



```http

GET /api/templates

X-API-Key: your-secret-api-key-here

```



### Health Check



```http

GET /api/health

```



## Email Templates



The API supports both inline Handlebars templates and HTML file templates for dynamic email content:



### HTML File Templates



HTML file templates are stored in the `templates/` directory and can be referenced by name. The system includes several pre-built templates:



- **welcome.html** - Welcome email template

- **password-reset.html** - Password reset email template

- **notification.html** - General notification template



#### Using HTML File Templates



```json

{

  "to": "user@example.com",

  "subject": "Welcome to HIMOSOFT Platform!",

  "templateName": "welcome",

  "variables": {

    "name": "John Doe",

    "platform": "HIMOSOFT Platform",

    "email": "user@example.com",

    "currentYear": 2024,

    "activationUrl": "https://himosoft.com.bd/activate?token=abc123"

  }

}

```



#### Password Reset Template



```json

{

  "to": "user@example.com",

  "subject": "Password Reset Request",

  "templateName": "password-reset",

  "variables": {

    "name": "Jane Smith",

    "platform": "HIMOSOFT Platform",

    "email": "user@example.com",

    "currentYear": 2024,

    "resetUrl": "https://himosoft.com.bd/reset?token=xyz789",

    "expiryTime": "24 hours"

  }

}

```



### Inline Templates



#### Simple HTML Email



```json

{

  "to": "user@example.com",

  "subject": "Welcome!",

  "html": "
Hello {{name}}!
Welcome to {{platform}}.
",

  "variables": {

    "name": "John",

    "platform": "Our Platform"

  }

}

```



#### Template with Both HTML and Text



```json

{

  "to": "user@example.com",

  "subject": "Password Reset",

  "template": {

    "html": "
Reset Your Password
Click here to reset.
",

    "text": "Reset Your Password\n\nClick here to reset: {{resetUrl}}"

  },

  "variables": {

    "resetUrl": "https://example.com/reset?token=abc123"

  }

}

```



## Configuration



### Environment Variables



| Variable                   | Description                                | Default                    |

| -------------------------- | ------------------------------------------ | -------------------------- |

| `PORT`                   | Server port                                | `3000`                   |

| `NODE_ENV`               | Node.js environment                        | `development`            |

| `REDIS_URL`              | Redis connection URL                       | `redis://localhost:6379` |

| `SMTP_HOST`              | SMTP server host                           | Required                   |

| `SMTP_PORT`              | SMTP server port                           | `587`                    |

| `SMTP_SECURE`            | Use SSL/TLS                                | `false`                  |

| `SMTP_USER`              | SMTP username                              | Required                   |

| `SMTP_PASS`              | SMTP password                              | Required                   |

| `FROM_EMAIL`             | Default from email                         | Required                   |

| `FROM_NAME`              | Default from name                          | Required                   |

| `API_KEYS`               | Multiple API keys with names (JSON format) | Required                   |

| `QUEUE_NAME`             | BullMQ queue name                          | `email-queue`            |

| `MAX_RETRIES`            | Maximum retry attempts                     | `3`                      |

| `RETRY_DELAY`            | Retry delay in ms                          | `5000`                   |

| `FAKE_EMAIL_FILTER`      | Enable disposable email filtering          | `true`                   |

| `ENABLE_MX_VERIFICATION` | Enable MX record verification              | `false`                  |



### Example .env file



```env

# Server Configuration

PORT=3000

NODE_ENV=development



# Redis Configuration

REDIS_URL=your redis URL here



# SMTP Configuration

SMTP_HOST=smtp.yourdomain.com

SMTP_PORT=587

SMTP_SECURE=false

SMTP_USER=your-smtp-username

SMTP_PASS=your-smtp-password



# Email Configuration

FROM_EMAIL=noreplay@yourdomain.com

FROM_NAME=Your Service Name



# API Security

API_KEYS={"himosoft":"himosoft-strong-api-key","admin":"admin-secret-key","client":"client-api-key"}



# Queue Configuration

QUEUE_NAME=email-queue

MAX_RETRIES=3

RETRY_DELAY=5000



# Temp Email Detection

FAKE_EMAIL_FILTER=false



# MX Record Verification

ENABLE_MX_VERIFICATION=false

```



### SMTP Providers



#### Gmail



```env

SMTP_HOST=smtp.gmail.com

SMTP_PORT=587

SMTP_SECURE=false

SMTP_USER=your-email@gmail.com

SMTP_PASS=your-app-password

```



#### SendGrid



```env

SMTP_HOST=smtp.sendgrid.net

SMTP_PORT=587

SMTP_SECURE=false

SMTP_USER=apikey

SMTP_PASS=your-sendgrid-api-key

```



#### Amazon SES



```env

SMTP_HOST=email-smtp.us-east-1.amazonaws.com

SMTP_PORT=587

SMTP_SECURE=false

SMTP_USER=your-ses-smtp-username

SMTP_PASS=your-ses-smtp-password

```



## Development



### Scripts



```bash

npm start          # Start the API server

npm run dev        # Start with nodemon (development)

npm run worker     # Start only the worker process

npm test           # Run tests

```



### Project Structure



```

src/

├── config/

│   └── database.js      # Redis configuration

├── middleware/

│   └── validation.js    # Request validation

├── routes/

│   └── emailRoutes.js   # API routes

├── services/

│   ├── emailService.js  # Email sending logic

│   └── queueService.js  # Queue management

├── server.js            # Express server

└── worker.js            # Standalone worker

```



## Monitoring



### Queue Status



Monitor your email queue with the `/api/queue-status` endpoint:



```bash

curl http://localhost:3000/api/queue-status

```



Response:



```json

{

  "success": true,

  "data": {

    "waiting": 5,

    "active": 2,

    "completed": 150,

    "failed": 3,

    "total": 160

  }

}

```



### Health Check



```bash

curl http://localhost:3000/api/health

```



## Error Handling



The application includes comprehensive error handling:



- **Validation Errors**: 400 Bad Request with detailed error messages

- **Queue Errors**: 500 Internal Server Error with retry mechanism

- **SMTP Errors**: Automatic retry with exponential backoff

- **Redis Errors**: Connection retry and graceful degradation



## Security



- **API Key Authentication**: All endpoints (except health check) require API key

- **Helmet.js**: Security headers

- **CORS**: Configurable cross-origin requests

- **Input Validation**: Joi schema validation

- **Rate Limiting**: Can be added via middleware

- **Environment Variables**: Secure configuration management



### API Key Authentication



All API endpoints (except `/api/health`) require authentication using an API key. You can provide the API key in two ways:



1. **X-API-Key header** (recommended):



   ```http

   X-API-Key: your-secret-api-key-here

   ```

2. **Authorization header**:



   ```http

   Authorization: Bearer your-secret-api-key-here

   ```



#### Multiple API Keys with Names



Set multiple API keys with names for reference in the `.env` file:



```env

API_KEYS={"himosoft":"himosoft-strong-api-key","admin":"admin-secret-key","client":"client-api-key"}

```



**Benefits of Multiple API Keys:**



- Different keys for different clients/services

- Easy to revoke individual keys

- Better security and access control

- Key names for reference and logging



## Scaling



### Horizontal Scaling



Run multiple worker instances:



```bash

# Start multiple workers

npm run worker  # Terminal 1

npm run worker  # Terminal 2

npm run worker  # Terminal 3

```



### Docker Scaling



```bash

# Scale workers

docker-compose up --scale worker=3

```



## Troubleshooting



### Common Issues



1. **Redis Connection Failed**



   - Ensure Redis is running: `redis-cli ping`

   - Check `REDIS_URL` in environment

2. **SMTP Authentication Failed**



   - Verify SMTP credentials

   - Check if 2FA is enabled (use app password for Gmail)

   - Ensure SMTP port is correct

3. **Emails Not Sending**



   - Check worker logs: `docker-compose logs worker`

   - Verify queue status: `GET /api/queue-status`

   - Check SMTP configuration



### Logs



```bash

# Docker logs

docker-compose logs -f api

docker-compose logs -f worker



# Node.js logs

npm start  # Check console output

```



## License



MIT License - see LICENSE file for details.



## Email Validation Features



### Disposable Email Detection



The API includes built-in protection against disposable/temporary email addresses using the FakeFilter API.



**Configuration:**



```env

FAKE_EMAIL_FILTER=true  # Enabled by default

```



**How it works:**



- Checks email addresses against the FakeFilter API before sending

- Blocks emails to known disposable email providers

- Graceful degradation - allows emails if API is unavailable

- Detailed logging in Docker logs



### MX Record Verification



The worker validates that email domains have proper MX records before sending emails.



**Configuration:**



```env

ENABLE_MX_VERIFICATION=true  # Disabled by default

```



**How it works:**



- DNS lookup to check if recipient domain has MX records

- Blocks emails to domains without valid MX records

- Prevents sending to non-existent or misconfigured domains

- Detailed logging showing MX record details



**Example logs:**



```

🔍 MX verification: ENABLED

🔍 Checking MX records for domain: gmail.com

✅ MX records found for gmail.com: gmail-smtp-in.l.google.com (priority: 5), alt1.gmail-smtp-in.l.google.com (priority: 10)

✅ MX verification: All domains have valid MX records

```



**When MX verification blocks an email:**



```

❌ Job 5 BLOCKED: No MX records for 'test@invalid-domain.xyz'

📧 Job details: {

  jobId: '5',

  to: 'test@invalid-domain.xyz',

  subject: 'Test Email',

  blockedAddress: 'test@invalid-domain.xyz'

}

❌ Job 5 failed: Blocked email 'test@invalid-domain.xyz' - no MX records found

```



**Benefits:**



- Reduces bounce rates

- Improves email deliverability

- Prevents sending to non-existent domains

- Better sender reputation