Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/linuxfoundation/lfx-v2-email-service

LFX v2 Platform Email Service
https://github.com/linuxfoundation/lfx-v2-email-service

Last synced: 4 days ago
JSON representation

LFX v2 Platform Email Service

Awesome Lists containing this project

README 

          
# LFX V2 Email Service



Thin transactional email relay for the LFX Self-Service platform. Receives

pre-rendered email payloads over NATS request/reply, delivers them via

Amazon SES SMTP, and tracks engagement events (opens, deliveries, bounces,

complaints) in NATS KV.



## Usage



### Send an email



**Subject:** `lfx.email-service.send_email`  

**Queue group:** `lfx.email-service.queue`



**Request payload fields:**



| Field | Type | Required | Description |

|---|---|---|---|

| `to` | string | yes | Recipient email address |

| `subject` | string | yes | Email subject line |

| `html` | string | yes | HTML body — callers render this before publishing |

| `text` | string | yes | Plain-text body — shown by clients that don't render HTML |

| `from` | string | no | Sender address (e.g. `newsletter@lfx.linuxfoundation.org`). When omitted the service default (`DEFAULT_SMTP_FROM`) is used. The domain must be in the service's allowed list — see [Configuring the sender address](#configuring-the-sender-address). |

| `from_display_name` | string | no | Display name shown in the From header (e.g. `LFX Newsletter`). When omitted the service default (`DEFAULT_SMTP_FROM_DISPLAY_NAME`, default: `"LFX Self Serve"`) is used. |

| `reply_to` | string | no | Email address set on the SMTP `Reply-To` header. When set, mail client replies go to this address instead of the `From` address. The domain must be in the service's reply-to allowlist (`SMTP_ALLOWED_REPLY_TO_DOMAINS`, default: `linuxfoundation.org`). Subdomain suffix matching applies — the default permits `@linuxfoundation.org` and `@*.linuxfoundation.org`. Omitted from the message when not provided. |

| `group_id` | string | no | Caller-supplied ID grouping related emails (e.g. an invite batch). Use it to query aggregate engagement counts via [`lfx.email-service.get_email_engagement_analytics`](#query-group-engagement-analytics). If omitted, a UUID is generated and returned but is not meaningful for analytics. |



```json

{

  "to": "user@example.com",

  "subject": "You've been added as a Writer on Demo Project",

  "html": "...",

  "text": "You've been added as a Writer on Demo Project.",

  "from": "newsletter@lfx.linuxfoundation.org",

  "from_display_name": "LFX Newsletter",

  "reply_to": "support@lfx.linuxfoundation.org",

  "group_id": "invite-batch-abc123"

}

```



**Success response:**

```json

{ "email_id": "", "group_id": "" }

```



`email_id` is a UUID generated by the service and injected as the `X-LFX-TRACKING-ID`

MIME header. Store it if you want to query delivery/open status later.



**Error response:**

```json

{ "error": "" }

```



| `error` value | Cause |

|---|---|

| `invalid request payload` | Request body is not valid JSON |

| `to, subject, html, and text are required` | One or more required fields are missing |

| `invalid from address` | `from` field is not a valid email address |

| `from address domain not allowed` | `from` domain is not in the service's allowed list |

| `invalid reply_to address` | `reply_to` field is not a valid email address |

| `reply_to address domain not allowed` | `reply_to` domain is not in the service's allowed list |

| `email delivery failed` | Service accepted the request but SMTP delivery failed |



**Examples (NATS CLI):**

```bash

# Default sender ("LFX Self Serve ")

nats req lfx.email-service.send_email \

  '{"to":"alice@example.com","subject":"Test","html":"
Hi
","text":"Hi"}'



# Custom sender address and display name

nats req lfx.email-service.send_email \

  '{"to":"alice@example.com","subject":"Test","html":"
Hi
","text":"Hi","from":"newsletter@lfx.linuxfoundation.org","from_display_name":"LFX Newsletter"}'

```



#### Configuring the sender address



The `from` field lets callers send from any address whose domain is in the service's

allowlist. The allowlist is configured via the `SMTP_ALLOWED_FROM_DOMAINS` env var

(comma-separated, default: `lfx.linuxfoundation.org`).



| Scenario | Behaviour |

|---|---|

| `from` omitted | Service default (`DEFAULT_SMTP_FROM`) is used |

| `from` domain is in the allowlist | Email is sent from the specified address |

| `from` domain is **not** in the allowlist | Request is rejected — `{"error":"from address domain not allowed"}` |

| `from_display_name` omitted | Service default (`DEFAULT_SMTP_FROM_DISPLAY_NAME`, default: `"LFX Self Serve"`) is used |



> **Note:** Because delivery goes through Amazon SES, the `from` domain must also be

> a verified SES sending identity. Contact the platform team to add a new domain to

> both the SES configuration and `SMTP_ALLOWED_FROM_DOMAINS`.



### Query email status



**Subject:** `lfx.email-service.get_email_status`



Returns the tracking record(s) for one or more emails. Only available when NATS KV

is configured (JetStream enabled and both KV buckets exist).



Exactly one of `email_id` or `group_id` must be provided.



**Request:**

```json

{ "email_id": "" }

```

```json

{ "group_id": "" }

```



**Success response — by `email_id`** — an `EmailRecipientRecord`:

```json

{

  "email_id": "550e8400-e29b-41d4-a716-446655440000",

  "group_id": "invite-batch-abc123",

  "to": "user@example.com",

  "subject": "You've been added as a Writer on Demo Project",

  "sent_at": "2025-01-15T10:30:00Z",

  "delivered": true,

  "delivered_at": "2025-01-15T10:30:02Z",

  "opened": true,

  "opened_at_list": [

    { "event_id": "abc-sns-message-id-1", "opened_at": "2025-01-15T11:05:33Z" },

    { "event_id": "abc-sns-message-id-2", "opened_at": "2025-01-15T14:22:10Z" }

  ],

  "last_opened_at": "2025-01-15T14:22:10Z",

  "failed": false

}

```



`opened_at_list` contains one entry per unique open event (keyed by SNS `MessageId` to survive replays). Use `len(opened_at_list)` for the open count.



**Success response — by `group_id`** — an array of `EmailRecipientRecord`:

```json

[

  {

    "email_id": "550e8400-e29b-41d4-a716-446655440000",

    "group_id": "invite-batch-abc123",

    "to": "user@example.com",

    "subject": "You've been added as a Writer on Demo Project",

    "sent_at": "2025-01-15T10:30:00Z",

    "delivered": true,

    "delivered_at": "2025-01-15T10:30:02Z",

    "opened": false,

    "failed": false

  }

]

```



`delivered`, `opened`, and `failed` are set by the SES engagement event poller

(see [SES Engagement Event Tracking](#ses-engagement-event-tracking) below).

They remain `false` until the poller is enabled and the corresponding SES event

arrives.



**Error response:**

```json

{ "error": "" }

```



| `error` value | Cause |

|---|---|

| `invalid request payload` | Request body is not valid JSON |

| `email_id or group_id is required` | Neither field was set |

| `only one of email_id or group_id may be set` | Both fields were set |

| `not found` | No record exists for the given `email_id` or `group_id` |



**Examples (NATS CLI):**

```bash

nats req lfx.email-service.get_email_status \

  '{"email_id":"550e8400-e29b-41d4-a716-446655440000"}'



nats req lfx.email-service.get_email_status \

  '{"group_id":"invite-batch-abc123"}'

```



### Query group engagement analytics



**Subject:** `lfx.email-service.get_email_engagement_analytics`



Returns aggregate counts across all emails in a group. Only available when

NATS KV is configured.



**Request:**

```json

{ "group_id": "invite-batch-abc123" }

```



**Success response:**

```json

{

  "group_id": "invite-batch-abc123",

  "total_sent": 42,

  "delivered": 40,

  "opened": 31,

  "unique_opened": 18,

  "failed": 2

}

```



**Error response:**

```json

{ "error": "" }

```



| `error` value | Cause |

|---|---|

| `invalid request payload` | Request body is not valid JSON or `group_id` is missing |

| `not found` | No emails have been sent under the given `group_id` |



**Example (NATS CLI):**

```bash

nats req lfx.email-service.get_email_engagement_analytics \

  '{"group_id":"invite-batch-abc123"}'

```



### Use with Go



The `pkg/api` package exports subject constants and request/response types.



```bash

go get github.com/linuxfoundation/lfx-v2-email-service/pkg/api

```



```go

package main



import (

	"context"

	"encoding/json"

	"fmt"

	"time"



	"github.com/nats-io/nats.go"

	emailapi "github.com/linuxfoundation/lfx-v2-email-service/pkg/api"

)



func main() {

	nc, err := nats.Connect(nats.DefaultURL)

	if err != nil {

		panic(err)

	}

	defer nc.Close()



	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)

	defer cancel()



	// Send an email with a custom sender address and display name.

	// From and FromDisplayName are optional — omit them to use the service defaults.

	req := emailapi.SendEmailRequest{

		To:              "user@example.com",

		Subject:         "You've been added",

		HTML:            "
Hello
",

		Text:            "Hello",

		From:            "newsletter@lfx.linuxfoundation.org", // optional

		FromDisplayName: "LFX Newsletter",                     // optional

		GroupID:         "my-batch-id",

	}

	data, _ := json.Marshal(req)



	reply, err := nc.RequestWithContext(ctx, emailapi.SendEmailSubject, data)

	if err != nil {

		panic(err)

	}



	// Check for an error response first — SendEmailErrorResponse and

	// SendEmailResponse are distinguished by the presence of the "error" field.

	var errResp emailapi.SendEmailErrorResponse

	if err := json.Unmarshal(reply.Data, &errResp); err == nil && errResp.Error != "" {

		fmt.Println("send failed:", errResp.Error)

		return

	}



	var sendResp emailapi.SendEmailResponse

	if err := json.Unmarshal(reply.Data, &sendResp); err != nil {

		panic(err)

	}

	fmt.Println("sent, email_id:", sendResp.EmailID)



	// Query the delivery/open status of the email we just sent.

	statusReq, _ := json.Marshal(emailapi.GetEmailStatusRequest{EmailID: sendResp.EmailID})

	statusReply, err := nc.RequestWithContext(ctx, emailapi.GetEmailStatusSubject, statusReq)

	if err != nil {

		panic(err)

	}

	var record emailapi.EmailRecipientRecord

	if err := json.Unmarshal(statusReply.Data, &record); err != nil {

		panic(err)

	}

	fmt.Printf("status: delivered=%v opened=%v failed=%v\n", record.Delivered, record.Opened, record.Failed)



	// Query status for all emails in the group.

	groupStatusReq, _ := json.Marshal(emailapi.GetEmailStatusRequest{GroupID: sendResp.GroupID})

	groupStatusReply, err := nc.RequestWithContext(ctx, emailapi.GetEmailStatusSubject, groupStatusReq)

	if err != nil {

		panic(err)

	}

	var groupRecords []emailapi.EmailRecipientRecord

	if err := json.Unmarshal(groupStatusReply.Data, &groupRecords); err != nil {

		panic(err)

	}

	fmt.Printf("group status: %d emails\n", len(groupRecords))



	// Query aggregate engagement counts for the whole group.

	analyticsReq, _ := json.Marshal(emailapi.GetEmailEngagementAnalyticsRequest{GroupID: sendResp.GroupID})

	analyticsReply, err := nc.RequestWithContext(ctx, emailapi.GetEmailEngagementAnalyticsSubject, analyticsReq)

	if err != nil {

		panic(err)

	}

	var analytics emailapi.GetEmailEngagementAnalyticsResponse

	if err := json.Unmarshal(analyticsReply.Data, &analytics); err != nil {

		panic(err)

	}

	fmt.Printf("group analytics: sent=%d delivered=%d opened=%d failed=%d\n",

		analytics.TotalSent, analytics.Delivered, analytics.Opened, analytics.Failed)

}

```



## Quick Start



### Prerequisites



- Go 1.24+

- [NATS Server](https://docs.nats.io/running-a-nats-service/introduction/installation) or Docker

- Local Kubernetes cluster with [OrbStack](https://orbstack.dev/) or similar

- Mailpit running in the cluster for local SMTP capture (UI at `http://localhost:8025`)



### Option 1 — Run directly with `make run`



```bash

cp .env.example .env

source .env && make run

```



`.env` is gitignored. `SMTP_USERNAME` and `SMTP_PASSWORD` can be left empty when

pointing at Mailpit (no auth required).



### Option 2 — Build and deploy to local cluster with Helm



```bash

cp charts/lfx-v2-email-service/values.local.example.yaml \

   charts/lfx-v2-email-service/values.local.yaml



make docker-build

make helm-install-local

```



`values.local.yaml` is gitignored.



## Environment Variables



| Variable | Default | Description |

|---|---|---|

| `NATS_URL` | `nats://localhost:4222` | NATS server URL |

| `PORT` | `8080` | HTTP health probe port |

| `EMAIL_ENABLED` | `false` | Set `true` to enable SMTP delivery; when `false` requests succeed but delivery is skipped via `NoOpSender` |

| `SMTP_HOST` | `localhost` | SMTP server hostname |

| `SMTP_PORT` | `587` | SMTP server port (STARTTLS) |

| `DEFAULT_SMTP_FROM` | `noreply@lfx.linuxfoundation.org` | Default envelope From address (falls back to legacy `SMTP_FROM` if unset) |

| `DEFAULT_SMTP_FROM_DISPLAY_NAME` | `LFX Self Serve` | Default display name in the From header; overridable per message via `from_display_name` |

| `SMTP_ALLOWED_FROM_DOMAINS` | `lfx.linuxfoundation.org` | Comma-separated domains permitted for per-message `from` overrides; set to `""` to disable overrides entirely |

| `SMTP_ALLOWED_REPLY_TO_DOMAINS` | `linuxfoundation.org` | Comma-separated base domains permitted for `reply_to`; subdomains are also permitted (e.g. `linuxfoundation.org` allows `lfx.linuxfoundation.org`); set to `""` to disable |

| `SMTP_ALLOWED_RECIPIENT_DOMAINS` | _(empty — permit all)_ | Comma-separated base domains permitted as recipients (subdomain suffix matching applies). When empty all recipient domains are permitted (production default). Set in non-prod (e.g. `linuxfoundation.org`) to prevent test mail from reaching real users' personal addresses. |

| `SMTP_USERNAME` | _(empty)_ | SMTP credential (from Kubernetes Secret in production) |

| `SMTP_PASSWORD` | _(empty)_ | SMTP credential (from Kubernetes Secret in production) |

| `SES_CONFIGURATION_SET` | _(empty)_ | SES configuration set name. When set, `X-SES-CONFIGURATION-SET` is added to every outbound email to route engagement events. Omitted when empty. |

| `SES_EVENTING_ENABLED` | `false` | Set `true` to start the SQS engagement event poller. Requires `SES_ENGAGEMENT_SQS_QUEUE_URL` and NATS KV — missing either is a fatal startup error. |

| `SES_ENGAGEMENT_SQS_QUEUE_URL` | _(empty)_ | SQS queue URL that receives SNS-wrapped SES engagement events. Required when `SES_EVENTING_ENABLED=true`. |

| `LOG_LEVEL` | `info` | Log level (`debug`, `info`, `warn`, `error`) |

| `LOG_ADD_SOURCE` | `false` | Set `true` to include source file/line in log entries |



In production, `SES_CONFIGURATION_SET` and `SES_ENGAGEMENT_SQS_QUEUE_URL` are

injected from a Kubernetes Secret managed by External Secrets Operator

(see `app.ses.engagementSecretName` in `charts/lfx-v2-email-service/values.yaml`).



## File Structure



```

lfx-v2-email-service/

├── cmd/email-service/

│   ├── main.go          # Entry point: NATS subscriptions, SQS poller, HTTP health, graceful shutdown

│   └── config.go        # Environment variable parsing

├── internal/

│   ├── domain/

│   │   └── email.go     # Sender interface

│   ├── infrastructure/

│   │   ├── smtp/

│   │   │   ├── sender.go    # SMTPSender — delivers via net/smtp, injects tracking headers

│   │   │   ├── noop.go      # NoOpSender — logs only (EMAIL_ENABLED=false)

│   │   │   └── message.go   # MIME message builder

│   │   └── sqs/

│   │       └── poller.go    # Long-polling SQS consumer (AWS SDK v2, IRSA credentials)

│   ├── logging/

│   │   └── logging.go   # Structured log helpers

│   └── service/

│       ├── send_email_handler.go                  # Handles send_email — sends and writes KV records

│       ├── engagement_event_handler.go            # Handles SES engagement events from SQS

│       ├── get_email_status_handler.go            # Handles get_email_status

│       ├── get_email_engagement_analytics_handler.go  # Handles get_email_engagement_analytics

│       └── mocks/

│           └── kv.go    # Thread-safe in-memory KeyValue mock for tests

├── pkg/

│   ├── api/

│   │   └── nats.go      # Public NATS subjects, request/response types, KV bucket constants

│   └── redaction/

│       └── redaction.go # Email address redaction for logs

└── charts/lfx-v2-email-service/

    ├── Chart.yaml

    ├── values.yaml

    └── templates/

        ├── deployment.yaml

        ├── externalsecret.yaml   # ESO secret for SES config set + SQS queue URL

        ├── nats-kv-buckets.yaml  # Declares email-recipients and email-group-index KV buckets

        └── service.yaml

```



## Development



Run the test suite:



```bash

make test

```



Run `make check` before committing — it verifies formatting, runs the linter,

and checks license headers:



```bash

make check

```



All commits must be signed off per the [DCO](https://developercertificate.org/):



```bash

git commit -s -m "feat: ..."

```



## SES Engagement Event Tracking



The service optionally captures SES engagement events (open, delivery, bounce,

complaint) and stores them in NATS KV so callers can query whether their emails

were opened or delivered.



### How it works



1. **Send time**: `SendEmailHandler` injects two MIME headers into every outbound email:

   - `X-SES-CONFIGURATION-SET: ` — routes SES events to the configured event destination

   - `X-LFX-TRACKING-ID: /` — a stable key SES echoes back in every engagement event



2. **KV write on send**: After each successful SMTP delivery the handler writes an

   `EmailRecipientRecord` to the `email-recipients` NATS KV bucket (key: `email_id`)

   and appends the `email_id` to the caller's group in the `email-group-index` bucket

   (key: `group_id`). Both writes use optimistic locking with a single retry on conflict.



3. **SES event pipeline**: SES → SNS topic → SQS queue. The email service polls

   the SQS queue in a background goroutine.



4. **Poller**: `internal/infrastructure/sqs.Poller` long-polls the queue (20-second

   wait, up to 10 messages per call) using AWS SDK v2 with IRSA credentials.

   On consecutive `ReceiveMessage` failures it applies exponential backoff (capped

   at 30 s) and aborts after 3 consecutive errors, triggering graceful shutdown.



5. **Event handler**: `EngagementEventHandler` parses each SNS-wrapped SES event,

   extracts the `email_id` from `X-LFX-TRACKING-ID`, looks up the KV record, updates

   the relevant fields using SES-provided RFC3339 timestamps, and writes back with

   optimistic locking. Unrecognised event types and missing records are silently skipped.



### Enabling the poller



Set `SES_EVENTING_ENABLED=true`. The service then requires both

`SES_ENGAGEMENT_SQS_QUEUE_URL` and a reachable NATS KV — missing either is a **fatal

startup error** (the pod exits without restarting rather than looping).



The AWS-side infrastructure (SES configuration set, SNS topic, SQS queue) is

provisioned separately in `lfx-v2-opentofu`. The Helm chart reads the configuration

set name and queue URL from a Kubernetes Secret created by External Secrets Operator

(secret name configured via `app.ses.engagementSecretName` in `values.yaml`).



## License



Copyright The Linux Foundation and each contributor to LFX.

SPDX-License-Identifier: MIT