https://github.com/criyle/caddy-notifier

Last synced: 5 months ago
JSON representation

Host: GitHub
URL: https://github.com/criyle/caddy-notifier
Owner: criyle
Created: 2025-04-29T02:58:43.000Z (about 1 year ago)
Default Branch: main
Last Pushed: 2025-08-11T02:47:30.000Z (10 months ago)
Last Synced: 2025-09-10T06:16:30.919Z (9 months ago)
Language: Go
Size: 174 KB
Stars: 1
Watchers: 1
Forks: 0
Open Issues: 0
Metadata Files:
- Readme: README.md

Awesome Lists containing this project

README

# Caddy Notifier

Design under construction...

## Design

The caddy-notifier works as WebSocket consolidator, that consolidate multiple WebSocket connections into logical channels.

1. To reduce the number reverse connection from caddy to backend
2. Simplifies message distribution workflows

```mermaid
flowchart TD
C[subscriber] <--websocket/subscribe--> A
A[caddy-notifier] <--authenticate/subscribe/publish--> B[backend endpoint]
subgraph backend
B
end
subgraph caddy-server
A
end
```

Subscribers are connected via WebSocket interface and subscriber is able to subscribe to specific channel via subscribe event. When subscriber subscribes a channel successfully, the subscriber will receive events from subscribed channel.

Backend endpoint are connected via WebSocket and responsible to authenticate subscribers and publish event to channels.

### Considerations

1. When subscriber disconnected from the notifier, it has a `resume_token` (`subscription_id`) to allow it re-connect to a subscription within `keep_alive` period. The resumed subscriber is able to receive buffered message during the reconnection with `seq` indication.
2. Metrics (prefix `caddy_websocket_notifier_*`)
1. Number of event sent (`event_sent_total`)
2. Number of event requested (`event_requested_total`)
3. number of subscribe event (`subscribe_requested_total`)
4. current connection count (`active_connection`)
5. number of subscriptions (`active_subscription`)
6. current channel count (`channel_count`)
7. upstream status (`upstream_healthy`)
8. inbound message size (`websocket_inbound_bytes_total`)
9. outbound message size (`websocket_outbound_bytes_total`)
10. outbound compressed message size (`websocket_outbound_compressed_bytes_total`)
11. number of subscriber in channel per category (defined by `channel_category`) (`subscriber_count`)
12. current message in worker channel (`worker_channel`)
13. message throttled by worker channel (`worker_throttled_total`)

### Safety Considerations

1. Maybe need to limit the number of channel single connection can connect to
2. Do we want to limit the rate of sending out events? Due to event fan-out nature, there must be write-amplification effect. In this case, do we consider certain event have higher priority or certain event could be dropped when rate limited.
3. Do we want to limit the subscribe rate for single connection?
4. Do we want to limit the number of `subscribe` a connection could sent?

### Packages

- [gorilla/websocket](https://github.com/gorilla/websocket) will be the key package to upgrade incoming and outbound connections, for its ease of use and reliability. Although [nbio](https://github.com/lesismal/nbio) claimed to be able to handle 1 M connections with non-blocking strategies. It needs special listener, which is essentially incompatible with the Caddy setup.

### Caddyfile

```caddyfile
{
metrics
log notifier {
level DEBUG
include http.handlers.websocket_notifier
}
}

:6080 {
websocket_notifier /ws "ws://localhost:6081/ws" {
write_wait 10s
pong_wait 60s
ping_interval 50s
max_message_size 256k # max incoming message size from subscriber
chan_size 16
recovery_wait 5s # how long should upstream maintainer try to reconnect if connection dropped

header_up +TEST_HEADER "value"
header_down -TEST_HEADER "value"

compression shorty
shorty_reset_count 1000
ping_type text # send text "ping" message to subscribers

metadata key "value"
channel_category "regexp" "category"

keep_alive 10s # time duration for subscription to be resume
max_event_buffer_size 1024

subscribe_retries 3 # request will retry maximum 3 times for every 2 second
subscribe_try_interval 2s
}
}
```

Because of the deduplication, some of the value will not fully update until a full restart:

- `channel_category`: previous categorized channel will be cached
- `subscribe_try_interval`: ticker interval will not be changed or enabled
- `chan_size`: channel will not be recreated with new size
- WebSocket related values will not be updated until disconnect and reconnection

If the retry is not enabled, the unresponsive requests will be removed after ~20s to avoid memory leak.

Debug Log Hierarchy:

- http.handlers.websocket_notifier
- subscriber {remote_addr}
- upstream
- maintainer
- conn
- hub

## TODO

- [x] De-authorize
- [x] Metrics
- [x] Message Buffer
- [ ] Rate / resource limit (?)

### Protocol

The client sent of `ping` message will be treated as `ping`, and the client sent of `shorty` message will be treated as `shorty` compression enable operation.

#### subscriber to caddy-notifier

##### subscribe

```json
{
"operation": "subscribe",
"request_id": "request id",
"credential": "a string token for authentication",
"channels": ["a list of string value for channel name"]
}
```

The caddy-notifier will request backend with credential to authenticate, whether subscribe request accepted or rejected. Channel specified channel to be subscribed. (? or reject directly if reach certain subscription limit). `request_id` will be pass through to the backend.

##### unsubscribe

```json
{
"operation": "unsubscribe",
"request_id": "request id",
"channels": ["a list of string values for channel name"]
}
```

The caddy-notifier will remove the subscriber from the channel subscription. If certain channel is not subscribed, it will be no-op. After unsubscribe, the subscriber will no longer receive message from that channel.

##### resume

```json
{
"operation": "resume",
"resume_token": "resume_token",
"seq": 123
}
```

If a connection disconnected from the notifier, and it tries to reconnect, the subscriber can send a `resume` request with a secret token to resume previous subscription.

#### caddy-notifier to subscriber

##### subscribe results

```json
{
"operation": "verify",
"request_id": "request id sent previously",
"accept": ["a list of channel_name"],
"reject": ["a list of channel_name"],
"message": "reason for reject",
"resume_token": "resume_token"
}
```

When subscriber successfully subscribe / rejected for certain subscribe request, the list of accepted / rejected channel will be returned with a `resume_token` to resume previous subscription if disconnected.

##### resume results

```json
{
"operation": "resume_success",
"channel": ["a list of channels in previous subscription"]
}
```

```json
{
"operation": "resume_failed"
}
```

##### unsubscribe results

The caddy-notifier notifies the subscriber on the decision from the authenticator.

```json
{
"operation": "unsubscribed",
"channels": ["a list of channel_name"],
}
```

De-authorized sent out `unsubscribed` events.

##### Events

```json
{
"operation": "event",
"channels": ["a list of channel_name"],
"seq": 123,
"payload": { "event content": "can be any valid JSON structure" }
}
```

When a subscriber subscribes to multiple channel in the list, the event will be sent out once. `seq` is the sequential increasing `id` to resume subscription and should be provided with `resume` request.

#### caddy-notifier to backend

##### subscribe

```json
{
"subscription_id": "identifier to distinct different connections",
"request_id": "request id",
"operation": "subscribe",
"channels": ["a list of channel_name"],
"credential": "xxx",
"metadata": {"metadata_key": "metadata_value"}
}
```

##### unsubscribe

When last subscriber of a list of subscriber desubscribed / deauthorized / disconnected, the backend will be notified with `unsubscibe` so that backend is acknowledge there's no longer a subscriber to a specific channel.

```json
{
"subscription_id": "identifier to distinct different connections",
"channels": ["a list of channel_name"],
}
```

##### resume

The caddy-notifier sends `resume` with list of channels that have been subscribed when reconnected to the back-end.

```json
{
"operation": "resume",
"channels": ["a list of channels that being subscribed"]
}
```

#### backend to caddy-notifier

##### subscribe result

```json
{
"subscription_id": "identifier to distinct different connections",
"request_id": "identifier sent previously via request",
"operation": "verify",
"accept": ["a list of channel_name of accepted channels"],
"reject": ["a list of channel_name of rejected channels"],
"message": "reason for reject"
}
```

##### event

```json
{
"operation": "event",
"channels": ["a list of channel_name"],
"payload": { "event contents" }
}
```

caddy-notifier will notifier all subscriber in the specific channel (? or list of channels)

##### de-authorize

```json
{
"operation": "deauthorize",
"credential": "xxx"
}
```

De-authorize certain credential = unsubscribe from all channels for that credential. Ensure no subscription retained when credential expires / invalidated.

ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Awesome

https://github.com/criyle/caddy-notifier

Awesome Lists containing this project

README