{"id":45366807,"url":"https://github.com/renatocron/howk","last_synced_at":"2026-02-21T14:37:05.761Z","repository":{"id":335618402,"uuid":"1146484742","full_name":"renatocron/howk","owner":"renatocron","description":"A reliable HTTP egress gateway with Lua scripting for payload transformation and secure token generation","archived":false,"fork":false,"pushed_at":"2026-02-18T23:38:46.000Z","size":10344,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-02-19T04:53:57.673Z","etag":null,"topics":["developer-tools","devtools","go","howk","infrastructure","kafka","outgoing-webhooks","redis","webhook","webhook-service","webhooks","webhooks-server"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/renatocron.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-01-31T06:55:41.000Z","updated_at":"2026-02-18T23:38:50.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/renatocron/howk","commit_stats":null,"previous_names":["renatocron/howk"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/renatocron/howk","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/renatocron%2Fhowk","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/renatocron%2Fhowk/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/renatocron%2Fhowk/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/renatocron%2Fhowk/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/renatocron","download_url":"https://codeload.github.com/renatocron/howk/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/renatocron%2Fhowk/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29683997,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-21T14:31:22.911Z","status":"ssl_error","status_checked_at":"2026-02-21T14:31:22.570Z","response_time":107,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["developer-tools","devtools","go","howk","infrastructure","kafka","outgoing-webhooks","redis","webhook","webhook-service","webhooks","webhooks-server"],"created_at":"2026-02-21T14:37:05.026Z","updated_at":"2026-02-21T14:37:05.753Z","avatar_url":"https://github.com/renatocron.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![codecov](https://codecov.io/gh/renatocron/howk/graph/badge.svg?token=UXH5PUHYH0)](https://codecov.io/gh/renatocron/howk)\n\n# HOWK - High Opinionated Webhook Kit\n\nA high-throughput, fault-tolerant webhook delivery system built on Kafka + Redis.\n\n## Philosophy\n\n- **Kafka is the source of truth** — every webhook and delivery result is a Kafka record\n- **Redis is rebuildable hot state** — if Redis dies, restore from compacted topic (zero maintenance)\n- **Zero maintenance recovery** — distributed lock + canary pattern enables automatic Redis rebuild on multi-instance deployments\n- **Last-Write-Wins (LWW)** — nanosecond timestamps prevent race conditions during concurrent reconciliation\n- **Circuit breakers protect endpoints** — failing endpoints don't burn your retry budget\n- **Penalty box isolates slow endpoints** — excess in-flight traffic is rate-limited to protect the fast lane\n- **At-least-once delivery** — we never lose a webhook, duplicates are the receiver's problem\n\n## Architecture\n\n### High-Level Overview\n\n```\n                    ┌─────────────────────────────────────┐\n                    │           API Gateway               │\n                    │  POST /webhooks/:config/enqueue     │\n                    │  validate → batch produce           │\n                    │  → 202 Accepted                     │\n                    └───────────────┬─────────────────────┘\n                                    │\n                          Kafka Produce (batched)\n                                    │\n                                    ▼\n                 ┌──────────────────────────────────────────┐\n                 │            Kafka Cluster                 │\n                 │                                          │\n                 │  howk.pending    → webhooks to deliver   │\n                 │  howk.slow       → rate-limited lane     │\n                 │  howk.results    → delivery outcomes     │\n                 │  howk.deadletter → exhausted retries     │\n                 │                                          │\n                 │  retention: 7 days                       │\n                 └────┬─────────────────────────┬──────────┘\n                      │                         │\n            ┌─────────┘                         └──────────┐\n            ▼                                              ▼\n  ┌──────────────────────┐                    ┌───────────────────────┐\n  │   Worker Pool        │                    │   Results Consumer    │\n  │   (N consumers)      │                    │                       │\n  │                      │                    │   • Update Redis      │\n  │   • Read pending     │                    │     status/stats      │\n  │   • Check circuit    │                    │   • Feed ClickHouse   │\n  │   • Fire HTTP        │                    │     (optional)        │\n  │   • Produce result   │                    │                       │\n  │   • Schedule retry   │                    └───────────────────────┘\n  │     if needed        │\n  └──────────┬───────────┘\n             │\n             ▼\n  ┌──────────────────────────────────────────────────────────────────┐\n  │                           Redis                                   │\n  │                                                                   │\n  │  Circuit Breaker (per endpoint):                                  │\n  │    HSET circuit:{endpoint_hash} state=OPEN failures=5 last=...   │\n  │                                                                   │\n  │  Concurrency Control (Penalty Box):                               │\n  │    INCR concurrency:{endpoint_hash}  (with TTL)                  │\n  │    Lua: DECR with floor at 0 to prevent drift                     │\n  │                                                                   │\n  │  Retry Scheduling:                                                │\n  │    ZADD retries \u003cnext_at_unix\u003e \u003cwebhook_id:attempt\u003e              │\n  │    SET retry_data:{id} \u003ccompressed_webhook\u003e                      │\n  │                                                                   │\n  │  Status (per webhook) - LWW Hash Structure:                       │\n  │    HSET status:{webhook_id}                                       │\n  │      data={json_blob}     ← WebhookStatus JSON                    │\n  │      ts={nanoseconds}     ← UpdatedAtNs for LWW resolution        │\n  │    EXPIRE status:{webhook_id} 7d                                  │\n  │                                                                   │\n  │  Stats (hourly buckets):                                          │\n  │    INCR stats:delivered:2026013015                               │\n  │    PFADD stats:hll:endpoints:2026013015 {endpoint}               │\n  │                                                                   │\n  │  System Keys:                                                     │\n  │    howk:system:initialized     ← Canary (Redis initialized?)     │\n  │    howk:reconciler:lock        ← Distributed lock for rebuild    │\n  │                                                                   │\n  │  ══════════════════════════════════════════════════════════════  │\n  │  ALL OF THIS IS REBUILDABLE FROM KAFKA REPLAY                    │\n  └──────────────────────────────────────────────────────────────────┘\n```\n\n### System Data Flow\n\n```mermaid\nflowchart TB\n    subgraph Kafka[\"Kafka Topics\"]\n        PENDING[(\"howk.pending\")]\n        SLOW[(\"howk.slow\u003cbr/\u003eRate-limited lane\")]\n        RESULTS[(\"howk.results\")]\n        DLQ[(\"howk.deadletter\")]\n        STATE[(\"howk.state\u003cbr/\u003eCompacted topic\u003cbr/\u003eActive webhook state\")]\n    end\n\n    subgraph Redis[\"Redis Keys\"]\n        DATA[(\"retry_data:{webhook_id}\u003cbr/\u003eCompressed JSON\u003cbr/\u003eTTL: 7 days\")]\n        ZSET[(\"retries ZSET\u003cbr/\u003escore: unix_timestamp\u003cbr/\u003emember: webhook_id:attempt\")]\n        META[(\"retry_meta:{id}:{attempt}\u003cbr/\u003ereason, scheduled_at\")]\n        CONC[(\"concurrency:{endpoint_hash}\u003cbr/\u003eIn-flight counter\u003cbr/\u003eTTL: 2min\")]\n    end\n\n    subgraph Worker[\"Worker Process (Fast Lane)\"]\n        W_CONSUME[\"1. Consume from\u003cbr/\u003ehowk.pending\"]\n        W_CHECK_CB{\"Circuit\u003cbr/\u003eAllows?\"}\n        W_CHECK_CONC{\"Inflight \u0026lt;\u003cbr/\u003ethreshold?\"}\n        W_INCR[\"IncrInflight()\u003cbr/\u003eINCR + EXPIRE\"]\n        W_DELIVER[\"2. Deliver HTTP\"]\n        W_DECR[\"DecrInflight()\u003cbr/\u003eLua: DECR ≥0\"]\n        W_SUCCESS{\"Success?\"}\n        W_RETRY{\"Should\u003cbr/\u003eRetry?\"}\n        W_STORE[\"StoreRetryData()\u003cbr/\u003eSET retry_data:{id}\"]\n        W_SCHEDULE[\"ScheduleRetry()\u003cbr/\u003eZADD + SET meta\"]\n        W_CLEANUP[\"DeleteRetryData()\u003cbr/\u003eDEL retry_data:{id}\"]\n        W_PUBLISH_OK[\"PublishResult()\"]\n        W_PUBLISH_DLQ[\"PublishDeadLetter()\"]\n        W_PUBLISH_SLOW[\"PublishToSlow()\u003cbr/\u003eDivert to slow lane\"]\n    end\n\n    subgraph SlowWorker[\"Slow Worker Process\"]\n        SW_CONSUME[\"1. Consume from\u003cbr/\u003ehowk.slow\"]\n        SW_RATE[\"2. Rate limit\u003cbr/\u003e(5/sec default)\"]\n        SW_REUSE[\"3. Same logic as\u003cbr/\u003efast lane\"]\n    end\n\n    subgraph Scheduler[\"Scheduler Process\"]\n        S_POLL[\"1. Poll every 1s\"]\n        S_POP[\"2. PopAndLockRetries()\u003cbr/\u003eLua: ZRANGEBYSCORE + ZADD\"]\n        S_PARSE[\"3. Parse reference\"]\n        S_FETCH[\"4. GetRetryData()\"]\n        S_PUBLISH[\"5. Publish to Kafka\"]\n        S_ACK[\"6. AckRetry()\u003cbr/\u003eZREM + DEL meta\"]\n    end\n\n    PENDING --\u003e W_CONSUME\n    W_CONSUME --\u003e W_CHECK_CB\n    W_CHECK_CB --\u003e|No| W_SCHEDULE\n    W_CHECK_CB --\u003e|Yes| W_INCR\n    W_INCR --\u003e CONC\n    W_INCR --\u003e W_CHECK_CONC\n    \n    W_CHECK_CONC --\u003e|Yes| W_DELIVER\n    W_CHECK_CONC --\u003e|No| W_PUBLISH_SLOW\n    W_PUBLISH_SLOW --\u003e SLOW\n    W_PUBLISH_SLOW -.-\u003e|decr| CONC\n    \n    W_DELIVER --\u003e W_SUCCESS\n    W_DELIVER -.-\u003e|defer| W_DECR\n    W_DECR --\u003e CONC\n    \n    W_SUCCESS --\u003e|Yes| W_CLEANUP\n    W_CLEANUP --\u003e W_PUBLISH_OK\n    W_PUBLISH_OK --\u003e RESULTS\n    \n    W_SUCCESS --\u003e|No| W_RETRY\n    W_RETRY --\u003e|Yes| W_STORE\n    W_STORE --\u003e DATA\n    W_STORE --\u003e W_SCHEDULE\n    W_SCHEDULE --\u003e ZSET\n    W_SCHEDULE --\u003e META\n    \n    W_RETRY --\u003e|No/Exhausted| W_CLEANUP\n    W_CLEANUP --\u003e W_PUBLISH_DLQ\n    W_PUBLISH_DLQ --\u003e DLQ\n\n    SLOW --\u003e SW_CONSUME\n    SW_CONSUME --\u003e SW_RATE\n    SW_RATE --\u003e SW_REUSE\n    SW_REUSE --\u003e W_CHECK_CB\n\n    S_POLL --\u003e S_POP\n    S_POP --\u003e ZSET\n    ZSET --\u003e S_PARSE\n    S_PARSE --\u003e S_FETCH\n    S_FETCH --\u003e DATA\n    S_FETCH --\u003e S_PUBLISH\n    S_PUBLISH --\u003e PENDING\n    S_PUBLISH --\u003e S_ACK\n    S_ACK --\u003e ZSET\n    S_ACK --\u003e META\n\n    classDef kafka fill:#ff9800,stroke:#e65100,color:#000\n    classDef redis fill:#dc382d,stroke:#a41e11,color:#fff\n    classDef worker fill:#2196f3,stroke:#1565c0,color:#fff\n    classDef slowworker fill:#ffeb3b,stroke:#f9a825,color:#000\n    classDef scheduler fill:#4caf50,stroke:#2e7d32,color:#fff\n    classDef decision fill:#ff9800,stroke:#e65100,color:#000\n\n    class PENDING,SLOW,RESULTS,DLQ kafka\n    class DATA,ZSET,META,CONC redis\n    class W_CONSUME,W_DELIVER,W_STORE,W_SCHEDULE,W_CLEANUP,W_PUBLISH_OK,W_PUBLISH_DLQ,W_INCR,W_DECR,W_PUBLISH_SLOW worker\n    class SW_CONSUME,SW_RATE,SW_REUSE slowworker\n    class S_POLL,S_POP,S_PARSE,S_FETCH,S_PUBLISH,S_ACK scheduler\n    class W_SUCCESS,W_RETRY,W_CHECK_CB,W_CHECK_CONC decision\n```\n\n### Retry Lifecycle Sequence\n\n```mermaid\nsequenceDiagram\n    autonumber\n    participant K as Kafka\u003cbr/\u003ehowk.pending\n    participant W as Worker\n    participant R as Redis\n    participant E as Endpoint\n    participant S as Scheduler\n\n    Note over K,S: INITIAL DELIVERY (Attempt 0)\n    \n    K-\u003e\u003eW: Consume webhook (attempt=0)\n    W-\u003e\u003eE: HTTP POST\n    E--\u003e\u003eW: 503 Service Unavailable\n    \n    Note over W,R: Store data \u0026 schedule retry\n    \n    W-\u003e\u003eR: SET retry_data:wh_123 [compressed]\n    W-\u003e\u003eR: ZADD retries score \"wh_123:1\"\n    W-\u003e\u003eR: SET retry_meta:wh_123:1\n\n    Note over K,S: SCHEDULER PICKS UP\n    \n    rect rgb(220, 240, 220)\n        Note over S,R: Visibility Timeout (Pop \u0026 Lock)\n        S-\u003e\u003eR: Lua: ZRANGEBYSCORE + ZADD future\n        R--\u003e\u003eS: [\"wh_123:1\"]\n    end\n    \n    S-\u003e\u003eR: GET retry_data:wh_123\n    S-\u003e\u003eK: Publish (attempt=1)\n    S-\u003e\u003eR: ZREM + DEL meta (NOT data)\n\n    Note over K,S: SECOND DELIVERY (Attempt 1)\n    \n    K-\u003e\u003eW: Consume (attempt=1)\n    W-\u003e\u003eE: HTTP POST\n    E--\u003e\u003eW: 503 Again\n    \n    W-\u003e\u003eR: SET retry_data:wh_123 (overwrite)\n    W-\u003e\u003eR: ZADD retries \"wh_123:2\"\n\n    Note over K,S: ... Scheduler cycle ...\n\n    Note over K,S: FINAL DELIVERY - SUCCESS\n    \n    K-\u003e\u003eW: Consume (attempt=2)\n    W-\u003e\u003eE: HTTP POST\n    E--\u003e\u003eW: 200 OK\n    \n    rect rgb(200, 255, 200)\n        Note over W,R: Terminal State - Cleanup\n        W-\u003e\u003eR: DEL retry_data:wh_123\n    end\n```\n\n### Redis Key Structure\n\n```mermaid\nflowchart LR\n    subgraph DataKeys[\"Data Keys (One Per Webhook)\"]\n        D1[\"\u003cb\u003eretry_data:wh_123\u003c/b\u003e\u003cbr/\u003e━━━━━━━━━━━━━━━\u003cbr/\u003eValue: gzip(webhook JSON)\u003cbr/\u003eTTL: 7 days\u003cbr/\u003eOverwritten each retry\"]\n        D2[\"\u003cb\u003eretry_data:wh_456\u003c/b\u003e\u003cbr/\u003e━━━━━━━━━━━━━━━\u003cbr/\u003eDifferent webhook\"]\n    end\n    \n    subgraph ZSet[\"Sorted Set (References)\"]\n        Z[\"\u003cb\u003eretries\u003c/b\u003e\u003cbr/\u003e━━━━━━━━━━━━━━━\u003cbr/\u003e1706745600 → wh_123:1\u003cbr/\u003e1706752800 → wh_123:2\u003cbr/\u003e1706760000 → wh_456:1\"]\n    end\n    \n    subgraph MetaKeys[\"Metadata (Per Attempt)\"]\n        M1[\"\u003cb\u003eretry_meta:wh_123:1\u003c/b\u003e\u003cbr/\u003e━━━━━━━━━━━━━━━\u003cbr/\u003e{reason, scheduled_at}\"]\n        M2[\"\u003cb\u003eretry_meta:wh_123:2\u003c/b\u003e\"]\n        M3[\"\u003cb\u003eretry_meta:wh_456:1\u003c/b\u003e\"]\n    end\n\n    Z --\u003e|\"parse id\"| D1\n    Z --\u003e|\"parse id\"| D2\n    Z -.-\u003e|\"same ref\"| M1\n    Z -.-\u003e|\"same ref\"| M2\n    Z -.-\u003e|\"same ref\"| M3\n\n    classDef data fill:#e3f2fd,stroke:#1565c0,color:#000\n    classDef zset fill:#fff3e0,stroke:#e65100,color:#000\n    classDef meta fill:#f3e5f5,stroke:#7b1fa2,color:#000\n\n    class D1,D2 data\n    class Z zset\n    class M1,M2,M3 meta\n```\n\n### Webhook State Machine\n\n```mermaid\nstateDiagram-v2\n    [*] --\u003e Pending: API Enqueue\n    \n    Pending --\u003e Delivering: Worker Consume\n    \n    state Delivering {\n        [*] --\u003e CheckCircuit: Receive message\n        CheckCircuit --\u003e CircuitOpen: circuit open\n        CheckCircuit --\u003e CheckConcurrency: circuit allows\n        \n        CheckConcurrency --\u003e FastLane: inflight \u003c threshold\n        CheckConcurrency --\u003e DivertToSlow: inflight ≥ threshold\n        \n        FastLane --\u003e HTTPDeliver: INCR concurrency\n        DivertToSlow --\u003e [*]: Publish to howk.slow\n        \n        HTTPDeliver --\u003e Success: 2xx response\n        HTTPDeliver --\u003e Failure: error/timeout\n        \n        Success --\u003e [*]: DECR + publish result\n        Failure --\u003e ScheduleRetry: retryable\n        Failure --\u003e DLQ: non-retryable\n        ScheduleRetry --\u003e [*]: DECR + schedule\n        DLQ --\u003e [*]: DECR + dead letter\n        \n        CircuitOpen --\u003e ScheduleRetryCircuit: schedule far future\n        ScheduleRetryCircuit --\u003e [*]\n    }\n    \n    state SlowLane {\n        [*] --\u003e RateLimitedConsume: Consume from howk.slow\n        RateLimitedConsume --\u003e ReCheck: Rate limited\n        ReCheck --\u003e RetryDeliver: Re-check concurrency\n        RetryDeliver --\u003e Delivering: Process normally\n    }\n    \n    Delivering --\u003e Delivered: HTTP 2xx\n    Delivering --\u003e Failed: HTTP 5xx/408/429\n    Delivering --\u003e Exhausted: HTTP 4xx\n    \n    Failed --\u003e Pending: Scheduler Re-enqueue\n    Failed --\u003e Exhausted: Max Attempts\n    \n    Delivered --\u003e [*]: ✓ Success\n    Exhausted --\u003e [*]: ✗ DLQ\n\n    note right of Delivering\n        Concurrency Control:\n        • INCR concurrency:{hash}\n        • Check against threshold\n        • DECR on all exits\n        • Floor at 0 (Lua script)\n    end note\n    \n    note right of SlowLane\n        Self-healing:\n        • Rate limited (5/sec)\n        • Re-checks concurrency\n        • Returns to fast lane\n          when endpoint recovers\n    end note\n    \n    note right of Failed\n        Redis State:\n        • retry_data:{id} = compressed\n        • retries ZSET = scheduled\n        • retry_meta:{ref} = metadata\n    end note\n    \n    note right of Delivered\n        Cleanup:\n        DEL retry_data:{id}\n    end note\n    \n    note right of Exhausted\n        Cleanup:\n        DEL retry_data:{id}\n        Publish to DLQ\n    end note\n```\n\n### Operations Summary\n\n| Operation | Component | Redis Commands | When Called |\n|-----------|-----------|----------------|-------------|\n| `IncrInflight()` | Worker | `INCR concurrency:{hash}`\u003cbr\u003e`EXPIRE concurrency:{hash} {ttl}` | Before delivery attempt |\n| `DecrInflight()` | Worker | `Lua: DECR if \u003e 0` | After delivery (success/fail/DLQ) |\n| `PublishToSlow()` | Worker | Kafka Produce to `howk.slow` | When inflight ≥ threshold |\n| `StoreRetryData()` | Worker | `SET retry_data:{id} compressed EX 604800` | Before scheduling retry |\n| `ScheduleRetry()` | Worker | `ZADD retries score member`\u003cbr\u003e`SET retry_meta:{ref}` | After storing data |\n| `PopAndLockRetries()` | Scheduler | `Lua: ZRANGEBYSCORE + ZADD future` | Poll loop (every 1s) |\n| `GetRetryData()` | Scheduler | `GET retry_data:{id}` | After parsing reference |\n| `AckRetry()` | Scheduler | `ZREM retries member`\u003cbr\u003e`DEL retry_meta:{ref}` | After Kafka publish |\n| `DeleteRetryData()` | Worker | `DEL retry_data:{id}` | Terminal state (success/DLQ) |\n\n## Circuit Breaker Design\n\nPer-endpoint circuit breaker with three states:\n\n```\n    ┌─────────────────────────────────────────────────────────────┐\n    │                                                             │\n    ▼                                                             │\n┌────────┐   failure_threshold    ┌────────┐   recovery_timeout  ┌───────────┐\n│ CLOSED │ ────────────────────▶  │  OPEN  │ ─────────────────▶  │ HALF_OPEN │\n│        │      exceeded          │        │     expired         │           │\n└────────┘                        └────────┘                     └───────────┘\n    ▲                                  ▲                              │\n    │                                  │                              │\n    │         success                  │        probe fails           │\n    └──────────────────────────────────┴──────────────────────────────┘\n                                       │\n                                       │ probe succeeds\n                                       └────────────────────▶ CLOSED\n```\n\n**When circuit is OPEN:**\n- Don't attempt delivery (save resources)\n- Schedule retry far in the future (respect the endpoint)\n- Periodically allow ONE probe request (HALF_OPEN)\n\n**Circuit state is per-endpoint, stored in Redis, rebuildable from Kafka results.**\n\n## Penalty Box / Slow Lane\n\nPrevents slow/timing-out endpoints from starving the fast delivery path by routing excess in-flight traffic to a rate-limited slow topic.\n\n### How It Works\n\n```\nFast Lane (howk.pending)          Slow Lane (howk.slow)\n┌─────────────────────┐          ┌─────────────────────┐\n│ Consume webhook     │          │ Rate-limited consume│\n│ INCR concurrency    │          │ (20/sec per worker) │\n│ Check threshold     │          │                     │\n│ (\u003c 50 by default)   │          │ Re-check concurrency│\n│                     │          │ If recovered → fast │\n│ If over threshold ──┼────────►│ If still slow ──────┼──► (backpressure)\n│                     │          │                     │\n│ HTTP POST           │          │ HTTP POST           │\n│ DECR concurrency    │          │ DECR concurrency    │\n└─────────────────────┘          └─────────────────────┘\n```\n\n**Key behaviors:**\n\n| Component | Failure Mode | Behavior |\n|-----------|-------------|----------|\n| **Concurrency Check** | Fail-open | If Redis is unavailable, delivery proceeds normally without throttling |\n| **Circuit Breaker** | Fail-closed | If Redis is unavailable, requests are blocked (safety over availability) |\n| **Idempotency Check** | Fail-open | If Redis is unavailable, duplicate delivery is possible |\n| **Slow Lane Divert** | Fail-open | If divert to slow lane fails, delivery proceeds in fast lane |\n| **Stats Recording** | Fail-silent | Stats errors are logged but don't block delivery |\n\n- **Fail-open vs Fail-closed**: \n  - **Fail-open** (concurrency, idempotency): Better to deliver duplicates than drop webhooks\n  - **Fail-closed** (circuit breaker): Better to pause delivery than overwhelm a failing endpoint\n- **Self-healing**: When endpoint recovers, traffic automatically returns to fast lane\n- **Crash recovery**: TTL on concurrency keys (2min default) auto-corrects leaked counts\n- **Floor protection**: Lua script ensures counter never goes below 0\n\n### Configuration\n\n| Setting | Default | Description |\n|---------|---------|-------------|\n| `concurrency.max_inflight_per_endpoint` | 50 | Threshold above which webhooks are diverted |\n| `concurrency.inflight_ttl` | 2m | TTL for concurrency counter (crash recovery) |\n| `concurrency.slow_lane_rate` | 20 | Max deliveries/sec from slow lane per worker |\n| `kafka.topics.slow` | howk.slow | Slow lane Kafka topic name |\n| `ttl.retry_data_ttl` | 7d | TTL for compressed retry data in Redis |\n| `ttl.status_ttl` | 7d | TTL for webhook status records |\n| `ttl.circuit_state_ttl` | 24h | TTL for circuit breaker state |\n| `ttl.stats_ttl` | 48h | TTL for hourly stats counters |\n| `ttl.idempotency_ttl` | 24h | TTL for idempotency keys |\n\nEnvironment variables:\n```bash\nexport HOWK_CONCURRENCY_MAX_INFLIGHT_PER_ENDPOINT=50\nexport HOWK_CONCURRENCY_INFLIGHT_TTL=2m\nexport HOWK_CONCURRENCY_SLOW_LANE_RATE=20\nexport HOWK_KAFKA_TOPICS_SLOW=howk.slow\nexport HOWK_TTL_RETRY_DATA_TTL=168h\nexport HOWK_TTL_STATUS_TTL=168h\nexport HOWK_TTL_CIRCUIT_STATE_TTL=24h\nexport HOWK_TTL_STATS_TTL=48h\nexport HOWK_TTL_IDEMPOTENCY_TTL=24h\n```\n\n## Domain Concurrency Limiter (NEW)\n\nAggregates in-flight requests by domain hostname to prevent overwhelming a single destination, regardless of how many different endpoint URLs point to it.\n\n### Problem It Solves\n\nWithout domain limiting:\n- `api.stripe.com/hook1` and `api.stripe.com/hook2` have independent inflight budgets\n- Could accidentally send 50 + 50 = 100 concurrent requests to `api.stripe.com`\n- Stripe (or any destination) may rate-limit or block the traffic\n\nWith domain limiting:\n- Both endpoints share a per-domain budget (e.g., 100 for `api.stripe.com`)\n- Total concurrent requests to stripe.com are capped\n\n### How It Works\n\n```\n┌─────────────────────────────────────────────────────────────┐\n│  Domain Limiter (Redis-backed)                              │\n│                                                             │\n│  INCR domain_concurrency:api.stripe.com  ──┐               │\n│  (check against max, default: disabled)    │               │\n│                                            ▼               │\n│  If under limit ──────────────────────────►  Proceed        │\n│  If over limit ───────────────────────────►  Divert to slow │\n│                                            │               │\n│  DECR domain_concurrency:api.stripe.com  ◄──┘  (on complete)│\n└─────────────────────────────────────────────────────────────┘\n```\n\n**Integration point:** Domain check runs **after** circuit breaker check, **before** endpoint inflight check.\n\n### Configuration\n\n| Setting | Default | Description |\n|---------|---------|-------------|\n| `concurrency.max_inflight_per_domain` | 0 | Max concurrent requests per domain (0 = disabled) |\n| `concurrency.domain_overrides` | {} | Per-domain limits: `{\"api.stripe.com\": 200}` |\n\n**Environment variables:**\n```bash\nexport HOWK_CONCURRENCY_MAX_INFLIGHT_PER_DOMAIN=100\nexport HOWK_CONCURRENCY_DOMAIN_OVERRIDES='{\"api.stripe.com\":200,\"hooks.slack.com\":30}'\n```\n\n**Safety features:**\n- **Fail-open**: On Redis error, allows the request (logs warning)\n- **TTL**: Uses same TTL as endpoint inflight counters (2min default)\n- **Lua DECR**: Never goes below zero (prevents counter drift)\n\n## Per-Key Parallelism (NEW)\n\nControls how many goroutines process messages for the same partition key (ConfigID) concurrently. \n\n### Trade-off\n\n| Value | Behavior | Use Case |\n|-------|----------|----------|\n| 1 (default) | Sequential per ConfigID | Need strict ordering per tenant |\n| N \u003e 1 | Parallel per ConfigID | Maximize throughput, idempotent webhooks |\n\n### Configuration\n\n```yaml\nkafka:\n  per_key_parallelism: 1  # Default: sequential\n```\n\n**Environment variable:**\n```bash\nexport HOWK_KAFKA_PER_KEY_PARALLELISM=4\n```\n\n**Important:** With N \u003e 1, messages from the same ConfigID may be delivered concurrently/out of order. This is safe if webhooks are idempotent (each has unique ID, receivers should handle duplicates).\n\n## Retry Strategy\n\nExponential backoff with circuit-aware delays:\n\n```\nBase delay: 10s\nMax delay: 24h\nMax attempts: 20\nJitter: ±20%\n\nCircuit CLOSED:  delay = base * (2 ^ min(attempt, 10)) + jitter\nCircuit OPEN:    delay = recovery_timeout (e.g., 5 minutes)\nCircuit HALF_OPEN: immediate (it's a probe)\n```\n\n## Components\n\n| Binary | Purpose |\n|--------|---------|\n| `howk-api` | HTTP API for enqueueing webhooks |\n| `howk-worker` | Consumes pending, delivers, produces results. Includes both fast lane and slow lane workers |\n| `howk-scheduler` | Pops due retries from Redis, re-enqueues to Kafka |\n| `howk-reconciler` | Rebuilds Redis state from Kafka replay |\n\n## Quick Start\n\n```bash\n# Start infrastructure\ndocker-compose up -d\n\n# Run all components\nmake run-api\nmake run-worker\nmake run-scheduler\n\n# Enqueue a webhook\ncurl -X POST http://localhost:8080/webhooks/tenant123/enqueue \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"endpoint\": \"https://example.com/webhook\",\n    \"payload\": {\"event\": \"user.created\", \"data\": {\"id\": 123}},\n    \"idempotency_key\": \"user-created-123\"\n  }'\n```\n\n## Configuration\n\nHOWK supports flexible configuration through:\n\n1. **Environment Variables** (highest priority) - `HOWK_*` prefixed\n2. **Config File** (YAML format) - specified via `--config` flag or auto-discovered\n3. **Defaults** (lowest priority) - sensible built-in defaults\n\n### Environment Variables\n\nOverride any configuration setting using environment variables with the `HOWK_` prefix:\n\n```bash\nexport HOWK_API_PORT=9090\nexport HOWK_KAFKA_BROKERS=kafka1:9092,kafka2:9092\nexport HOWK_REDIS_ADDR=redis.example.com:6379\nexport HOWK_REDIS_PASSWORD=secret\nexport HOWK_TTL_STATUS_TTL=72h\nbin/howk-api\n```\n\nSee `.env.example` for a complete list of environment variables.\n\n### Config File\n\nUse a YAML config file for complex configurations:\n\n```bash\nbin/howk-api --config=/etc/howk/config.yaml\n```\n\nExample `config.yaml`:\n\n```yaml\napi:\n  port: 8080\n  read_timeout: 10s\n  write_timeout: 10s\n\nkafka:\n  brokers:\n    - localhost:19092\n  topics:\n    pending: howk.pending\n    slow: howk.slow\n    results: howk.results\n    deadletter: howk.deadletter\n  consumer_group: howk-workers\n  retention: 168h\n  # per_key_parallelism: 1  # Uncomment for per-key parallelism (default: 1 = sequential)\n\nredis:\n  addr: \"localhost:6379\"\n  password: \"\"\n  pool_size: 100\n\ndelivery:\n  timeout: 30s\n  max_idle_conns: 100\n  max_conns_per_host: 10\n\nretry:\n  base_delay: 10s\n  max_delay: 24h\n  max_attempts: 20\n  jitter: 0.2\n\ncircuit_breaker:\n  failure_threshold: 5\n  failure_window: 60s\n  recovery_timeout: 5m\n  probe_interval: 60s\n  success_threshold: 2\n\nconcurrency:\n  max_inflight_per_endpoint: 50\n  inflight_ttl: 2m\n  slow_lane_rate: 20\n  # max_inflight_per_domain: 0  # Uncomment to enable domain limiting (0 = disabled)\n  # domain_overrides:           # Optional per-domain limits\n  #   api.stripe.com: 200\n  #   hooks.slack.com: 30\n\nscheduler:\n  poll_interval: 1s\n  batch_size: 500\n\nttl:\n  circuit_state_ttl: 24h\n  status_ttl: 168h\n  stats_ttl: 48h\n  idempotency_ttl: 24h\n  retry_data_ttl: 168h  # Compressed webhook data for retries\n```\n\n### Configuration Precedence\n\nEnvironment variables override config file settings, which override defaults:\n\n```bash\n# config.yaml has: api.port: 7070\n# Environment variable overrides it:\nexport HOWK_API_PORT=9090\nbin/howk-api --config=config.yaml\n# Result: API listens on port 9090\n```\n\n## API\n\n### Enqueue Webhook\n\n```\nPOST /webhooks/:config/enqueue\n```\n\nRequest:\n```json\n{\n  \"endpoint\": \"https://customer.com/webhook\",\n  \"payload\": {\"event\": \"order.completed\"},\n  \"headers\": {\"X-Custom\": \"value\"},\n  \"idempotency_key\": \"order-123-completed\",\n  \"signing_secret\": \"whsec_...\"\n}\n```\n\nResponse: `202 Accepted`\n```json\n{\n  \"webhook_id\": \"wh_01HQXYZ...\",\n  \"status\": \"pending\"\n}\n```\n\n### Get Status\n\n```\nGET /webhooks/:webhook_id/status\n```\n\nResponse:\n```json\n{\n  \"webhook_id\": \"wh_01HQXYZ...\",\n  \"state\": \"delivered\",\n  \"attempts\": 2,\n  \"last_attempt_at\": \"2026-01-30T10:00:00Z\",\n  \"last_status_code\": 200,\n  \"next_retry_at\": null\n}\n```\n\n### Get Stats\n\n```\nGET /stats\n```\n\nResponse:\n```json\n{\n  \"last_1h\": {\n    \"enqueued\": 7200,\n    \"delivered\": 7150,\n    \"failed\": 50,\n    \"unique_endpoints\": 1200\n  },\n  \"last_24h\": {\n    \"enqueued\": 172800,\n    \"delivered\": 170000,\n    \"failed\": 2800,\n    \"unique_endpoints\": 45000\n  }\n}\n```\n\n### Incoming Webhook Transformer\n\n```\nPOST /incoming/:script_name\n```\n\nExecute a Lua transformer script to fan out incoming webhooks. See [docs/transformers.md](docs/transformers.md) for full documentation.\n\nRequest:\n```bash\ncurl -X POST http://localhost:8080/incoming/stripe-router \\\n  -H \"Content-Type: application/json\" \\\n  -u admin:password \\\n  -d '{\"type\": \"charge.succeeded\", \"amount\": 1000}'\n```\n\nResponse: `200 OK`\n```json\n{\n  \"webhooks\": [\n    {\"id\": \"wh_01HQXYZ...\", \"endpoint\": \"https://billing.internal/webhook\"},\n    {\"id\": \"wh_01HQABC...\", \"endpoint\": \"https://analytics.internal/track\"}\n  ],\n  \"count\": 2\n}\n```\n\n**Features:**\n- Lua scripting for payload transformation and routing\n- Fan-out: 1 incoming request → N outgoing webhooks\n- Basic Auth support (bcrypt or plaintext)\n- Domain allowlists for security\n- Hot-reload via SIGHUP\n- KV store access for deduplication/state\n- HTTP client for external API calls\n\n## Recovery\n\n### Redis Dies (Zero Maintenance Recovery)\n\nHOWK implements **automatic self-healing** for Redis loss using a distributed coordination pattern:\n\n#### Automatic Recovery (Multi-Instance)\n\nWhen Redis is lost in a multi-instance deployment (e.g., 3 workers + 2 publishers):\n\n```\n┌─────────────┐     ┌─────────────┐     ┌─────────────┐\n│   Worker    │     │   Worker    │     │   Worker    │\n│  Instance 1 │     │  Instance 2 │     │  Instance N │\n└──────┬──────┘     └──────┬──────┘     └──────┬──────┘\n       │                   │                   │\n       └─────────┬─────────┴─────────┬─────────┘\n                 ▼                   ▼\n       ┌───────────────────────────────────────┐\n       │         Distributed Lock              │\n       │   SET howk:reconciler:lock NX EX 60   │\n       │   (Only 1 instance wins)              │\n       └───────────────┬───────────────────────┘\n                       ▼\n       ┌───────────────────────────────────────┐\n       │         Reconciler (Winner)           │\n       │   1. Flush Redis                      │\n       │   2. Consume howk.state → HWM         │\n       │   3. Restore status + retries         │\n       │   4. SET howk:system:initialized      │\n       └───────────────┬───────────────────────┘\n                       ▼\n       ┌───────────────────────────────────────┐\n       │        All Other Instances            │\n       │   WaitForCanary() then resume         │\n       └───────────────────────────────────────┘\n```\n\n**Startup Sequence (Per Instance):**\n1. Check `howk:system:initialized` canary key\n2. If missing: try to acquire `howk:reconciler:lock`\n3. If lock acquired: run reconciler, set canary, release lock\n4. If lock NOT acquired: `WaitForCanary()` until peer finishes\n5. Proceed with normal operation\n\n**Runtime Sentinel (Auto-Recovery):**\n- Background goroutine checks canary every 30s\n- If canary missing (Redis flushed/lost):\n  - Pause Kafka consumer\n  - Try to acquire lock → reconcile → set canary\n  - Resume consumer\n\n#### Last-Write-Wins (LWW) Conflict Resolution\n\nTo prevent race conditions during concurrent reconciliation:\n\n```lua\n-- Every SetStatus uses Lua script with nanosecond timestamp check\nlocal old_ts = tonumber(redis.call('HGET', KEYS[1], 'ts') or '0')\nif new_ts \u003e old_ts then\n    redis.call('HSET', key, 'data', data, 'ts', new_ts)\n    return 1  -- updated\nend\nreturn 0  -- skipped (old data)\n```\n\n- **Workers** set `UpdatedAtNs = time.Now().UnixNano()` on every status change\n- **Reconciler** restores timestamps from Kafka snapshots\n- **Result:** Even if reconciler replays stale data, it won't overwrite newer writes\n\n#### Manual Recovery (Optional)\n\nFor single-instance deployments or forced rebuild:\n\n```bash\n# Stop workers, flush Redis, run reconciler, start workers\nredis-cli FLUSHDB\n./bin/howk-reconciler\n./bin/howk-worker \u0026\n```\n\n**Why this works:**\n- Workers continuously publish state snapshots to `howk.state` topic\n- Failed webhooks (pending retry) → full state snapshot with `UpdatedAtNs`\n- Terminal webhooks (delivered/exhausted) → tombstone published\n- Kafka compaction retains only the latest state per webhook\n- LWW ensures safe concurrent writes during recovery\n\n**No data loss:** Redis state is fully reconstructible from Kafka's compacted topic.\n\n### Kafka Broker Dies\n\nKafka handles this internally via replication. If you lose all replicas... you have bigger problems.\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frenatocron%2Fhowk","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frenatocron%2Fhowk","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frenatocron%2Fhowk/lists"}