{"id":51222226,"url":"https://github.com/dodopayments/durable-webhook-architecture-demo","last_synced_at":"2026-06-28T08:02:49.102Z","repository":{"id":340745327,"uuid":"1166425152","full_name":"dodopayments/durable-webhook-architecture-demo","owner":"dodopayments","description":"Production-grade reference implementation of Dodo Payments’ reliable webhook delivery architecture.","archived":false,"fork":false,"pushed_at":"2026-02-26T09:18:43.000Z","size":63,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-02-26T14:59:41.545Z","etag":null,"topics":["dodopayments","kafka","postgresql","sequin","svix","webhooks"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dodopayments.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-02-25T07:59:12.000Z","updated_at":"2026-02-26T14:38:22.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/dodopayments/durable-webhook-architecture-demo","commit_stats":null,"previous_names":["dodopayments/durable-webhook-architecture-demo"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/dodopayments/durable-webhook-architecture-demo","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dodopayments%2Fdurable-webhook-architecture-demo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dodopayments%2Fdurable-webhook-architecture-demo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dodopayments%2Fdurable-webhook-architecture-demo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dodopayments%2Fdurable-webhook-architecture-demo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dodopayments","download_url":"https://codeload.github.com/dodopayments/durable-webhook-architecture-demo/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dodopayments%2Fdurable-webhook-architecture-demo/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34881384,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-28T02:00:05.809Z","response_time":54,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dodopayments","kafka","postgresql","sequin","svix","webhooks"],"created_at":"2026-06-28T08:02:48.362Z","updated_at":"2026-06-28T08:02:49.092Z","avatar_url":"https://github.com/dodopayments.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Reliable Webhook Architecture Demo\n\nThis is a webhook replication of the Dodo Payments production architecture. Read the full blog post here: [Building Reliable Webhooks at Scale](https://medium.com/dodopayments/building-webhooks-that-never-fail-our-journey-to-99-99-delivery-reliability-f69ed069cf00)\n\nProduction-grade webhook delivery system replicating Dodo Payments' reliable webhook infrastructure at scale.\n\n#### Quick Start **Local setup:** See [LOCAL_SETUP.md](./LOCAL_SETUP.md)\n\n\n## The Problem\n\nWhen Joe runs his T-shirt shop, he needs instant notifications when customers pay. If your webhook system crashes mid-delivery, those payment notifications vanish forever. Customers get charged, but Joe never fulfills orders.\n\nThis demo shows how to build webhooks that never lose events, even during service crashes, network failures, database downtime, or deployment rollouts.\n\n## Architecture Overview\n\n```\nPayment Created → PostgreSQL → Sequin (CDC) → Kafka → Restate → Svix Cloud → Merchant\n     ↓              (WAL)         (Stream)    (Queue)  (Durable)  (Delivery)   (Joe's Shop)\n   Atomic         Capture       Reliable    Ordering  Execution   Retries      Receives\n   Writes         Changes       Transport   Preserved Guaranteed  Signing      Webhook\n```\n\n## Real-World Example: Joe's T-Shirt Shop\n\n### The Setup\n\n- **Dodo Payments**: Payment processor\n- **Joe's T-Shirt Shop**: Merchant using Dodo to accept payments\n- **Customer**: Buys a t-shirt for $25\n\n### The Flow\n\n#### 1. Payment is Created (Atomic Event Capture)\n\n```bash\nPOST /payments\n{\n  \"amount\": 2500,\n  \"currency\": \"USD\",\n  \"merchant_id\": \"joes-tshirt-shop\"\n}\n```\n\nThe API service writes to PostgreSQL with a **database trigger** that atomically creates both the payment record and domain event record in a single transaction.\n\n**Why atomic?** If we crash after writing the payment but before writing the event, the webhook is lost forever.\n\n#### 2. Change Data Capture (Sequin)\n\nSequin monitors PostgreSQL's Write-Ahead Log (WAL) and detects new events:\n\n```sql\nSELECT * FROM domain_events WHERE merchant_id = 'joes-tshirt-shop';\n-- event_type: 'payment.succeeded'\n-- object_id: '550e8400-e29b-41d4-a716-446655440000'\n```\n\n**Why CDC?** Reading the WAL is more reliable than polling tables. It captures every change with exactly-once delivery guarantees.\n\n#### 3. Kafka Streaming\n\nSequin publishes events to Kafka topic `webhook-events` with ordering preserved per merchant.\n\n**Why Kafka?** Provides reliable, ordered delivery with events persisted to disk and replicated across brokers.\n\n#### 4. Restate (Durable Execution)\n\nRestate consumes from Kafka and invokes `SvixCaller.process()`:\n\n```rust\nasync fn process(event: DomainEvent) -\u003e Result\u003cString\u003e {\n    // 1. Fetch enriched payload from data-service\n    let payment = fetch_payment_details(event.object_id);\n\n    // 2. Send to Svix Cloud\n    svix.message().create(\n        event.merchant_id,  // \"joes-tshirt-shop\"\n        MessageIn {\n            event_type: \"payment.succeeded\",\n            payload: payment\n        }\n    );\n}\n```\n\n**Why Restate?** If the service crashes mid-execution, Restate automatically retries from the last successful step. It's like a database transaction for your entire workflow.\n\n**Crash recovery example:**\n- Crashes after fetching payload but before Svix API call → Restate retries just the Svix call\n- Crashes after Svix API call succeeds → Restate marks complete, moves to next event\n- Network timeout → Restate retries with exponential backoff\n\n#### 5. Svix Cloud (Webhook Delivery)\n\nSvix receives the event and handles delivery to Joe's shop with automatic retries, cryptographic signing, and monitoring.\n\n```http\nPOST https://joes-tshirt-shop.com/webhooks\nContent-Type: application/json\nSvix-Signature: v1,g0hM9SsE+OTPJTGt...\n\n{\n  \"event_id\": \"550e8400-e29b-41d4-a716-446655440000\",\n  \"event_type\": \"payment.succeeded\",\n  \"payment\": {\n    \"id\": \"550e8400-e29b-41d4-a716-446655440000\",\n    \"amount\": 2500,\n    \"currency\": \"USD\",\n    \"status\": \"succeeded\"\n  }\n}\n```\n\n**Svix handles:**\n- Cryptographic signing (HMAC-SHA256)\n- Automatic retries with exponential backoff\n- Delivery monitoring and alerting\n- Customer portal for debugging webhooks\n\n#### 6. Joe's Shop Receives Webhook\n\n```javascript\napp.post('/webhooks', (req, res) =\u003e {\n  // Verify signature\n  svix.webhooks.verify(req.body, req.headers);\n\n  // Process event\n  const { payment } = req.body;\n  fulfillOrder(payment.id);\n\n  res.status(200).send('OK');\n});\n```\n\n## Key Components\n\n| Component | Role |\n|-----------|------|\n| **PostgreSQL + Triggers** | Atomic event capture |\n| **Sequin (CDC)** | Reliable event extraction |\n| **Kafka** | Durable event streaming |\n| **Restate** | Durable workflow execution |\n| **Svix** | Webhook delivery platform |\n\n## Reliability Guarantees\n\n| Failure Scenario | How It's Handled |\n|-----------------|------------------|\n| API crashes after payment | Trigger ensures event is written atomically |\n| Sequin crashes | Resumes from last WAL position |\n| Kafka broker fails | Replication keeps events safe |\n| Restate crashes mid-processing | Resumes from last journal entry |\n| Svix API timeout | Restate retries with backoff |\n| Merchant endpoint down | Svix retries for 3 days |\n\n## Architecture Comparison\n\n### Svix for Webhook Delivery\n\n**Why Svix?**\n- Automatic webhook signing (HMAC-SHA256)\n- Delivery monitoring \u0026 alerts\n- Customer self-service portal\n- Advanced features (transformations, filtering, rate limiting)\n- Reliable retries for 3 days with exponential backoff\n\nFocus on your core product while Svix handles webhook delivery infrastructure.\n\n## Architecture Principles\n\n### 1. Atomic Event Capture\n\n```sql\nCREATE OR REPLACE FUNCTION notify_payment_created()\nRETURNS TRIGGER AS $$\nBEGIN\n  INSERT INTO domain_events (event_type, object_id, merchant_id, payload)\n  VALUES ('payment.succeeded', NEW.id, NEW.merchant_id, row_to_json(NEW));\n  RETURN NEW;\nEND;\n$$ LANGUAGE plpgsql;\n```\n\n### 2. Change Data Capture\n\n```\nAVOID - Traditional Polling: SELECT * FROM events WHERE created_at \u003e last_poll\n  - Misses events during high load\n  - Adds load to database\n\nBETTER - CDC (Sequin): Read from PostgreSQL WAL\n  - Zero impact on database performance\n  - Captures every change\n  - Exactly-once delivery\n```\n\n### 3. Durable Execution\n\n```rust\n// WITHOUT Restate: Crash = restart from beginning\nasync fn deliver_webhook(event) {\n  let payload = fetch_payload(event.id);\n  send_to_svix(payload);\n}\n\n// WITH Restate: Crash = resume from last step\n#[restate_sdk::service]\nasync fn deliver_webhook(ctx, event) {\n  let payload = ctx.run(|| fetch_payload(event.id)).await;  // Cached\n  ctx.run(|| send_to_svix(payload)).await;                   // Idempotent\n}\n```\n\n### 4. Separation of Concerns\n\nEach component has a single responsibility and can fail independently without data loss:\n\n```\nPayment API:      Accepts payments, writes to database\nPostgreSQL:       Source of truth for payment data\nSequin:           Reliable event extraction\nKafka:            Durable event streaming\nRestate:          Durable workflow execution\nSvix:             Webhook delivery infrastructure\n```\n\n## Real-World Example: Dodo Payments\n\nThis demo replicates the Dodo Payments production architecture:\n- PostgreSQL + triggers for atomicity\n- Sequin for CDC (Change Data Capture)\n- Kafka for event streaming\n- Restate for durable execution\n- **Svix for webhook delivery** ← This exact architecture\n\nThese principles are battle-tested in production at Dodo Payments.\n\n## Common Pitfalls Avoided\n\n### AVOID: Event Creation in Application Code\n```javascript\n// Race condition - crash between writes = lost event\nawait db.payments.create(payment);\nawait db.events.create(event);\n```\n\n### BETTER: Event Creation in Database Trigger\n```sql\n-- Atomic - both succeed or both fail\nCREATE TRIGGER payment_created AFTER INSERT ON payments\nFOR EACH ROW EXECUTE FUNCTION notify_payment_created();\n```\n\n### AVOID: Direct HTTP Calls Without Durability\n```javascript\n// Crash = lost event\napp.post('/payments', async (req, res) =\u003e {\n  await db.payments.create(req.body);\n  await axios.post('https://merchant.com/webhook', event);\n});\n```\n\n### BETTER: Async Processing with Durability\n```javascript\n// Event persisted, delivery guaranteed\napp.post('/payments', async (req, res) =\u003e {\n  await db.payments.create(req.body);  // Trigger creates event\n  res.json({ success: true });\n  // Sequin → Kafka → Restate handles delivery\n});\n```\n\n## Documentation\n\n- **[LOCAL_SETUP.md](./LOCAL_SETUP.md)** - Local development setup guide\n- **[SVIX_SETUP.md](./SVIX_SETUP.md)** - Svix Cloud integration guide\n- **[ARCHITECTURE.md](./ARCHITECTURE.md)** - Detailed architecture explanation (if exists)\n\n**External Resources:**\n- Sequin Docs: https://docs.sequinstream.com\n- Restate Docs: https://docs.restate.dev\n- Svix Docs: https://docs.svix.com\n\n## Blog Post\n\nRead the full story: [Building Reliable Webhooks: How Dodo Payments Delivers 100% of Events](https://dodo.dev/blog/reliable-webhooks)\n\n---\n\n**Built to demonstrate production-grade webhook architecture using battle-tested open source tools.**\n\n**Interested in building your own webhook system instead of using Svix?** Checkout branch [`inhouse-webhook-no-svix`](https://github.com/your-repo/tree/inhouse-webhook-no-svix) for an in-house webhook implementation.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdodopayments%2Fdurable-webhook-architecture-demo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdodopayments%2Fdurable-webhook-architecture-demo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdodopayments%2Fdurable-webhook-architecture-demo/lists"}