{"id":50454226,"url":"https://github.com/mizcausevic-dev/webhook-ingestion-pipeline","last_synced_at":"2026-06-01T01:05:46.441Z","repository":{"id":356165524,"uuid":"1231315759","full_name":"mizcausevic-dev/webhook-ingestion-pipeline","owner":"mizcausevic-dev","description":"Webhook Ingestion Pipeline","archived":false,"fork":false,"pushed_at":"2026-05-06T21:38:04.000Z","size":698,"stargazers_count":0,"open_issues_count":4,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-06T23:33:53.022Z","etag":null,"topics":["backend","event-driven","express","nodejs","openapi","swagger","typescript","webhooks"],"latest_commit_sha":null,"homepage":"https://mizcausevic.com/skills/","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mizcausevic-dev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-06T21:02:34.000Z","updated_at":"2026-05-06T21:38:08.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/mizcausevic-dev/webhook-ingestion-pipeline","commit_stats":null,"previous_names":["mizcausevic-dev/webhook-ingestion-pipeline"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/mizcausevic-dev/webhook-ingestion-pipeline","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mizcausevic-dev%2Fwebhook-ingestion-pipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mizcausevic-dev%2Fwebhook-ingestion-pipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mizcausevic-dev%2Fwebhook-ingestion-pipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mizcausevic-dev%2Fwebhook-ingestion-pipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mizcausevic-dev","download_url":"https://codeload.github.com/mizcausevic-dev/webhook-ingestion-pipeline/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mizcausevic-dev%2Fwebhook-ingestion-pipeline/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33755379,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-31T02:00:06.040Z","response_time":95,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backend","event-driven","express","nodejs","openapi","swagger","typescript","webhooks"],"created_at":"2026-06-01T01:05:46.379Z","updated_at":"2026-06-01T01:05:46.433Z","avatar_url":"https://github.com/mizcausevic-dev.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Webhook Ingestion Pipeline\n\n\u003e **TypeScript webhook operations portfolio project** demonstrating multi-provider event ingestion, shared-secret verification, idempotency handling, retry workflows, and production-style operational visibility.\n\n**Recruiter takeaway:** *\"This person can design reliable backend services around real SaaS integration patterns, not just basic CRUD APIs.\"*\n\n---\n\n## Project Overview\n\n| Attribute | Detail |\n|---|---|\n| **Runtime** | Node.js + TypeScript |\n| **Framework** | Express 5 |\n| **Domain** | SaaS webhook ingestion and event operations |\n| **Providers Modeled** | Stripe · HubSpot · Salesforce · Internal App |\n| **Core Behaviors** | Secret verification · Idempotency · Failure handling · Retry operations · Metrics |\n| **Docs** | Swagger UI at `/docs` |\n\n---\n\n## Executive Summary\n\nWebhook Ingestion Pipeline models the kind of internal platform service revenue, product, and operations teams rely on when third-party systems need to push event data into a centralized processing layer. Instead of stopping at \"receive a POST body,\" the project demonstrates the operational concerns that matter in production: duplicate delivery protection, event state transitions, retry workflows, failure visibility, and clear observability endpoints.\n\nThe result is a recruiter-facing backend project that feels like a real integration service sitting between vendors such as Stripe or HubSpot and a broader internal event-processing stack.\n\n---\n\n## Architecture\n\n```text\nProvider Webhook\n    |\n    v\nPOST /webhooks/:provider\n    |\n    +--\u003e Shared-secret verification\n    +--\u003e Payload validation\n    +--\u003e Idempotency lookup\n    +--\u003e Event record creation\n    +--\u003e Processing attempt\n             |\n             +--\u003e processed\n             +--\u003e failed\n                      |\n                      v\n              GET /api/failures\n              POST /api/events/:id/retry\n```\n\n### Request Flow\n\n1. A provider sends an event to `POST /webhooks/:provider`.\n2. The service verifies the provider secret from `x-webhook-secret`.\n3. The payload is validated with Zod.\n4. The service checks whether the `(provider, eventId)` pair has already been seen.\n5. New events move through `received -\u003e validated -\u003e processed/failed`.\n6. Failed events remain operator-visible via `/api/failures`.\n7. Retries create a new processing attempt and can promote the event into `retried`.\n\n---\n\n## Event Lifecycle\n\n| State | Meaning |\n|---|---|\n| `received` | Inbound delivery has been captured |\n| `validated` | Provider and payload checks passed |\n| `processed` | First-pass processing succeeded |\n| `failed` | Processing failed and is awaiting operator action |\n| `retried` | A later attempt succeeded after a prior failure |\n\n### Idempotency Model\n\nEach event is keyed by provider and external event ID, with an optional explicit `idempotencyKey`. Duplicate deliveries are accepted but not reprocessed. Instead, the service increments a `duplicateDeliveries` counter and returns the existing event record.\n\nThat mirrors the defensive patterns needed when Stripe, HubSpot, or internal event buses redeliver the same event more than once.\n\n---\n\n## API Endpoints\n\n| Method | Endpoint | Purpose |\n|---|---|---|\n| `GET` | `/health` | Service status and uptime |\n| `POST` | `/webhooks/:provider` | Receive a provider webhook |\n| `GET` | `/api/events` | List webhook event records |\n| `GET` | `/api/events/:id` | Fetch a single event |\n| `GET` | `/api/failures` | View failed deliveries / dead-letter style records |\n| `POST` | `/api/events/:id/retry` | Retry a failed event |\n| `GET` | `/api/metrics` | Operational metrics summary |\n| `GET` | `/docs` | Swagger / OpenAPI UI |\n\n---\n\n## Sample Webhook Request\n\n```json\n{\n  \"eventId\": \"evt_stripe_test_200\",\n  \"eventType\": \"invoice.paid\",\n  \"occurredAt\": \"2026-05-06T14:00:00.000Z\",\n  \"data\": {\n    \"customerId\": \"cus_test_77\",\n    \"amount\": 42000\n  }\n}\n```\n\nExample request:\n\n```bash\ncurl -X POST http://localhost:3000/webhooks/stripe \\\n  -H \"Content-Type: application/json\" \\\n  -H \"x-webhook-secret: stripe-demo-secret\" \\\n  -d @payload.json\n```\n\n## Sample Processed Event Response\n\n```json\n{\n  \"accepted\": true,\n  \"duplicate\": false,\n  \"event\": {\n    \"id\": \"evt_8d4b9d56-5f4f-4fd4-9a8b-4d8c0f583777\",\n    \"provider\": \"stripe\",\n    \"eventType\": \"invoice.paid\",\n    \"status\": \"processed\",\n    \"idempotencyKey\": \"stripe:evt_stripe_test_200\",\n    \"duplicateDeliveries\": 0,\n    \"processingAttempts\": 1,\n    \"latestSummary\": \"Event validated, routed, and processed successfully.\"\n  }\n}\n```\n\n---\n\n## Screenshots\n\n### Hero Capture\n\n![Swagger UI](https://raw.githubusercontent.com/mizcausevic-dev/webhook-ingestion-pipeline/main/screenshots/01-hero.png)\n\n### Processing Workflow\n\n![Event workflow](https://raw.githubusercontent.com/mizcausevic-dev/webhook-ingestion-pipeline/main/screenshots/02-feature.png)\n\n### Validation Proof\n\n![Test proof](https://raw.githubusercontent.com/mizcausevic-dev/webhook-ingestion-pipeline/main/screenshots/03-proof.png)\n\n---\n\n## Getting Started\n\n### Prerequisites\n\n- Node.js 20+\n- npm\n\n### Setup\n\n```bash\ngit clone https://github.com/mizcausevic-dev/webhook-ingestion-pipeline.git\ncd webhook-ingestion-pipeline\nnpm install\ncp .env.example .env\nnpm run dev\n```\n\nVisit:\n\n- `http://localhost:3000/docs`\n- `http://localhost:3000/api/events`\n- `http://localhost:3000/api/metrics`\n\n### Run Tests\n\n```bash\nnpm test\n```\n\n---\n\n## What This Demonstrates\n\n- backend API design beyond CRUD\n- webhook hardening with provider verification\n- idempotency and duplicate-delivery handling\n- event lifecycle modeling and operator-friendly retry flows\n- production-minded observability through failures and metrics\n- clean TypeScript service structure and documentation discipline\n\n---\n\n## Future Enhancements\n\n- persist events and attempts in PostgreSQL\n- introduce Redis or SQS-backed asynchronous workers\n- add HMAC signature verification for each provider\n- emit OpenTelemetry traces and structured logs\n- add dead-letter replay policies and alerting hooks\n\n---\n\n## Tech Stack\n\n- Node.js\n- TypeScript\n- Express\n- Zod\n- Swagger / OpenAPI\n- Helmet\n- CORS\n- Morgan\n- Node test runner + Supertest\n\n### Portfolio Links\n\n- [LinkedIn](https://www.linkedin.com/in/mirzacausevic)\n- [Skills Page](https://mizcausevic.com/skills/)\n- [Medium](https://medium.com/@mizcausevic)\n- [GitHub](https://github.com/mizcausevic-dev)\n\n---\n\n*Part of [mizcausevic-dev's GitHub portfolio](https://github.com/mizcausevic-dev) — demonstrating webhook operations design, event reliability workflows, and production-aware backend delivery.*\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmizcausevic-dev%2Fwebhook-ingestion-pipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmizcausevic-dev%2Fwebhook-ingestion-pipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmizcausevic-dev%2Fwebhook-ingestion-pipeline/lists"}