{"id":49849547,"url":"https://github.com/zeybek/ulak","last_synced_at":"2026-05-14T14:09:35.256Z","repository":{"id":353205189,"uuid":"1211156427","full_name":"zeybek/ulak","owner":"zeybek","description":"PostgreSQL extension for reliable async message delivery to HTTP, Kafka, MQTT, Redis, AMQP \u0026 NATS — atomically committed with your transaction, using the transactional outbox pattern","archived":false,"fork":false,"pushed_at":"2026-05-12T21:57:47.000Z","size":517,"stargazers_count":14,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-05-12T23:31:36.836Z","etag":null,"topics":["amqp","background-worker","event-driven","extension","http","kafka","message-queue","mqtt","postgres","postgresql","redis","transactional-outbox"],"latest_commit_sha":null,"homepage":null,"language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zeybek.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":".github/SECURITY.md","support":".github/SUPPORT.md","governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-15T05:53:21.000Z","updated_at":"2026-05-12T21:57:42.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/zeybek/ulak","commit_stats":null,"previous_names":["zeybek/ulak"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/zeybek/ulak","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zeybek%2Fulak","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zeybek%2Fulak/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zeybek%2Fulak/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zeybek%2Fulak/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zeybek","download_url":"https://codeload.github.com/zeybek/ulak/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zeybek%2Fulak/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33028258,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-13T13:14:54.681Z","status":"online","status_checked_at":"2026-05-14T02:00:06.663Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["amqp","background-worker","event-driven","extension","http","kafka","message-queue","mqtt","postgres","postgresql","redis","transactional-outbox"],"created_at":"2026-05-14T14:09:33.225Z","updated_at":"2026-05-14T14:09:35.238Z","avatar_url":"https://github.com/zeybek.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ulak\n\n[![CI](https://github.com/zeybek/ulak/actions/workflows/ci.yml/badge.svg)](https://github.com/zeybek/ulak/actions/workflows/ci.yml)\n[![PostgreSQL 14-18](https://img.shields.io/badge/PostgreSQL-14--18-336791?logo=postgresql\u0026logoColor=white)](https://www.postgresql.org)\n[![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE.md)\n[![GitHub Release](https://img.shields.io/github/v/release/zeybek/ulak)](https://github.com/zeybek/ulak/releases/latest)\n\n`ulak` is a PostgreSQL extension for the transactional outbox pattern.\n\nIt solves the dual-write problem by inserting messages into `ulak.queue` **inside the same transaction** as your business data, then dispatching them asynchronously through PostgreSQL background workers. You get **exactly-once writes to the local queue** and **at-least-once delivery** to external systems.\n\n`ulak` means \"messenger\". The point of the project is to keep enqueue atomic with your data, while moving retries, backoff, circuit breaking, DLQ handling, and redrive out of application code.\n\n## Why ulak\n\n- **Atomic enqueue**: `ulak.send()` and `ulak.publish()` write to the queue inside your transaction. If the transaction rolls back, the message never exists.\n- **Database-native execution model**: background workers poll `ulak.queue` with `FOR UPDATE SKIP LOCKED` and dispatch without requiring a separate CDC stack.\n- **Operational safety built in**: retry policy, stale-processing recovery, circuit breaker, DLQ, archive, health checks, and redrive are part of the engine.\n- **Protocol adapters, one queue model**: HTTP is built in; Kafka, MQTT, Redis Streams, AMQP, and NATS are optional adapters behind the same lifecycle.\n\n## Use ulak when\n\n- PostgreSQL is your source of truth and you want outbox semantics close to the data.\n- You need reliable webhook or broker delivery without rebuilding retry and DLQ logic in every service.\n- You prefer database-native operations over an external CDC pipeline.\n\n## Do not use ulak when\n\n- You already operate a CDC stack such as Debezium and are satisfied with that model.\n- Your main problem is large-scale event streaming infrastructure rather than transactional outbox delivery.\n- You do not want background worker activity, queue state, and delivery policy managed inside PostgreSQL.\n\n## How ulak compares\n\n| Approach | Atomic enqueue with business transaction | Built-in retry / DLQ / circuit breaker | Runs inside PostgreSQL | Best fit |\n|----------|-------------------------------------------|----------------------------------------|------------------------|----------|\n| **ulak** | Yes | Yes | Yes | PostgreSQL-centric systems that want native outbox delivery |\n| **App-level outbox** | Usually yes | Usually custom | No | Teams that prefer delivery logic in application services |\n| **Debezium / CDC** | Yes | Broker or consumer dependent | No | Existing Kafka + CDC estates |\n| **Direct broker publish** | No | Broker dependent | No | Fire-and-forget or eventually consistent integrations |\n\n## Supported protocols\n\n- **HTTP / HTTPS**: built in, always available\n- **Kafka**: compile with `ENABLE_KAFKA=1`\n- **MQTT**: compile with `ENABLE_MQTT=1`\n- **Redis Streams**: compile with `ENABLE_REDIS=1`\n- **AMQP**: compile with `ENABLE_AMQP=1`\n- **NATS**: compile with `ENABLE_NATS=1`\n\nAll non-HTTP protocols depend on their client libraries at build time.\n\n## 5-Minute HTTP Quick Start\n\n`ulak` is a PostgreSQL background worker extension. It will not run unless PostgreSQL starts with:\n\n```conf\nshared_preload_libraries = 'ulak'\n```\n\nThe shortest path is to start with HTTP only.\n\n```bash\ngit clone https://github.com/zeybek/ulak.git\ncd ulak\n\n# Start PostgreSQL only\ndocker compose up -d postgres\n\n# Build and install HTTP-only ulak\ndocker exec ulak-postgres-1 bash -c \\\n  \"cd /src/ulak \u0026\u0026 make clean \u0026\u0026 make \u0026\u0026 make install\"\n\n# Preload ulak and point workers at the test database\ndocker exec ulak-postgres-1 psql -U postgres -c \\\n  \"ALTER SYSTEM SET shared_preload_libraries = 'ulak';\n   ALTER SYSTEM SET ulak.database = 'ulak_test';\n   ALTER SYSTEM SET ulak.capture_response = 'on';\"\n\ndocker restart ulak-postgres-1\n\n# Create the extension\ndocker exec ulak-postgres-1 psql -U postgres -d ulak_test -c \\\n  \"CREATE EXTENSION ulak;\"\n```\n\nCreate one endpoint and send one message:\n\n```sql\nSELECT ulak.create_endpoint(\n  'httpbin',\n  'http',\n  '{\"url\": \"https://httpbin.org/post\", \"method\": \"POST\"}'::jsonb\n);\n\nBEGIN;\n  SELECT ulak.send(\n    'httpbin',\n    '{\"event\": \"order.created\", \"order_id\": 123, \"total\": 99.99}'::jsonb\n  );\nCOMMIT;\n```\n\nCheck delivery state:\n\n```sql\nSELECT id, status, retry_count, completed_at, last_error\nFROM ulak.queue\nORDER BY id DESC\nLIMIT 1;\n\nSELECT response\nFROM ulak.queue\nORDER BY id DESC\nLIMIT 1;\n\nSELECT * FROM ulak.health_check();\nSELECT * FROM ulak.get_worker_status();\n```\n\nIf the target is reachable, the newest row should move from `pending` to `completed`. If delivery fails, `retry_count` and `last_error` show why, and the worker retries according to policy.\n\n## Message lifecycle\n\n```mermaid\nflowchart TD\n    A[Application transaction] --\u003e B[ulak.send / ulak.publish]\n    B --\u003e C[Insert into ulak.queue]\n    C --\u003e D{Transaction commits?}\n    D -- No --\u003e E[No message exists]\n    D -- Yes --\u003e F[Background worker claims pending row]\n    F --\u003e G{Dispatch result}\n    G -- Success --\u003e H[status = completed]\n    H --\u003e I[Archived by maintenance]\n    G -- Retryable failure --\u003e J[status = pending with next_retry_at]\n    J --\u003e F\n    G -- Permanent failure or max retries --\u003e K[Move to ulak.dlq]\n    K --\u003e L[Operator redrive]\n    L --\u003e C\n```\n\n## Architecture\n\n```mermaid\nflowchart LR\n    App[Application SQL transaction] --\u003e API[SQL API: send publish subscribe]\n    API --\u003e Queue[(ulak.queue)]\n    Queue --\u003e Workers[Background workers 1..32]\n    Workers --\u003e CB[Circuit breaker and retry policy]\n    Workers --\u003e Dispatch[Dispatcher factory]\n    Dispatch --\u003e HTTP[HTTP]\n    Dispatch --\u003e Kafka[Kafka]\n    Dispatch --\u003e MQTT[MQTT]\n    Dispatch --\u003e Redis[Redis Streams]\n    Dispatch --\u003e AMQP[AMQP]\n    Dispatch --\u003e NATS[NATS]\n    Workers --\u003e DLQ[(ulak.dlq)]\n    Workers --\u003e Archive[(ulak.archive)]\n    Workers --\u003e Metrics[health_check worker_status endpoint_health metrics]\n```\n\n## Delivery model and guarantees\n\n- **Exactly-once write to the queue**: the enqueue happens inside the same transaction as your business data.\n- **At-least-once delivery**: messages are only terminal after confirmed delivery or explicit failure handling.\n- **Retryable failures stay in the queue**: the worker updates `retry_count`, schedules `next_retry_at`, and tries again.\n- **Permanent failures move to the DLQ**: exhausted or permanent failures are archived into `ulak.dlq`.\n- **Crash recovery is built in**: stale `processing` rows are reset back to `pending`.\n- **Per-endpoint circuit breaker**: endpoints move through `closed`, `open`, and `half_open`.\n\nWhat `ulak` does **not** claim is exactly-once delivery to remote systems. Remote consumers should still be idempotent.\n\n## Core SQL API\n\n### Queueing\n\n```sql\nSELECT ulak.send('endpoint_name', '{\"event\":\"user.created\"}'::jsonb);\n\nSELECT ulak.send_with_options(\n  'endpoint_name',\n  '{\"event\":\"user.created\"}'::jsonb,\n  5,\n  NOW() + INTERVAL '10 minutes',\n  'user-42-created',\n  '550e8400-e29b-41d4-a716-446655440000'::uuid,\n  NOW() + INTERVAL '1 hour',\n  'user-42'\n);\n\nSELECT ulak.send_batch('endpoint_name', ARRAY[\n  '{\"id\":1}'::jsonb,\n  '{\"id\":2}'::jsonb\n]);\n```\n\n### Endpoints\n\n```sql\nSELECT ulak.create_endpoint('orders-http', 'http',\n  '{\"url\":\"https://example.com/webhook\",\"method\":\"POST\"}'::jsonb);\n\nSELECT * FROM ulak.get_endpoint_health();\n```\n\n### Pub/Sub\n\n```sql\nSELECT ulak.create_event_type('order.created', 'Order created');\nSELECT ulak.subscribe('order.created', 'orders-http');\nSELECT ulak.publish('order.created', '{\"order_id\":123}'::jsonb);\n```\n\n### Operations\n\n```sql\nSELECT * FROM ulak.health_check();\nSELECT * FROM ulak.get_worker_status();\nSELECT * FROM ulak.dlq_summary();\nSELECT * FROM ulak.metrics();\n\nSELECT ulak.redrive_message(42);\nSELECT ulak.redrive_endpoint('orders-http');\nSELECT ulak.redrive_all();\n\nSELECT ulak.replay_message(100);\nSELECT ulak.replay_range(\n  1,\n  date_trunc('month', now()) - interval '1 month',\n  date_trunc('month', now())\n);\n```\n\n## Reliability and operations\n\n### Built-in behaviors\n\n- **Retry policies**: configurable fixed, linear, or exponential backoff\n- **Circuit breaker**: configurable threshold and cooldown per endpoint\n- **Stale-processing recovery**: recovers messages left in `processing` after worker failure\n- **Backpressure**: queue depth protection via `ulak.max_queue_size`\n- **Archive management**: completed messages can be moved out of the hot queue into `ulak.archive`\n- **DLQ retention and redrive**: failed messages stay inspectable and can be replayed into the queue\n- **Event log**: internal lifecycle and operational events are recorded in `ulak.event_log`\n\n### Operational checklist\n\n- Set `shared_preload_libraries = 'ulak'`\n- Set `ulak.database` to the database the workers should connect to\n- Size `ulak.workers`, `ulak.poll_interval`, and `ulak.batch_size` for your workload\n- Decide whether `ulak.capture_response` should be on in production\n- Monitor `ulak.health_check()`, `ulak.get_worker_status()`, `ulak.get_endpoint_health()`, `ulak.dlq_summary()`, and `ulak.metrics()`\n- Review `ulak.dlq_retention_days`, `ulak.archive_retention_months`, and `ulak.stale_recovery_timeout`\n\n## Security and access model\n\n`ulak` includes:\n\n- **RBAC roles**: `ulak_admin`, `ulak_application`, `ulak_monitor`\n- **HTTP SSRF protection**: internal URLs are blocked unless explicitly allowed\n- **TLS / mTLS support**\n- **HTTP auth helpers**: OAuth2 and AWS SigV4 validation paths are present in the repo\n- **Webhook signing / CloudEvents support**\n\nFor the full protocol-specific security surface, see the wiki pages linked below.\n\n## Installation\n\n### Prerequisites\n\n| Dependency | Required | Build flag |\n|------------|----------|------------|\n| PostgreSQL 14–18 | Yes | — |\n| libcurl | Yes | — |\n| librdkafka | Optional | `ENABLE_KAFKA=1` |\n| libmosquitto | Optional | `ENABLE_MQTT=1` |\n| hiredis | Optional | `ENABLE_REDIS=1` |\n| librabbitmq | Optional | `ENABLE_AMQP=1` |\n| libnats / cnats | Optional | `ENABLE_NATS=1` |\n\n### Build from source\n\n```bash\n# HTTP only\nmake \u0026\u0026 make install\n\n# All adapters\nmake ENABLE_KAFKA=1 ENABLE_MQTT=1 ENABLE_REDIS=1 ENABLE_AMQP=1 ENABLE_NATS=1 \u0026\u0026 make install\n```\n\nThen preload the extension and restart PostgreSQL:\n\n```conf\nshared_preload_libraries = 'ulak'\n```\n\nCreate the extension in the target database:\n\n```sql\nCREATE EXTENSION ulak;\n```\n\n### Docker\n\n```bash\n# Default PostgreSQL major\ndocker compose up -d\n\n# Specific PostgreSQL version\nPG_MAJOR=15 docker compose up -d\n```\n\nThe compose file includes PostgreSQL, Kafka, Redis, Mosquitto, RabbitMQ, and NATS for local development and e2e testing.\n\n## Configuration essentials\n\nAll settings use the `ulak.` prefix.\n\n| Parameter | Default | Purpose |\n|-----------|---------|---------|\n| `ulak.workers` | `4` | Number of background workers |\n| `ulak.database` | unset | Database workers connect to |\n| `ulak.poll_interval` | `500ms` | Queue polling interval |\n| `ulak.batch_size` | `200` | Messages claimed per cycle |\n| `ulak.default_max_retries` | `10` | Default retry budget |\n| `ulak.retry_base_delay` | `10s` | Retry backoff base |\n| `ulak.circuit_breaker_threshold` | `10` | Failures before opening the breaker |\n| `ulak.circuit_breaker_cooldown` | `30s` | Cooldown before half-open probe |\n| `ulak.capture_response` | `false` | Store protocol response payloads |\n| `ulak.max_queue_size` | `1000000` | Backpressure limit |\n| `ulak.dlq_retention_days` | `30` | DLQ retention |\n| `ulak.archive_retention_months` | `6` | Archive retention |\n\nSee the [Configuration Reference](https://github.com/zeybek/ulak/wiki/Configuration-Reference) for the full GUC surface.\n\n## Testing\n\nThe repository includes:\n\n- **TAP tests** in [`t/`](/Users/ahmet/Code/ulak/t:1) for worker startup, reload, and stale recovery\n- **Regression tests** in [`tests/regress`](/Users/ahmet/Code/ulak/tests/regress:1)\n- **Isolation tests** in [`tests/isolation`](/Users/ahmet/Code/ulak/tests/isolation:1)\n- **End-to-end protocol tests** in [`tests/e2e`](/Users/ahmet/Code/ulak/tests/e2e:1)\n\nRun the core regression suite:\n\n```bash\ndocker exec ulak-postgres-1 bash -c \\\n  \"cd /src/ulak \u0026\u0026 make installcheck\"\n```\n\nFor local code quality:\n\n```bash\nmake tools-install\nmake tools-versions\nmake format\nmake lint\nmake hooks-install\nmake hooks-run\n```\n\n## Documentation\n\nFull documentation lives in the **[Wiki](https://github.com/zeybek/ulak/wiki)**.\n\n| Category | Pages |\n|----------|-------|\n| Getting Started | [Quick Start](https://github.com/zeybek/ulak/wiki/Getting-Started) |\n| Architecture | [System Architecture](https://github.com/zeybek/ulak/wiki/Architecture) |\n| Reliability | [Reliability](https://github.com/zeybek/ulak/wiki/Reliability) |\n| Monitoring | [Monitoring](https://github.com/zeybek/ulak/wiki/Monitoring) |\n| Security | [Security](https://github.com/zeybek/ulak/wiki/Security) |\n| Protocols | [HTTP](https://github.com/zeybek/ulak/wiki/Protocol-HTTP) · [Kafka](https://github.com/zeybek/ulak/wiki/Protocol-Kafka) · [MQTT](https://github.com/zeybek/ulak/wiki/Protocol-MQTT) · [Redis](https://github.com/zeybek/ulak/wiki/Protocol-Redis) · [AMQP](https://github.com/zeybek/ulak/wiki/Protocol-AMQP) · [NATS](https://github.com/zeybek/ulak/wiki/Protocol-NATS) |\n| API Reference | [SQL API](https://github.com/zeybek/ulak/wiki/SQL-API-Reference) · [Configuration](https://github.com/zeybek/ulak/wiki/Configuration-Reference) |\n\n## License\n\n`ulak` is licensed under the [Apache License 2.0](LICENSE.md).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzeybek%2Fulak","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzeybek%2Fulak","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzeybek%2Fulak/lists"}