{"id":49994832,"url":"https://github.com/clerk/jack-courier-driver-pglg","last_synced_at":"2026-05-19T07:01:52.526Z","repository":{"id":358741565,"uuid":"1159377706","full_name":"clerk/jack-courier-driver-pglg","owner":"clerk","description":"A Postgres logical replication based driver for JacK Courier Library","archived":false,"fork":false,"pushed_at":"2026-05-18T21:06:53.000Z","size":41,"stargazers_count":0,"open_issues_count":2,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-18T23:42:15.860Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/clerk.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-02-16T16:53:27.000Z","updated_at":"2026-05-18T21:04:41.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/clerk/jack-courier-driver-pglg","commit_stats":null,"previous_names":["clerk/jack-courier-driver-pglg"],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/clerk/jack-courier-driver-pglg","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clerk%2Fjack-courier-driver-pglg","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clerk%2Fjack-courier-driver-pglg/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clerk%2Fjack-courier-driver-pglg/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clerk%2Fjack-courier-driver-pglg/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/clerk","download_url":"https://codeload.github.com/clerk/jack-courier-driver-pglg/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clerk%2Fjack-courier-driver-pglg/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33205417,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-18T09:27:30.708Z","status":"online","status_checked_at":"2026-05-19T02:00:06.763Z","response_time":58,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-05-19T07:01:51.328Z","updated_at":"2026-05-19T07:01:52.519Z","avatar_url":"https://github.com/clerk.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# jack-courier-driver-pglg\n\nA [jack-courier-lib](https://github.com/clerk/jack-courier-lib) driver that uses PostgreSQL logical replication (WAL streaming) to source jobs from an outbox table and deliver them to jack-service.\n\n## How It Works\n\n```\n┌─────────────────┐    INSERT (same tx)     ┌──────────────────┐\n│  Domain Service │ ────────────────────►   │  Outbox Table    │\n│ (business logic)│                         │  (partitioned)   │\n└─────────────────┘                         └────────┬─────────┘\n                                                     │ WAL stream\n                                                     ▼\n                                            ┌──────────────────┐\n                                            │  pglg Driver     │\n                                            │  (this library)  │\n                                            └────────┬─────────┘\n                                                     │ submit([]Job)\n                                                     ▼\n                                            ┌──────────────────┐\n                                            │  jack-service    │\n                                            │  (gRPC)          │\n                                            └──────────────────┘\n```\n\n1. The domain service INSERTs a row into the outbox table **within the same transaction** as its business logic (transactional outbox pattern).\n2. The pglg driver streams WAL changes via PostgreSQL logical replication, parses INSERT events, batches them into `courier.Job` objects, and calls `submit()` to deliver them to jack-service.\n3. On success, the driver advances its cursor (LSN). On gRPC failure, it reconnects and re-streams from the last cursor position (at-least-once delivery).\n\n## Setup\n\n### 1. Prerequisites\n\nThe PostgreSQL instance must have logical decoding enabled:\n\n```\nwal_level = logical\nmax_replication_slots \u003e= 1   (one per service using this driver)\nmax_wal_senders \u003e= 1\n```\n\nOn Cloud SQL, set the `cloudsql.logical_decoding` flag to `on`.\n\n### 2. Create outbox tables, publication, and replication slot\n\nRun the setup tool once per service (not at runtime):\n\n```bash\ngo run github.com/clerk/jack-courier-driver-pglg/cmd/pglg-setup create \\\n    --conn-string=\"postgres://user:pass@host:5432/db\" \\\n    --schema=public \\\n    --prefix=billing_outbox \\\n    --publication=billing_outbox_pub \\\n    --slot=billing_outbox_slot\n```\n\nThis creates:\n- `{schema}.{prefix}_jobs` — partitioned outbox table\n- `{schema}.{prefix}_cursor` — LSN progress tracking\n- `{schema}.{prefix}_partition_meta` — partition registry\n- A PostgreSQL publication (INSERT-only) on the outbox table\n- A logical replication slot using the `pgoutput` plugin\n- An initial partition covering the current hour\n\nTo tear down (dev/test only):\n\n```bash\ngo run github.com/clerk/jack-courier-driver-pglg/cmd/pglg-setup destroy \\\n    --conn-string=\"...\" --prefix=billing_outbox \\\n    --publication=billing_outbox_pub --slot=billing_outbox_slot\n```\n\n### 3. Service: write to the outbox table\n\nIn your service's transaction, INSERT into the outbox table alongside your business logic:\n\n```sql\nINSERT INTO billing_outbox_jobs (producer_id, job_type, payload, run_at, trace_id)\nVALUES ($1, $2, $3, $4, $5);\n```\n\nExample with clerk_go's `PerformTx`:\n\n```go\ntxErr := db.PerformTx(ctx, func(tx database.Tx) (bool, error) {\n    // Business logic\n    if err := userRepo.Insert(ctx, tx, user); err != nil {\n        return true, err\n    }\n\n    // Outbox write — same transaction guarantees atomicity\n    _, err := tx.ExecContext(ctx,\n        `INSERT INTO billing_outbox_jobs (producer_id, job_type, payload, trace_id)\n         VALUES ($1, $2, $3, $4)`,\n        \"billing-service\", \"charge_customer\", payloadJSON, traceID,\n    )\n    return err != nil, err\n})\n```\n\n### 4. Service: register and run the driver\n\n```go\npackage main\n\nimport (\n    \"log\"\n    \"os\"\n\n    courier \"github.com/clerk/jack-courier-lib\"\n    pglg \"github.com/clerk/jack-courier-driver-pglg\"\n)\n\nfunc main() {\n    driver, err := pglg.New(pglg.Config{\n        ConnString:      os.Getenv(\"PGLG_CONN_STRING\"),\n        SlotName:        \"billing_outbox_slot\",\n        PublicationName: \"billing_outbox_pub\",\n        TablePrefix:     \"billing_outbox\",\n    })\n    if err != nil {\n        log.Fatal(err)\n    }\n\n    courier.RegisterDriver(driver)\n    courier.Main()\n}\n```\n\nOr use `ConfigFromEnv()` to read all config from `PGLG_*` environment variables.\n\n## Outbox Table Schema\n\n```sql\nCREATE TABLE {schema}.{prefix}_jobs (\n    id          BIGINT GENERATED ALWAYS AS IDENTITY,\n    created_at  TIMESTAMPTZ NOT NULL DEFAULT now(),\n    producer_id TEXT        NOT NULL,\n    job_type    TEXT        NOT NULL,\n    payload     BYTEA       NOT NULL DEFAULT '',\n    run_at      TIMESTAMPTZ,\n    trace_id    TEXT        NOT NULL DEFAULT '',\n    PRIMARY KEY (created_at, id)\n) PARTITION BY RANGE (created_at);\n```\n\n| Column | courier.Job Field | Notes |\n|--------|-------------------|-------|\n| `id` | `CorrelationID` | Auto-generated, stringified by driver |\n| `producer_id` | `ProducerID` | Must match registered producer in jack-service |\n| `job_type` | `JobType` | Must match registered job type in jack-service |\n| `payload` | `Payload` | Opaque bytes, typically JSON |\n| `run_at` | `RunAt` | NULL = immediate execution |\n| `trace_id` | `TraceID` | Distributed tracing correlation ID |\n\n## Configuration\n\n### Config struct\n\n| Field | Required | Default | Description |\n|-------|----------|---------|-------------|\n| `ConnString` | Yes | — | PostgreSQL connection string |\n| `SlotName` | Yes | — | Logical replication slot name |\n| `PublicationName` | Yes | — | Publication name |\n| `Schema` | No | `public` | Database schema |\n| `TablePrefix` | No | `outbox` | Table name prefix |\n| `MaxBatchSize` | No | `100` | Max jobs per submit() call |\n| `BatchTimeout` | No | `1s` | Max wait before flushing partial batch |\n| `StandbyInterval` | No | `10s` | Keepalive interval to Postgres |\n| `PartitionInterval` | No | `1h` | Duration of each partition |\n| `PartitionLookAhead` | No | `12h` | How far ahead to pre-create partitions |\n| `PartitionRetention` | No | `3h` | How long to keep old partitions |\n| `PartitionMaintInterval` | No | `5m` | Partition maintenance loop interval |\n| `ReconnectInitialDelay` | No | `1s` | Initial reconnection delay |\n| `ReconnectMaxDelay` | No | `30s` | Max reconnection delay |\n| `Logger` | No | `slog.Default()` | Structured logger |\n\n### Environment variables (via ConfigFromEnv)\n\n| Variable | Maps to |\n|----------|---------|\n| `PGLG_CONN_STRING` | `ConnString` |\n| `PGLG_SLOT_NAME` | `SlotName` |\n| `PGLG_PUBLICATION_NAME` | `PublicationName` |\n| `PGLG_SCHEMA` | `Schema` |\n| `PGLG_TABLE_PREFIX` | `TablePrefix` |\n| `PGLG_MAX_BATCH_SIZE` | `MaxBatchSize` |\n| `PGLG_BATCH_TIMEOUT` | `BatchTimeout` |\n| `PGLG_STANDBY_INTERVAL` | `StandbyInterval` |\n\n## Delivery Guarantees\n\n- **At-least-once delivery.** If the gRPC call to jack-service fails, the cursor is not advanced. The driver reconnects and re-streams from the last saved position.\n- **Per-job rejections** (unknown job type, invalid payload) are logged as warnings and the cursor advances past them. These are permanent failures that retrying won't fix.\n\n## Partition Management\n\nThe driver automatically manages partitions in a background goroutine:\n- Creates partitions up to 12 hours ahead (configurable)\n- Drops partitions older than 3 hours (configurable)\n- Runs every 5 minutes (configurable)\n- Errors are logged but don't crash the driver\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fclerk%2Fjack-courier-driver-pglg","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fclerk%2Fjack-courier-driver-pglg","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fclerk%2Fjack-courier-driver-pglg/lists"}