{"id":48614009,"url":"https://github.com/angusgyoung/gox","last_synced_at":"2026-04-09T01:04:11.143Z","repository":{"id":163699910,"uuid":"628582684","full_name":"angusgyoung/gox","owner":"angusgyoung","description":"A small outbox publisher for Postgres/Kafka, with support for kafka-backed parallelisation and rebalancing.","archived":false,"fork":false,"pushed_at":"2025-05-21T12:19:11.000Z","size":171,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-21T12:48:29.742Z","etag":null,"topics":["confluent-kafka","event-sourcing","go","golang","kafka","sidecar"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/angusgyoung.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-04-16T12:06:01.000Z","updated_at":"2025-05-21T11:51:33.000Z","dependencies_parsed_at":"2023-12-17T13:24:20.529Z","dependency_job_id":"854ea309-d0e2-4e60-aad6-8837207d28e3","html_url":"https://github.com/angusgyoung/gox","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/angusgyoung/gox","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/angusgyoung%2Fgox","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/angusgyoung%2Fgox/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/angusgyoung%2Fgox/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/angusgyoung%2Fgox/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/angusgyoung","download_url":"https://codeload.github.com/angusgyoung/gox/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/angusgyoung%2Fgox/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31580509,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-08T14:31:17.711Z","status":"ssl_error","status_checked_at":"2026-04-08T14:31:17.202Z","response_time":54,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["confluent-kafka","event-sourcing","go","golang","kafka","sidecar"],"created_at":"2026-04-09T01:04:06.651Z","updated_at":"2026-04-09T01:04:10.903Z","avatar_url":"https://github.com/angusgyoung.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Gox\n\n![Build](https://github.com/angusgyoung/gox/actions/workflows/build.yml/badge.svg?branch=main)\n[![Go Reference](https://pkg.go.dev/badge/github.com/angusgyoung/gox.svg)](https://pkg.go.dev/github.com/angusgyoung/gox)\n\nA small outbox publisher for Postgres/Kafka, with support for kafka-backed parallelisation and rebalancing. Intended to work nicely as a sidecar to a container performing transactional publication.\n\nInspired by [a fantastic blog post by Alexander Morley and Richard Noble](https://medium.com/babylon-technology-blog/distributed-outbox-event-publishing-pattern-with-kafka-and-sidecars-c57350c0ff7c).\n\nSupports\n\n- postgres \u003e= 11.x\n- kafka \u003e= 2.1.1\n\n## Features\n\n- Batched delivery\n- Automatic rebalancing and leader election\n  across instances using Kafka consumer groups\n- Multi-topic support\n- Structured logging\n- Telemetry\n\n## Getting Started\n\n### Running Gox\n\n```shell\ndocker run ghcr.io/angusgyoung/gox:latest --brokers=broker:9092 --topics=client-events-v1 --db=postgres://outbox:outbox@postgres:5432/outbox\n```\n\nThis will start gox, which will begin to poll the broker. Once the broker has assigned it some partitions, the instance will begin publishing events from the outbox table.\n\n```\ntime=\"2023-05-14T20:46:45Z\" level=info msg=\"Starting gox...\"\ntime=\"2023-05-14T20:46:45Z\" level=info msg=\"Operator initialised\" instanceId=6f1008e8-a5c7-40c2-a0a3-9a64c10ff65e\ntime=\"2023-05-14T20:46:45Z\" level=info msg=\"Polling for events...\"\n```\n\nAs instances of gox enter and exit the consumer group, responsibility for each partition of the configured topics will be redistributed amoung the remaining instances.\n\n### Writing Events\n\nTo publish an event, insert a new row into the outbox table. The producing application is responsible for serialising the message content.\n\nApplications should populate the `topic`, `partition`, `key` and `message` columns on inserts:\n\n```sql\ninsert into outbox \n\t(topic, partition, key, message)\nvalues \n\t(\n\t\t'client-events-v1', \n\t\t'0', \n\t\tgen_random_uuid(), \n\t\t'message'\n\t);\n```\n\nOnce the event is written and committed, an instance of gox assigned to the event's partition will publish the\nmessage to the topic, and update its status from `PENDING` to `SENT`, setting `updated_timestamp`\nto the timestamp of batch completion.\n\n\nOptionally, the producing application can insert an array of JSON key/value pairs into the `headers` column to publish the message with headers. \n\n```sql\ninsert into outbox \n\t(topic, partition, key, message, headers)\nvalues \n\t(\n\t\t'client-events-v1', \n\t\t'0', \n\t\tgen_random_uuid(), \n\t\t'message',\n\t\t'[\n\t\t\t{\n\t\t\t\t\"key\": \"x-client-name\",\n\t\t\t\t\"value\": \"client-a\"\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"key\": \"x-client-version\",\n\t\t\t\t\"value\": 1\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"key\": \"x-client-id\",\n\t\t\t\t\"value\": \"abc\"\n\t\t\t}\n\t\t]'::json\n\t);\n```\n\n### Database Setup\n\nBy default, gox will attempt to create the following tables on startup, and as such the database should be configured to allow gox to create tables:\n\n```sql\nCREATE TABLE IF NOT EXISTS outbox (\n\t\tid uuid UNIQUE NOT NULL DEFAULT gen_random_uuid(),\n\t\tcreated_timestamp timestamp WITHOUT TIME ZONE DEFAULT (now() at time zone 'utc'),\n\t\tupdated_timestamp timestamp,\n\t\tstatus varchar(8) NOT NULL DEFAULT 'PENDING',\n\t\ttopic varchar(128) NOT NULL,\n\t\tpartition smallint NOT NULL,\n\t\tkey varchar(64) NOT NULL,\n\t\tmessage bytea NOT NULL,\n\t\theaders json,\n\t\tinstance_id uuid\n);\n\n```\n\nAn additional table, `schema-migrations` will also be created to track schema changes.\n\n## Configuration\n\n### Database connection\n\n`--db`/`GOX_DB_URL`\n\nDatabase connection URL.\n\n### Broker connection\n\n`--brokers`/`GOX_BROKERS`\n\nComma-separated list of brokers to connect to.\n\n### Topics\n\n`--topics`/`GOX_TOPICS`\n\nComma-separated list of topics to subscribe to. Topics present in the outbox table that are not part of this list will **not** be published by gox.\n\n### Poll Interval (optional)\n\n`--pollInterval`/`GOX_POLL_INTERVAL`\n\nSets the interval between consumer polls in milliseconds. Defaults to `100` (ms).\n\n### Batch Size (optional)\n\n`--batchSize`/`GOX_BATCH_SIZE`\n\nSets the maximum number of events to be published on each poll. Defaults to `50`.\n\n### Log Level (optional)\n\n`--logLevel`/`GOX_LOG_LEVEL`\n\nSets the log level. Defaults to `info`.\n\n### Log Format (optional)\n\n`--logFormat`/`GOX_LOG_FORMAT`\n\nSets the log format. Available options are `json` and `text`. Defaults to `text`.\n\n### Enable Telemetry (optional)\n\n`--enableTelemetry`/`GOX_ENABLE_TELEMETRY`\n\n\u003e [!WARNING]\n\u003e **Telemetry support is experimental.** Things may not work as expected, and\n\u003e configuration/capabilities may change.\n\nEnables metric telemetry over OTLP. See [go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp](https://pkg.go.dev/go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp)\nfor configuration options.\n\nSo far, the following metrics are available:\n\n- gox_published_messages\n- gox_rebalance_events\n\n### Completion Mode (optional)\n\n`--completionMode`/`GOX_COMPLETION_MODE`\n\n\nSets the strategy for updating the table after a batch of events have been published. \n\n- `UPDATE` will retain published events, setting `status` to `SENT` and setting `updated_timestamp` to the current time.\n- `DELETE` will delete published events from the table. \n\nDefaults to `UPDATE`.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fangusgyoung%2Fgox","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fangusgyoung%2Fgox","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fangusgyoung%2Fgox/lists"}