{"id":29116813,"url":"https://github.com/oagudo/outbox","last_synced_at":"2026-02-02T00:24:14.841Z","repository":{"id":292120788,"uuid":"979886154","full_name":"oagudo/outbox","owner":"oagudo","description":"Lightweight library for the transactional outbox pattern in Go, not tied to any specific relational database or broker.","archived":false,"fork":false,"pushed_at":"2026-01-22T21:47:24.000Z","size":5965,"stargazers_count":114,"open_issues_count":1,"forks_count":3,"subscribers_count":3,"default_branch":"main","last_synced_at":"2026-01-23T14:23:59.975Z","etag":null,"topics":["go","golang","golang-library","outbox","outbox-example","outbox-pattern"],"latest_commit_sha":null,"homepage":"https://pkg.go.dev/github.com/oagudo/outbox?utm_source=godoc","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oagudo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-05-08T08:16:25.000Z","updated_at":"2026-01-22T21:47:28.000Z","dependencies_parsed_at":"2026-02-02T00:19:24.564Z","dependency_job_id":null,"html_url":"https://github.com/oagudo/outbox","commit_stats":null,"previous_names":["oagudo/outbox"],"tags_count":28,"template":false,"template_full_name":null,"purl":"pkg:github/oagudo/outbox","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oagudo%2Foutbox","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oagudo%2Foutbox/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oagudo%2Foutbox/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oagudo%2Foutbox/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oagudo","download_url":"https://codeload.github.com/oagudo/outbox/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oagudo%2Foutbox/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28996596,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-01T23:10:54.274Z","status":"ssl_error","status_checked_at":"2026-02-01T23:10:47.298Z","response_time":56,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["go","golang","golang-library","outbox","outbox-example","outbox-pattern"],"created_at":"2025-06-29T11:13:48.561Z","updated_at":"2026-02-02T00:24:14.829Z","avatar_url":"https://github.com/oagudo.png","language":"Go","funding_links":[],"categories":["Messaging","Distributed Systems","分布式系统"],"sub_categories":["Search and Analytic Databases","检索及分析资料库"],"readme":"\u003cp align=\"center\" class=\"disable-logo\"\u003e\n\u003ca href=\"#\"\u003e\u003cimg src=\"assets/logo.png\" width=\"200\"/\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n\n[![Release](https://img.shields.io/github/release/oagudo/outbox.svg?style=flat-square)](https://github.com/oagudo/outbox/releases/latest)\n[![Software License](https://img.shields.io/badge/license-MIT-brightgreen.svg?style=flat-square)](LICENSE)\n![GitHub Actions](https://github.com/oagudo/outbox/actions/workflows/ci.yml/badge.svg)\n[![codecov](https://codecov.io/gh/oagudo/outbox/graph/badge.svg?token=KH1GUAV4VR)](https://codecov.io/gh/oagudo/outbox)\n[![Go Report Card](https://goreportcard.com/badge/github.com/oagudo/outbox?style=flat-square)](https://goreportcard.com/report/github.com/oagudo/outbox)\n[![Go Reference](https://pkg.go.dev/badge/github.com/oagudo/outbox.svg)](https://pkg.go.dev/github.com/oagudo/outbox)\n\nLightweight library for the [transactional outbox pattern](https://microservices.io/patterns/data/transactional-outbox.html) in Go, not tied to any specific relational database or broker.\n\n## Key Features\n\n- **Lightweight:** Adds only one external dependency: [google/uuid](https://github.com/google/uuid)\n- **Database agnostic:** Works with PostgreSQL, MySQL, Oracle and other relational databases.\n- **Message broker agnostic:** Works with any message broker or external system.\n- **Easy integration:** Designed for easy integration into your own projects.\n- **Observability:** Exposes channels for processing errors and discarded messages that you can connect to your metrics and alerting systems.\n- **Fast publishing:** Optional immediate async message publishing after transaction commit for reduced latency, with guaranteed delivery fallback.\n- **Configurable retry and backoff policies:** Fixed, exponential or custom backoff strategies when delivery fails.\n- **Max attempts safeguard:** Automatically discards poison messages that exceed a configurable `maxAttempts` threshold.\n- **Scheduled messages:** Delay message publishing to a future time.\n\n## Usage\n\nThe library consists of two main components:\n\n1. **Writer**: Stores your business objects and corresponding messages atomically within a single transaction\n2. **Reader**: Publishes stored messages to your message broker in the background\n\n### The Writer\n\nSupports two transaction management models, depending on how much control you need.\n\n#### 1. Library managed transactions\n\nThe Writer handles the entire transaction lifecycle (begin, commit, rollback) for you. This is the recommended approach for most use cases, as it reduces boilerplate and avoids common transactional pitfalls.\n\n```go\n// Initialise Writer\ndb, _ := sql.Open(\"pgx\", \"postgres://...\")\ndbCtx := outbox.NewDBContext(db, outbox.SQLDialectPostgres)\nwriter := outbox.NewWriter(dbCtx)\n\n// Use Write function for conditional or multiple message publishing.\n// If the callback returns an error the transaction is rolled back\nerr = writer.Write(ctx,\n        func(ctx context.Context, tx outbox.TxQueryer, msgWriter outbox.MessageWriter) error {\n\n    result, err := tx.ExecContext(ctx,\n        \"UPDATE orders SET status = 'confirmed' WHERE id = $1 AND status = 'pending'\", id)\n    if err != nil {\n        return err\n    }\n    if rows, _ := result.RowsAffected(); rows == 0 {\n        return ErrOrderNotPending // no message, rollback\n    }\n\n    // Create and store outbox messages\n    payload, _ := json.Marshal(order)\n    msg := outbox.NewMessage(payload,\n        outbox.WithCreatedAt(order.CreatedAt),\n        outbox.WithMetadata(json.RawMessage(`{\"trace_id\":\"abc123\"}`)))\n\n    return msgWriter.Store(ctx, msg)\n})\n\n// Use WriteOne function for simple cases that store a single message unconditionally\nmsg := outbox.NewMessage(payload)\nerr = writer.WriteOne(ctx, msg, func(ctx context.Context, tx outbox.TxQueryer) error {\n    _, err := tx.ExecContext(ctx,\n        \"INSERT INTO entity (id, created_at) VALUES ($1, $2)\",\n        entity.ID, entity.CreatedAt)\n    return err\n})\n```\n\n#### 2. User managed transactions\n\nYou control the transaction lifecycle yourself, giving full flexibility to integrate the outbox into existing transaction management patterns.\n\n```go\n// For users who want to manage the transaction lifecycle themselves\n// and only need to persist outbox messages\nunmanagedWriter := writer.Unmanaged()\ntx, _ := db.BeginTx(ctx, nil)\ndefer tx.Rollback()\n\nerr = unmanagedWriter.Store(ctx, tx, msg)\n_, _ = tx.ExecContext(ctx, \"INSERT INTO entity (...) VALUES (...)\", ...)\n\ntx.Commit()\n```\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003e🚀 Optimistic Publishing (Optional)\u003c/strong\u003e\u003c/summary\u003e\n\nPublishes messages immediately after transaction commit for lower latency. The background reader acts as fallback if optimistic publishing fails.\n\n#### How It Works\n\n1. Transaction commits (entity + outbox message stored)\n2. Immediate publish attempt to broker (asynchronously, will not block the incoming request)\n3. On success message is removed from outbox\n4. On failure background reader handles delivery later\n\n#### Configuration\n\n```go\n// Create publisher (see Reader section below)\npublisher := \u0026messagePublisher{}\n\n// Enable optimistic publishing in writer\nwriter := outbox.NewWriter(dbCtx, outbox.WithOptimisticPublisher(publisher))\n```\n\n**Important considerations:**\n- Publishing happens asynchronously after transaction commit\n- Message consumers must be idempotent as messages could be published twice - by the optimistic publisher and by the reader (Note: consumer idempotency is a good practice regardless of optimistic publishing, though some brokers also provide deduplication features)\n- Publishing failures don't affect your transactions - they don't cause `Write()` to fail\n- Optimistic publisher is not triggered for user managed transactions - `Writer.Unmanaged()`\n\n\u003c/details\u003e\n\n### The Reader\n\nThe Reader periodically checks for unsent messages and publishes them to your message broker:\n\n```go\n// Create a message publisher implementation\ntype messagePublisher struct {\n    // Your message broker client (e.g., Kafka, RabbitMQ)\n}\nfunc (p *messagePublisher) Publish(ctx context.Context, msg *outbox.Message) error {\n    // Publish the message to your broker. See examples below for specific implementations\n    return nil\n}\n\n// Create and start the reader\nreader := outbox.NewReader(\n    dbCtx,                              // Database context\n    \u0026messagePublisher{},                // Publisher implementation\n    outbox.WithInterval(5*time.Second), // Polling interval (default: 10s)\n    outbox.WithReadBatchSize(200),      // Read batch size (default: 100)\n    outbox.WithDeleteBatchSize(50),     // Delete batch size (default: 20)\n    outbox.WithMaxAttempts(300),        // Discard after 300 attempts (default: MaxInt32)\n    outbox.WithExponentialDelay(        // Delay between attempts (default: Exponential; can also use Fixed or Custom)\n        500*time.Millisecond,           // Initial delay (default: 200ms)\n        30*time.Minute),                // Maximum delay (default: 1h)\n)\nreader.Start()\ndefer reader.Stop(context.Background()) // Stop during application shutdown\n```\n\n#### Monitoring\n\nThe reader exposes channels for errors and discarded messages:\n\n```go\n// Monitor processing errors\ngo func() {\n    for err := range reader.Errors() {\n        switch e := err.(type) {\n        case *outbox.PublishError:\n            log.Printf(\"Failed to publish message | ID: %s | Error: %v\",\n                e.Message.ID, e.Err)\n\n        case *outbox.UpdateError:\n            log.Printf(\"Failed to update message | ID: %s | Error: %v\",\n                e.Message.ID, e.Err)\n\n        case *outbox.DeleteError:\n            log.Printf(\"Batch message deletion failed | Count: %d | Error: %v\",\n                len(e.Messages), e.Err)\n            for _, msg := range e.Messages {\n                log.Printf(\"Failed to delete message | ID: %s\", msg.ID)\n            }\n\n        case *outbox.ReadError:\n            log.Printf(\"Failed to read outbox messages | Error: %v\", e.Err)\n\n        default:\n            log.Printf(\"Unexpected error occurred | Error: %v\", e)\n        }\n    }\n}()\n\n// Monitor poison messages (exceeded max attempts)\ngo func() {\n    for msg := range reader.DiscardedMessages() {\n        log.Printf(\"outbox message %s discarded after %d attempts\",\n            msg.ID, msg.TimesAttempted)\n        // Example next steps:\n        //   • forward to a dead-letter topic\n        //   • raise an alert / metric\n        //   • persist for manual inspection\n    }\n}()\n```\n\n### Database Setup\n\n#### 1. Choose Your Database Dialect\n\nThe library supports multiple relational databases. Configure the appropriate `SQLDialect` when creating the `DBContext`. Supported dialects are PostgreSQL, MySQL, MariaDB, SQLite, Oracle and SQL Server.\n\n```go\ndbCtx := outbox.NewDBContext(db, outbox.SQLDialectMySQL)\n```\n\n#### 2. Create the Outbox Table\n\nThe outbox table stores messages that need to be published to your message broker. Choose your database below:\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003e🐘 PostgreSQL\u003c/strong\u003e\u003c/summary\u003e\n\n```sql\nCREATE TABLE IF NOT EXISTS outbox (\n    id UUID PRIMARY KEY,\n    created_at TIMESTAMP WITH TIME ZONE NOT NULL,\n    scheduled_at TIMESTAMP WITH TIME ZONE NOT NULL,\n    metadata BYTEA,\n    payload BYTEA NOT NULL,\n    times_attempted INTEGER NOT NULL\n);\n\nCREATE INDEX IF NOT EXISTS idx_outbox_created_at ON outbox (created_at);\nCREATE INDEX IF NOT EXISTS idx_outbox_scheduled_at ON outbox (scheduled_at);\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003e📊 MySQL\u003c/strong\u003e\u003c/summary\u003e\n\n```sql\nCREATE TABLE IF NOT EXISTS outbox (\n    id BINARY(16) PRIMARY KEY,\n    created_at TIMESTAMP(3) NOT NULL,\n    scheduled_at TIMESTAMP(3) NOT NULL,\n    metadata BLOB,\n    payload BLOB NOT NULL,\n    times_attempted INT NOT NULL\n);\n\nCREATE INDEX idx_outbox_created_at ON outbox (created_at);\nCREATE INDEX idx_outbox_scheduled_at ON outbox (scheduled_at);\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003e🐬 MariaDB\u003c/strong\u003e\u003c/summary\u003e\n\n```sql\nCREATE TABLE IF NOT EXISTS outbox (\n    id UUID PRIMARY KEY,\n    created_at TIMESTAMP(3) NOT NULL,\n    scheduled_at TIMESTAMP(3) NOT NULL,\n    metadata BLOB,\n    payload BLOB NOT NULL,\n    times_attempted INT NOT NULL\n);\n\nCREATE INDEX idx_outbox_created_at ON outbox (created_at);\nCREATE INDEX idx_outbox_scheduled_at ON outbox (scheduled_at);\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003e🗃️ SQLite\u003c/strong\u003e\u003c/summary\u003e\n\n```sql\nCREATE TABLE IF NOT EXISTS outbox (\n    id TEXT PRIMARY KEY,\n    created_at DATETIME NOT NULL,\n    scheduled_at DATETIME NOT NULL,\n    metadata BLOB,\n    payload BLOB NOT NULL,\n    times_attempted INTEGER NOT NULL\n);\n\nCREATE INDEX IF NOT EXISTS idx_outbox_created_at ON outbox (created_at);\nCREATE INDEX IF NOT EXISTS idx_outbox_scheduled_at ON outbox (scheduled_at);\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003e🏛️ Oracle\u003c/strong\u003e\u003c/summary\u003e\n\n```sql\nCREATE TABLE outbox (\n    id RAW(16) PRIMARY KEY,\n    created_at TIMESTAMP WITH TIME ZONE NOT NULL,\n    scheduled_at TIMESTAMP WITH TIME ZONE NOT NULL,\n    metadata BLOB,\n    payload BLOB NOT NULL,\n    times_attempted NUMBER(10) NOT NULL\n);\n\nCREATE INDEX idx_outbox_created_at ON outbox (created_at);\nCREATE INDEX idx_outbox_scheduled_at ON outbox (scheduled_at);\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003e🪟 SQL Server\u003c/strong\u003e\u003c/summary\u003e\n\n```sql\nCREATE TABLE outbox (\n    id UNIQUEIDENTIFIER PRIMARY KEY,\n    created_at DATETIMEOFFSET(3) NOT NULL,\n    scheduled_at DATETIMEOFFSET(3) NOT NULL,\n    metadata VARBINARY(MAX),\n    payload VARBINARY(MAX) NOT NULL,\n    times_attempted INT NOT NULL\n);\n\nCREATE INDEX idx_outbox_created_at ON outbox (created_at);\nCREATE INDEX idx_outbox_scheduled_at ON outbox (scheduled_at);\n```\n\u003c/details\u003e\n\n## Examples\n\nComplete working examples for different databases and message brokers:\n\n- [Postgres \u0026 Kafka](./examples/postgres-kafka/service.go)\n- [Oracle \u0026 NATS](./examples/oracle-nats/service.go)\n- [MySQL \u0026 RabbitMQ](./examples/mysql-rabbitmq/service.go)\n\nTo run an example:\n\n```bash\ncd examples/postgres-kafka # or examples/oracle-nats or examples/mysql-rabbitmq\n../../scripts/up-and-wait.sh\ngo run service.go\n\n# In another terminal trigger a POST to trigger entity creation\ncurl -X POST http://localhost:8080/entity\n```\n\n## FAQ\n\n### What happens when multiple instances of my service use the library?\n\nWhen running multiple instances of your service, each with its own reader, be aware that:\n\n- Multiple readers will independently retrieve messages. This can result in messages published more than once. To handle this you can either:\n  1. Ensure your consumers are idempotent and accept duplicates\n  2. Use broker deduplication features if available (e.g. NATS JetStream's Msg-Id)\n  3. Run the reader in a single instance only (e.g. single replica deployment in k8s with reader)\n\nThe optimistic publisher feature can significantly reduce the number of duplicates. With optimistic publisher messages are delivered as soon as they are committed, so readers will usually see no messages in the outbox table.\n\nAlso note that even in single instance deployments, message duplicates can still occur (e.g. if the service crashes right after successfully publishing to the broker). However, these duplicates are less frequent than when you are running multiple reader instances.\n\n### How to instantiate a `DBContext` when using `pgxpool` ?\n\nYou can use `stdlib.OpenDBFromPool` [function](https://pkg.go.dev/github.com/jackc/pgx/v5/stdlib#OpenDBFromPool) to get a `*sql.DB` from a `*pgxpool.Pool`.\n\n```go\nimport (\n    \"github.com/jackc/pgx/v5/pgxpool\"\n    \"github.com/jackc/pgx/v5/stdlib\"\n    \"github.com/oagudo/outbox\"\n)\n\n// ...\npool, _ := pgxpool.New(context.Background(), os.Getenv(\"DATABASE_URL\"))\ndb := stdlib.OpenDBFromPool(pool)\ndbCtx := outbox.NewDBContext(db, outbox.SQLDialectPostgres)\n```\n\n## Contributing\n\nContributions welcome! Bug fix PRs are always appreciated. For new features or bigger changes, consider opening an issue first so we can discuss the approach together.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foagudo%2Foutbox","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foagudo%2Foutbox","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foagudo%2Foutbox/lists"}