{"id":50794307,"url":"https://github.com/squall-chua/go-event-pubsub","last_synced_at":"2026-06-12T13:31:52.740Z","repository":{"id":342485044,"uuid":"1174137324","full_name":"squall-chua/go-event-pubsub","owner":"squall-chua","description":"A lightweight, production-ready Go library for building event-driven architectures. It provides a high-level abstraction over popular message brokers like Kafka and RabbitMQ, featuring background delivery, automatic retries with backoff, and robust Dead Letter Queue (DLQ) support.","archived":false,"fork":false,"pushed_at":"2026-03-27T03:02:24.000Z","size":104,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-03-27T15:47:21.953Z","etag":null,"topics":["event-driven","golang","kafka","pubsub","rabbitmq"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/squall-chua.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-06T05:28:56.000Z","updated_at":"2026-03-27T07:24:18.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/squall-chua/go-event-pubsub","commit_stats":null,"previous_names":["squall-chua/go-event-pubsub"],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/squall-chua/go-event-pubsub","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/squall-chua%2Fgo-event-pubsub","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/squall-chua%2Fgo-event-pubsub/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/squall-chua%2Fgo-event-pubsub/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/squall-chua%2Fgo-event-pubsub/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/squall-chua","download_url":"https://codeload.github.com/squall-chua/go-event-pubsub/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/squall-chua%2Fgo-event-pubsub/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34247461,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-12T02:00:06.859Z","response_time":109,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["event-driven","golang","kafka","pubsub","rabbitmq"],"created_at":"2026-06-12T13:31:52.509Z","updated_at":"2026-06-12T13:31:52.730Z","avatar_url":"https://github.com/squall-chua.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Go Event PubSub\n\nA lightweight, production-ready Go library for building event-driven architectures. It provides a high-level abstraction over popular message brokers like Kafka and RabbitMQ, featuring background delivery, automatic retries with backoff, and robust Dead Letter Queue (DLQ) support.\n\n## Key Features\n\n- **Standardized Payloads**: Uses a uniform `Event` struct for all communications, ensuring consistency across services.\n- **Non-Blocking Publishing**: Integrated worker pool and task channel for asynchronous event delivery.\n- **Robust Retries**: Built-in exponential backoff (v5) for handling transient broker failures.\n- **Advanced Routing**: Decouples application logic from physical topics using a schema-based Router.\n- **Observability in DLQ**: Automatically routes failed messages to DLQs with full diagnostic metadata (`fail_reason`, `original_destination`).\n- **Multiple Backends**: Pluggable support for Kafka, RabbitMQ, and an In-Memory broker for testing.\n- **Thread Safe**: Automatic deep cloning of event state ensuring zero data races between callers and background workers.\n\n---\n\n## Installation\n\n```bash\ngo get github.com/squall-chua/go-event-pubsub\n```\n\n---\n\n## Core Concepts\n\n### 1. The Event Struct\n\nEvery message in the system is an `Event`. It includes standard fields for correlation and traceability.\n\n```go\ntype Event struct {\n    EventId    string         `json:\"eventId\"`    // Unique tracking ID\n    EventType  string         `json:\"eventType\"`  // e.g., \"order.created\"\n    EventTime  time.Time      `json:\"eventTime\"`  // UTC occurrence time\n    User       string         `json:\"user\"`       // Triggering user ID\n    Source     string         `json:\"source\"`     // Originating service\n    Schema     string         `json:\"schema\"`     // Routing domain\n    ResourceID string         `json:\"resourceId\"` // Primary entity ID\n    Data       any            `json:\"data\"`       // Actual payload\n    Metadata   map[string]any `json:\"metadata\"`   // Key-value headers\n}\n```\n\n### 2. Routing Configuration\n\nThe `Router` maps logical events (Schema + EventType) to physical destinations and defines their delivery behavior. Destinations can be defined at the schema level to provide a default for all events.\n\n```go\nregistry := event.SchemaRegistry{\n    \"order_domain\": {\n        QueueType:    \"kafka\",             // Schema-level default broker\n        Destinations: []string{\"orders\"},  // Schema-level default destinations\n        DLQPostfix:   \".failed\",          // Schema-level default DLQ postfix\n        Events: map[string]event.TopicConfig{\n            \"order.created\": {\n                // Inherits QueueType, Destinations, and DLQPostfix\n            },\n            \"order.internal.*\": { // Wildcard for all internal sub-events\n                QueueType:    \"memory\", // Explicit override\n                Destinations: []string{\"internal-logs\"}, // Explicit override\n            },\n        },\n    },\n}\nrouter := event.NewStaticRouter(registry)\n```\n\n#### Wildcard Event Types\n\nThe `Router` and `Subscriber` support simple prefix-based wildcards using the `*` suffix:\n\n- **Prefix Match**: `domain.*` matches `domain.created`, `domain.deleted`, etc.\n- **Global Match**: `*` matches every event type within the schema.\n\nThis is particularly useful for building domain-wide event listeners or routing related events to the same topic without explicit registration for every single subtype.\n\n#### Loading from YAML\n\nThe registry is compatible with standard YAML/JSON tags.\n\n**config.yaml**:\n\n```yaml\norder_domain:\n  queue_type: \"kafka\"\n  destinations: [\"orders\"] # Default for all events in this schema\n  dlq_postfix: \".failed\"\n  events:\n    order.created:\n      # Automatically inherits from order_domain\n    order.internal:\n      queue_type: \"memory\"\n      destinations: [\"internal-logs\"] # Specific override\n```\n\n**Go**:\n\n```go\nimport \"gopkg.in/yaml.v3\"\n\nvar registry event.SchemaRegistry\n_ = yaml.Unmarshal(yamlData, \u0026registry)\nrouter := event.NewStaticRouter(registry)\n```\n\n**config.json**:\n\n```json\n{\n  \"order_domain\": {\n    \"queueType\": \"kafka\",\n    \"destinations\": [\"orders\"],\n    \"dlqPostfix\": \".failed\",\n    \"events\": {\n      \"order.created\": {},\n      \"order.internal\": {\n        \"queueType\": \"memory\",\n        \"destinations\": [\"internal-logs\"]\n      }\n    }\n  }\n}\n```\n\n---\n\n## Usage Examples\n\n### Initializing Brokers\n\n#### Kafka\n\n```go\nimport \"github.com/squall-chua/go-event-pubsub/pkg/broker/kafka\"\n\nkBroker, err := kafka.NewBroker(kafka.Config{\n    Brokers: []string{\"localhost:9092\"},\n    Writer: kafka.WriterConfig{ BatchSize: 100 },\n})\nif err != nil {\n    log.Fatal(err) // e.g. no brokers configured\n}\n```\n\n#### RabbitMQ\n\n```go\nimport \"github.com/squall-chua/go-event-pubsub/pkg/broker/rabbitmq\"\n\nrBroker, _ := rabbitmq.NewBroker(\"amqp://guest:guest@localhost:5672/\")\n```\n\n---\n\n### Publishing Events\n\nPublishing is non-blocking. **Validation happens immediately during the `Publish` call.** If the event type is not registered in the router, the call will return an error synchronously.\n\n```go\n// 1. Configure the publisher\npubCfg := \u0026event.PublisherConfig{\n    Workers:    10,  // Concurrent delivery units\n    BufferSize: 500, // Internal queue size\n    RetryConfig: \u0026event.RetryConfig{\n        InitialInterval: 500 * time.Millisecond,\n        MaxElapsedTime:  30 * time.Second,\n    },\n}\n\n// 2. Create the publisher\nbrokers := map[string]event.Broker{\n    \"kafka\": kBroker,\n}\npub := event.NewPublisher(router, brokers, pubCfg)\ndefer pub.Close() // Wait for pending tasks before shutdown\n\n// 3. Publish\nevt := \u0026event.Event{\n    EventId:   uuid.NewString(),\n    EventType: \"order.created\",\n    User:      \"user_123\",\n    Schema:    \"order_domain\",\n    Data:      map[string]any{\"order_id\": \"123\"},\n}\n\n// Publish returns an error if routing fails (e.g. unregistered EventType)\nif err := pub.Publish(ctx, evt); err != nil {\n    log.Printf(\"Rejected: %v\", err)\n}\n```\n\n---\n\n### Subscribing to Events\n\nSubscribers handle all event types within a specific schema context. `Start()` is **non-blocking** — it validates routing synchronously, launches consumer goroutines in the background, and returns immediately.\n\n```go\nsub := event.NewSubscriber(router, brokers, nil)\n\n// Register a handler with a wildcard\nsub.Subscribe(\"order_domain\", \"order.*\", func(ctx context.Context, evt *event.Event) error {\n    log.Printf(\"Received %s event for user %s\", evt.EventType, evt.User)\n    return nil\n})\n\n// Start is non-blocking: validates config synchronously, runs consumers in the background.\nerrCh, err := sub.Start(ctx)\nif err != nil {\n    // Config error: bad routing or missing broker — nothing has been started.\n    log.Fatal(err)\n}\n\n// Optionally watch for fatal runtime consumer errors\ngo func() {\n    for err := range errCh {\n        log.Printf(\"subscriber runtime error: %v\", err)\n    }\n}()\n```\n\n#### Graceful Shutdown\n\nTo wait for all consumer goroutines to finish before exiting, drain the error channel after cancelling the context — it is closed once all goroutines have exited.\n\n```go\nctx, cancel := context.WithCancel(context.Background())\nerrCh, err := sub.Start(ctx)\nif err != nil {\n    log.Fatal(err)\n}\n\n// ... wait for SIGTERM ...\n\ncancel() // propagate shutdown to all consumer goroutines\n\nfor err := range errCh { // blocks until all goroutines have exited\n    log.Printf(\"subscriber error during shutdown: %v\", err)\n}\nlog.Println(\"subscriber fully stopped\")\n```\n\n##### Multiple Subscribers\n\nFor complex applications with many subscribers, we recommend using `golang.org/x/sync/errgroup`. It provides a unified way to wait for all background tasks and handle fatal errors.\n\n```go\nimport \"golang.org/x/sync/errgroup\"\n\n// Create a group and derived context\ng, ctx := errgroup.WithContext(mainCtx)\n\nsubs := []event.Subscriber{sub1, sub2}\n\nfor _, s := range subs {\n    sub := s // capture loop var\n    g.Go(func() error {\n        errCh, err := sub.Start(ctx)\n        if err != nil {\n            return err\n        }\n\n        // Draining errCh ensures we wait for all internal consumers to finish\n        for err := range errCh {\n            log.Printf(\"Consumer runtime error: %v\", err)\n            // Note: In an errgroup, returning an error here would cancel all other subscribers.\n            // Only return the error if you want a complete system stop.\n        }\n        return nil\n    })\n}\n\n// Blocks until all subscribers have stopped or one returned a fatal error\nif err := g.Wait(); err != nil {\n    log.Printf(\"System stopped with error: %v\", err)\n}\n```\n\n---\n\n### Testing with Memory Broker\n\nThe `memory` package provides a blazing-fast, local implementation perfect for unit tests.\n\n```go\nimport \"github.com/squall-chua/go-event-pubsub/pkg/broker/memory\"\n\nfunc TestMyLogic(t *testing.T) {\n    memBroker := memory.NewBroker()\n    // ... use exactly like the production brokers\n}\n```\n\n---\n\n## Dead Letter Queue (DLQ)\n\nWhen an event fails (either background delivery retries are exhausted, or a subscriber handler returns an error), the event is wrapped and sent to the configured DLQ topic **only if `DLQPostfix` is specified in the routing configuration**.\n\nIf `DLQPostfix` is omitted:\n\n- **Publishers** will log a warning and drop the event after all retries fail.\n- **Subscribers** will return an error from the internal dispatcher and the event will be dropped (or retried by the broker depending on broker-specific ack settings).\n\nWhen enabled, the DLQ message will have:\n\n- **EventType**: Original event type + postfix (e.g., `order.created.failed`).\n- **Data**: The full original event.\n- **Metadata**:\n  - `fail_reason`: The error string describing the failure.\n  - `original_destination`: Where the event was supposed to go.\n\n### Recovery from DLQ\n\nThe library provides a `DLQProcessor` to help automate the recovery of failed events. It unwraps the original event, strips failure metadata, and republishes it back into the main pipeline.\n\n```go\nprocessor := event.NewDLQProcessor(broker, pub)\n\n// Process all events from a specific DLQ topic\nerr := processor.Process(ctx, \"orders-topic.failed\", func(evt *event.Event) bool {\n    // Optional filter: only reprocess transient errors\n    reason, _ := evt.Metadata[\"fail_reason\"].(string)\n    return strings.Contains(reason, \"connection timeout\")\n})\n```\n\n### Unreachable DLQ Fallback\n\nIn extreme cases (e.g., the broker cluster is completely down), even publishing to the DLQ might fail. To prevent data loss, both `PublisherConfig` and `SubscriberConfig` provide a `DLQFallbackHandler` hook.\n\n```go\n// For Publisher\nconfig := \u0026event.PublisherConfig{\n    DLQFallbackHandler: func(ctx context.Context, evt *event.Event, dlqErr error) {\n        log.Printf(\"EMERGENCY: Broker down. Event: %s\", evt.EventId)\n    },\n}\n\n// For Subscriber\nconfig := \u0026event.SubscriberConfig{\n    DLQFallbackHandler: func(ctx context.Context, evt *event.Event, dlqErr error) {\n        // Handle failed subscriber DLQ moves here\n    },\n}\n```\n\nIf no handler is provided, the library defaults to logging a highly visible error to standard out\n---\n\n## Examples\n\nThe library includes several runnable examples under the `examples/` directory:\n\n- [Basic In-Memory](examples/basic_memory/main.go): Simplest loop.\n- [Kafka Integration](examples/kafka_producer_consumer/main.go): Performance tuning and groups.\n- [RabbitMQ Integration](examples/rabbitmq_producer_consumer/main.go): Reliable AMQP usage.\n- [DLQ Diagnostics](examples/dlq_diagnostics/main.go): Failure wrapping and metadata inspection.\n- [DLQ Recovery](examples/dlq_recovery/main.go): Reprocessing events using the `DLQProcessor`.\n- [Publisher Fallback](examples/dlq_fallback/main.go): Emergency handling for unreachable brokers.\n- [Subscriber Fallback](examples/subscriber_dlq_fallback/main.go): Emergency handling for failed DLQ routing during consumption.\n- [Custom Broker](examples/custom_broker/main.go): How to extend the library with your own messaging backend.\n\nTo run an example:\n\n```bash\ngo run examples/basic_memory/main.go\n```\n\n---\n\n## Agent Skill\n\nThis repository ships an **agent skill** that provides a structured reference for AI coding assistants (e.g. Antigravity, Claude Code, GitHub Copilot Workspace) to reason about and work with this library without re-reading raw source files.\n\n**Location**: [`skills/go-event-pubsub/SKILL.md`](skills/go-event-pubsub/SKILL.md)\n\n### Using the skill in your project\n\nTo make the skill immediately available to your AI agent, copy it into your own project's local skills directory:\n\n```bash\n# Create the skills directory in your project (if it doesn't exist)\nmkdir -p \u003cyour-project\u003e/.agents/skills/go-event-pubsub\n\n# Copy the skill file\ncp path/to/go-event-pubsub/skills/go-event-pubsub/SKILL.md \\\n   \u003cyour-project\u003e/.agents/skills/go-event-pubsub/SKILL.md\n```\n\nMost agent runtimes (Antigravity, Claude Code, GitHub Copilot Workspace) auto-discover skill files placed under a `.agents/skills/` directory at the project root. Once copied, your agent can answer questions about this library without requiring any additional configuration.\n\n### What the skill covers\n\n| Section | Contents |\n| --- | --- |\n| Documentation access | How to find the README, `pkg.go.dev` URL, `go doc` commands, examples index |\n| Core concepts | `Event` struct, `SchemaRegistry`, wildcard routing rules |\n| Usage patterns | Publisher setup, publishing, subscriber, graceful shutdown, multi-subscriber with `errgroup` |\n| Testing | In-memory broker patterns for unit tests |\n| DLQ | Trigger conditions, metadata fields, `DLQProcessor` recovery, fallback handler |\n| Custom broker | How to implement the `Broker` interface |\n| Quick reference | Key behaviours table, common error causes and fixes |\n\n### How agents use it\n\nAI agents that support the skill format will automatically discover and load `skills/go-event-pubsub/SKILL.md` when working inside this repository. The skill is triggered by queries such as:\n\n- *\"How do I publish an event?\"*\n- *\"Set up a subscriber with graceful shutdown.\"*\n- *\"How do I test this library without Kafka?\"*\n- *\"What happens when my subscriber handler returns an error?\"*\n\n### Keeping the skill up to date\n\nUpdate `skills/go-event-pubsub/SKILL.md` in these two situations:\n\n1. **Adding features or changing API behaviour** — update the skill alongside your code changes so agents always reflect the current API.\n2. **Upgrading the library version in a consumer project** — fetch the updated `SKILL.md` from the new release tag and replace the copy in your project's skills directory. Running against a stale skill from an older version can cause agents to suggest outdated patterns or missing APIs.\n\n```bash\n# Example: refresh the skill after upgrading to vX.Y.Z\ncurl -sSL https://raw.githubusercontent.com/squall-chua/go-event-pubsub/vX.Y.Z/skills/go-event-pubsub/SKILL.md \\\n  -o .agents/skills/go-event-pubsub/SKILL.md\n```\n\n---\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsquall-chua%2Fgo-event-pubsub","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsquall-chua%2Fgo-event-pubsub","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsquall-chua%2Fgo-event-pubsub/lists"}