{"id":22287581,"url":"https://github.com/botchris/go-pubsub","last_synced_at":"2025-06-10T22:37:35.733Z","repository":{"id":45415439,"uuid":"250112530","full_name":"botchris/go-pubsub","owner":"botchris","description":"multi-topic pub/sub package for Go with pluggable providers and statically typed.","archived":false,"fork":false,"pushed_at":"2025-03-27T09:42:01.000Z","size":382,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-27T10:35:07.929Z","etag":null,"topics":["aws","go","golang","kubemq","pubsub","sns","sqs"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/botchris.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-03-25T23:13:20.000Z","updated_at":"2025-03-27T09:42:03.000Z","dependencies_parsed_at":"2024-06-20T21:52:05.200Z","dependency_job_id":"5f869b92-f181-41c3-b6b3-14386bc7704b","html_url":"https://github.com/botchris/go-pubsub","commit_stats":null,"previous_names":["christophercastro/go-pubsub"],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/botchris%2Fgo-pubsub","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/botchris%2Fgo-pubsub/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/botchris%2Fgo-pubsub/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/botchris%2Fgo-pubsub/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/botchris","download_url":"https://codeload.github.com/botchris/go-pubsub/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248142367,"owners_count":21054630,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","go","golang","kubemq","pubsub","sns","sqs"],"created_at":"2024-12-03T17:00:40.392Z","updated_at":"2025-04-10T01:43:36.108Z","avatar_url":"https://github.com/botchris.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Go PubSub\n\nThe `go-pubsub` package is a simple package for implementing publish-subscribe\nasynchronous tasks in Golang. It allows writing publishers and subscribers fully\nstatically typed, and swap out Broker implementations (e.g. Memory, AWS SQS, \netc.) as required.\n\nThis package imposes no restrictions on how messages should be represented, the \nidea is to keep subscribers agnostic to transport concerns and be fully typed\nusing golang definitions. How messages are encoded/decoded in order to be\ntransported over the network is up to the provider implementation in \ncombination with Codec middlewares.\n\n![broker overview][broker-overview]\n\n## What is a PubSub system\n\nA PubSub system is a messaging system that has, as its name implies, two\ncomponents: Publisher of messages and subscriber to messages. In contrast to\nsynchronous communication, the publisher doesn't have to wait for a message to\nbe received, as well as the receiver doesn't have to be online to retrieve\nmessages sent earlier. As such, a PubSub system acts like a buffer for\nasynchronous messaging.\n\n## Features\n\n- Multi-topic support, the same subscriber may listen for messages on\n  multiple topics at the same time.\n- [Hybrid message filtering][hybrid-filtering], subscriber are free to\n  decide whether they want to receive messages for a concrete type or not\n  (content-based), or just receive everything that is pushed to a given\n  topic/s (topic-based).\n- Pluggable providers. Just implement the `Broker` interface. See below for \n  a list of built-in providers\n\n## Providers\n\nProviders are concrete implementations of the Broker interface. Examples of\nproviders could be messaging services such as Google's PubSub, Amazon's SNS\nor Nats.io. The Broker interface acts as a generalization for such services.\n\nThe `go-pubsub` package comes with a set of built-in providers:\n\n- `memory`: A simple, synchronous Broker for in-process message communication, \n  acting as a straightforward \"Message Bus\" or \"Event Dispatcher\" replacement. \n  It ensures message processing in publication order and allows error feedback \n  to the consumer if a subscriber fails to process a message. Notably, it executes \n  all subscribers, requiring idempotent handling for retry functionality, as \n  subscribers may be invoked multiple times until successful processing.\n- `nop`: a simple NO-OP broker implementation that can be used for testing.\n- `redis`: a broker that uses redis streams as PubSub mechanism.\n- `snssqs`: a Broker that uses AWS SNS and AWS SQS.\n- `kmq`: a KubeMQ implementation of the Broker interface.\n\n### Creating your own provider\n\nThis packages moves around the `Broker` interface definition, which is the \ncentral piece for dealing with PubSub systems. The `Broker` interface is a \ncomposition of three independent interfaces which can be used in order to keep\nyou application concerns clean and separated:\n\n```go\ntype Broker interface {\n\tPublisher\n\tSubscriber\n\tShutdowner\n}\n\ntype Publisher interface {\n\tPublish(ctx context.Context, topic Topic, m interface{}) error\n}\n\ntype Subscriber interface {\n\tSubscribe(ctx context.Context, topic Topic, handler Handler, option ...SubscribeOption) (Subscription, error)\n}\n\ntype Shutdowner interface {\n\tShutdown(ctx context.Context) error\n}\n```\n\nCreating your own provider is as simple as implementing the Broker interface \ndescribed above.\n\n## Middleware\n\nA middleware acts as a wrapper for a Broker implementation. It can be used \nto intercept each message being published or being delivered to subscribers. \nUsers can use middleware to do logging, metrics collection, and many other \nfunctionalities that can be shared across PubSub Providers.\n\nTo use middleware capabilities you must simply wrap your broker using any of \nthe provided middlewares, example:\n\n```go\nbroker := printer.NewPrinterMiddleware(myProvider, os.Stdout)\n```\n\nIncluded middlewares are:\n\n- `codec`: a middleware that encodes and decodes messages using the given codec.\n- `lifecycle`: a middleware that allows to bind to broker lifecycle events.\n- `printer`: a simple middleware that prints each message to the given output.\n- `recover`: a middleware that recovers from panics.\n- `retry`: a middleware that retries publishing messages if the broker fails.\n\nMiddlewares can be combined by wrapping each other, for example:\n\n```go\nbroker := memory.NewMemoryBroker() \nbroker = printer.NewPrinterMiddleware(myProvider, os.Stdout)\nbroker = codec.NewCodecMiddleware(broker, codec.JSON)\nbroker = recovery.NewRecoveryMiddleware(broker, func(ctx context.Context, p interface{}) error {\n    println(\"panic:\", p)\n\t\n\treturn nil \n})\n```\n\nPlease note that middlewares are applied in reverse order they are wrapped, so in\nthe example above, the `recovery` middleware will be applied first, then the\n`codec` middleware, and so on:\n\nRecovery -\u003e Codec -\u003e Printer -\u003e Memory-Broker\n\n## TODO\n\n- [ ] Kafka provider\n- [ ] Google's Pub/Sub provider\n- [ ] Nats.io provider\n- [x] Redis provider\n- [x] Add protobuf support as a middleware codec\n- [x] Recovery middleware for dealing with panics\n- [x] Retry middleware for dealing with unreliable providers/handlers\n\n## Example\n\n```go\npackage main\n\nimport (\n  \"context\"\n  \"fmt\"\n  \"time\"\n\n  \"github.com/botchris/go-pubsub\"\n  \"github.com/botchris/go-pubsub/provider/memory\"\n)\n\nvar myTopic pubsub.Topic = \"my-topic\"\n\ntype myMessage struct {\n  Body string\n}\n\nfunc main() {\n  ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)\n  defer cancel()\n\n  broker := memory.NewBroker()\n\n  // Define handlers \n  h1 := pubsub.NewHandler(func(ctx context.Context, t pubsub.Topic, m myMessage) error {\n    fmt.Printf(\"%s -\u003e %+v -\u003e [s1]\\n\", t, m)\n\n    return nil\n  })\n\n  h2 := pubsub.NewHandler(func(ctx context.Context, t pubsub.Topic, m *myMessage) error {\n    fmt.Printf(\"%s -\u003e %+v -\u003e [s2]\\n\", t, m)\n\n    return nil\n  })\n\n  h3 := pubsub.NewHandler(func(ctx context.Context, t pubsub.Topic, m string) error {\n    fmt.Printf(\"%s -\u003e %+v -\u003e [s3]\\n\", t, m)\n\n    return nil\n  })\n\n  // Subscribe to topic\n  var s1 pubsub.Subscription\n  {\n    s1l, err := broker.Subscribe(ctx, myTopic, h1)\n    if err != nil {\n      panic(err)\n    }\n\n    s1 = s1l\n\n    if _, sErr := broker.Subscribe(ctx, myTopic, h2); sErr != nil {\n      panic(sErr)\n    }\n\n    if _, sErr := broker.Subscribe(ctx, myTopic, h3); sErr != nil {\n      panic(sErr)\n    }\n  }\n\n  // Publish test messages\n  {\n    if err := broker.Publish(ctx, myTopic, myMessage{Body: \"value(hello world)\"}); err != nil {\n      panic(err)\n    }\n\n    if err := broker.Publish(ctx, myTopic, \u0026myMessage{Body: \"pointer(hello world)\"}); err != nil {\n      panic(err)\n    }\n\n    if err := broker.Publish(ctx, myTopic, \"string(hello world)\"); err != nil {\n      panic(err)\n    }\n\n    // Unsubscribe S1\n    if err := s1.Unsubscribe(); err != nil {\n      panic(err)\n    }\n\n    // This will noop\n    if err := broker.Publish(ctx, myTopic, myMessage{Body: \"value(hello world)\"}); err != nil {\n      panic(err)\n    }\n  }\n\n  // Output:\n  //  {Body:value(hello world)} -\u003e my-topic -\u003e [s1]\n  //  \u0026{Body:pointer(hello world)} -\u003e my-topic -\u003e [s2]\n  //  string(hello world) -\u003e my-topic -\u003e [s3]\n  //  \u003cnothing\u003e\n}\n```\n\n[broker-overview]: doc/broker.overview.png\n[hybrid-filtering]: https://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_pattern#Message_filtering\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbotchris%2Fgo-pubsub","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbotchris%2Fgo-pubsub","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbotchris%2Fgo-pubsub/lists"}