{"id":20900555,"url":"https://github.com/dataphos/lib-brokers","last_synced_at":"2026-05-22T20:34:31.186Z","repository":{"id":257809068,"uuid":"867004801","full_name":"dataphos/lib-brokers","owner":"dataphos","description":"lib-brokers is a Go library which contains the interfaces used to interact with messaging systems without relying on a specific technology or client library. This library attempts to solve the issue of properly abstracting away the interaction between applications and messaging systems.","archived":false,"fork":false,"pushed_at":"2026-04-24T08:08:53.000Z","size":205,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2026-04-24T09:44:23.047Z","etag":null,"topics":["cloud-native","data-stream","data-streaming","go","jetstream","kafka","library","messaging","pubsub","pulsar","servicebus"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dataphos.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-10-03T09:26:22.000Z","updated_at":"2026-04-24T08:08:46.000Z","dependencies_parsed_at":null,"dependency_job_id":"0a9707fb-8a47-49cd-b56f-4c361387eaba","html_url":"https://github.com/dataphos/lib-brokers","commit_stats":null,"previous_names":["dataphos/lib-brokers"],"tags_count":8,"template":false,"template_full_name":null,"purl":"pkg:github/dataphos/lib-brokers","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dataphos%2Flib-brokers","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dataphos%2Flib-brokers/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dataphos%2Flib-brokers/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dataphos%2Flib-brokers/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dataphos","download_url":"https://codeload.github.com/dataphos/lib-brokers/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dataphos%2Flib-brokers/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33366523,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-21T12:23:38.849Z","status":"online","status_checked_at":"2026-05-22T02:00:06.671Z","response_time":265,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cloud-native","data-stream","data-streaming","go","jetstream","kafka","library","messaging","pubsub","pulsar","servicebus"],"created_at":"2024-11-18T11:20:42.702Z","updated_at":"2026-05-22T20:34:26.176Z","avatar_url":"https://github.com/dataphos.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"lib-brokers - a universal consumer/producer library\n===\n\nlib-brokers is a Go library which contains the interfaces used to interact with messaging systems without relying on a\nspecific technology or client library.\nThis library attempts to solve the issue of properly abstracting away the interaction between applications and\nmessaging systems.\n\nThis is because these interfaces are unexpectedly difficult to get right, leading to issues like:\n\n- Complicated interfaces which force implementations to duplicate code which has nothing to do with the actual operation\n  the interfaces are supposed to abstract\n- Business logic creeping into interface definition (things like message publishers sharing the message type with\n  other, unrelated business logic)\n- Concrete implementation creeping into interface definition, leading to interfaces which are too limiting and can’t be\n  used in all cases, as the author of the interface had one specific implementation in mind, so the interface still\n  contains implementation details of that initial implementation\n\nFurthermore, there is the issue of the added cost since the time spent on implementing these interfaces is multiplied by\nthe number of teams which require an implementation. Therefore, not having a unified approach to interacting with\ndifferent messaging systems leads to the following issues as well:\n\n- Time inefficiency\n- Lack of testing\n- Lack of proper documentation\n- Poor knowledge sharing\n- Possibility that the author of the implementation was not aware of some caveat of the technology in question\n- Learning curve when switching/joining a product team\n- Greater possibility of improper use of the interface (as a result of all the things mentioned above)\n\nThis is the motivation for designing a universal consumer/producer API: a Go library which would allow all existing and all\nfuture products a unified view of a messaging system, pulling together resources and greatly improving knowledge\nretention.\n\n## Installation\n\n`go get github.com/dataphos/lib-brokers`\n\nNote that due to the external dependencies, Go version 1.18 or higher is required.\n\n\n## Getting Started\n\nApplications can either produce or consume messages from a messaging system, so this library is split into\ntwo parts: the producer (publisher) API and the consumer API.\n\n### Publishing\n\nProducing (publishing) messages to a messaging system is rather straightforward, with the entire\nAPI shown in the following snippet:\n\n  ```go\ntype Publisher interface {\n        // Topic constructs a new Topic.\n        Topic(string) (Topic, error)\n}\n\ntype Topic interface {\n        // Publish publishes a Message.\n        //\n        // Blocks until the Message is received by the broker, the context is canceled or an error occurs.\n        Publish(context.Context, OutboundMessage) error\n        // BatchPublish publishes a batch.\n        // \n        // Blocks until all the messages in the batch are received by the broker, the context is canceled or an error occurs.\n        BatchPublish(context.Context, ...OutboundMessage) error\n}\n\ntype OutboundMessage struct {\n        // Key identifies the Data part of the message (and not the message itself).\n        // It doesn't have to be unique across the topic and is mostly used for ordering guarantees.\n        Key string\n        \n        // Data the payload of the message.\n        Data []byte\n        \n        // Attributes holds the properties commonly used by brokers to pass metadata.\n        Attributes map[string]interface{}\n}\n  ```\n\nThe `Publisher` is basically a factory of `Topic` instances. This is because most client libraries implement some sort\nof internal batching mechanism. For example, Pub/Sub has a configurable batch size for publishing messages, so that not\nevery publish triggers an individual network request. This batch is kept at a topic level, so constructing new topic\ninstances for each message has substantial overhead.\n\nFor Kafka and Service Bus, BatchPublish may be more efficient than calling Publish for multiple messages in\nseparate goroutines.\n\nAn example of using this API can be found in the [examples](/examples) folder.\n\n### Consuming\n\nConsuming from messaging system, on the other hand, is a lot more complex; some messaging systems\nhave the concept of a negative acknowledge (Pub/Sub), while others don't (Kafka). The presence\nof this concept also implies that that messaging system treats messages as individual entities, \nmeaning acknowledgment operations have no side effect on any other message. Log-based messaging systems\nlike Kafka don't have this concept, so committing offsets in Kafka has side effects on all messages\nin that partition prior to the \"committed\" one (they get implicitly committed).\n\nBecause of this, the consumer part of the API is split into two disjunctive groups:\nthe first concerns Pub/Sub-like messaging systems which have both positive and negative acknowledgment\noperations (which means they trivially support concurrent processing as order is not important) and\nKafka-like messaging systems, which don't.  \nBecause of this, these two groups don't share the API and use different interfaces and a different\nmessage type.\n\n#### Individual Message Processing\n\nAt the core of the first group (the one where messages are individual entities) is the `Message` struct:\n\n```go\ntype Message struct {\n    // ID identifies the message (unique across the source topic).\n    ID string\n    \n    // Key identifies the payload part of the message.\n    // Unlike ID, it doesn't have to be unique across the source topic and is mostly used\n    // for ordering guarantees.\n    Key string\n    \n    // Data holds the payload of the message.\n    Data []byte\n    \n    // Attributes holds the properties commonly used by brokers to pass metadata.\n    Attributes map[string]interface{}\n    \n    // PublishTime the time this message was published.\n    PublishTime time.Time\n    \n    // IngestionTime the time this message was received.\n    IngestionTime time.Time\n    \n    // AckFunc must be called after successful processing (and not before).\n    // This signals to the broker that this Message is safe to delete.\n    AckFunc func()\n    \n    // NackFunc must be called if the processing has failed.\n    // This signals to the broker that this Message is not safe to delete.\n    NackFunc func()\n}\n\n```\n\nThere are six different consumer interfaces which, in some way, return the `Message` struct.  \nHowever, the number of different interfaces is only there to keep the complexity of the broker-specific\nimplementations at a minimum; each individual messaging system implementation only needs to implement a single, \"native\"\ninterface (the one which is most alike the underlying client the implementation uses). \nThe other interface implementations can be achieved through carefully written adapters found in the `brokerutil` package.\n\nFor example, the \"main\" interface for consuming individual messages is the `Receiver`, whose definition is shown below:\n\n```go\ntype Receiver interface {\n    // Receive runs the pulling process, blocking for the entire duration of the pull.\n    //\n    // Receive executes the provided callback for each pulled Message, meaning it is the responsibility\n    // of the Receiver to provide the callbacks with multithreading.\n    //\n    // Receive blocks until either an error is returned, or when the provided ctx is canceled, in which case, nil is returned.\n    Receive(context.Context, func(context.Context, Message)) error\n}\n```\n\nYou're used to consuming from Pub/Sub, so you use the `Receiver` interface, but now you want to also support ServiceBus,\nwhich only implements `BatchedMessageIterator`.\nYou can achieve this in just a few lines of code, by first initializing the \"native\" ServiceBus consumer\nobject, and then pass it on to the appropriate `brokerutil` adapter, like so:\n\n```go\niterator, err := servicebus.NewBatchIteratorFromEnv()\n  if err != nil {\n    log.Fatal(err)\n  }\n  defer iterator.Close()\n\nreceiver := brokerutil.BatchedMessageIteratorIntoReceiver(\n        iterator,\n        brokerutil.IntoReceiverSettings{\n        NumGoroutines: 10,\n        },\n)\n\nif err := receiver.Receive(ctx, func(ctx context.Context, message broker.Message) {\n        // do something here\n        \n        message.Ack()\n}); err != nil {\n        log.Println(err)\n}\n```\n\n### Ordered Log Processing\n\nAt the core of the Kafka-like group are the `Record` and `Batch` message types:\n\n```go\ntype Record struct {\n  // Offset is the index of the record stored in a specific Partition.\n  Offset int64\n\n  // Partition is the index of the partition this record belongs to.\n  // Partitions are a common way brokers store topics.\n  Partition int64\n\n  // Key identifies the payload part of the record (and not the record itself).\n  // It doesn't have to be unique across the source topic and is mostly used\n  // for ordering guarantees.\n  Key string\n\n  // Value holds the payload of the record.\n  Value []byte\n\n  // Attributes holds the properties commonly used by brokers to pass metadata.\n  Attributes map[string]interface{}\n\n  // PublishTime the time this message was published.\n  PublishTime time.Time\n\n  // IngestionTime the time this record was received.\n  IngestionTime time.Time\n\n  // CommitFunc must be called after successful processing (and not before).\n  // This signals to the broker that this Record is safe to delete.\n  CommitFunc func()\n}\n\ntype Batch struct {\n  // Records the records of the Batch. BatchRecord is identical to Record, but without the ability \n  // to commit itself (without CommitFunc).\n        Records []BatchRecord\n    //CommitFunc must be called after successful processing of the entire Batch, signaling to the \n    //broker that this batch is safe to delete. \n    CommitFunc func()\n}\n```\n\nThere are only two different consumer interfaces which use this structures. These interfaces\nabstract what is effectively an iterator; `RecordIterator` returns a single record at a time, while `BatchedIterator` returns multiple.\nOnce again, there are two interfaces to simplify things for concrete implementations, and also because some\nmessaging systems can be used more efficiently if they return batches at a time.\n\n\n\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdataphos%2Flib-brokers","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdataphos%2Flib-brokers","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdataphos%2Flib-brokers/lists"}