{"id":20900562,"url":"https://github.com/dataphos/lib-streamproc","last_synced_at":"2026-02-25T22:32:26.430Z","repository":{"id":257809067,"uuid":"867089693","full_name":"dataphos/lib-streamproc","owner":"dataphos","description":"A Go library that exposes executors, interfaces, data structures, and utility functions which combined a universal stream processor, invariant to any specific messaging system.","archived":false,"fork":false,"pushed_at":"2024-10-11T09:23:05.000Z","size":84,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-08-29T03:09:08.771Z","etag":null,"topics":["cloud-native","data-stream","data-streaming","go","library","messaging"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dataphos.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-10-03T12:39:28.000Z","updated_at":"2024-10-13T10:35:08.000Z","dependencies_parsed_at":"2024-10-04T06:41:44.335Z","dependency_job_id":"0ea187aa-668a-4691-9db5-4277347b2e8c","html_url":"https://github.com/dataphos/lib-streamproc","commit_stats":null,"previous_names":["dataphos/lib-streamproc"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/dataphos/lib-streamproc","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dataphos%2Flib-streamproc","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dataphos%2Flib-streamproc/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dataphos%2Flib-streamproc/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dataphos%2Flib-streamproc/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dataphos","download_url":"https://codeload.github.com/dataphos/lib-streamproc/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dataphos%2Flib-streamproc/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29843434,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-25T21:18:31.832Z","status":"ssl_error","status_checked_at":"2026-02-25T21:18:29.265Z","response_time":61,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cloud-native","data-stream","data-streaming","go","library","messaging"],"created_at":"2024-11-18T11:20:44.299Z","updated_at":"2026-02-25T22:32:26.413Z","avatar_url":"https://github.com/dataphos.png","language":"Go","readme":"# lib-streamproc\n\nlib-streamproc is a Go library that exposes executors, interfaces, data structures, and utility functions which\ncombined a universal stream processor, invariant to any specific messaging system, while still being as performant as\npossible, taking advantage of different approaches messaging systems take.\n\nThe source systems are exposed through integration with the [lib-brokers](https://github.com/dataphos/lib-brokers) library,\nthe \"parent\" library of lib-streamproc.\n\n## Installation\n\n`go get github.com/dataphos/lib-streamproc`\n\nNote that due to the external dependencies of [lib-brokers](https://github.com/dataphos/lib-brokers), Go version 1.18 or higher is required.\n\n## Important Notice\n\nThe latest version of lib-streamproc is unstable, so there might be breaking changes with every release before v1.0.0 is\nreached. If you experience bugs or would like to start a discussion about the soundness of the API or would like to\nsee some additional feature added, contact the Labs team.\n\n## Getting Started\n\nThe focus point of lib-streamproc are *executors*, wrappers around [lib-brokers](https://github.com/dataphos/lib-brokers)-compliant consumers,\nwhich abstract away details such as properly (and efficiently) polling the consumer, acknowledging the messages, retry logic and\nmore sophisticated error handling.\n\nSince the consumer API of the [lib-brokers](https://github.com/dataphos/lib-brokers) library is split into two \n(message-based brokers and log-based brokers), there are multiple\nexecutor implementations, purpose-built for both of these groups.  \nFor example, `ReceiverExecutor` is specialized for `broker.Receiver` implementations.  \nThese executors allow for some specific optimizations like [double buffering](https://en.wikipedia.org/wiki/Multiple_buffering),\nor *lazy acknowledges*, which minimize (or completely remove) the time the system is blocked waiting for\nthe broker to respond to acknowledge requests.\n\n### The Message Structure\n\nThe core of lib-streamproc is the `Message` structure; although the consumer API is split into two,\nthe fact lib-streamproc exposes specialized executors which take care of things like acknowledges,\nallow them to pass a *unified* message structures onto user-defined functions.\n\nThe `Message` struct is defined as follows:\n\n```go\ntype Message struct {\n  // ID identifies the message (unique across the source topic).\n  ID string\n\n  // Key identifies the payload part of the message.\n  // Unlike ID, it doesn't have to be unique across the source topic and is mostly used\n  // for ordering guarantees.\n  Key string\n\n  // Data holds the payload of the message.\n  Data []byte\n\n  // Attributes holds the properties commonly used by brokers to pass metadata.\n  Attributes map[string]interface{}\n\n  // PublishTime the time this message was published.\n  PublishTime time.Time\n\n  // IngestionTime the time this message was received.\n  IngestionTime time.Time\n}\n```\n\nUsers need to only define the `MessageHandler` or `BatchHandler`, depending on if\nthey perform batch processing or not. The definition of these two interfaces is shown below:\n\n```go\ntype MessageHandler interface {\n  // HandleMessage handles the processing of a single Message.\n  // The method is assumed to be safe to call concurrently.\n  HandleMessage(context.Context, Message) error\n}\n\ntype BatchHandler interface {\n        // HandleBatch handles the processing of a batch of messages.\n        HandleBatch(context.Context, []Message) error\n}\n```\n\nThese two interfaces and the `Message` type is what allows lib-streamproc to\nabstract over the [lib-brokers](https://github.com/dataphos/lib-brokers) library and form\na universal processor over messaging systems.\n\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdataphos%2Flib-streamproc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdataphos%2Flib-streamproc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdataphos%2Flib-streamproc/lists"}