{"id":15294885,"url":"https://github.com/lileio/pubsub","last_synced_at":"2025-04-13T05:13:51.656Z","repository":{"id":41070634,"uuid":"111601729","full_name":"lileio/pubsub","owner":"lileio","description":" A Publish \u0026 Subscribe library, with pluggable providers and middleware for Golang","archived":false,"fork":false,"pushed_at":"2025-03-19T17:17:02.000Z","size":259,"stargazers_count":143,"open_issues_count":9,"forks_count":22,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-04-13T05:13:39.361Z","etag":null,"topics":["golang","google","publish","pubsub","queue","sqs","subscribe","workers"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lileio.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-11-21T21:10:17.000Z","updated_at":"2025-03-07T19:03:29.000Z","dependencies_parsed_at":"2024-06-18T15:52:09.132Z","dependency_job_id":null,"html_url":"https://github.com/lileio/pubsub","commit_stats":null,"previous_names":[],"tags_count":39,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lileio%2Fpubsub","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lileio%2Fpubsub/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lileio%2Fpubsub/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lileio%2Fpubsub/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lileio","download_url":"https://codeload.github.com/lileio/pubsub/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248665746,"owners_count":21142123,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["golang","google","publish","pubsub","queue","sqs","subscribe","workers"],"created_at":"2024-09-30T17:07:46.324Z","updated_at":"2025-04-13T05:13:51.635Z","avatar_url":"https://github.com/lileio.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"![logo](./docs/logo.png)\n--\n![Actions Status](https://github.com/lileio/pubsub/workflows/Test/badge.svg) [![](https://godoc.org/github.com/lileio/pubsub?status.svg)](http://godoc.org/github.com/lileio/pubsub)\n\nPubSub provides a simple helper library for doing publish and subscribe style asynchronous tasks in Go, usually in a web or micro service. PubSub allows you to write publishers and subscribers, fully typed, and swap out providers (Google Cloud PubSub, AWS SQS etc) as required. \n\nPubSub also abstracts away the creation of the queues and their subscribers, so you shouldn't have to write any cloud specific code, but still gives you [options](https://godoc.org/github.com/lileio/pubsub#HandlerOptions) to set concurrency, deadlines, error handling etc.\n\n[Middleware](https://github.com/lileio/pubsub/tree/master/middleware) is also included, including logging, tracing and error handling!\n\n\n## Table of Contents\n\n- [Example](#example)\n  * [Publisher](#publisher)\n  * [Subscriber](#subscriber)\n  * [Full Example](#full-example)\n- [Middleware](#middleware)\n  * [Default](#default)\n  * [Opentracing](#opentracing)\n  * [Prometheus](#prometheus)\n- [Providers](#providers)\n  * [Google PubSub](#google-cloud-pubsub)\n  * [Nats Streaming](#nats-streaming-server)\n  * [AWS SQS/SNS](#aws-sqssns)\n  * [Kakfa](#kafka)\n\n## Example\n\nHere's a basic example using [Nats streaming server](https://nats-io.github.io/docs/nats_streaming/intro.html) and a basic subscriber function that prints hello.\n\n\nTo publish messages, you can call `Publish`, you can publish a [Protobuf](https://developers.google.com/protocol-buffers/docs/gotutorial) or JSON serializable object (i.e, most Go objects).\n\nPublish is Protobuf by default..\n\n### Publisher\n\n``` go\npubsub.Publish(ctx, \"topic-name\", \u0026User{Id: \"usr_0001\"})\n```\n\nto publish a JSON object\n\n``` go\npubsub.PublishJSON(ctx, \"topic-name\", \u0026User{Id: \"usr_0001\"})\n```\n\nThis can be useful if the application subscribing isn't good with Protobuf or is external to your company for example. However Protobuf is recommended for speed, type safety and forwards compatability.\n\n### Subscriber\n\nSubscribing to a topic is done with a single function, you'll receive a context, the object that was in the queue and the pubsub message, which includes some metadata and timing information, should you need it.\n\n``` go\nfunc PrintHello(ctx context.Context, msg *HelloMsg, m *pubsub.Msg) error {\n\tfmt.Printf(\"Message received %+v\\n\\n\", m)\n\n\tfmt.Printf(msg.Greeting + \" \" + msg.Name + \"\\n\")\n\n\treturn nil\n}\n```\n\nFirst though, you need to \"Setup\" your subscribers\n\n``` go\ntype Subscriber struct{}\n\nfunc (s *Subscriber) Setup(c *pubsub.Client) {\n\tc.On(pubsub.HandlerOptions{\n\t\tTopic:   HelloTopic,\n\t\tName:    \"print-hello\",\n\t\tHandler: PrintHello,\n\t\tAutoAck: true,\n\t\tJSON:    true,\n\t})\n}\n\npubsub.Subscribe(\u0026Subscriber{})\n```\n\n### Full Example\n\nYou can see a full example in the [example folder](https://github.com/lileio/pubsub/tree/master/example).\n\n## Middleware\n\n### Default\n\nPubSub provides a helper to setup the [default middleware](https://github.com/lileio/pubsub/blob/master/middleware/defaults/defaults.go).\n\nAt the time of writing this includes, [Logrus](https://github.com/sirupsen/logrus), [Opentracing](https://github.com/opentracing/opentracing-go), [Prometheus](https://github.com/lileio/pubsub/blob/master/middleware/prometheus/prometheus.go), [Recovery (Handles panics)](https://github.com/lileio/pubsub/blob/master/middleware/recover/recover.go) and [Audit Logging](https://github.com/lileio/pubsub/blob/master/middleware/audit/audit.go)\n\nTo use this, simple include it when initialising PubSub\n\n```go\npubsub.SetClient(\u0026pubsub.Client{\n\tServiceName: \"my-service-name\",\n\tProvider:    provider,\n\tMiddleware:  defaults.Middleware,\n})\n```\n\nYou can optionaly provide a recovery handler too.\n\n```go\npubsub.SetClient(\u0026pubsub.Client{\n\tServiceName: \"my-service-name\",\n\tProvider:    provider,\n\tMiddleware:  defaults.MiddlewareWithRecovery(func(p interface{}) (err error){\n\t\t// log p or report to an error reporter\n\t}),\n})\n```\n\n### Logrus\n\nWhen enabled, the Logrus middleware will output something similar to below. Note that the level is `DEBUG` by default. To see the logs, you'll need to set `logrus.SetLevel(logrus.DebugLevel)` or use something like [github.com/lileio/Logr](https://github.com/lileio/logr/blob/master/logr.go#L44) which can set it from ENV variables.\n\n```\ntime=\"2019-09-23T12:46:13Z\" level=debug msg=\"Google Pubsub: Publishing\"\ntime=\"2019-09-23T12:46:13Z\" level=debug msg=\"Google Pubsub: Publish confirmed\"\ntime=\"2019-09-23T12:46:13Z\" level=debug msg=\"Published PubSub Msg\" component=pubsub duration=143.545203ms metadata=\"map[x-b3-parentspanid:622cff2be9102141 x-b3-sampled:1 x-b3-flags:0 x-audit-user:xxxx@example.co.uk x-b3-traceid:4275176f2f7f729257887d1e4853498d x-b3-spanid:017e28147c6c3704]\" topic=hello.world\ntime=\"2019-09-23T12:46:16Z\" level=debug msg=\"Processed PubSub Msg\" component=pubsub duration=1.702259988s handler=function_name id=734207593944188 metadata=\"map[x-b3-traceid:4275176f2f7f729257887d1e4853498d x-b3-spanid:017e28147c6c3704 x-audit-user:xxxx@example.co.uk x-b3-parentspanid:622cff2be9102141 x-b3-flags:0 x-b3-sampled:1]\" topic=hello.work\n```\n\n### Opentracing\n\nThe Opentracing middle adds tags ands logs to spans which will later be sent to something like [Zipkin](https://zipkin.io/) or [Jaeger](https://www.jaegertracing.io/) when setup in the application. Note that the Opentracing middleware only adds things to the context but isn't responsible for setting up Opentracing and it's reporting, for that, [see here](https://github.com/opentracing/opentracing-go).\n\n### Prometheus\n\nThe Prometheus middleware includes some counters and histograms to help with monitoring, you can see there [here](https://github.com/lileio/pubsub/blob/master/middleware/prometheus/prometheus.go#L13) but these include.\n\n```\npubsub_message_published_total{topic,service}\npubsub_outgoing_bytes{topic,service}\npubsub_publish_durations_histogram_seconds\npubsub_server_handled_total{\"topic\", \"service\", \"success\"}\npubsub_incoming_bytes{\"topic\", \"service\"}\npubsub_subscribe_durations_histogram_seconds{\"topic\", \"service\"}\n```\n\nHere's an example query to get messages handled (by a subscriber) every minute, make sure your prometheus step is also `1m`\n\n```\nsum(increase(pubsub_server_handled_total[1m])) by (topic, success)\n```\n\n## Providers\n\n\n### Google Cloud PubSub\n\nTo setup a Google Cloud client, you can do the following..\n\n``` go\npubsub.SetClient(\u0026pubsub.Client{\n\tServiceName: \"my-service-name\",\n\tProvider:    google.NewGoogleCloud('projectid'),\n\tMiddleware:  defaults.Middleware,\n})\n```\n\nIf you're on Google Cloud vms you're environment likely already has credentials setup, but locally you can set them up with [default credentials](https://cloud.google.com/sdk/gcloud/reference/auth/application-default/), if you're on Kubernetes, I reccomend setting up service account and then making a secret file and setting the `GOOGLE_APPLICATION_CREDENTIALS` to the filepath of that JSON secret key.\n\nThe Google PubSub provider is tested heavily in production by [Echo](http://echo.co.uk) and works well, we have however noticed some strange behaviour from Google subscribers, as they try to be clever and balance traffic and other strange things. For example, if you want to only process 2 messages at a time, and don't process the two you're given, then can often result in a pause before more messages are sent to you, this can be hard to debug as a queue builds up, but often fixes itself.\n\n\n### Nats Streaming Server\n\nTo setup a Nats Streaming client, you can do the following. Optionally passing options for the original [client](github.com/nats-io/stan.go)\n\n``` go\npubsub.SetClient(\u0026pubsub.Client{\n\tServiceName: \"my-service-name\",\n\tProvider:    nats.NewNats('clustername', opts),\n\tMiddleware:  defaults.Middleware,\n})\n```\n\nNote this driver is for Nats Streaming, and not for plain Nats.\n\n### AWS SQS/SNS\n\nCurrently there is no provider for AWS SNS and SQS. Please feel free to make a pull request!\n\n### Kafka\n\nThere's an experimental provider for Kafka available [here](https://github.com/lileio/pubsub/blob/master/providers/kafka/kafka.go), but it's limiting in options you can override. I'd love to see someone take this on and help it become more bullet proof. But things like retries are hard.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flileio%2Fpubsub","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flileio%2Fpubsub","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flileio%2Fpubsub/lists"}