{"id":19288524,"url":"https://github.com/samber/go-amqp-pubsub","last_synced_at":"2025-04-22T04:33:37.071Z","repository":{"id":45285657,"uuid":"384807622","full_name":"samber/go-amqp-pubsub","owner":"samber","description":"Fault tolerant Pub/Sub library for RabbitMQ","archived":false,"fork":false,"pushed_at":"2024-11-09T18:03:57.000Z","size":656,"stargazers_count":21,"open_issues_count":10,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-11-09T19:17:46.580Z","etag":null,"topics":["amqp","broker","consumer","go","message-queue","producer","publish","pubsub","queue","rabbitmq","reactive","retry","sink","source","subscribe"],"latest_commit_sha":null,"homepage":"https://pkg.go.dev/github.com/samber/go-amqp-pubsub","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/samber.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["samber"]}},"created_at":"2021-07-10T22:28:28.000Z","updated_at":"2024-11-09T18:02:23.000Z","dependencies_parsed_at":"2024-06-27T22:17:19.732Z","dependency_job_id":"7876632f-1094-4d7c-b0ca-451ac33401d8","html_url":"https://github.com/samber/go-amqp-pubsub","commit_stats":null,"previous_names":[],"tags_count":24,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/samber%2Fgo-amqp-pubsub","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/samber%2Fgo-amqp-pubsub/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/samber%2Fgo-amqp-pubsub/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/samber%2Fgo-amqp-pubsub/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/samber","download_url":"https://codeload.github.com/samber/go-amqp-pubsub/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223888365,"owners_count":17220083,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["amqp","broker","consumer","go","message-queue","producer","publish","pubsub","queue","rabbitmq","reactive","retry","sink","source","subscribe"],"created_at":"2024-11-09T22:09:16.328Z","updated_at":"2024-11-09T22:09:16.865Z","avatar_url":"https://github.com/samber.png","language":"Go","funding_links":["https://github.com/sponsors/samber"],"categories":[],"sub_categories":[],"readme":"# Resilient Pub/Sub framework for RabbitMQ and Go\n\n[![tag](https://img.shields.io/github/tag/samber/go-amqp-pubsub.svg)](https://github.com/samber/go-amqp-pubsub/releases)\n![Go Version](https://img.shields.io/badge/Go-%3E%3D%201.20.3-%23007d9c)\n[![GoDoc](https://godoc.org/github.com/samber/go-amqp-pubsub?status.svg)](https://pkg.go.dev/github.com/samber/go-amqp-pubsub)\n![Build Status](https://github.com/samber/go-amqp-pubsub/actions/workflows/test.yml/badge.svg)\n[![Go report](https://goreportcard.com/badge/github.com/samber/go-amqp-pubsub)](https://goreportcard.com/report/github.com/samber/go-amqp-pubsub)\n[![Coverage](https://img.shields.io/codecov/c/github/samber/go-amqp-pubsub)](https://codecov.io/gh/samber/go-amqp-pubsub)\n[![Contributors](https://img.shields.io/github/contributors/samber/go-amqp-pubsub)](https://github.com/samber/go-amqp-pubsub/graphs/contributors)\n[![License](https://img.shields.io/github/license/samber/go-amqp-pubsub)](./LICENSE)\n\n- Based on github.com/rabbitmq/amqp091-go driver\n- Resilient to network failure\n- Auto reconnect: recreate channels, bindings, producers, consumers...\n- Hot update of queue bindings (thread-safe)\n- Optional retry queue on message rejection\n- Optional dead letter queue on message rejection\n- Optional deferred message consumption\n\n## How to\n\nDuring your tests, feel free to restart Rabbitmq. This library will reconnect automatically.\n\n### Connection\n\n```go\nimport pubsub \"github.com/samber/go-amqp-pubsub\"\n\nconn, err := pubsub.NewConnection(\"connection-1\", pubsub.ConnectionOptions{\n    URI: \"amqp://dev:dev@localhost:5672\",\n    Config: amqp.Config{\n        Dial:      amqp.DefaultDial(time.Second),\n    },\n})\n\n// ...\n\nconn.Close()\n```\n\n### Producer\n\n```go\nimport (\n    pubsub \"github.com/samber/go-amqp-pubsub\"\n    \"github.com/samber/lo\"\n    \"github.com/samber/mo\"\n)\n\n// `err` can be ignored since it will connect lazily to rabbitmq\nconn, err := pubsub.NewConnection(\"connection-1\", pubsub.ConnectionOptions{\n    URI: \"amqp://dev:dev@localhost:5672\",\n    LazyConnection: mo.Some(true),\n})\n\nproducer := pubsub.NewProducer(conn, \"producer-1\", pubsub.ProducerOptions{\n    Exchange: pubsub.ProducerOptionsExchange{\n        Name: \"product.event\",\n        Kind: pubsub.ExchangeKindTopic,\n    },\n})\n\nerr := producer.Publish(routingKey, false, false, amqp.Publishing{\n    ContentType:  \"application/json\",\n    DeliveryMode: amqp.Persistent,\n    Body:         []byte(`{\"hello\": \"world\"}`),\n})\n\nproducer.Close()\nconn.Close()\n```\n\n### Consumer\n\n```go\nimport (\n    pubsub \"github.com/samber/go-amqp-pubsub\"\n    \"github.com/samber/lo\"\n    \"github.com/samber/mo\"\n)\n\n// `err` can be ignore since it will connect lazily to rabbitmq\nconn, err := pubsub.NewConnection(\"connection-1\", pubsub.ConnectionOptions{\n    URI: \"amqp://dev:dev@localhost:5672\",\n    LazyConnection: mo.Some(true),\n})\n\nconsumer := pubsub.NewConsumer(conn, \"consumer-1\", pubsub.ConsumerOptions{\n    Queue: pubsub.ConsumerOptionsQueue{\n        Name: \"product.onEdit\",\n    },\n    Bindings: []pubsub.ConsumerOptionsBinding{\n        {ExchangeName: \"product.event\", RoutingKey: \"product.created\"},\n        {ExchangeName: \"product.event\", RoutingKey: \"product.updated\"},\n    },\n    Message: pubsub.ConsumerOptionsMessage{\n        PrefetchCount: mo.Some(100),\n    },\n    EnableDeadLetter: mo.Some(true),     // will create a \"product.onEdit.deadLetter\" DL queue\n})\n\nfor msg := range consumer.Consume() {\n    lo.Try0(func() { // handle exceptions\n        // ...\n        msg.Ack(false)\n    })\n}\n\nconsumer.Close()\nconn.Close()\n```\n\n### Consumer with pooling and batching\n\nSee [examples/consumer-with-pool-and-batch](examples/consumer-with-pool-and-batch/main.go).\n\n### Consumer with retry strategy\n\n![Retry architecture](doc/retry.png)\n\nSee [examples/consumer-with-retry](examples/consumer-with-retry/main.go).\n\n3 retry strategies are available:\n\n- Exponential backoff\n- Constant interval\n- Lazy retry\n\n#### Examples\n\nExponential backoff:\n\n```go\nconsumer := pubsub.NewConsumer(conn, \"example-consumer-1\", pubsub.ConsumerOptions{\n    Queue: pubsub.ConsumerOptionsQueue{\n        Name: \"product.onEdit\",\n    },\n    // ...\n    RetryStrategy:    mo.Some(pubsub.NewExponentialRetryStrategy(3, 3*time.Second, 2)), // will create a \"product.onEdit.retry\" queue\n})\n\nfor msg := range consumer.Consume() {\n    // ...\n    msg.Reject(false)   // will retry 3 times with exponential backoff\n}\n```\n\nLazy retry:\n\n```go\nconsumer := pubsub.NewConsumer(conn, \"example-consumer-1\", pubsub.ConsumerOptions{\n    Queue: pubsub.ConsumerOptionsQueue{\n        Name: \"product.onEdit\",\n    },\n    // ...\n    RetryStrategy:    mo.Some(pubsub.NewLazyRetryStrategy(3)), // will create a \"product.onEdit.retry\" queue\n})\n\nfor msg := range consumer.Consume() {\n    // ...\n    \n\terr := json.Unmarshal(body, \u0026object)\n    if err != nil {\n        // retry is not necessary\n        msg.Reject(false)\n        continue\n    }\n\n    // ...\n\n    err = sql.Exec(query)\n    if err != nil {\n        // retry on network error\n        pubsub.RejectWithRetry(msg, 10*time.Second)\n        continue\n    }\n\n    // ...\n    msg.Ack(false)\n}\n```\n\n#### Custom retry strategy\n\nCustom strategies can be provided to the consumer.\n\n```go\ntype MyCustomRetryStrategy struct {}\n\nfunc NewMyCustomRetryStrategy() RetryStrategy {\n\treturn \u0026MyCustomRetryStrategy{}\n}\n\nfunc (rs *MyCustomRetryStrategy) NextBackOff(msg *amqp.Delivery, attempts int) (time.Duration, bool) {\n    // retries every 10 seconds, until message get older than 5 minutes\n    if msg.Timestamp.Add(5*time.Minute).After(time.Now()) {\n        return 10 * time.Second, true\n    }\n\n    return time.Duration{}, false\n}\n```\n\n#### Consistency\n\nOn retry, the message is published into the retry queue then is acked from the initial queue. This 2 phases delivery is unsafe, since connection could drop during operation. With the `ConsistentRetry` policy, the steps will be embbeded into a transaction. Use it carefully because the delivery rate will be reduced by an order of magnitude.\n\n```go\nconsumer := pubsub.NewConsumer(conn, \"example-consumer-1\", pubsub.ConsumerOptions{\n    Queue: pubsub.ConsumerOptionsQueue{\n        Name: \"product.onEdit\",\n    },\n    // ...\n    RetryStrategy:    mo.Some(pubsub.NewExponentialRetryStrategy(3, 3*time.Second, 2)),\n    RetryConsistency: mo.Some(pubsub.ConsistentRetry),\n})\n```\n\n### Defer message consumption\n\n![](./doc/defer.png)\n\nSee [examples/consumer-with-delay](examples/consumer-with-delay/main.go).\n\nOn publishing, the first consumption of the message can be delayed. The message will instead be sent to the .defer queue, expire, and then go to the initial queue.\n\n```go\nconsumer := pubsub.NewConsumer(conn, \"example-consumer-1\", pubsub.ConsumerOptions{\n    Queue: pubsub.ConsumerOptionsQueue{\n        Name: \"product.onEdit\",\n    },\n    // ...\n    Defer:            mo.Some(5 * time.Second),\n})\n```\n\n## Run examples\n\n```sh\n# run rabbitmq\ndocker-compose up rabbitmq\n```\n\n```sh\n# run producer\ncd examples/producer/\ngo mod download\ngo run main.go --rabbitmq-uri amqp://dev:dev@localhost:5672\n```\n\n```sh\n# run consumer\ncd examples/consumer/\ngo mod download\ngo run main.go --rabbitmq-uri amqp://dev:dev@localhost:5672\n```\n\nThen trigger network failure, by restarting rabbitmq:\n\n```sh\ndocker-compose restart rabbitmq\n```\n\n## 🤝 Contributing\n\n- Ping me on Twitter [@samuelberthe](https://twitter.com/samuelberthe) (DMs, mentions, whatever :))\n- Fork the [project](https://github.com/samber/oops)\n- Fix [open issues](https://github.com/samber/oops/issues) or request new features\n\nDon't hesitate ;)\n\n```bash\n# Install some dev dependencies\nmake tools\n\n# Run tests\nmake test\n# or\nmake watch-test\n```\n\n### Todo\n\n- Connection pooling (eg: 10 connections, 100 channels per connection)\n- Better documentation\n- Testing + CI\n- BatchPublish + PublishWithConfirmation + BatchPublishWithConfirmation\n\n## 👤 Contributors\n\n![Contributors](https://contrib.rocks/image?repo=samber/oops)\n\n## 💫 Show your support\n\nGive a ⭐️ if this project helped you!\n\n[![GitHub Sponsors](https://img.shields.io/github/sponsors/samber?style=for-the-badge)](https://github.com/sponsors/samber)\n\n## 📝 License\n\nCopyright © 2023 [Samuel Berthe](https://github.com/samber).\n\nThis project is [MIT](./LICENSE) licensed.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsamber%2Fgo-amqp-pubsub","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsamber%2Fgo-amqp-pubsub","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsamber%2Fgo-amqp-pubsub/lists"}