{"id":13614623,"url":"https://github.com/streamdal/plumber","last_synced_at":"2025-05-14T06:13:07.416Z","repository":{"id":37840193,"uuid":"283047400","full_name":"streamdal/plumber","owner":"streamdal","description":"A swiss army knife CLI tool for interacting with Kafka, RabbitMQ and other messaging systems.","archived":false,"fork":false,"pushed_at":"2024-12-24T19:58:16.000Z","size":53012,"stargazers_count":2126,"open_issues_count":14,"forks_count":71,"subscribers_count":21,"default_branch":"master","last_synced_at":"2025-05-10T13:02:17.897Z","etag":null,"topics":["event-bus","event-driven","golang","hacktoberfest","kafka","message-bus","message-queue","protobuf","rabbitmq"],"latest_commit_sha":null,"homepage":"https://streamdal.com","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/streamdal.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-07-28T00:01:05.000Z","updated_at":"2025-05-08T13:44:40.000Z","dependencies_parsed_at":"2024-01-14T09:47:53.870Z","dependency_job_id":"f7ebb2be-198f-450f-a781-a8634e1d2579","html_url":"https://github.com/streamdal/plumber","commit_stats":{"total_commits":956,"total_committers":25,"mean_commits":38.24,"dds":"0.38075313807531386","last_synced_commit":"085244887c2d361eeeb35951a13d2fb34504d3fc"},"previous_names":["batchcorp/plumber"],"tags_count":117,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/streamdal%2Fplumber","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/streamdal%2Fplumber/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/streamdal%2Fplumber/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/streamdal%2Fplumber/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/streamdal","download_url":"https://codeload.github.com/streamdal/plumber/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254081049,"owners_count":22011545,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["event-bus","event-driven","golang","hacktoberfest","kafka","message-bus","message-queue","protobuf","rabbitmq"],"created_at":"2024-08-01T20:01:03.861Z","updated_at":"2025-05-14T06:13:07.363Z","avatar_url":"https://github.com/streamdal.png","language":"Go","readme":"![Brief Demo](./assets/plumber_logo_full.png)\n\n[![Master build status](https://github.com/streamdal/plumber/workflows/master/badge.svg)](https://github.com/streamdal/plumber/actions/workflows/master-test.yaml) [![Go Report Card](https://goreportcard.com/badge/github.com/streamdal/plumber)](https://goreportcard.com/report/github.com/streamdal/plumber) [![slack](https://img.shields.io/badge/slack-streamdal-brightgreen.svg?logo=slack)](https://launchpass.com/streamdal)\n\nplumber is a CLI devtool for inspecting, piping, messaging and redirecting data\nin message systems like Kafka, RabbitMQ , GCP PubSub and \n[many more](#supported-messaging-systems). \\[1]\n\nThe tool enables you to:\n\n* Safely view the contents of your data streams\n* Write plain or encoded data to any system\n* Route data from one place to another\n* Decode protobuf/avro/thrift/JSON data in real-time\n  * Support for both Deep and Shallow protobuf envelope types\n  * Support for google.protobuf.Any fields\n* Relay data to the [Streamdal platform](https://streamdal.com)\n* Ship change data capture events to [Streamdal platform](https://streamdal.com)\n* [Replay events into a message system on your local network](https://docs.streamdal.com/what-are/what-are-destinations/plumber-as-a-destination)\n* And _many_ other features (for a full list: `plumber -h`)\n\n\u003csub\u003e\\[1] It's like `curl` for messaging systems.\u003c/sub\u003e\n\n## Why do you need it?\n\nMessaging systems are black boxes - gaining visibility into what is passing\nthrough them is an involved process that requires you to write brittle consumer\ncode that you will eventually throw away.\n\n`plumber` enables you to stop wasting time writing throw-away code - use it to\nlook into your queues and data streams, use it to connect disparate systems \ntogether or use it for debugging your event driven systems.\n\n## Demo\n\n![Brief Demo](./assets/demo.gif)\n\n## Install\n\n### Via brew\n\n```bash\n$ brew tap streamdal/public\n$ brew install plumber\n```\n\n### Manually\n\n* [macOS](https://github.com/streamdal/plumber/releases/latest/download/plumber-darwin)\n* [Linux](https://github.com/streamdal/plumber/releases/latest/download/plumber-linux)\n* [Windows](https://github.com/streamdal/plumber/releases/latest/download/plumber-windows.exe)\n\nPlumber is a single binary, to install you simply need to download it, give it executable\npermissions and call it from your shell. Here's an example set of commands to do this:\n\n```bash\n$ curl -L -o plumber https://github.com/streamdal/plumber/releases/latest/download/plumber-darwin\n$ chmod +x plumber\n$ mv plumber /usr/local/bin/plumber\n```\n\n## Usage\n\n### Write messages\n\n```bash\n❯ plumber write kafka --topics test --input foo\nINFO[0000] Successfully wrote message to topic 'test'    backend=kafka\nINFO[0000] Successfully wrote '1' message(s)             pkg=plumber\n```\n\n### Read message(s)\n\n```bash\n❯ plumber read kafka --topics test\nINFO[0000] Initializing (could take a minute or two) ...  backend=kafka\n\n------------- [Count: 1 Received at: 2021-11-30T12:51:32-08:00] -------------------\n\n+----------------------+------------------------------------------+\n| Key                  |                                     NONE |\n| topic                |                                     test |\n| Offset               |                                        8 |\n| Partition            |                                        0 |\n| Header(s)            |                                     NONE |\n+----------------------+------------------------------------------+\n\nfoo\n```\n\nNOTE: Add `-f` to perform a continuous read (like `tail -f`)\n\n### Write messages via pipe\n\n**Write multiple messages**\n\nNOTE: Multiple messages are separated by a newline.\n\n```bash\n$ cat mydata.txt\nline1\nline2\nline3\n\n$ cat mydata.txt | plumber write kafka --topics foo\n\nINFO[0000] Successfully wrote message to topic 'foo'  pkg=kafka/write.go\nINFO[0000] Successfully wrote message to topic 'foo'  pkg=kafka/write.go\nINFO[0000] Successfully wrote message to topic 'foo'  pkg=kafka/write.go\n```\n\n**Write each element of a JSON array as a message**\n\n```bash\n$ cat mydata.json\n[{\"key\": \"value1\"},{\"key\": \"value2\"}]\n\n$ cat mydata.json | plumber write kafka --topics foo --input-as-json-array\n\nINFO[0000] Successfully wrote message to topic 'foo'  pkg=kafka/write.go\nINFO[0000] Successfully wrote message to topic 'foo'  pkg=kafka/write.go\n```\n\n## Documentation\n\n* [docs/examples.md](https://github.com/streamdal/plumber/blob/master/docs/examples.md) for more usage examples\n* [docs/env.md](https://github.com/streamdal/plumber/blob/master/docs/env.md) for list of supported environment variables\n* [docs/metrics.md](https://github.com/streamdal/plumber/blob/master/docs/metrics.md) for information on metrics that plumber exposes\n* [docs/server.md](https://github.com/streamdal/plumber/blob/master/docs/server.md) for examples on running plumber in server mode\n\n## Getting Help\n\nA full list of available flags can be displayed by using the `--help` flag after\ndifferent parts of the command:\n\n```bash\n$ plumber --help\n$ plumber read --help\n$ plumber read kafka --help\n```\n\n## Features\n\n* Encode \u0026 decode for multiple formats\n  * Protobuf (Deep and [Shallow envelope](https://www.confluent.io/blog/spring-kafka-protobuf-part-1-event-data-modeling/#shallow-envelope))\n  * Avro\n  * Thrift\n  * Flatbuffer\n  * GZip\n  * JSON\n  * JSONPB (protobuf serialized as JSON)\n  * Base64\n* `--continuous` support (ie. `tail -f`)\n* Support for **most** messaging systems\n* Supports writing via string, file or pipe\n* Observe, relay and archive messaging data\n* Single-binary, zero-config, easy-install\n\n## Hmm, what is this Streamdal thing?\n\nWe are distributed system enthusiasts that started a company called\n[Streamdal](https://streamdal.com). \n\nOur company focuses on solving data stream observability for complex systems\nand workflows. Our goal is to allow _everyone_ to build asynchronous systems,\nwithout the fear of introducing too much complexity.\n\nWhile working on our company, we built a tool for reading and writing messages\nfrom our messaging systems and realized that there is a serious lack of tooling\nin this space.\n\nWe wanted a swiss army knife type of tool for working with messaging systems\n(we use Kafka and RabbitMQ internally), so we created `plumber`.\n\n## Why the name `plumber`?\n\nWe consider ourselves \"internet plumbers\" of sort - so the name seemed to fit :)\n\n## Supported Messaging Systems\n\n* Kafka\n* RabbitMQ\n* RabbitMQ Streams \n* Google Cloud Platform PubSub\n* MQTT\n* Amazon Kinesis Streams \n* Amazon SQS\n* Amazon SNS (Publishing)\n* ActiveMQ (STOMP protocol)\n* Azure Service Bus\n* Azure Event Hub\n* NATS\n* NATS Streaming (Jetstream) \n* Redis-PubSub\n* Redis-Streams\n* Postgres CDC (Change Data Capture)\n* MongoDB CDC (Change Data Capture)\n* Apache Pulsar\n* NSQ\n* KubeMQ\n* Memphis - **NEW!**\n\nNOTE: If your messaging tech is not supported - submit an issue and we'll do\nour best to make it happen!\n\n### Kafka\nYou need to ensure that you are using the same consumer group on all plumber\ninstances.\n\n### RabbitMQ\nMake sure that all instances of `plumber` are pointed to the same queue.\n\n### Note on boolean flags\nIn order to flip a boolean flag to `false`, prepend `--no` to the flag.\n\nie. `--queue-declare` is `true` by default. To make it false, use `--no-queue-declare`.\n\n## Tunnels\n\n`plumber` can now act as a replay destination (tunnel). Tunnel mode allows you to\nrun an instance of plumber, on your local network, which will then be available\nin the Streamdal platform as a _replay destination_.\n\nThis mitigates the need make firewall changes to replay messages from a Streamdal\ncollection back to your message bus.\n\nSee https://docs.streamdal.com/what-are/what-are-destinations/plumber-as-a-destination\nfor full documentation.\n\n## High Performance \u0026 High Availability\n`plumber` comes with a \"server\" mode which will cause plumber to operate as a\nhighly available cluster.\n\nYou can read more about \"server mode\" [here](https://docs.streamdal.com/plumber/server-mode).\n\nServer mode examples can be found in [docs/server.md](https://github.com/streamdal/plumber/blob/master/docs/server.md)\n\n## Acknowledgments\n\n**Huge** shoutout to [jhump](https://github.com/jhump) and for his excellent\n[protoreflect](https://github.com/jhump/protoreflect) library, without which\n`plumber` would not be anywhere *near* as easy to implement. _Thank you!_\n\n## Release\n\nTo push a new plumber release:\n\n1. `git tag v0.18.0 master`\n1. `git push origin v0.18.0`\n1. Watch the github action\n1. New release should be automatically created under https://github.com/streamdal/plumber/releases/\n1. Update release to include any relevant info\n1. Update [homebrew](https://github.com/streamdal/homebrew-public/blob/master/plumber.rb) SHA and version references\n\n## Contribute\n\nWe love contributions! Prior to sending us a PR, open an issue to discuss what\nyou intend to work on. When ready to open PR - add good tests and let's get this\nthing merged! For further guidance check out our [contributing guide](https://github.com/streamdal/plumber/blob/master/CONTRIBUTING.md).\n","funding_links":[],"categories":["Go"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstreamdal%2Fplumber","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstreamdal%2Fplumber","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstreamdal%2Fplumber/lists"}