{"id":20285419,"url":"https://github.com/mramshaw/pubsub","last_synced_at":"2026-05-11T10:32:54.446Z","repository":{"id":92905697,"uuid":"136110068","full_name":"mramshaw/pubsub","owner":"mramshaw","description":"Adventures in messaging","archived":false,"fork":false,"pushed_at":"2019-08-05T15:04:28.000Z","size":17,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-04T03:44:34.132Z","etag":null,"topics":["google-cloud-pub","kafka","message-broker","messaging","mosquitto","mqtt","publish-subscribe","pubsub","rabbitmq","redis","sns","zeromq","zookeeper"],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mramshaw.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-06-05T02:44:48.000Z","updated_at":"2019-08-05T18:51:40.000Z","dependencies_parsed_at":"2023-04-29T00:55:13.878Z","dependency_job_id":null,"html_url":"https://github.com/mramshaw/pubsub","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mramshaw/pubsub","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mramshaw%2Fpubsub","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mramshaw%2Fpubsub/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mramshaw%2Fpubsub/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mramshaw%2Fpubsub/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mramshaw","download_url":"https://codeload.github.com/mramshaw/pubsub/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mramshaw%2Fpubsub/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32890653,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-10T13:40:02.631Z","status":"online","status_checked_at":"2026-05-11T02:00:05.975Z","response_time":120,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["google-cloud-pub","kafka","message-broker","messaging","mosquitto","mqtt","publish-subscribe","pubsub","rabbitmq","redis","sns","zeromq","zookeeper"],"created_at":"2024-11-14T14:26:33.718Z","updated_at":"2026-05-11T10:32:54.428Z","avatar_url":"https://github.com/mramshaw.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Adventures in Messaging\n\nNoodling around with [pub/sub](http://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_pattern) systems.\n\nIn the examples below, the instructions will be for the _simplest possible configuration_. It is possible\nto run both Redis and Kafka clusters, but for learning purposes it will suffice to use a single instance\nof either.\n\n* [Concepts](#concepts)\n* [MQTT and Mosquitto](#mqtt-and-mosquitto)\n   * [QoS](#qos)\n   * [Message Retention](#message-retention)\n* [Google Cloud Pub/Sub](#google-cloud-pubsub)\n* [Amazon Simple Notification Service](#amazon-simple-notification-service)\n* [RabbitMQ](#rabbitmq)\n* [ZeroMQ](#zeromq)\n* [Redis](#redis)\n    * [Start Redis](#start-redis)\n    * [Register a subscriber](#register-a-subscriber)\n    * [Publish a message](#publish-a-message)\n    * [Useful commands](#useful-commands)\n    * [Close down the publisher and subscriber](#close-down-the-publisher-and-subscriber)\n    * [Close down the broker](#close-down-the-broker)\n* [Kafka](#kafka)\n    * [Java](#java)\n    * [Zookeeper](#zookeeper)\n    * [Start Kafka](#start-kafka)\n    * [Create a topic](#create-a-topic)\n    * [Pass messages](#pass-messages)\n    * [Cleanup](#cleanup)\n* [Reference](#reference)\n* [To Do](#to-do)\n\n## Concepts\n\nThe general concept is that __publishers__ create messages and ___publish___ them to a given\n__channel__ or __topic__. And __Subscribers__ receive messages by ___subscribing___ to specific channel(s)\nor topic(s). __Message Brokers__ may (or may not) handle the enqueueing and/or dispersal of the messages.\n\n## MQTT and Mosquitto\n\nI'm starting here because the ground is familiar:\n\n    http://github.com/mramshaw/MQTT_and_mosquitto\n\n__MQTT__ is the protocol and __Mosquitto__ is a message broker.\n\nMQTT was specifically designed for unreliable networks, and so includes a __Last Will and Testament__\n(LWT) feature. This is to handle the situation where clients unexpectedly drop or get disconnected - so\nthat closedown handling can be invoked (for a clean shutdown, for instance). The __Keep Alive__ feature\nis of course important for determining the timing of when the LWT processing is invoked.\n\nDepending on the specified [QoS](#qos) and [Message Retention](#message-retention), messages may or may\nnot be queued; as with [redis](#redis) listeners may need to be actively subscribed in order to receive\nmessages.\n\n#### QoS\n\nValid values for Quality of Service are:\n\n- 0 [at most once]\n- 1 [at least once]\n- 2 [exactly once]\n\nQoS 0 is also referred to as \"fire and forget\" meaning these messages may or may not actually be delivered.\n\nMessages published with QoS greater than zero will be queued, however the broker will respect the QoS\nrequested by the client - in which case the QoS may be downgraded.\n\nQoS 2 is the most costly delivery mechanism. With QoS 1 the client must be capable of handling duplicated\nmessages.\n\n#### Message Retention\n\nPublished messages may (or may not) be specified for retention.\n\nIf specified, the message is saved by the broker as the last known good value for the specific topic.\n\nWhen a new client subscribes to a topic, they receive the last message that is retained on that topic.\n\n[The broker stores only one retained message per topic.]\n\n## Google Cloud Pub/Sub\n\nLike MQTT QoS 0 or [redis](#redis), [Google Cloud Pub/Sub](http://cloud.google.com/pubsub/docs/) is also\napparently \"fire and forget\" meaning messages may or may not actually be delivered.\n\nSupports large-scale messaging over HTTP (REST) or [gRPC](http://www.grpc.io/).\n\nNote that, as with most messaging solutions, Cloud Pub/Sub doesn't guarantee the order of the messages.\n\nAfter successfully pulling a message and processing it, you are supposed to __acknowledge__  the message\n(this means notifying Google Cloud Pub/Sub that you successfully received the message). You may also\n`--auto-ack` the message or messages, which automatically pulls and acknowledges the message or messages.\n\nFailure to acknowledge the message within the __acknowledgement deadline__ period will result in the\nmessage being re-sent.\n\n## Amazon Simple Notification Service\n\nSupported protocols:\n\n1. [Amazon SQS](http://aws.amazon.com/sqs/)\n2. HTTP/S\n3. email\n4. [SMS](http://en.wikipedia.org/wiki/SMS)\n5. Lambda\n\n[From: http://docs.aws.amazon.com/sns/latest/dg/welcome.html]\n\n## RabbitMQ\n\nSupported protocols:\n\n1. AMQP 0-9-1\n2. [STOMP](http://stomp.github.io/)\n3. MQTT\n4. AMQP 1.0\n5. HTTP and WebSockets\n\n[From: http://www.rabbitmq.com/protocols.html]\n\n## ZeroMQ\n\nUnusually, [ZeroMQ](http://zeromq.org/) is more of a protocol than middleware. Think sockets on steroids.\n\n## Redis\n\n[Perhaps not the ideal use case for [Redis](http://redis.io/), but as Redis is widely\n deployed perhaps worth considering. In any case, a useful tool for exploring how pub/sub works.]\n\nMessages are not queued; as with [MQTT and Mosquitto](#mqtt-and-mosquitto) listeners\nneed to be actively subscribed in order to receive them.\n\nOffers no persistence and no delivery guarantees.\n\nCheck out the following link for Salvatore Sanfilippo's thoughts on Redis and Pub/Sub:\n\n    http://oldblog.antirez.com/post/redis-weekly-update-3-publish-submit.html\n\n#### Start Redis\n\nIn a console, execute the following command to start Redis:\n\n```\n$ docker-compose up\n```\n\n#### Register a subscriber\n\nIn another console, use the `redis-cli` command to subscribe to a topic, as follows:\n\n```\n$ redis-cli\n127.0.0.1:6379\u003e subscribe news\nReading messages... (press Ctrl-C to quit)\n1) \"subscribe\"\n2) \"news\"\n3) (integer) 1\n```\n\n[In Redis, a return code of 1 usually indicates success while 0 usually indicates failure.]\n\n#### Publish a message\n\nIn a third console, use the `redis-cli` command to publish a message to the news topic, as follows:\n\n```\n$ redis-cli\n127.0.0.1:6379\u003e publish news \"redis is up!\"\n(integer) 1\n127.0.0.1:6379\u003e\n```\n\nIn the second console, the subscriber should echo the message sent:\n\n```\n$ redis-cli\n127.0.0.1:6379\u003e subscribe news\nReading messages... (press Ctrl-C to quit)\n1) \"subscribe\"\n2) \"news\"\n3) (integer) 1\n1) \"message\"\n2) \"news\"\n3) \"redis is up!\"\n```\n\n#### Useful commands\n\nIn the second console, the following commands may be interesting:\n\n```\n127.0.0.1:6379\u003e pubsub channels\n1) \"news\"\n127.0.0.1:6379\u003e pubsub numsub news\n1) \"news\"\n2) (integer) 1\n127.0.0.1:6379\u003e\n```\n\n[List the channels \u0026 List the number of subscribers to the news channel.]\n\n#### Close down the publisher and subscriber\n\nEither type `exit` or Ctrl-C in the second and third console to terminate.\n  \n#### Close down the broker\n\nShut down Redis (first console) with Ctrl-C.\n\nAnd:\n\n```\ndocker-compose down\n```\n\n## Kafka\n\n[Apache Kafka](http://kafka.apache.org/) describes itself as _a distributed\nstreaming platform_.\n\nLike [redis](#redis), Kafka is a multi-use storage application that can be\nused (among other things) for pub/sub messaging.\n\nIt is written in [Scala](http://www.scala-lang.org/), which requires a JVM\n(Java Virtual Machine) to run.\n\nBuilds on [Apache Zookeeper](http://zookeeper.apache.org/), which is much like\n[etcd](http://github.com/coreos/etcd).\n\nThere is also a pure Golang implementation, [Jocko](http://github.com/travisjeffery/jocko).\n\n#### Java\n\nVerify `java` is installed:\n\n```\n$ java -version\nopenjdk version \"1.8.0_171\"\nOpenJDK Runtime Environment (build 1.8.0_171-8u171-b11-0ubuntu0.16.04.1-b11)\nOpenJDK 64-Bit Server VM (build 25.171-b11, mixed mode)\n$\n```\n\n#### Zookeeper\n\nStart `zookeeper` (bundled with Kafka) as follows:\n\n```\n$ bin/zookeeper-server-start.sh config/zookeeper.properties\n\u003c...\u003e\n```\n\n#### Start Kafka\n\nIn a new console, start `kafka` as follows:\n\n```\n$ bin/kafka-server-start.sh config/server.properties\n\u003c...\u003e\n[2018-06-05 15:12:16,412] INFO [Kafka Server 0], started (kafka.server.KafkaServer)\n```\n\n#### Create a topic\n\nIn a third console, create a topic as follows:\n\n```\n$ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic news\nCreated topic \"news\".\n$\n```\n\nAnd verify the topic was created, as follows:\n\n```\n$ bin/kafka-topics.sh --list --zookeeper localhost:2181\nnews\n$\n```\n\n#### Pass messages\n\nIn the third console, open a publisher as follows:\n\n```\n$ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic news\n\u003eKafka is UP!\n\u003e\n```\n\n[And type a message for broadcasting.]\n\nIn a fourth console, open a subscriber as follows:\n\n```\n$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic news\n\n```\n\nNote that the recently published message is not received. Let's fix that:\n\n```\n$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic news\n^CProcessed a total of 0 messages\n$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic news --from-beginning\nKafka is UP!\n\n```\n\n[First we entered Ctrl-C to kill the subscriber. Then we restarted the subscriber\n with the `--from-beginning` option so as to get all previously published messages.]\n\nAnd now our publisher (third console) operates as expected:\n\n```\n$ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic news\n\u003eKafka is UP!\n\u003eHello?\n\u003e\n```\n\nAnd our subscriber (fourth console) receives the latest message:\n\n```\n$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic news --from-beginning\nKafka is UP!\nHello?\n```\n\n[From the above it should be apparent that Kafka caches (stores) messages. Also, it does not\n seem to track which messages have been _consumed_ either - this seems to be a _subscriber_\n responsibility. For instance, specifying `from-beginning` (or any other constant offset)\n will mean that the subscriber messages will always start with the same message.]\n\n#### Cleanup\n\nNow kill everything with Ctrl-C.\n\nSubscriber:\n\n```\n$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic news --from-beginning\nKafka is UP!\nHello?\n^CProcessed a total of 2 messages\n$\n```\n\nPublisher:\n\n```\n$ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic news\n\u003eKafka is UP!\n\u003eHello?\n\u003e^C$\n```\n\nKafka:\n\n```\n\u003c...\u003e\n[2018-06-05 15:51:45,855] INFO [Kafka Server 0], shut down completed (kafka.server.KafkaServer)\n$\n```\n\nZookeeper:\n\n```\n\u003c...\u003e\n[2018-06-05 15:51:45,854] INFO Closed socket connection for client /127.0.0.1:45648 which had sessionid 0x163d20186a50000 (org.apache.zookeeper.server.NIOServerCnxn)\n^C$\n```\n\n## Reference\n\nSalvatore Sanfilippo (antirez) on Redis and Pub/Sub:\n\n    http://oldblog.antirez.com/post/redis-weekly-update-3-publish-submit.html\n\n[For anyone interested in Redis, the antirez weblog is a great resource.]\n\n## To Do\n\n- [ ] Explore ZeroMQ\n- [x] Add QoS and retention notes for MQTT \n- [x] Flesh out the Kafka messaging details\n- [x] Investigate [Google Cloud Pub/Sub](http://cloud.google.com/pubsub/docs/)\n- [ ] Investigate [Amazon SNS](http://docs.aws.amazon.com/sns/latest/dg/welcome.html)\n- [ ] Investigate [RabbitMQ](http://www.rabbitmq.com/) and [RabbitMQ as a Service](http://www.cloudamqp.com/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmramshaw%2Fpubsub","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmramshaw%2Fpubsub","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmramshaw%2Fpubsub/lists"}