{"id":41692850,"url":"https://github.com/wkennedy/pubsubly","last_synced_at":"2026-01-24T20:24:33.458Z","repository":{"id":39951107,"uuid":"220436198","full_name":"wkennedy/pubsubly","owner":"wkennedy","description":"Pubsubly is an application to track messages as they flow through a pub/sub backed message driven architecture.","archived":false,"fork":false,"pushed_at":"2023-01-05T00:42:34.000Z","size":3032,"stargazers_count":1,"open_issues_count":25,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-07-13T22:52:33.159Z","etag":null,"topics":["activemq","event","eventsource","eventsourcing","eventstream","kafka","publish-subscribe","spring-kafka"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wkennedy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-11-08T09:47:06.000Z","updated_at":"2020-11-03T22:55:24.000Z","dependencies_parsed_at":"2023-02-03T01:46:32.103Z","dependency_job_id":null,"html_url":"https://github.com/wkennedy/pubsubly","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/wkennedy/pubsubly","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wkennedy%2Fpubsubly","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wkennedy%2Fpubsubly/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wkennedy%2Fpubsubly/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wkennedy%2Fpubsubly/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wkennedy","download_url":"https://codeload.github.com/wkennedy/pubsubly/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wkennedy%2Fpubsubly/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28736509,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-24T19:23:36.361Z","status":"ssl_error","status_checked_at":"2026-01-24T19:23:28.966Z","response_time":89,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["activemq","event","eventsource","eventsourcing","eventstream","kafka","publish-subscribe","spring-kafka"],"created_at":"2026-01-24T20:24:33.397Z","updated_at":"2026-01-24T20:24:33.450Z","avatar_url":"https://github.com/wkennedy.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pubsubly\n\n![Java CI](https://github.com/wkennedy/pubsubly/workflows/Java%20CI/badge.svg?branch=dynamic_topics)\n\n## What is it?\n\nPubsubly is an application to track messages as they flow through a pub/sub backed message driven architecture.\n\nIt does this by allowing users to group messages by metadata through the use of pluggable Spring auto-configurations. As each message enters the system it is processed and grouped according to one of the user defined tags.\n\n## Limitations\n\nCurrently, the only messaging platforms supported are Kafka, ActiveMQ and Redis. Another limitation is that the rendering of the message payload is limited as media types aren't currently supported. Probably the biggest limitation is that it's not scalable beyond one instance.\n\n## Quickstart - Demo\nIf all you want to do is see a demo of the application, you can run Pubsubly in demo mode. \n\nFirst of all, you'll need to clone this project:\n\n    git clone https://github.com/wkennedy/pubsubly.git\n\nIn order to run the platform all you need is Docker installed in order to use Docker Compose,\n\nhttps://docs.docker.com/v17.09/engine/installation/\n\nThen from pubsubly/docker run:\n\n    docker-compose -f docker-compose-demo.yml up\n\nOnce everything is up and running you can navigate to http://localhost:3001 and see the Pubsubly UI interface.\n\n## Quickstart - Local\nOnce you clone the repo, if you don't already have access to Kafka brokers you can start Kafka with the docker-compose-kafka.yml file in pubsubly/docker.\nFirst you need to create a network that is used by Kafka and Pubsubly, then you need to start the services:\n\n    docker network create pubsubly-network\n    \n    docker-compose -f docker-compose-kafka.yml -f docker-compose.yml up\n    \nIf you aren't running Kafka locally, then you'll have update the application.yml with the broker URLs. In the application.yml in the pubsubly/docker/ folder resources, update:\n\n    spring:\n      kafka:\n        bootstrap-servers: \u003cIP or hostname of Kafka broker\u003e:9092    \n        \nAlso update the topics you want to listen to either with a pattern OR topic names:\n\n    kafka.topic.pattern: DEMO-.*\n\nor\n\n    kafka.topic.names:\n    \nThen you can start the server through docker-compose:\n\n    docker-compsose -f docker-compose.yml up\n    \nThen navigate to localhost:3001 in your browser.\n\n## Common issues\n\n- If you have trouble with ports due to applications on your machine already utilizing those ports, please edit the ports in the docker-compose-demo.yml (or docker-compose.yml) file appropriately.\n\n- One way to fix the offsets if they get screwed up in your container is to do the following:\n\n  Get the container id of the Kafka service using\n\n       docker ps\n       \n  Then login into that service with:\n\n      docker exec -it \u003ccontainer id\u003e /bin/bash\n    \n  Then go to usr/bin\n\n      cd /usr/bin\n     \n  And execute\n\n      ./kafka-consumer-groups --bootstrap-server localhost:9092 --reset-offsets --to-latest --group pubsubly-group --all-topics --execute\n\n  You can also modify topics and delete consumer groups from here.\n\n- If you are facing issues with Kafka brokers and old volume data and not all the data is showing up in the UI, try this:  \n\n      docker-compose -f .\\docker-compose-demo.yml up -d --force-recreate --renew-anon-volumes\n  then\n      docker-compose -f docker-compose-demo.yml up\n\n  Then you might have to use volumes and uncomment the Zookeeper and Kafka volumes in the docker-compose-demo.yml file.\n\n      #    volumes:\n      #      - ./zk-single-kafka-single/zoo1/data:/data\n      #      - ./zk-single-kafka-single/zoo1/datalog:/datalog\n      and\n      #    volumes:\n      #      - ./zk-single-kafka-single/kafka1/data:/var/lib/kafka/data\n   \n\n## Demo Walk-through\nThis demo demonstrates the use of 3 different messaging platforms and the ability of Pubsubly to track the correlation of messages across the different systems (Kafka, Redis, ActiveMQ).\n\nThe demo runs a basic simulator that produces events. The event chains are triggered every 60 seconds and they simulate the creation of a user, the user getting a session stored in Redis and the sending of user data to a legacy system using ActiveMQ.\nThe simulation also shows a series of bids and asks on a product, eventually resulting in a match which creates an order.\n\nTo run the demo execute the following from pubsubly/docker:\n\n    docker-compose -f docker-compose-demo.yml up\n\n![Image](images/home_page.PNG?raw=true)\nThis image shows the home page. The home page consists of a bar graph that shows the topics and the amount of messages from each topic. It also has a bubble chart that depicts the amount of messages processed per day per hour. This is to bring any attention to times of the day where message flow is high.\nThe \"Latest messages\" section is a live display of the last 50 messages processed. The charts do not automatically update, so the page needs to be refreshed to reflect the latest numbers. \n\n\nClick on the menu button and select \"Topics\". This will bring you to a list of all the topics available.\n\n![Image](images/topics.PNG?raw=true)\n\nThe topics page displays the topic names along with the amount of messages processed for each topic. If you click the \"eye\" icon, you'll be able to see messages processed from that topic. Go ahead and click on \"DEMO-KAFKA_ORDER_TOPIC\".\n\n![Image](images/topic.PNG?raw=true)\n\nIn the topic page for the \"DEMO-KAFKA_ORDER_TOPIC\" you can view all the order events with the newest order first. If you click the drop down error to the left of the message you can traverse through the message headers and payload. Now let's click on the correlationId values for one of the orders. This will allow us to view all the messages all the way to the message that was the root cause.\n\n![Image](images/tags.PNG?raw=true)\n\nThe \"Message Hops for correlationId\" shows the topics and time from the root cause message to the final event. The \"Time Since Last Topic\" displays the time between when the correlationId was last seen in a topic.\nThe \"Message for correlationId\" displays the message starting with the root cause message and the events triggered by it.\n\nNow let's go to the menu in the upper left corner and click the search button in order to go to the search screen.\n\n![Image](images/search.PNG?raw=true)\n\nThe search page allows you to search by specific tag values, header keys and values, or text within the payload. In this example we performed a search for messages that contained the text \"Yeezy\".\n\n![Image](images/kafka_listeners.PNG?raw=true)\n\nThe listeners page allows you to add and remove additional Kafka topic names or topic patterns. \nMessages will automatically start being consumed from topics matching the new names or patterns.\n\n## How to build\n\n#### Configuration\n\nConfiguration for the pubsubly-service is done through the application.yml file in pubsubly-service main resources (pubsubly\\pubsubly-service\\src\\main\\resources\\application.yml).\n\nConfigurations for Kafka, ActiveMQ, and Redis utilize Spring's autoconfigurations for these services, so use the Spring common applications to configure your application for those platforms. The properties can be found here:\n\nhttps://docs.spring.io/spring-boot/docs/current/reference/html/appendix-application-properties.html\n\nKafka:  In order to listen to topics on Kafka, you can set the topics either as a pattern or a list of topics:\n\n    kafka.topic.pattern: \u003cPREFIX\u003e.*\n    or\n    kafka.topic.names: TOPIC1,TOPIC2\n\nRedis:  Redis topic pattern configuration is (configure one or the other):\n\n    redis.topic.pattern: REDISTOPIC\n    or\n    redis.topic.names: REDISTOPIC1,REDISTOPIC2\n    \nFor Redis also remove this from the autoconfigure exclude:\n\n    org.springframework.boot.autoconfigure.data.redis.RedisAutoConfiguration\n\nActiveMQ:  ActiveMQ requires a comma separated list:\n\n    activemq.topic.names: TOPIC1,TOPIC2\n        \nThe main configuration for Pubsubly is setting the plugin-processors. The processors are used to define what data you want to track. Currently there is only one built-in plugin, which is the HeaderProcessorPlugin. It allows you to track messages by specified header values.\n\n    plugin-processors:\n      tags:\n        - id: eventId\n          value: eventId\n          description: This is the event ID\n          isPrimaryMessageId: true\n          display: false\n        - id: correlationId\n          value: correlationId\n          description: This is the correlation ID\n          isMessageCorrelationId: true\n        - id: eventName\n          value: eventName\n          description: This is the event name\n          patternValueMonitors:\n            - regexPattern: The regex pattern you want to match for this value, example '.*ERROR.*'\n              priority: Possible values - NORMAL, LOW, MEDIUM, HIGH\n        - id: sku\n          value: sku\n          description: ID of the product\n      processors:\n        - id: headerProcessorPlugin\n        - id: redisMessageHeaderProcessorPlugin\n\nLet's go over the demo configuration as an example. The tags have the following properties:\n\n- id: this is a unique identifier for the tag and is used internally for tracking messages\n- value: this is the value in the message that you want to tag and track, in this example it's a header key.\n- description: this is the description of the data you are tagging. It can be used for clients such as the UI.\n- display: whether this tag should display in the UI\n- isPrimaryMessageId: specifies if this value is used a the unique identification of the message\n- isMessageCorrelationId: this is the id of the event that caused this message to be raised. Allows for tracking the root cause of the event.\n\nThe processors are the plugins you want to apply the tags too. In this example we have to processors, the headerProcessorPlugin and the redisMessageHeaderProcessorPlugin. The redisMessageHeaderProcessorPlugin is a simple example of creating a plugin for custom tag processing.\nThe idea of the redisMessageHeaderProcessorPlugin is to support sending a message with headers and a payload to Redis and be able to analyze the headers, even though headers aren't supported in Redis.\n\n#### pubsubly-api\nIn order to build the pubsubly-api you need Java 8 (and set JAVA_HOME) or above installed and Maven. In the pubsubly-api (pubsubly/pubsubly-api) root execute:\n\n    mvn install\n\nIf you don't or can't install Maven, you can use the included Maven wrapper:\n\n    ./mvnw install\n    \n#### pubsubly-service\nNOTE: the pubsubly-api is not yet pushed to Maven Central, so you need to build it yourself before building pubsubly-service. Go to pubsubly/pubsubly-api and do:\n\n    mvn install\n\nIf you don't or can't install Maven, you can use the included Maven wrapper:\n\n    ./mvnw install\n   \n\nNow build pubsubly-service. In order to build the pubsubly-service you need Java 8 (and set JAVA_HOME) or above installed and Maven. In the pubsubly-service (pubsubly/pubsubly-service) root execute:\n\n    mvn install\n\nIf you don't or can't install Maven, you can use the included Maven wrapper:\n\n    ./mvnw install\n\nFrom here you can build your own docker images. In the pubsubly/pubsubly-service directory execute (replace \u003cYOUR PREFIX\u003e with your docker hub name):\n\n     docker build -t \u003cYOUR PREFIX\u003e/pubsubly-service .\n\n#### pubsubly-ui\nIn order to use the UI you need NodeJS installed (https://nodejs.org/en/download/). You don't have to build the UI in order to run it. From the pubsubly/pubsubly-ui directory you can execute:\n\n    npm run dev\n\nPlease note that you'll need the pubsubly-service running in order for the UI to operate correctly.\n\nFrom here you can build your own docker images. In the pubsubly/pubsubly-ui directory execute (replace \u003cYOUR PREFIX\u003e with your docker hub name):\n\n     docker build -t \u003cYOUR PREFIX\u003e/pubsubly-ui .\n\n## How to run\n\nNow that your docker images are built, you can update the docker-compose.yml file in pubsubly/docker with your image names. For example, \n\n    pubsubly-service:\n      image: \"\u003cYOUR PREFIX\u003e/pubsubly-service\"\n    pubsubly-ui:\n      image: \"\u003cYOUR PREFIX\u003e/pubsubly-ui\"\n\nOnce that's done you can run:\n\n    docker-compose up\n\nand then navigate to http://localhost:3001 in your browser.\n\n## Custom Plugins\n\nUsing the pubsubly-api you can create a new plugin to process message in order to track based on specific data. For an example see: https://github.com/wkennedy/pubsubly/tree/master/redis-message-header-plugin\n\nThere are three basic parts to the plugin:\n\n1. Create a plugin that implements com.github.wkennedy.pubsubly.api.Processor. The processor will return the value that you want to track. In the case of RedisMessageHeaderProcessorPlugin we are looking header values that are actually in the payload.\n\n2. Create a @Configuration for your plugin. The bean name will be used as the id for your plugin-processors -\u003e processor configuration as mentioned above.\n\n3. Create spring.factories in /resources/META-INF of your project. This is used by Spring to autoconfigure your plugin when you include it as a dependency in the pubsubly-service.\n\n        org.springframework.boot.autoconfigure.EnableAutoConfiguration=\\\n        com.github.wkennedy.pubsubly.rmhpp.plugins.RedisMessageHeaderProcessorConfig\n        \n## Internal processing overview\n\n![Image](images/pubsubly-service-processing.png?raw=true)\n\n## Useful Endpoints\n\nhttp://localhost:9000/swagger-ui.html\n\nThis is the swagger endpoint to display the API provided by the pubsubly-service.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwkennedy%2Fpubsubly","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwkennedy%2Fpubsubly","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwkennedy%2Fpubsubly/lists"}