{"id":29698814,"url":"https://github.com/moshdev2213/springboot-kafka-pubsub","last_synced_at":"2026-05-15T12:32:57.449Z","repository":{"id":303735323,"uuid":"1010663661","full_name":"moshdev2213/springboot-kafka-pubsub","owner":"moshdev2213","description":"This repository offers a Spring Boot Kafka Pub/Sub quickstart 🚀, demonstrating asynchronous, loosely coupled communication ✨. It provides separate publisher and consumer services with shared DTOs, plus local Kafka setup instructions 💻. Future plans include Dockerization 🐳, Docker Compose, CI/CD integration 🚀, and cloud deployment to EC2/VMs ☁️.","archived":false,"fork":false,"pushed_at":"2025-07-13T06:50:33.000Z","size":13624,"stargazers_count":1,"open_issues_count":3,"forks_count":1,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-07-23T10:56:28.405Z","etag":null,"topics":["kafka","pubsub","shared-module","spring-boot"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/moshdev2213.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-29T14:54:32.000Z","updated_at":"2025-07-09T17:10:58.000Z","dependencies_parsed_at":"2025-07-09T06:51:06.290Z","dependency_job_id":null,"html_url":"https://github.com/moshdev2213/springboot-kafka-pubsub","commit_stats":null,"previous_names":["moshdev2213/springboot-kafka-pubsub-quickstart"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/moshdev2213/springboot-kafka-pubsub","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/moshdev2213%2Fspringboot-kafka-pubsub","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/moshdev2213%2Fspringboot-kafka-pubsub/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/moshdev2213%2Fspringboot-kafka-pubsub/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/moshdev2213%2Fspringboot-kafka-pubsub/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/moshdev2213","download_url":"https://codeload.github.com/moshdev2213/springboot-kafka-pubsub/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/moshdev2213%2Fspringboot-kafka-pubsub/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33067253,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-15T11:35:32.926Z","status":"ssl_error","status_checked_at":"2026-05-15T11:35:31.362Z","response_time":103,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kafka","pubsub","shared-module","spring-boot"],"created_at":"2025-07-23T10:39:11.197Z","updated_at":"2026-05-15T12:32:57.435Z","avatar_url":"https://github.com/moshdev2213.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Spring Boot, Kafka Quickstart\n\n\u003cimg width=\"5600\" height=\"3152\" alt=\"Image\" src=\"https://github.com/user-attachments/assets/89393edf-e163-48df-b0d5-08be9be8d672\" /\u003e\n\n---\n\n## 🌟 Features \u0026 Concepts Demonstrated\n\nThis repository provides a hands-on quickstart guide to implementing **Apache Kafka**'s publish-subscribe messaging pattern using **Spring Boot**. It demonstrates how to set up separate producer (publisher) and consumer applications, communicate through a shared data transfer object, and manage Kafka locally.\n\n* **Apache Kafka Fundamentals:** Understand topics, producers, and consumers.\n* **Spring for Apache Kafka:** Seamless integration of Kafka messaging into Spring Boot applications.\n* **Publish-Subscribe Pattern:** Clear separation of concerns between message publishers and consumers.\n* **Asynchronous Communication:** Real-time data processing without blocking.\n* **Loose Coupling:** Independent services communicating via events.\n* **Shared DTO Module:** Best practice for defining common data structures across services.\n* **Local Kafka Setup:** Instructions for running Kafka using both Open Source Apache Kafka and Confluent Community Edition.\n\n---\n\n## 🚀 Getting Started\n\nTo get this project up and running locally, follow these steps.\n\n### Prerequisites\n\nBefore you begin, ensure you have the following installed:\n\n* **Java Development Kit (JDK) 17 or higher**\n* **Maven 3.6.0 or higher** (or Gradle if you prefer, though this project uses Maven)\n* **Apache Kafka:** You'll need a running Kafka instance. Instructions for local setup are provided below.\n\n### ⚙️ Setting Up Kafka Locally\n\nYou have two main options for running Kafka locally, for this project i followed the option 01:\n\n#### Option 1: Open Source Apache Kafka\n\n1.  **Download Kafka:**\n    Visit the [Apache Kafka downloads page](https://kafka.apache.org/downloads) and download the latest stable release.\n2.  **Extract the Archive:**\n    Unzip the downloaded `kafka_*.tgz` file to a directory of your choice (e.g., `C:\\kafka`, `~/kafka`).\n3.  **Start Zookeeper:**\n    Kafka relies on Zookeeper. Open a new terminal in your Kafka installation directory and run:\n    ```bash\n    # For Windows\n    .\\bin\\windows\\zookeeper-server-start.bat .\\config\\zookeeper.properties\n\n    # For macOS/Linux\n    ./bin/zookeeper-server-start.sh ./config/zookeeper.properties\n    ```\n4.  **Start Kafka Server:**\n    Open *another* new terminal in your Kafka installation directory and run:\n    ```bash\n    # For Windows\n    .\\bin\\windows\\kafka-server-start.bat .\\config\\server.properties\n\n    # For macOS/Linux\n    ./bin/kafka-server-start.sh ./config/server.properties\n    ```\n\n#### Option 2: Confluent Community Edition (Recommended for ease of use)\n\nConfluent Community Edition simplifies local Kafka setup significantly.\n\n1.  **Download Confluent CLI:**\n    Follow the instructions on the [Confluent documentation](https://docs.confluent.io/platform/current/installation/install/cli-install.html) to install the Confluent CLI.\n2.  **Start Confluent Platform (including Kafka and Zookeeper):**\n    Open a terminal and run:\n    ```bash\n    confluent local services start\n    ```\n    This command will start Zookeeper, Kafka, Schema Registry, and other components.\n\n### 📝 Creating Kafka Topic(s)\n\nOnce Kafka is running, you need to create the topic(s) that your publisher will write to and your consumer will read from.\n\nFor this project, let's assume we'll use a topic named `orders-topic`.\n\n```bash\n# Using Apache Kafka CLI (from your Kafka installation directory)\n# For Windows\n.\\bin\\windows\\kafka-topics.bat --create --topic orders-topic --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1\n\n# For macOS/Linux\n./bin/kafka-topics.sh --create --topic orders-topic --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1\n\n# Using Confluent CLI\nconfluent kafka topic create orders-topic\n````\n\nYou can verify the topic creation:\n\n```bash\n# Apache Kafka CLI\n# For Windows\n.\\bin\\windows\\kafka-topics.bat --describe --topic orders-topic --bootstrap-server localhost:9092\n\n# For macOS/Linux\n./bin/kafka-topics.sh --describe --topic orders-topic --bootstrap-server localhost:9092\n\n# Confluent CLI\nconfluent kafka topic describe orders-topic\n```\n\n### 🏃 Running the Spring Boot Applications\n\nEach component (`consumer`, `publisher`, `shared-dto`) is a separate Maven module.\n\n1.  **Clone the Repository:**\n\n    ```bash\n    git clone [https://github.com/moshdev2213/springboot-kafka-pubsub-quickstart.git](https://github.com/moshdev2213/springboot-kafka-pubsub-quickstart.git)\n    cd springboot-kafka-pubsub-quickstart\n    ```\n\n2.  **Build the Project:**\n    Build all modules to ensure dependencies are resolved and `shared-dto` is available for `publisher` and `consumer`.\n\n    ```bash\n    mvn clean install\n    ```\n\n3.  **Run the Consumer Application:**\n    Open a new terminal, navigate to the `consumer` directory, and run the application.\n\n    ```bash\n    cd consumer\n    mvn spring-boot:run\n    ```\n\n    The consumer will start listening for messages on the `orders-topic`.\n\n4.  **Run the Publisher Application:**\n    Open *another* new terminal, navigate to the `publisher` directory, and run the application.\n\n    ```bash\n    cd publisher\n    mvn spring-boot:run\n    ```\n\n    The publisher application will expose an endpoint to send messages. Check the `publisher` module's Controller code for how to trigger message sending.\n\n-----\n\n## 💡 How It Works\n---\n\u003cimg width=\"4320\" height=\"2172\" alt=\"Image\" src=\"https://github.com/user-attachments/assets/721f7ded-c723-4709-8441-472ac93e63fa\" /\u003e\n\n\n---\n\n  * The **`publisher`** application is responsible for creating and sending messages (events) to the Kafka `orders-topic`. These messages typically represent an \"order\" or a similar business event.\n  * The **`consumer`** application continuously listens to the `orders-topic`. Whenever a new message arrives, the consumer processes it (e.g., logs it, stores it in a database, triggers another business logic).\n  * The **`shared-dto`** module defines the common data structure (e.g., `OrderDTO`) that both the publisher sends and the consumer expects, ensuring type safety and consistency across the message flow.\n\n-----\n\n## 📂 Project Structure\n\n```\nspringboot-kafka-pubsub-quickstart/\n├── consumer/                 # Spring Boot application for consuming Kafka messages\n│   └── src/main/java/.../ConsumerApplication.java\n│   └── ...\n├── publisher/                # Spring Boot application for publishing Kafka messages\n│   └── src/main/java/.../PublisherApplication.java\n│   └── ...\n├── shared-dto/               # Common Data Transfer Objects (DTOs) shared by consumer and publisher\n│   └── src/main/java/.../OrderDTO.java\n│   └── ...\n└── README.md                 # This file\n```\n\n-----\n\n## 🛠️ Technologies Used\n\n  * **Java 17+**\n  * **Spring Boot 3.x**\n  * **Apache Kafka**\n  * **Maven**\n\n-----\n\n## 🚀 What's Next? (Future Plans)\n\nThis quickstart is just the beginning\\! We have exciting plans to enhance this project:\n\n  * **Dockerization:** Containerize both the `publisher` and `consumer` Spring Boot applications using Docker, making them portable and easy to deploy.\n  * **Docker Compose Integration:** Introduce a `docker-compose.yml` file to orchestrate the entire environment, including Kafka, Zookeeper, and our Spring Boot applications, for simplified local development and testing.\n  * **CI/CD Pipeline:** Implement a Continuous Integration/Continuous Deployment (CI/CD) pipeline (e.g., using GitHub Actions, Jenkins, GitLab CI) to automate testing, building, and deployment processes.\n  * **Cloud Deployment:** Explore deploying the Dockerized applications to a cloud environment like an AWS EC2 instance or any other Virtual Machine (VM) running Docker, demonstrating a real-world deployment scenario.\n\n-----\n\n## 🤝 Contributing\n\nContributions are welcome\\! If you have suggestions for improvements, new features, or bug fixes, please feel free to:\n\n1.  Fork the repository.\n2.  Create a new branch (`git checkout -b feature/your-feature-name`).\n3.  Make your changes.\n4.  Commit your changes (`git commit -m 'feat: Add new feature'`).\n5.  Push to the branch (`git push origin feature/your-feature-name`).\n6.  Open a Pull Request.\n\n-----\n\n\n## 🙏 Acknowledgements\n\n  * Apache Kafka Community\n  * Spring Framework Contributors\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmoshdev2213%2Fspringboot-kafka-pubsub","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmoshdev2213%2Fspringboot-kafka-pubsub","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmoshdev2213%2Fspringboot-kafka-pubsub/lists"}