{"id":18048046,"url":"https://github.com/ivangfr/spring-cloud-stream-kafka-elasticsearch","last_synced_at":"2025-09-11T13:12:48.021Z","repository":{"id":111328515,"uuid":"144392735","full_name":"ivangfr/spring-cloud-stream-kafka-elasticsearch","owner":"ivangfr","description":"The goal of this project is to implement a \"News\" processing pipeline composed of five Spring Boot applications: producer-api, categorizer-service, collector-service, publisher-api and news-client.","archived":false,"fork":false,"pushed_at":"2025-07-17T15:08:22.000Z","size":20519,"stargazers_count":96,"open_issues_count":0,"forks_count":51,"subscribers_count":12,"default_branch":"master","last_synced_at":"2025-07-17T17:44:29.996Z","etag":null,"topics":["docker","elasticsearch","eureka","java","jib","kafka","spring-boot","spring-cloud-openfeign","spring-cloud-stream","spring-data-elasticsearch","spring-web-mvc","springdoc-openapi","thymeleaf","zipkin"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ivangfr.png","metadata":{"files":{"readme":"README.adoc","changelog":"news-client/pom.xml","contributing":null,"funding":".github/FUNDING.yml","license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":"ivangfr"}},"created_at":"2018-08-11T14:39:23.000Z","updated_at":"2025-07-17T15:08:26.000Z","dependencies_parsed_at":"2024-08-17T16:32:10.910Z","dependency_job_id":"40b03bda-cd6a-40b0-9995-a387089e5b24","html_url":"https://github.com/ivangfr/spring-cloud-stream-kafka-elasticsearch","commit_stats":{"total_commits":105,"total_committers":1,"mean_commits":105.0,"dds":0.0,"last_synced_commit":"baef5c86bf95963081c885e9b78338d4c3ac5c48"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ivangfr/spring-cloud-stream-kafka-elasticsearch","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ivangfr%2Fspring-cloud-stream-kafka-elasticsearch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ivangfr%2Fspring-cloud-stream-kafka-elasticsearch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ivangfr%2Fspring-cloud-stream-kafka-elasticsearch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ivangfr%2Fspring-cloud-stream-kafka-elasticsearch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ivangfr","download_url":"https://codeload.github.com/ivangfr/spring-cloud-stream-kafka-elasticsearch/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ivangfr%2Fspring-cloud-stream-kafka-elasticsearch/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":274640924,"owners_count":25322843,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-11T02:00:13.660Z","response_time":74,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","elasticsearch","eureka","java","jib","kafka","spring-boot","spring-cloud-openfeign","spring-cloud-stream","spring-data-elasticsearch","spring-web-mvc","springdoc-openapi","thymeleaf","zipkin"],"created_at":"2024-10-30T20:11:05.116Z","updated_at":"2025-09-11T13:12:48.010Z","avatar_url":"https://github.com/ivangfr.png","language":"Java","funding_links":["https://github.com/sponsors/ivangfr"],"categories":[],"sub_categories":[],"readme":"= spring-cloud-stream-kafka-elasticsearch\n\nThe goal of this project is to implement a \"News\" processing pipeline composed of five https://docs.spring.io/spring-boot/index.html[`Spring Boot`] applications: `producer-api`, `categorizer-service`, `collector-service`, `publisher-api` and `news-client`.\n\n== Proof-of-Concepts \u0026 Articles\n\nOn https://ivangfr.github.io[ivangfr.github.io], I have compiled my Proof-of-Concepts (PoCs) and articles. You can easily search for the technology you are interested in by using the filter. Who knows, perhaps I have already implemented a PoC or written an article about what you are looking for.\n\n== Additional Readings\n\n* [Medium]: https://medium.com/@ivangfr/implementing-a-kafka-producer-and-consumer-using-spring-cloud-stream-d4b9a6a9eab1[**Implementing a Kafka Producer and Consumer using Spring Cloud Stream**]\n* [Medium]: https://medium.com/@ivangfr/implementing-unit-tests-for-a-kafka-producer-and-consumer-that-uses-spring-cloud-stream-f7a98a89fcf2[**Implementing Unit Tests for a Kafka Producer and Consumer that uses Spring Cloud Stream**]\n* [Medium]: https://medium.com/@ivangfr/implementing-end-to-end-testing-for-a-kafka-producer-and-consumer-that-uses-spring-cloud-stream-fbf5e666899e[**Implementing End-to-End testing for a Kafka Producer and Consumer that uses Spring Cloud Stream**]\n* [Medium]: https://medium.com/@ivangfr/configuring-distributed-tracing-with-zipkin-in-a-kafka-producer-and-consumer-that-uses-spring-cloud-9f1e55468b9e[**Configuring Distributed Tracing with Zipkin in a Kafka Producer and Consumer that uses Spring Cloud Stream**]\n* [Medium]: https://medium.com/@ivangfr/using-cloudevents-in-a-kafka-producer-and-consumer-that-uses-spring-cloud-stream-9c51670b5566[**Using CloudEvents in a Kafka Producer and Consumer that uses Spring Cloud Stream**]\n\n== Technologies used\n\n* https://docs.spring.io/spring-cloud-stream/docs/current/reference/html/[`Spring Cloud Stream`] to build highly scalable event-driven applications connected with shared messaging systems;\n* https://docs.spring.io/spring-cloud-schema-registry/docs/current/reference/html/spring-cloud-schema-registry.html[`Spring Cloud Schema Registry`] that supports schema evolution so that the data can be evolved over time; besides, it lets you store schema information in a textual format (typically JSON) and makes that information accessible to various applications that need it to receive and send data in binary format;\n* https://docs.spring.io/spring-data/elasticsearch/reference/[`Spring Data Elasticsearch`] to persist data in https://www.elastic.co/elasticsearch[`Elasticsearch`];\n* https://docs.spring.io/spring-cloud-openfeign/docs/current/reference/html/[`Spring Cloud OpenFeign`] to write web service clients easily;\n* https://www.thymeleaf.org/[`Thymeleaf`] as HTML template;\n* https://zipkin.io[`Zipkin`] to visualize traces between and within applications;\n* https://github.com/Netflix/eureka[`Eureka`] as service registration and discovery.\n\nNOTE: In https://github.com/ivangfr/docker-swarm-environment[`docker-swarm-environment`] repository, it is shown how to deploy this project into a cluster of Docker Engines in swarm mode.\n\n== Project Architecture\n\nimage::documentation/project-diagram.jpeg[]\n\n== Applications\n\n* *producer-api*\n+\n`Spring Boot` Web Java application that creates news and pushes news events to `producer.news` topic in `Kafka`.\n\n* *categorizer-service*\n+\n`Spring Boot` Web Java application that listens to news events in `producer.news` topic in `Kafka`, categorizes and pushes them to `categorizer.news` topic.\n\n* *collector-service*\n+\n`Spring Boot` Web Java application that listens for news events in `categorizer.news` topic in `Kafka`, saves them in `Elasticsearch` and pushes the news events to `collector.news` topic.\n\n* *publisher-api*\n+\n`Spring Boot` Web Java application that reads directly from `Elasticsearch` and exposes a REST API. It doesn't listen from `Kafka`.\n\n* *news-client*\n+\n`Spring Boot` Web java application that provides a User Interface to see the news. It implements a `Websocket` that consumes news events from the topic `collector.news`. So, news are updated on the fly on the main page. Besides, `news-client` communicates directly with `publisher-api` whenever search for a specific news or news update are needed.\n+\nThe `Websocket` operation is shown in the short gif below. News is created in `producer-api` and, immediately, it appears in `news-client`.\n+\nimage::documentation/websocket-operation.gif[]\n\n== Prerequisites\n\n* https://www.oracle.com/java/technologies/downloads/#java21[`Java 21`] or higher;\n* A containerization tool https://www.docker.com[`Docker`], https://podman.io[`Podman`], etc.\n\n== Generate NewsEvent\n\n* Open a terminal and navigate to the `spring-cloud-stream-kafka-elasticsearch` root folder;\n\n* Run the following command to generate `NewsEvent`:\n+\n[source,bash]\n----\n./mvnw clean install --projects commons-news\n----\n+\nIt will install `commons-news-1.0.0.jar` in your local `Maven` repository, so that it can be visible by all services.\n\n== Start Environment\n\n* In a terminal, navigate to the `spring-cloud-stream-kafka-elasticsearch` root folder, and run:\n+\n[source,bash]\n----\ndocker compose up -d\n----\n\n* Wait for Docker containers to be up and running. To check it, run:\n+\n[source,bash]\n----\ndocker ps -a\n----\n\n== Running Applications with Maven\n\nInside the `spring-cloud-stream-kafka-elasticsearch` root folder, run the following `Maven` commands in different terminals:\n\n* *eureka-server*\n+\n[source,bash]\n----\n./mvnw clean spring-boot:run --projects eureka-server\n----\n\n* *producer-api*\n+\n[source,bash]\n----\n./mvnw clean spring-boot:run --projects producer-api -Dspring-boot.run.jvmArguments=\"-Dserver.port=9080\"\n----\n\n* *categorizer-service*\n+\n[source,bash]\n----\n./mvnw clean spring-boot:run --projects categorizer-service -Dspring-boot.run.jvmArguments=\"-Dserver.port=9081\"\n----\n\n* *collector-service*\n+\n[source,bash]\n----\n./mvnw clean spring-boot:run --projects collector-service -Dspring-boot.run.jvmArguments=\"-Dserver.port=9082\"\n----\n\n* *publisher-api*\n+\n[source,bash]\n----\n./mvnw clean spring-boot:run --projects publisher-api -Dspring-boot.run.jvmArguments=\"-Dserver.port=9083\"\n----\n\n* *news-client*\n+\n[source,bash]\n----\n./mvnw clean spring-boot:run --projects news-client\n----\n\n== Running Applications as Docker containers\n\n=== Build Application's Docker Image\n\n* In a terminal, make sure you are in the `spring-cloud-stream-kafka-elasticsearch` root folder;\n\n* To build the application's Docker images, run the following script:\n+\n[source,bash]\n----\n./build-docker-images.sh\n----\n\n=== Application's Environment Variables\n\n* *producer-api*\n+\n|===\n|Environment Variable | Description\n\n|`KAFKA_HOST`\n|Specify host of the `Kafka` message broker to use (default `localhost`)\n\n|`KAFKA_PORT`\n|Specify port of the `Kafka` message broker to use (default `29092`)\n\n|`SCHEMA_REGISTRY_HOST`\n|Specify host of the `Schema Registry` to use (default `localhost`)\n\n|`SCHEMA_REGISTRY_PORT`\n|Specify port of the `Schema Registry` to use (default `8081`)\n\n|`EUREKA_HOST`\n|Specify host of the `Eureka` service discovery to use (default `localhost`)\n\n|`EUREKA_PORT`\n|Specify port of the `Eureka` service discovery to use (default `8761`)\n\n|`ZIPKIN_HOST`\n|Specify host of the `Zipkin` distributed tracing system to use (default `localhost`)\n\n|`ZIPKIN_PORT`\n|Specify port of the `Zipkin` distributed tracing system to use (default `9411`)\n\n|===\n\n* *categorizer-service*\n+\n|===\n|Environment Variable | Description\n\n|`KAFKA_HOST`\n|Specify host of the `Kafka` message broker to use (default `localhost`)\n\n|`KAFKA_PORT`\n|Specify port of the `Kafka` message broker to use (default `29092`)\n\n|`SCHEMA_REGISTRY_HOST`\n|Specify host of the `Schema Registry` to use (default `localhost`)\n\n|`SCHEMA_REGISTRY_PORT`\n|Specify port of the `Schema Registry` to use (default `8081`)\n\n|`EUREKA_HOST`\n|Specify host of the `Eureka` service discovery to use (default `localhost`)\n\n|`EUREKA_PORT`\n|Specify port of the `Eureka` service discovery to use (default `8761`)\n\n|`ZIPKIN_HOST`\n|Specify host of the `Zipkin` distributed tracing system to use (default `localhost`)\n\n|`ZIPKIN_PORT`\n|Specify port of the `Zipkin` distributed tracing system to use (default `9411`)\n\n|===\n\n* *collector-service*\n+\n|===\n|Environment Variable | Description\n\n|`ELASTICSEARCH_HOST`\n|Specify host of the `Elasticsearch` search engine to use (default `localhost`)\n\n|`ELASTICSEARCH_NODES_PORT`\n|Specify nodes port of the `Elasticsearch` search engine to use (default `9300`)\n\n|`ELASTICSEARCH_REST_PORT`\n|Specify rest port of the `Elasticsearch` search engine to use (default `9200`)\n\n|`KAFKA_HOST`\n|Specify host of the `Kafka` message broker to use (default `localhost`)\n\n|`KAFKA_PORT`\n|Specify port of the `Kafka` message broker to use (default `29092`)\n\n|`SCHEMA_REGISTRY_HOST`\n|Specify host of the `Schema Registry` to use (default `localhost`)\n\n|`SCHEMA_REGISTRY_PORT`\n|Specify port of the `Schema Registry` to use (default `8081`)\n\n|`EUREKA_HOST`\n|Specify host of the `Eureka` service discovery to use (default `localhost`)\n\n|`EUREKA_PORT`\n|Specify port of the `Eureka` service discovery to use (default `8761`)\n\n|`ZIPKIN_HOST`\n|Specify host of the `Zipkin` distributed tracing system to use (default `localhost`)\n\n|`ZIPKIN_PORT`\n|Specify port of the `Zipkin` distributed tracing system to use (default `9411`)\n\n|===\n\n* *publisher-api*\n+\n|===\n|Environment Variable | Description\n\n|`ELASTICSEARCH_HOST`\n|Specify host of the `Elasticsearch` search engine to use (default `localhost`)\n\n|`ELASTICSEARCH_NODES_PORT`\n|Specify nodes port of the `Elasticsearch` search engine to use (default `9300`)\n\n|`ELASTICSEARCH_REST_PORT`\n|Specify rest port of the `Elasticsearch` search engine to use (default `9200`)\n\n|`EUREKA_HOST`\n|Specify host of the `Eureka` service discovery to use (default `localhost`)\n\n|`EUREKA_PORT`\n|Specify port of the `Eureka` service discovery to use (default `8761`)\n\n|`ZIPKIN_HOST`\n|Specify host of the `Zipkin` distributed tracing system to use (default `localhost`)\n\n|`ZIPKIN_PORT`\n|Specify port of the `Zipkin` distributed tracing system to use (default `9411`)\n\n|===\n\n* *news-client*\n+\n|===\n|Environment Variable | Description\n\n|`KAFKA_HOST`\n|Specify host of the `Kafka` message broker to use (default `localhost`)\n\n|`KAFKA_PORT`\n|Specify port of the `Kafka` message broker to use (default `29092`)\n\n|`SCHEMA_REGISTRY_HOST`\n|Specify host of the `Schema Registry` to use (default `localhost`)\n\n|`SCHEMA_REGISTRY_PORT`\n|Specify port of the `Schema Registry` to use (default `8081`)\n\n|`EUREKA_HOST`\n|Specify host of the `Eureka` service discovery to use (default `localhost`)\n\n|`EUREKA_PORT`\n|Specify port of the `Eureka` service discovery to use (default `8761`)\n\n|`ZIPKIN_HOST`\n|Specify host of the `Zipkin` distributed tracing system to use (default `localhost`)\n\n|`ZIPKIN_PORT`\n|Specify port of the `Zipkin` distributed tracing system to use (default `9411`)\n\n|===\n\n=== Run Application's Docker Container\n\n* In a terminal, make sure you are inside the `spring-cloud-stream-kafka-elasticsearch` root folder;\n\n* Run following script:\n+\n[source,bash]\n----\n./start-apps.sh\n----\n\n== Applications URLs\n\n|===\n|Application |URL\n\n|producer-api\n|http://localhost:9080/swagger-ui.html\n\n|publisher-api\n|http://localhost:9083/swagger-ui.html\n\n|news-client\n|http://localhost:8080\n\n|===\n\n== Useful links\n\n* *Eureka*\n+\n`Eureka` can be accessed at http://localhost:8761\n+\nimage::documentation/eureka.jpg[]\n\n* *Zipkin*\n+\n`Zipkin` can be accessed at http://localhost:9411\n+\nimage::documentation/zipkin.jpg[]\n\n* *Kafdrop*\n+\n`Kafdrop` can be accessed at http://localhost:9000\n\n* *Schema Registry UI*\n+\n`Schema Registry UI` can be accessed at http://localhost:8001\n\n* *Elasticsearch REST API*\n+\nCheck if ES is up and running\n+\n[source,bash]\n----\ncurl localhost:9200\n----\n+\nCheck indexes\n+\n[source,bash]\n----\ncurl \"localhost:9200/_cat/indices?v\"\n----\n+\nCheck _news_ index mapping\n+\n[source,bash]\n----\ncurl \"localhost:9200/news/_mapping?pretty\"\n----\n+\nSimple search\n+\n[source,bash]\n----\ncurl \"localhost:9200/news/_search?pretty\"\n----\n+\nDelete _news_ index\n+\n[source,bash]\n----\ncurl -X DELETE localhost:9200/news\n----\n\n== Shutdown\n\n* To stop applications:\n** If they were started with `Maven`, go to the terminals where they are running and press `Ctrl+C`.\n** If they were started as Docker containers, in a terminal and inside the `spring-cloud-stream-kafka-elasticsearch` root folder, run the script below:\n+\n[source,bash]\n----\n./stop-apps.sh\n----\n\n* To stop and remove Docker Compose containers, network and volumes, in a terminal, navigate to the `spring-cloud-stream-kafka-elasticsearch` root folder, and run the following command:\n+\n[source,bash]\n----\ndocker compose down -v\n----\n\n== Cleanup\n\nTo remove the Docker images created by this project, in a terminal and inside the `spring-cloud-stream-kafka-elasticsearch` root folder, run the script below:\n[source,bash]\n----\n./remove-docker-images.sh\n----","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fivangfr%2Fspring-cloud-stream-kafka-elasticsearch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fivangfr%2Fspring-cloud-stream-kafka-elasticsearch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fivangfr%2Fspring-cloud-stream-kafka-elasticsearch/lists"}