https://github.com/ravening/cloudstack-events-kafka
Project to read events from cloudstack kafka broker and send it to elasticsearch
https://github.com/ravening/cloudstack-events-kafka
cloudstack elasticsearch events java kafka kafka-consumer spring-boot
Last synced: 2 months ago
JSON representation
Project to read events from cloudstack kafka broker and send it to elasticsearch
- Host: GitHub
- URL: https://github.com/ravening/cloudstack-events-kafka
- Owner: ravening
- Created: 2021-02-15T16:48:05.000Z (over 4 years ago)
- Default Branch: main
- Last Pushed: 2021-02-15T16:48:17.000Z (over 4 years ago)
- Last Synced: 2025-02-01T18:44:58.997Z (4 months ago)
- Topics: cloudstack, elasticsearch, events, java, kafka, kafka-consumer, spring-boot
- Language: Java
- Homepage:
- Size: 11.7 KB
- Stars: 0
- Watchers: 3
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# kafka-elasticsearch-connector
Java application to read messages from Apache Kafka topics and stream it to\
Elasticsearch cluster
## Prerequisites
Running Apache Kafka server
Elasticsearch cluster
Java 11 or docker installed
## Assumptions
First setup beans in Cloudstack and connect it to kafka by following steps under `Kafka Configuration` section in\
https://docs.cloudstack.apache.org/en/latest/adminguide/events.html
It is assumed that both Apache Kafka and Elasticsearch are running in the same machine\
as this application. If not you need to specify the hostname in `application.yml` file.
It is also assumed that all messages will be sent and read from `cloudstack` topic on kafka\
and 'cloudstack-kafka' index name is used as default
If you want to receive messages on different topics then change the topic name in\
`KafkaTopicListener.java` file.\
If you want to change the index name for elasticsearch then change the value in\
`Message.java` file.
Once all these changes are done, you can build the entire project or docker image.
## Building project
The application can be built using\
`mvn clean project -DskipTests`
Docker image can be built using\
`./mvnw spring-boot:build-image`
## Running the project
You can run the jar file using\
`java -jar target/kafka-elasticsearch-connector.jar`
or start the docker container using\
`docker run -d -it -p 8080:8080 --network host --name kafka-elasticksearch-connector /kafka-elasticsearch-connector`