https://github.com/wavemaker/kafka-connector
https://github.com/wavemaker/kafka-connector
wavemaker-connector
Last synced: 7 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/wavemaker/kafka-connector
- Owner: wavemaker
- License: apache-2.0
- Created: 2020-07-02T09:41:29.000Z (over 5 years ago)
- Default Branch: master
- Last Pushed: 2024-10-21T06:46:31.000Z (over 1 year ago)
- Last Synced: 2025-04-11T21:37:59.267Z (12 months ago)
- Topics: wavemaker-connector
- Language: Java
- Homepage:
- Size: 38.1 KB
- Stars: 2
- Watchers: 5
- Forks: 2
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
## Connector Introduction
Connector is a Java based backend extension for WaveMaker applications. Connectors are built as Java modules & exposes java based SDK to interact with the connector implementation.
Each connector is built for a specific purpose and can be integrated with one of the external services. Connectors are imported & used in the WaveMaker application. Each connector runs on its own container thereby providing the ability to have it’s own version of the third party dependencies.
## Features of Connectors
1. Connector is a java based extension which can be integrated with external services and reused in many Wavemaker applications.
1. Each connector can work as an SDK for an external system.
1. Connectors can be imported once in a WaveMaker application and used many times in the applications by creating multiple instances.
1. Connectors are executed in its own container in the WaveMaker application, as a result there are no dependency version conflict issues between connectors.
## About kafka-connector
This Connector exposes APIs to intracts with Apacahe Kafka Cluster which can be integrated into WaveMaker Application.
## Kafka Introduction
Apache Kafka is a distributed publish-subscribe messaging system.Kafka is fast, scalable, durable, fault-tolerant and distributed messaging system by design.
It was originally developed by Linked later it became of part of Apache project.
## Kafka Fundamentals
### Topic
Messages belongs to a particular category is known as topic.
### Partition
Topics are split into partitions, for each topic Kafka will keep at least one partition.
### Broker
A Kafka cluster consists of one or more servers (Kafka brokers) running Kafka. A Kafka server, a Kafka broker and a Kafka node all refer to the same concept in cluster.
### Kafka Controller
Within a Kafka cluster, a single broker serves as the active controller which is responsible for state management of partitions and replicas. So in your case, if you have a cluster with 100 brokers, one of them will act as the controller.
### Zoo Keeper
Zookeeper is a top-level software developed by Apache that acts as a centralized service and is used to maintain naming and configuration data.
Zookeeper keeps track of status of the Kafka cluster nodes and it also keeps track of Kafka topics, partitions etc.
### Consumer
Consumers read data from brokers. Consumers subscribes to one or more topics and consume published messages by pulling data from the brokers.
### Consumer Group
When multiple consumers are subscribed to a topic and belong to the same consumer group.
Each Consumer group has unique Id and each group can register for one or more topics.
## Build
You can build this connector using following command
```
mvn clean install
```
## Deploy
You can import connector dist/kafka.zip artifact in WaveMaker studio application using file upload option.
On after deploying Kafka-Connector in the WaveMaker studio application, make sure to update connector properties in the profile properties.Such as
zookeeper IP, Kafka brokers IPs...etc
## Using Connector in WaveMaker
This connector has three connector interfaces which are used for different operations in Kafka cluster.
### TopicConnector
This connector Interface apis are used to manages topics in the Kafka cluster.Such as creating, Listing, Deleting, Updating Topics in the cluster.
```
@Autowired
private KafkaTopicConnector kafkaTopicConnector;
String topicName = "demoTopic";
kafkaTopicConnector.createTopic(topicName, 2, 1, new Properties());
```
### ProducerConnector
This connector interface apis are used to send massages to specific topic in the kafka cluster
```
@Autowired
private KafkaProducerConnector kafkaProducerConnector;
KafkaProducer kafkaProducer = kafkaProducerConnector.getKafkaProducer(topicName, Integer.class, String.class);
kafkaProducer.send(1,"Hello, Welcome to kafka connector");
```
### ConsumerConnector
This connector interface apis are used to consume message from a specific topic using consumer groups in the kafka cluster.
```
@Autowired
private KafkaConsumerConnector kafkaConsumerConnector;
KafkaConsumer kafkaConsumer = kafkaConsumerConnector.getKafkaConsumer("group1", 2, Integer.class, String.class);
kafkaConsumer.start(topicName, new KafkaConsumerCallback() {
@Override
public void onMessage(ConsumerRecord record, Acknowledgment acknowledgment) {
logger.info("Consumer receiving messages : " + record.key() + ":" + record.value() + " in partition [ " + record.partition() + " ]" + " with offset no [ " + record.offset() + " ]");
}
});
```
Apart from above apis, there are other apis in each connector interface, please visit connector interfaces in the kafka-api module.