Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/marouane-elgoumiri/kafka_initiation
Setting up Kafka in Linux Ubuntu 22.04 & Windows with Intellij IDEA
https://github.com/marouane-elgoumiri/kafka_initiation
apache-kafka java maven
Last synced: 2 days ago
JSON representation
Setting up Kafka in Linux Ubuntu 22.04 & Windows with Intellij IDEA
- Host: GitHub
- URL: https://github.com/marouane-elgoumiri/kafka_initiation
- Owner: Marouane-Elgoumiri
- Created: 2024-04-29T09:57:54.000Z (9 months ago)
- Default Branch: main
- Last Pushed: 2024-05-12T19:39:12.000Z (9 months ago)
- Last Synced: 2024-11-29T21:57:50.865Z (about 2 months ago)
- Topics: apache-kafka, java, maven
- Language: Java
- Homepage:
- Size: 8.18 MB
- Stars: 3
- Watchers: 1
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Set up Kafka with Intellij idea
## I. Download latest version of Kafka from the official docs:
## II. Extract the file and run Zookeper and kafka-server
### open the extracted folder:
```bash
cd kafka_2.12-3.7.0
```
### Debian Linux
#### run zookeper:
```bash
bin/zookeeper-server-start.sh config/zookeeper.properties
```
#### run Kafka-server client:
```bash
bin/kafka-server-start.sh config/server.properties
```
### Windows
#### Start Zookeeper
```bash
start bin\windows\zookeeper-server-start config\zookeeper.properties
```
#### Start Kafka server
```bash
start bin\windows\kafka-server-start config\server.properties
```
#### Create Topic
```bash
bin\windows\kafka-topics --create --topic quickstart-events --bootstrap-server localhost:9092
bin\windows\kafka-topics --create --topic quickstart-events2 --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1
```
#### Create Producer
```bash
Start bin\windows\kafka-console-producer –topic quickstart-events --bootstrap-server localhost:9092
```
#### Create Consumer
```bash
Start bin\windows\kafka-console-consumer –topic quickstart-events --bootstrap-server localhost:9092
```
## III. Create a maven project:
### Use Kafka Plugin in the Marketplace to create Topics directly from the IDE:
### Create Topics:
### 1. Add the necessary Dependencies from Apache Kafka docs (Client & Streams)
```bash
org.apache.kafka
kafka-streams
3.7.0
org.apache.kafka
kafka_2.13
3.7.0
org.apache.kafka
kafka-clients
3.7.0
org.slf4j
slf4j-log4j12
1.7.25
log4j
log4j
1.2.17
```
### 2. Create Producer class:
```java
public class ProducerApp {
public static void main(String[] args) {
ProducerApp producerApp = new ProducerApp();
producerApp.sendMessage();
}
public void sendMessage() {
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("acks", "all");
props.put("retries", 0);
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("GROUP_ID_CONFIG", "test-group-1");
KafkaProducer producer = new KafkaProducer<>(props);
int i = 0;
while(true){
ProducerRecord message = new ProducerRecord<>("test-topic", Integer.toString(i));
producer.send(message);
try {
i++;
System.out.println("message" + i);
Thread.sleep(1000);
}catch (Exception e) {
System.out.println(e.getMessage());
}
}
}
}
```
### 3. Create Consumer class:
```java
public class ConsumerApp {
public static void main(String[] args) {
ConsumerApp newConsumer = new ConsumerApp();
newConsumer.consume();
}
public static void consume() {
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "test");
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
props.put(ConsumerConfig.GROUP_ID_CONFIG, "test");
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
props.put(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, "1000");
props.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, "30000");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("GROUP_ID_CONFIG", "test-group-1");
KafkaConsumer consumer = new KafkaConsumer<>(props);
consumer.subscribe(Collections.singletonList("test-topic"));
Executors.newScheduledThreadPool(1).scheduleAtFixedRate(()-> {
ConsumerRecords records = consumer.poll(Duration.ofMillis(1000));
records.forEach(record -> {
System.out.println("Key "+record.key()+" value "+record.value()+" topic:"+record.topic());
});
},1000, 1000, TimeUnit.MILLISECONDS);
}
}
```
### 4. Result:
