https://github.com/fabogit/node-kafka-conduktor

Kafka Consumer/Producer microservices
https://github.com/fabogit/node-kafka-conduktor

fastify kafka kafka-consumer kafka-producer mongoose

Last synced: about 1 month ago
JSON representation

Kafka Consumer/Producer microservices

• Host: GitHub
• URL: https://github.com/fabogit/node-kafka-conduktor
• Owner: fabogit
• Created: 2023-01-08T12:14:41.000Z (about 2 years ago)
• Default Branch: main
• Last Pushed: 2023-01-08T13:56:54.000Z (about 2 years ago)
• Last Synced: 2024-11-06T20:37:59.506Z (3 months ago)
• Topics: fastify, kafka, kafka-consumer, kafka-producer, mongoose
• Language: TypeScript
• Homepage:
• Size: 52.7 KB
• Stars: 1
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# node-kafka-conduktor

`cd conductor && ./launch.sh`

login into conduktor @ `http://localhost:8080/home/`

`cd services/messages && npm i` and `cd services/notifications && npm i`

have mongodb running or set up a docker container

- run producer service `services/messages` `npm run dev`

- run consumer service `services/notifications` `npm run dev`

create messages using postman collection or thr conduktor ui

shout down docker `cd conductor && docker-compose down`

### Kafka terms and definitions

Producer - A producer is a client that sends messages to a topic on the Kafka cluster

Consumer - A consumer is a client that reads messages from a topic on the Kafka cluster

Topic - A topic is a category or feed name to which records are published. Topics in Kafka are always multi-subscriber

Broker - A broker is a server that hosts a topic. Each broker is identified by a unique id

Partition - A partition is an ordered, immutable sequence of records that is continually appended to a file. The records in the partitions are each assigned a sequential id number called the offset that uniquely identifies each record within the partition.

Replication - Replication is the process of duplicating data across multiple brokers. Replication provides fault tolerance and high availability. A topic can have multiple partitions, and each partition can have multiple replicas.

Consumer Group - A consumer group is a group of consumers that cooperate to consume messages from a set of topics. Each consumer in a group can process the messages in parallel. The Kafka cluster ensures that each message is delivered to one consumer in each subscribing consumer group.

Offset - The offset is a unique identifier for a message within a partition. The offset is assigned when a producer publishes a message to a partition. The offset is used by the consumer to determine the message to process next.

Leader - The leader is the server that is currently the active controller for a partition. The leader handles all read and write requests for the partition. Only the leader can become a follower.

Follower - The follower is a passive server that replicates the log of the leader. The follower can become the leader if the current leader fails