https://github.com/haarismemon/conduktortask
https://github.com/haarismemon/conduktortask
Last synced: about 1 month ago
JSON representation
- Host: GitHub
- URL: https://github.com/haarismemon/conduktortask
- Owner: haarismemon
- Created: 2025-02-05T12:28:08.000Z (3 months ago)
- Default Branch: main
- Last Pushed: 2025-02-06T11:48:10.000Z (3 months ago)
- Last Synced: 2025-02-06T12:32:13.043Z (3 months ago)
- Language: Scala
- Size: 109 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Conduktor Task
The chosen topic name for this project is `people-records`.
## Setup
First run the kafka docker container:
```
docker run -p 9092:9092 apache/kafka:3.7.0
```
To get container id for after running the above:
```
docker ps
```
Then create the three partitions for the `people-records` topic:
```
docker exec -it /opt/kafka/bin/kafka-topics.sh \
--bootstrap-server localhost:9092 \
--create \
--topic people-records \
--partitions 3 \
--replication-factor 1 \
--config cleanup.policy=delete
```
Then run the kafka producer, by selecting `ProducerService` when running `sbt run`.
This will load the people records from the json file into the `people-records` topic.
Finally, to obtain the people records with the kafka consumer, run the REST API server by selecting `ConsumerApiServer` when running `sbt run`.
The endpoint to hit to obtain the people records is:
```
localhost:8080/topic/people-records/{offset-value}?count={count-value}
```
## Useful Debugging Commands
To check for existing topics in the kafka container:
```
docker exec -it /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --list
```
To confirm the topic has been configured correctly with the correct number of partitions:
```
docker exec -it /opt/kafka/bin/kafka-topics.sh \
--bootstrap-server localhost:9092 \
--describe \
--topic people-records
```
To check how many messages in each partition for the topic:
```
docker exec -it /opt/kafka/bin/kafka-log-dirs.sh \
--bootstrap-server localhost:9092 \
--describe
```
To read messages from the topic as they come in:
```
docker exec -it /opt/kafka/bin/kafka-console-consumer.sh \
--bootstrap-server localhost:9092 \
--topic people-records
```
Add `--from-beginning` flag to see all messages.