https://github.com/makism/stream-stack-playground

https://github.com/makism/stream-stack-playground

flink kafka kafka-connect

Last synced: 2 months ago
JSON representation

• Host: GitHub
• URL: https://github.com/makism/stream-stack-playground
• Owner: makism
• Created: 2023-03-06T18:46:40.000Z (almost 2 years ago)
• Default Branch: main
• Last Pushed: 2023-10-04T19:31:31.000Z (over 1 year ago)
• Last Synced: 2024-04-19T05:44:22.633Z (9 months ago)
• Topics: flink, kafka, kafka-connect
• Language: Python
• Homepage:
• Size: 35.9 MB
• Stars: 0
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
The repository contains docker-compose files to run a Kafka cluster with different configurations.

---

# 1 - Kafka with Connect & S3 support

This is a docker-compose file to run a Kafka cluster with Connect and S3 (MinIO) support. There's a sample connector that reads from a topic and writes to S3.

```bash

docker-compose -f cp-docker-compose.yml build

docker-compose -f cp-docker-compose.yml up

```

This image is based on the [Confluent All-In-One Images](https://github.com/confluentinc/cp-all-in-one).

## TODO

* Integrate Flink.




# 2 - Kafka and Flink

This is a docker-compose file to run a Kafka cluster with Flink support.

```bash

docker-compose -f kafka-flink.docker-compose.yml build

docker-compose -f kafka-flink.docker-compose.yml up

```

- - - 

# Example jobs

# 1. Read lines

Start the producer and give in some text as input (`src/read_lines/producer.sh`):

```bash

docker exec -it broker kafka-console-producer.sh --bootstrap-server localhost:9092 --topic users

```

At the same time, start a consumer:

```bash

docker exec -it stream-stack-playground_jobmanager_1 flink run -py /opt/src/read_lines/consumer.py

```

# 2a. JSON events (datastream to table)

Produce data with:

```bash

python src/flink_kafka/json_events/producer.py --num-events 1000

```

Datastream to datastream:

```bash

docker exec -it stream-stack-playground_jobmanager_1 flink run -py /opt/src/json_events/consumer_table.py

```

Datastream to table:

```bash

docker exec -it stream-stack-playground_jobmanager_1 flink run -py /opt/src/json_events/consumer_stream.py

```

# 2b. JSON events (table to table)

```bash

docker exec -it stream-stack-playground_jobmanager_1 flink run -py /opt/src/tables/consumer.py

```