https://github.com/gordonmurray/apache_flink_and_docker_compose

Running Apache Flink containers using Docker Compose is a convenient way to get up and running to try out some Flink workloads.
https://github.com/gordonmurray/apache_flink_and_docker_compose

apache-flink cdc mariadb redis

Last synced: about 2 months ago
JSON representation

Running Apache Flink containers using Docker Compose is a convenient way to get up and running to try out some Flink workloads.

• Host: GitHub
• URL: https://github.com/gordonmurray/apache_flink_and_docker_compose
• Owner: gordonmurray
• License: mit
• Created: 2023-11-02T21:27:11.000Z (about 1 year ago)
• Default Branch: main
• Last Pushed: 2023-11-15T21:25:31.000Z (about 1 year ago)
• Last Synced: 2024-11-30T08:42:31.750Z (2 months ago)
• Topics: apache-flink, cdc, mariadb, redis
• Homepage: https://gordonmurray.com/data/2023/11/02/deploying-flink-cdc-jobs-with-docker-compose.html
• Size: 25 MB
• Stars: 8
• Watchers: 2
• Forks: 2
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# Apache Flink containers using Docker Compose

Running Apache Flink containers using Docker Compose is a convenient way to get up and running to try out some Flink workloads.

There is a post to go along with this project here [https://gordonmurray.com/data/2023/11/02/deploying-flink-cdc-jobs-with-docker-compose.html](https://gordonmurray.com/data/2023/11/02/deploying-flink-cdc-jobs-with-docker-compose.html)

Assuming docker compose is installed you can start the containers using the following command in the same folder as the docker compose file:

```

docker compose up -d

```

This will start the containers in the background and you can check that the containers are running using

```

docker ps

```

You should see something like the following showing a jobmanager and 2 task managers.

```

CONTAINER ID   IMAGE             COMMAND                  CREATED          STATUS          PORTS                                                 NAMES

eb87408560be   flink:1.17.1      "/docker-entrypoint.…"   35 minutes ago   Up 35 minutes   6123/tcp, 8081/tcp                                    apache_flink_and_docker_compose-taskmanager-2

565fd52d250a   flink:1.17.1      "/docker-entrypoint.…"   35 minutes ago   Up 35 minutes   6123/tcp, 8081/tcp                                    apache_flink_and_docker_compose-taskmanager-1

0b3e3eaa5c06   flink:1.17.1      "/docker-entrypoint.…"   35 minutes ago   Up 35 minutes   6123/tcp, 0.0.0.0:8081->8081/tcp, :::8081->8081/tcp   jobmanager

```

To start adding some work to Flink you can access the Flink console using the following command and from there you can try out various jobs like creating tables.

```bash

docker exec -it jobmanager /opt/flink/bin/sql-client.sh

```

To run a Flink job to CDC from a Mariadb container and sink some data in to Redis, use the following:

```bash

docker exec -it jobmanager /opt/flink/bin/sql-client.sh embedded -f job.sql

```