Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/j3-signalroom/linux_flink_with_iceberg
Apache Flink Docker image with Apache Iceberg support for Linux (i.e., non-Mac M1, M2, and M3 chips).
https://github.com/j3-signalroom/linux_flink_with_iceberg
apache-flink apache-iceberg flink iceberg
Last synced: 24 days ago
JSON representation
Apache Flink Docker image with Apache Iceberg support for Linux (i.e., non-Mac M1, M2, and M3 chips).
- Host: GitHub
- URL: https://github.com/j3-signalroom/linux_flink_with_iceberg
- Owner: j3-signalroom
- License: apache-2.0
- Created: 2024-08-08T00:22:00.000Z (3 months ago)
- Default Branch: main
- Last Pushed: 2024-09-15T04:21:59.000Z (2 months ago)
- Last Synced: 2024-10-12T23:39:15.639Z (about 1 month ago)
- Topics: apache-flink, apache-iceberg, flink, iceberg
- Language: Dockerfile
- Homepage: https://linkedin.com/in/jeffreyjonathanjennings
- Size: 26.4 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE.md
Awesome Lists containing this project
README
# Apache Flink with Apache Iceberg support for Linux machines
This repository hosts the [`Dockerfile`](https://github.com/j3-signalroom/linux_flink_with_iceberg/blob/main/Dockerfile), which defines the Apache Flink Docker image with Hadoop and Apache Iceberg for Linux machines. The official Apache Flink image on [Docker Hub](https://hub.docker.com/_/flink/) does not contain Hadoop and Apache Iceberg dependencies. This image extends the official image by adding the Flink-provided Hadoop Uber JAR and Apache Iceberg. You can use this Docker image to deploy Flink in [Session Mode](https://github.com/j3-signalroom/j3-techstack-lexicon/blob/main/apache-flink-glossary.md#flink-session-mode) or [Application Mode](https://github.com/j3-signalroom/j3-techstack-lexicon/blob/main/apache-flink-glossary.md#flink-application-mode) cluster on Docker, which allows you to read and write to Apache Iceberg tables.
**Table of Contents**
+ [1.0 Let's get started!](#10-lets-get-started)
+ [2.0 Resources](#20-resources)
## 1.0 Let's get started!
To run the Docker container locally:
1. Clone the repo:
```shell
git clone https://github.com/j3-signalroom/linux_flink_with_iceberg.git
```
2. Login in to your docker desktop.
3. We'll run the [Dockerfile](Dockerfile) via [docker-compose](docker-compose.yml), so please follow these steps:
a. Navigate to the root folder of the `linux_flink_with_iceberg/` repository that you cloned.
b. Open a terminal in this directory.
c. Execute the following [script](scripts/run-flink-locally.sh):
```bash
scripts/run-flink-locally.sh --profile= [--aws_s3_bucket=]
```
Argument placeholder|Replace with
-|-
``|your AWS SSO profile name for your AWS infrastructue that host your AWS Secrets Manager.
``|**[Optional]** can specify the name of the AWS S3 bucket used to store Apache Iceberg files.
## 2.0 Resources
[Apache Flink 1.20.0](https://nightlies.apache.org/flink/flink-docs-release-1.20/)
[Apache Iceberg 1.6.1](https://github.com/apache/iceberg/releases/tag/apache-iceberg-1.6.1)