{"id":21125500,"url":"https://github.com/j3-signalroom/supercharge_streamlit-apache_flink","last_synced_at":"2026-05-22T14:21:28.689Z","repository":{"id":262708969,"uuid":"886457885","full_name":"j3-signalroom/supercharge_streamlit-apache_flink","owner":"j3-signalroom","description":"Engaging, interactive visualizations crafted with Streamlit, seamlessly powered by Apache Flink in batch mode to reveal deep insights from data.","archived":false,"fork":false,"pushed_at":"2024-12-01T16:14:15.000Z","size":666,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-21T05:28:30.520Z","etag":null,"topics":["apache-flink","apache-iceberg","aws-glue-data-catalog","flink","flink-sql","iceberg","kafka","pyflink","streamlit","streamlit-dashboard"],"latest_commit_sha":null,"homepage":"https://linkedin.com/in/jeffreyjonathanjennings","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/j3-signalroom.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-11-11T02:22:32.000Z","updated_at":"2024-12-01T16:11:28.000Z","dependencies_parsed_at":"2024-11-13T21:35:53.259Z","dependency_job_id":"16045461-31bc-46d4-b53d-65724c4efaed","html_url":"https://github.com/j3-signalroom/supercharge_streamlit-apache_flink","commit_stats":null,"previous_names":["j3-signalroom/supercharge_streamlit-apache_flink"],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/j3-signalroom%2Fsupercharge_streamlit-apache_flink","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/j3-signalroom%2Fsupercharge_streamlit-apache_flink/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/j3-signalroom%2Fsupercharge_streamlit-apache_flink/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/j3-signalroom%2Fsupercharge_streamlit-apache_flink/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/j3-signalroom","download_url":"https://codeload.github.com/j3-signalroom/supercharge_streamlit-apache_flink/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243573163,"owners_count":20312879,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-flink","apache-iceberg","aws-glue-data-catalog","flink","flink-sql","iceberg","kafka","pyflink","streamlit","streamlit-dashboard"],"created_at":"2024-11-20T04:34:20.208Z","updated_at":"2026-05-22T14:21:28.616Z","avatar_url":"https://github.com/j3-signalroom.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Supercharge Your Streamlit Visualizations: Batch Processing Iceberg Data with Apache Flink\nAfter you have all your fun running the Java-based Flink application [`DataGeneratorApp`](https://github.com/j3-signalroom/apache_flink-kickstarter/blob/main/java/README.md) to kickstart the data pipeline. This app powers up your Kafka topics—`airline.skyone` and `airline.sunset`—by generating sample records that fuel the rest of the process. Once the data flows into Kafka, launch the [`FlightImporterApp`](https://github.com/j3-signalroom/apache_flink-kickstarter/blob/main/python/README.md) Flink application. This crucial step reads the enriched data from Kafka and writes it into the `apache_kickstarter.airlines.flight` Apache Iceberg table, seamlessly preparing your data for advanced analytics and insight generation.\n\n![datageneratorapp-flightimporterapp](.blog/images/datageneratorapp-flightimporterapp.png)\n\nGet ready to see the magic of Flink in action! Now, it's time to share those insights with the world! One fantastic way to do that is with [`Streamlit`](https://streamlit.io/), which allows you to easily create interactive visualizations. Streamlit is intuitive, powerful, and designed with Python developers in mind, making it a breeze to turn raw data into captivating dashboards. 😉  \n\n![iceberg-flink-streamlit-drawing](.blog/images/iceberg-flink-streamlit-drawing.png)\n\n**Table of Contents**\n\n\u003c!-- toc --\u003e\n+ [1.0 Power up the Apache Flink Docker containers](#10-power-up-the-apache-flink-docker-containers)\n+ [2.0 Supercharge Your Streamlit Visualizations](#20-supercharge-your-streamlit-visualizations)\n    - [2.1 Did you notice we prepended `uv run` to `streamlit run`?](#21-did-you-notice-we-prepended-uv-run-to-streamlit-run)\n+ [3.0 Local Integration: How This App Harnesses Apache Flink](#30-local-integration-how-this-app-harnesses-apache-flink)\n+ [4.0 Resources](#40-resources)\n\u003c!-- tocstop --\u003e\n\n## 1.0 Power up the Apache Flink Docker containers\n\n\u003e **Prerequisite**\n\u003e \n\u003e Before you can run `scripts/run-flink-locally.sh` Bash script, you need to install the [`aws2-wrap`](https://pypi.org/project/aws2-wrap/#description) utility.  If you have a Mac machine, run this command from your Terminal:\n\u003e ````bash\n\u003e brew install aws2-wrap\n\u003e ````\n\u003e\n\u003e If you are not using a Mac, make sure you have Python3.x installed on your machine, and run this command from your Terminal:\n\u003e ```bash\n\u003e pip install aws2-wrap\n\u003e ```\n\nThis section guides you through the local setup (on one machine but in separate containers) of the Apache Flink cluster in [Session mode](https://github.com/j3-signalroom/j3-techstack-lexicon/blob/main/apache-flink-glossary.md#flink-session-mode) using Docker containers with support for Apache Iceberg.  Run the `bash` script below to start the Apache Flink cluster in Session Mode on your machine:\n\n```bash\nscripts/run-flink-locally.sh \u003cDOCKER_SWITCH\u003e --profile=\u003cAWS_SSO_PROFILE_NAME\u003e\n                                             --chip=\u003camd64 | arm64\u003e\n                                             --aws-s3-bucket=\u003cAWS_S3_BUCKET_NAME\u003e\n```\n\u003e Argument placeholder|Replace with\n\u003e -|-\n\u003e `\u003cDOCKER_SWITCH\u003e`|`on` to start up your very own local Apache Cluster running in Docker containers, otherwise `off` to stop the Docker containers.\n\u003e `\u003cAWS_SSO_PROFILE_NAME\u003e`|your AWS SSO profile name for your AWS infrastructue that host your AWS Secrets Manager.\n\u003e `\u003cCHIP\u003e`|if you are running on a Mac with M1, M2, or M3 chip, use `arm64`.  Otherwise, use `amd64`.\n\u003e `\u003cAWS_S3_BUCKET_NAME\u003e`|specify the name of the AWS S3 bucket used to store Apache Iceberg files.\n\nTo learn more about this script, click [here](.blog/run-flink-locally-script-explanation.md).\n\n## 2.0 Supercharge Your Streamlit Visualizations\nTo access the Flink JobManager (`supercharge_streamlit-apache_flink-jobmanager-1`) container, open the interactive shell by running:\n\n```bash\ndocker exec -it -w /opt/flink/python_apps supercharge_streamlit-apache_flink-jobmanager-1 /bin/bash\n```\n\nJump right into the container and take charge! You’ll have full control to run commands, explore the file system, and tackle any tasks you need. You’ll land directly in the `/opt/flink/python_apps` directory—this is the headquarters for the Python script in the repo.\n\nTo illustrate, I created a Streamlit script that queries the `apache_kickstarter.airlines.flight` Apache Iceberg Table, harnessing Flink SQL to extract valuable insights. These insights are then brought to life through a Streamlit dashboard, transforming raw data into an accessible, visual experience.\n\nHere you go, run this in the docker container terminal command line:\n\n```bash\nuv run streamlit run app.py -- --aws-s3-bucket \u003cAWS_S3_BUCKET\u003e --aws-region \u003cAWS_REGION_NAME\u003e\n```\n\u003e _Notice the extra `--` between `streamlit run app.py` and the actual script arguments.  This is necessary to pass arguments to the Streamlit script without causing conflicts with Streamlit's own CLI options._\n\nWhen you run the script, for instance, it produces the following output:\n\n![streamlit-run-from-terminal-screenshot](.blog/images/streamlit-run-from-terminal-screenshot.png)\n\nOpen your host web browser, enter the local URL, `localhost:8501`, and in a few moments this web page will be displayed:\n\n![streamlit-screenshot](.blog/images/streamlit-screenshot.png)\n\n\u003e _**\"After many years in this industry, I’m still amazed by what we can achieve today!  The possibilities are endless—enjoy the ride!\"**_\n\u003e \n\u003e _**---J3**_\n\n### 2.1 Did you notice we prepended `uv run` to `streamlit run`?\nYou maybe asking yourself why.  Well, `uv` is an incredibly fast Python package installer and dependency resolver, written in [**Rust**](https://github.blog/developer-skills/programming-languages-and-frameworks/why-rust-is-the-most-admired-language-among-developers/), and designed to seamlessly replace `pip`, `pipx`, `poetry`, `pyenv`, `twine`, `virtualenv`, and more in your workflows. By prefixing `uv run` to a command, you're ensuring that the command runs in an optimal Python environment.\n\nNow, let's go a little deeper into the magic behind `uv run`:\n- When you use it with a file ending in `.py` or an HTTP(S) URL, `uv` treats it as a script and runs it with a Python interpreter. In other words, `uv run file.py` is equivalent to `uv run python file.py`. If you're working with a URL, `uv` even downloads it temporarily to execute it. Any inline dependency metadata is installed into an isolated, temporary environment—meaning zero leftover mess! When used with `-`, the input will be read from `stdin`, and treated as a Python script.\n- If used in a project directory, `uv` will automatically create or update the project environment before running the command.\n- Outside of a project, if there's a virtual environment present in your current directory (or any parent directory), `uv` runs the command in that environment. If no environment is found, it uses the interpreter's environment.\n\nSo what does this mean when we put `uv run` before `streamlit run`? It means `uv` takes care of all the setup—fast and seamless—right in your local Docker container. If you think AI/ML is magic, the work the folks at [Astral](https://astral.sh/) have done with `uv` is pure wizardry!\n\nCurious to learn more about [Astral](https://astral.sh/)'s `uv`? Check these out:\n- Documentation: Learn about [`uv`](https://docs.astral.sh/uv/).\n- Video: [`uv` IS THE FUTURE OF PYTHON PACKING!](https://www.youtube.com/watch?v=8UuW8o4bHbw).\n\n## 3.0 Local Integration: How This App Harnesses Apache Flink\nThis [`app.py`](app.py) Python script streamlines Apache Flink integration by leveraging PyFlink directly in the [`app.py`](app.py) script. Running the Flink job using the same Python process as the [`Streamlit`](https://streamlit.io/) app offers an intuitive setup for local testing and debugging. It's an ideal solution for quickly iterating on data processing logic without additional infrastructure.\n\n_However, a more advanced setup is recommended for production environments where scalability is critical. Running Flink jobs in a separate process and enabling communication through Kafka or a REST API provides greater scalability, albeit at the cost of added complexity._\n\nThe simplicity of this app’s current approach makes it perfect for prototyping and development while offering a foundation to explore more sophisticated integration methods as your requirements grow.\n\n## 4.0 Resources\n\n[Flink Python Docs](https://nightlies.apache.org/flink/flink-docs-master/api/python/)\n\n[PyFlink API Reference](https://nightlies.apache.org/flink/flink-docs-release-1.20/api/python/reference/index.html)\n\n[Apache Iceberg in Action with Apache Flink using Python](https://github.com/j3-signalroom/apache_flink-kickstarter/blob/main/.blog/apache-iceberg-in-action-with-apache-flink-using-python.md)\n\n[Streamlit Documentation](https://docs.streamlit.io/)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fj3-signalroom%2Fsupercharge_streamlit-apache_flink","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fj3-signalroom%2Fsupercharge_streamlit-apache_flink","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fj3-signalroom%2Fsupercharge_streamlit-apache_flink/lists"}