{"id":24396243,"url":"https://github.com/abeltavares/real-time-data-pipeline","last_synced_at":"2025-10-13T18:32:17.604Z","repository":{"id":272126326,"uuid":"915481370","full_name":"abeltavares/real-time-data-pipeline","owner":"abeltavares","description":"📡 Real-time data pipeline with Kafka, Flink, Iceberg, Trino, MinIO, and Superset. Ideal for learning data systems.","archived":false,"fork":false,"pushed_at":"2025-01-18T18:52:26.000Z","size":1033,"stargazers_count":41,"open_issues_count":0,"forks_count":3,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-29T18:02:58.832Z","etag":null,"topics":["apache-flink","apache-iceberg","apache-kafka","apache-superset","aws","big-data","data-engineering","data-pipeline","data-visualization","docker","etl","lakehouse","minio","open-source","real-time-data","s3","sql-analytics","streaming-analytics","trino"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/abeltavares.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-01-12T00:22:41.000Z","updated_at":"2025-03-28T11:21:25.000Z","dependencies_parsed_at":"2025-01-12T11:17:57.038Z","dependency_job_id":"af569f1f-69ec-4b69-907e-4329f2b84333","html_url":"https://github.com/abeltavares/real-time-data-pipeline","commit_stats":null,"previous_names":["abeltavares/e2e-real-time-data-pipeline"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abeltavares%2Freal-time-data-pipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abeltavares%2Freal-time-data-pipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abeltavares%2Freal-time-data-pipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abeltavares%2Freal-time-data-pipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/abeltavares","download_url":"https://codeload.github.com/abeltavares/real-time-data-pipeline/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250306601,"owners_count":21408927,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-flink","apache-iceberg","apache-kafka","apache-superset","aws","big-data","data-engineering","data-pipeline","data-visualization","docker","etl","lakehouse","minio","open-source","real-time-data","s3","sql-analytics","streaming-analytics","trino"],"created_at":"2025-01-19T21:25:46.086Z","updated_at":"2025-10-13T18:32:12.560Z","avatar_url":"https://github.com/abeltavares.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"**E2E Real-Time Data Pipeline with Kafka, Flink, Iceberg, Trino, MinIO, and Superset**\n======================================================================================\n\n![Docker](https://img.shields.io/badge/Docker-Enabled-blue?logo=docker)\n![Apache Kafka](https://img.shields.io/badge/Apache%20Kafka-Event%20Streaming-black?logo=apachekafka)\n![Apache Flink](https://img.shields.io/badge/Apache%20Flink-Real%20Time%20Processing-orange?logo=apacheflink)\n![Apache Iceberg](https://img.shields.io/badge/Apache%20Iceberg-Table%20Format-blue?logo=apache)\n![Trino](https://img.shields.io/badge/Trino-SQL%20Query%20Engine-green?logo=trino)\n![Apache Superset](https://img.shields.io/badge/Apache%20Superset-Visualization-ff69b4?logo=apache)\n\n\n**📖 Overview**\n---------------\n\nThis project demonstrates a **real-time end-to-end (E2E) data pipeline** designed to handle clickstream data. It shows how to ingest, process, store, query, and visualize streaming data using open-source tools, all containerized with Docker for easy deployment.\n\n🔎 **Technologies Used:**\n\n-   **Data Ingestion:** [Apache Kafka](https://kafka.apache.org/)  \n-   **Stream Processing:** [Apache Flink](https://flink.apache.org/)  \n-   **Object Storage:** [MinIO (S3-compatible)](https://min.io/)\n-   **Data Lake Table Format:** [Apache Iceberg](https://iceberg.apache.org/)  \n-   **Query Engine:** [Trino](https://trino.io/)  \n-   **Visualization:** [Apache Superset](https://superset.apache.org/)    \n\n\nThis pipeline is perfect for **data engineers** and **students** interested in learning how to design real-time data systems.\n\n* * * * *\n\n**🏗  Architecture**\n-----------------------------------\n\n![Architecture Diagram](img/e2e-pipeline.png)\n\n1.  **Clickstream Data Generator** simulates real-time user events and pushes them to **Kafka** topic.\n2.  **Apache Flink** processes Kafka streams and writes clean data to **Iceberg tables** stored on **MinIO**.\n3.  **Trino** connects to Iceberg for querying the processed data.\n4.  **Apache Superset** visualizes the data by connecting to Trino.\n\n\n🛠 **Tech Stack**\n-----------------\n\n| **Component**       | **Technology**                                                                 | **Purpose**                                     |\n|--------------------|-------------------------------------------------------------------------------|-------------------------------------------------|\n| **Data Generator**  | [Python (Faker)](https://faker.readthedocs.io/)                              | Simulate clickstream events                      |\n| **Data Ingestion**  | [Apache Kafka](https://kafka.apache.org/)                                    | Real-time event streaming                        |\n| **Coordination Service** | [Apache ZooKeeper](https://zookeeper.apache.org/) | Kafka broker coordination and metadata management |\n| **Stream Processing** | [Apache Flink](https://flink.apache.org/)                                  | Real-time data processing and transformation     |\n| **Data Lake Storage** | [Apache Iceberg](https://iceberg.apache.org/)                               | Data storage and schema management              |\n| **Object Storage**  | [MinIO](https://min.io/)                                                      | S3-compatible storage for Iceberg tables         |\n| **Query Engine**    | [Trino](https://trino.io/)                                                    | Distributed SQL querying on Iceberg data         |\n| **Visualization**   | [Apache Superset](https://superset.apache.org/)                               | Interactive dashboards and data visualization    |\n\n* * * * *\n\n\n**📦 Project Structure**\n------------------------\n\n```bash\ne2e-data-pipeline/\n├── docker-compose.yml   # Docker setup for all services\n├── flink/               # Flink SQL client and streaming jobs\n├── producer/            # Clickstream data producer using Faker\n├── superset/            # Superset setup and configuration\n└── trino/               # Trino configuration for Iceberg \n```\n\n* * * * *\n\n**🔧 Setup Instructions**\n-------------------------\n\n### **1\\. Prerequisites**\n\n-   **Docker** and **Docker Compose** installed.\n-   Minimum **16GB RAM** recommended.\n\n### **2\\. Clone the Repository**\n\n```bash\ngit clone https://github.com/abeltavares/real-time-data-pipeline.git\ncd real-time-data-pipeline\n```\n\n### **3\\. Start All Services**\n\n```bash\ndocker-compose up -d\n```\n\n⚠️ **Note:** All components (Kafka, Flink, Iceberg, Trino, MinIO, and Superset) are containerized using Docker for easy deployment and scalabilit\n\n### **4\\. Access the Services**\n\n| **Service** | **URL** | **Credentials** |\n| --- | --- | --- |\n| **Kafka Control Center** | `http://localhost:9021` | *No Auth* |\n| **Flink Dashboard** | `http://localhost:18081` | *No Auth* |\n| **MinIO Console** | `http://localhost:9001` | `admin` / `password` |\n| **Trino UI** | `http://localhost:8080/ui` | *No Auth* |\n| **Superset** | `http://localhost:8088` | `admin` / `admin` |\n\n\n📥 **Data Ingestion**\n---------------------\n\n### 1\\. **Clickstream Data Generation**\n\nClickstream events are simulated using Python's **Faker** library. Here's the event structure:\n\n```python\n{\n  \"event_id\": fake.uuid4(),\n  \"user_id\": fake.uuid4(),\n  \"event_type\": fake.random_element(elements=(\"page_view\",        \"add_to_cart\", \"purchase\", \"logout\")),\n  \"url\": fake.uri_path(),\n  \"session_id\": fake.uuid4(),\n  \"device\": fake.random_element(elements=(\"mobile\", \"desktop\", \"tablet\")),\n  \"timestamp\": time.strftime(\"%Y-%m-%dT%H:%M:%SZ\", time.gmtime()),\n  \"geo_location\": {\n      \"lat\": float(fake.latitude()),\n      \"lon\": float(fake.longitude())\n  },\n  \"purchase_amount\": float(random.uniform(0.0, 500.0)) if fake.boolean(chance_of_getting_true=30) else None\n}\n```\n\n⚠️ **Note:** The **Clickstream Producer** runs automatically when Docker Compose is up. No manual execution is needed.\n\n### 2\\. **Kafka Consumer**\n\nThe Kafka consumer reads the clickstream events and pushes them to **Apache Flink** for real-time processing.\n\nYou can monitor the Kafka topic through the **Kafka Control Center**:\n\n-   **Kafka Control Center URL:** \u003chttp://localhost:9021\u003e\n\n![Kafka Topic](img/topic-clickstream.png)\n\n* * * * *\n\n⚡ **Real-Time Data Processing with Apache Flink**\n-------------------------------------------------\n\n### 1\\. **Flink Configuration**\n\n-   **State Backend:** RocksDB\n-   **Checkpointing:** Enabled for fault tolerance\n-   **Connectors:** Kafka → Iceberg (via Flink SQL)\n\n### 2\\. **Flink SQL Job Execution**\n\nThe `sql-client` service in Docker Compose automatically submits the Flink SQL job after the JobManager and TaskManager are running. It uses the `clickstream-filtering.sql` script to process Kafka streams and write to Iceberg.\n\n```bash\n/opt/flink/bin/sql-client.sh -f /opt/flink/clickstream-filtering.sql\n```\n\n### 2\\. **Flink Dashboard**\n\nMonitor real-time data processing jobs at:\\\n📊 http://localhost:18081\n\n![Flink Job](img/flink-job.png)\n\n* * * * *\n\n🗄️ **Data Lakehouse with Apache Iceberg**\n------------------------------------------\n\nProcessed data from Flink is stored in **Iceberg tables** on **MinIO**. This enables:\n\n-   **Efficient Querying** with Trino\n-   **Schema Evolution** and **Time Travel**\n\nTo list the contents of the MinIO warehouse, you can use the following command:\n\n```bash\ndocker exec mc bash -c \"mc ls -r minio/warehouse/\"\n```\n\nAlternatively, you can access the MinIO console via the web at \u003chttp://localhost:9001\u003e.\n\n-   **Username:** `admin`\n-   **Password:** `password`\n\n![Warehouse Bucket](img/warehouse-bucket.png)\n\n**🔍 Query Data with Trino**\n----------------------------\n\n **1\\. Run Trino CLI**\n\n```bash\ndocker-compose exec trino trino\n```\n\n**2\\. Connect to Iceberg Catalog**\n\n```sql\nUSE iceberg.db;\n```\n\n**3\\. Query Processed Data**\n\n```sql\nSELECT * FROM iceberg.db.clickstream_sink\nWHERE purchase_amount \u003e 100\nLIMIT 10;\n```\n\n![Trino Query](img/trino-query.png)\n\n📊 **Data Visualization with Apache Superset**\n----------------------------------------------\n\n1.  **Access Superset:** \u003chttp://localhost:8088\u003e\n\n    -   **Username:** `admin`\n    -   **Password:** `admin`\n2.  **Connect Superset to Trino:**\n\n-   **SQLAlchemy URI:**\n\n    ```bash\n    trino://trino@trino:8080/iceberg/db\n    ```\n-   **Configure in Superset:**\n\n    1.  Open `http://localhost:8088`\n    2.  Go to **Data** → **Databases** → **+**\n    3.  Use the above SQLAlchemy URI.\n\n3.  **Create Dashboards:**\n\n![Superset](img/superset_dashboard.png)\n\n🏆 **Key Features**\n-------------------\n\n### 🔄 **Real-Time Data Processing**\n\n-   Stream processing with **Apache Flink**.\n-   Clickstream events are transformed and filtered in real-time.\n\n### 📂 **Modern Data Lakehouse**\n\n-   Data is stored in **Apache Iceberg** on **MinIO**, S3 compatible, supporting schema evolution and time travel.\n\n### ⚡ **Fast SQL Analytics**\n\n-   **Trino** provides fast, distributed SQL queries on Iceberg data.\n\n### 📊 **Interactive Dashboards**\n\n-   **Apache Superset** delivers real-time visual analytics.\n\n### 📦 **Fully Containerized Setup**\n\n-   Simplified deployment using **Docker** and **Docker Compose** for seamless integration across all services.\n\n* * * * *\n\n📈 **Future Enhancements**\n--------------------------\n\n-   Implement **alerting** and **monitoring** with **Grafana** and **Prometheus**.\n-   Introduce **machine learning pipelines** for predictive analytics.\n-   Optimize **Iceberg partitioning** for faster queries.\n\n* * * * *\n\n📎 **Quick Reference Commands**\n-------------------------------\n\n| **Component** | **Command** |\n| --- | --- |\n| **Start Services** | `docker-compose up --build -d` |\n| **Stop Services** | `docker-compose down` |\n| **View Running Containers** | `docker ps` |\n| **Check Logs** | `docker-compose logs -f` |\n| **Rebuild Containers** | `docker-compose up --build --force-recreate -d` |\n\n* * * * *\n\n🙌 **Get Involved**\n-------------------\n\nContributions are welcome! Feel free to submit issues or pull requests to improve this project.\n\n* * * * *\n\n📜 License\n--------------\n\nThis project is licensed under the [MIT License](LICENSE).\n\n* * * * *\n\nEnjoy exploring real-time data pipelines!","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabeltavares%2Freal-time-data-pipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fabeltavares%2Freal-time-data-pipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabeltavares%2Freal-time-data-pipeline/lists"}