{"id":20638447,"url":"https://github.com/ahmetfurkandemir/flink-example","last_synced_at":"2025-08-20T00:20:07.157Z","repository":{"id":208081651,"uuid":"720408701","full_name":"AhmetFurkanDEMIR/Flink-Example","owner":"AhmetFurkanDEMIR","description":"Flink Example","archived":false,"fork":false,"pushed_at":"2023-11-19T14:22:49.000Z","size":5868,"stargazers_count":15,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-15T21:52:07.533Z","etag":null,"topics":["apache","apache-flink","data-engineering","data-stream-processing","debezium","debezium-client","debezium-connector","debeziumkafkaconnector","docker","flink","flink-sql","flink-stream-processing","flink-streaming","kafka","kafka-streams","scala","scala2","ubuntu"],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AhmetFurkanDEMIR.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-11-18T11:54:58.000Z","updated_at":"2025-03-20T00:31:48.000Z","dependencies_parsed_at":"2023-11-19T16:42:06.532Z","dependency_job_id":null,"html_url":"https://github.com/AhmetFurkanDEMIR/Flink-Example","commit_stats":null,"previous_names":["ahmetfurkandemir/flink-example"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/AhmetFurkanDEMIR/Flink-Example","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AhmetFurkanDEMIR%2FFlink-Example","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AhmetFurkanDEMIR%2FFlink-Example/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AhmetFurkanDEMIR%2FFlink-Example/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AhmetFurkanDEMIR%2FFlink-Example/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AhmetFurkanDEMIR","download_url":"https://codeload.github.com/AhmetFurkanDEMIR/Flink-Example/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AhmetFurkanDEMIR%2FFlink-Example/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271242759,"owners_count":24725109,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-19T02:00:09.176Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache","apache-flink","data-engineering","data-stream-processing","debezium","debezium-client","debezium-connector","debeziumkafkaconnector","docker","flink","flink-sql","flink-stream-processing","flink-streaming","kafka","kafka-streams","scala","scala2","ubuntu"],"created_at":"2024-11-16T15:18:03.110Z","updated_at":"2025-08-20T00:20:06.989Z","avatar_url":"https://github.com/AhmetFurkanDEMIR.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"![](https://img.shields.io/badge/Docker-2CA5E0?style=for-the-badge\u0026logo=docker\u0026logoColor=white) ![](https://img.shields.io/badge/PostgreSQL-316192?style=for-the-badge\u0026logo=postgresql\u0026logoColor=white) ![](https://img.shields.io/badge/Scala-DC322F?style=for-the-badge\u0026logo=scala\u0026logoColor=white) ![](https://img.shields.io/badge/Apache_Kafka-231F20?style=for-the-badge\u0026logo=apache-kafka\u0026logoColor=white)\n\n# Flink Example\n\n![](/images/schema.png)\n\nProject to create flink data architecture with products previously taken from [Trendyol.com](https://www.trendyol.com/) e-commerce site.\n\nDataset link: [https://www.kaggle.com/datasets/ahmetfurkandemr/trendyol-product-comments](https://www.kaggle.com/datasets/ahmetfurkandemr/trendyol-product-comments)\n\n[**Apache Flink**](https://flink.apache.org/) is an open-source, unified stream-processing and batch-processing framework developed by the Apache Software Foundation. The core of Apache Flink is a distributed streaming data-flow engine written in Java and Scala. Flink executes arbitrary dataflow programs in a data-parallel and pipelined (hence task parallel) manner. Flink's pipelined runtime system enables the execution of bulk/batch and stream processing programs. Furthermore, Flink's runtime supports the execution of iterative algorithms natively.\n\n## **Run the project**\n\nFirst, it is necessary to start the docker services by running the start.sh file.\n\n```bash\nsudo bash start.sh\n```\n\nAfter running this command, all services will start. The services automatically connect and data is entered into the postgresql database, then the data is written to the kafka topic with debezium.\n\n* **PostgreSQL | postgresql:5432 or 0.0.0.0:5432**\n\n    - POSTGRES_DB=postgres\n    - POSTGRES_USER=demir\n    - POSTGRES_PASSWORD=demir*/895/\n\n\n* **Pgadmin | [0.0.0.0:8888](http://0.0.0.0:8888)**\n\n    - PGADMIN_DEFAULT_EMAIL: demir@ahmetfurkandemir.com\n    - PGADMIN_DEFAULT_PASSWORD: demir*/895/\n\n    ![](/images/pgadmin.png)\n\n\n* **Debezium Ui | [0.0.0.0:8086](http://0.0.0.0:8086/)**\n\n    ![](/images/debezium_ui.png)\n\n\n* **Kafka Ui | [0.0.0.0:8080](http://0.0.0.0:8080)**\n\n    ![](/images/kafka_ui.png)\n\n    - If you do not receive any errors, you can see that data is flowing to [debezium_connector_table.public.trendyol_product](http://0.0.0.0:8080/ui/clusters/kafka/all-topics/debezium_connector_table.public.trendyol_product/messages?keySerde=String\u0026valueSerde=String\u0026limit=100) topic by debezium.\n\n### **Flink Run**\n\n**Flink Ui | [0.0.0.0:8081](http://0.0.0.0:8081)**\n\n![](/images/flink_ui.png)\n\nFlink starts up automatically like other services, but we need to make some configurations in Flink.\n\nFirst, download the jar file from the link below to the [/flink-jar](/flink-jar/) folder.\n\nLink: [https://github.com/AhmetFurkanDEMIR/Flink-Example/releases/download/jar/untitled-1.0-SNAPSHOT-jar-with-dependencies.jar](https://github.com/AhmetFurkanDEMIR/Flink-Example/releases/download/jar/untitled-1.0-SNAPSHOT-jar-with-dependencies.jar)\n\nThis .jar file reads the data in the kafka topic and simplifies this data, makes it useful and prints it on a new topic. [source codes of the file](/flink-scala-source/).\n\nFind the container address of Flink jobmanager with the following command.\n\n```bash\ndocker ps\n```\n\nAnd again, enter the container with the following command.\n\n```bash\ndocker exec -it xxxxxx /bin/bash\n```\n\n```bash\nflink run -c org.example.App /flink-example/untitled-1.0-SNAPSHOT-jar-with-dependencies.jar\n```\n\nIf the command runs successfully, you can see the job running via Flink Ui as shown in the picture below. And never close this window. [Check link](http://0.0.0.0:8081/#/job/running)\n\n![](/images/flink_run_jar.png)\n\nAfter the job runs without errors, you need to insert the data in the sql file into the database to ensure data flow. [SQL file](/sql/insert_data.sql)\n\n```bash\npsql -h 0.0.0.0 -d postgres -U demir -f sql/insert_data.sql\n```\n\nAfter the data is inserted, you can see that a new topic named [debezium_connector_table.public.trendyol_product-clear-out](http://0.0.0.0:8080/ui/clusters/kafka/all-topics/debezium_connector_table.public.trendyol_product-clear-out/messages?keySerde=String\u0026valueSerde=String\u0026limit=100) has been opened on Kafka.\n\nAfter these operations, open a new terminal and enter the Flink Jobmanager container, which is the process you did before.\n\n```bash\ndocker exec -it xxxxxx /bin/bash\n```\n\nThen run Flink SQL Client.\n\n```bash\n./bin/sql-client.sh -j /flink-example/flink-sql-connector-kafka-3.0.1-1.18.jar\n```\n\nIf everything is OK, you should connect to a command interface like the one below.\n\n![](/images/flink_sql_client.png)\n\nCreate a table from the kafka topic with the following command in Flink SQL.\n\n```sql\nCREATE TABLE trendyol_product_flink_table(\n  `Product_Id` STRING,\n  `Product_Name` STRING,\n  `Product_Brand` STRING,\n  `Product_Link` STRING\n) WITH (\n  'connector' = 'kafka',\n  'topic' = 'debezium_connector_table.public.trendyol_product-clear-out',\n  'properties.bootstrap.servers' = 'kafka:9092',\n  'properties.group.id' = 'demir-flink-sql',\n  'scan.startup.mode' = 'earliest-offset',\n  'format' = 'json'\n);\n```\n\nAnd finally, run an SQL query on this table and watch with pleasure as your data flows :)\n\n```sql\nselect * from trendyol_product_flink_table;\n```\n\n![](/images/sql_1.png)\n\n![](/images/sql_2.png)\n\nCheck the jobs and SQL commands you run from the interface.\n\n![](/images/flink_run_jobs.png)\n\n[**Ahmet Furkan Demir**](https://ahmetfurkandemir.com/)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fahmetfurkandemir%2Fflink-example","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fahmetfurkandemir%2Fflink-example","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fahmetfurkandemir%2Fflink-example/lists"}