{"id":19819786,"url":"https://github.com/dongminlee94/data-pipeline-kafka-docker","last_synced_at":"2026-04-12T11:42:24.619Z","repository":{"id":172949644,"uuid":"649230594","full_name":"dongminlee94/data-pipeline-kafka-docker","owner":"dongminlee94","description":"A data pipeline example using Kafka on Docker","archived":false,"fork":false,"pushed_at":"2023-07-08T05:30:29.000Z","size":928,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-11-27T09:17:03.584Z","etag":null,"topics":["data-pipeline","docker","docker-compose","glue","kafka","kafka-cluster","stream-glue"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dongminlee94.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-06-04T07:48:52.000Z","updated_at":"2023-10-04T12:03:53.000Z","dependencies_parsed_at":null,"dependency_job_id":"3ecf9636-c28f-4e1c-835c-f209406ca7b5","html_url":"https://github.com/dongminlee94/data-pipeline-kafka-docker","commit_stats":null,"previous_names":["dongminlee94/data-pipeline-kafka-docker"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/dongminlee94/data-pipeline-kafka-docker","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dongminlee94%2Fdata-pipeline-kafka-docker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dongminlee94%2Fdata-pipeline-kafka-docker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dongminlee94%2Fdata-pipeline-kafka-docker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dongminlee94%2Fdata-pipeline-kafka-docker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dongminlee94","download_url":"https://codeload.github.com/dongminlee94/data-pipeline-kafka-docker/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dongminlee94%2Fdata-pipeline-kafka-docker/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31713876,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-12T06:22:27.080Z","status":"ssl_error","status_checked_at":"2026-04-12T06:21:52.710Z","response_time":58,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-pipeline","docker","docker-compose","glue","kafka","kafka-cluster","stream-glue"],"created_at":"2024-11-12T10:19:57.395Z","updated_at":"2026-04-12T11:42:24.571Z","avatar_url":"https://github.com/dongminlee94.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Data Pipeline Kafka Docker\n\n[![License: Apache 2.0](https://img.shields.io/badge/license-Apache--2.0-green.svg)](https://opensource.org/licenses/Apache-2.0)\n[![Python 3.10](https://img.shields.io/badge/python-3.10-blue.svg)](https://www.python.org/downloads/release/python-3100)\n[![pdm: managed](https://img.shields.io/badge/pdm-managed-blueviolet)](https://pdm.fming.dev)\n[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)\n[![Imports: isort](https://img.shields.io/badge/imports-isort-white)](https://pycqa.github.io/isort)\n[![Type Checking: mypy](https://img.shields.io/badge/type%20checking-mypy-red)](https://github.com/python/mypy)\n[![Linting: ruff](https://img.shields.io/badge/linting-ruff-purple)](https://github.com/astral-sh/ruff)\n\n\u003cp align=\"center\"\u003e\u003cimg src=\"asset/landscape.png\" width=\"800\"\u003e\u003c/p\u003e\n\n## Prerequisites\n\n- Install [Docker](https://docs.docker.com/engine/install/).\n\n## Preparation\n\nInstall [Python 3.10](https://www.python.org/downloads/release/python-3100/) on [Pyenv](https://github.com/pyenv/pyenv#installation) or [Anaconda](https://docs.anaconda.com/anaconda/install/index.html) and execute the following commands:\n\n```bash\n$ make init             # setup packages (need only once)\n```\n\n## How To Play\n\n### 1. Create Infra\n\n```bash\n$ make compose          # create all the containers\n```\n\nYou can delete the containers.\n\n```bash\n$ make compose-clean    # delete the containers\n```\n\nYou can divide the containers and create them.\n\n```bash\n$ make db               # create a postgres (source DB), a minio (target DB), a data generator, and a bucket creator (need only once)\n$ make kafka            # create a kafka cluster (need only once)\n$ make glue             # create a glue process that ETLs data (need only once)\n```\n\nYou can delete the divided containers.\n\n```bash\n$ make glue-clean       # delete the glue process\n$ make kafka-clean      # delete the kafka cluster\n$ make db-clean         # delete the databases and the rest.\n```\n\n### 2. Check Source DB\n\nYou can access the source DB (PostgreSQL DB) and check that the data is being added.\n\nYou can check the data in source DB by installing `psql`.\n\n```bash\n# MacOS\n$ brew install libpq\n\n# Ubuntu \u0026 Debian\n$ apt-get install postgresql-client\n```\n\nYou can use the `psql` command to access the source DB below.\n\n```bash\n$ PGPASSWORD=postgres psql -h localhost -p 5432 -U postgres -d postgres\npsql (15.3, server 14.0 (Debian 14.0-1.pgdg110+1))\nType \"help\" for help.\n\npostgres=# \\d\n                List of relations\n Schema |       Name       |   Type   |  Owner\n--------+------------------+----------+----------\n public | iris_data        | table    | postgres\n public | iris_data_id_seq | sequence | postgres\n(2 rows)\n\npostgres=# select * from iris_data limit 5;\n id |      timestamp      | sepal_length | sepal_width | petal_length | petal_width | target\n----+---------------------+--------------+-------------+--------------+-------------+--------\n  1 | 2023-06-19 00:22:30 |          7.6 |           3 |          6.6 |         2.1 |      2\n  2 | 2023-06-19 00:22:32 |            5 |         3.2 |          1.2 |         0.2 |      0\n  3 | 2023-06-19 00:22:34 |          6.7 |           3 |          5.2 |         2.3 |      2\n  4 | 2023-06-19 00:22:36 |          6.7 |         3.1 |          4.4 |         1.4 |      1\n  5 | 2023-06-19 00:22:38 |          5.9 |         3.2 |          4.8 |         1.8 |      1\n(5 rows)\n```\n\n### 3. Check Kafka Topics\n\nYou can access `localhost:9009` on a web browser and see the topics in Kafka through Kafdrop.\n\n\u003cp align=\"center\"\u003e\u003cimg src=\"asset/kafdrop_main_screen.png\" width=\"800\"\u003e\u003c/p\u003e\n\nYou can click the `iris_data` topic on the main screen of Kafdrop and click `View Messages` to see the data being produced to the `iris_data` topic through the JDBC source connector.\n\n\u003cp align=\"center\"\u003e\u003cimg src=\"asset/kafdrop_topic_iris_data.png\" width=\"800\"\u003e\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\u003cimg src=\"asset/kafdrop_topic_iris_data_topic_messages.png\" width=\"800\"\u003e\u003c/p\u003e\n\nSimilarly, you can click the `new_iris_data` topic on the main screen and click `View Messages` to see the data being produced to the `new_iris_data` topic through the Glue process.\n\n\u003cp align=\"center\"\u003e\u003cimg src=\"asset/kafdrop_topic_new_iris_data.png\" width=\"800\"\u003e\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\u003cimg src=\"asset/kafdrop_topic_new_iris_data_topic_messages.png\" width=\"800\"\u003e\u003c/p\u003e\n\n### 4. Check Target DB\n\nYou can access `localhost:9900` on the web browser and see the data being consumed in MinIO.\n\n\u003cp align=\"center\"\u003e\u003cimg src=\"asset/minio_main_screen.png\" width=\"800\"\u003e\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\u003cimg src=\"asset/minio_data_check.png\" width=\"800\"\u003e\u003c/p\u003e\n\nYon can download and see the data, and you can check that it is in a different type from the data in the source DB.\n\n```json\n{\n  \"id\":12,\n  \"timestamp\":\"2023-06-19 10:32:56\",\n  \"sepal_length\":6.1,\n  \"sepal_width\":3.0,\n  \"sepal_sum\":9.1,\n  \"sepal_mean\":4.55,\n  \"petal_length\":4.9,\n  \"petal_width\":1.8,\n  \"petal_sum\":6.7,\n  \"petal_mean\":3.35,\n  \"target\":2\n}\n```\n\nAs shown in the landscape, you can confirm that the Glue process consumes the data in the `iris_data` topic, updates the type and value of the data, and then produces it to the `new_iris_data` topic.\n\nLet's make the data pipeline using Kafka together with this repository!\n\n## For Developers\n\n```bash\n$ make check          # run all static analyses\n$ make format         # format scripts\n$ make lint           # lint scripts\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdongminlee94%2Fdata-pipeline-kafka-docker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdongminlee94%2Fdata-pipeline-kafka-docker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdongminlee94%2Fdata-pipeline-kafka-docker/lists"}