{"id":15208146,"url":"https://github.com/chukwuemekaaham/data-engineering-zoomcamp","last_synced_at":"2026-02-24T06:34:02.318Z","repository":{"id":247904293,"uuid":"826020761","full_name":"ChukwuemekaAham/data-engineering-zoomcamp","owner":"ChukwuemekaAham","description":"Datatalks Club Free Data Engineering Zoomcamp Project","archived":false,"fork":false,"pushed_at":"2024-07-14T06:02:25.000Z","size":4857,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-06T17:15:41.508Z","etag":null,"topics":["bigquery","dbt","docker-compose","duckdb","gcp","gcp-cloud-storage","github-actions","jupyter-notebook","kafka","linux","looker-studio","mageai","pandas","postgresql","prefect","python","redpanda","risingwave","spark","terraform"],"latest_commit_sha":null,"homepage":"https://github.com/ChukwuemekaAham/uber-gcp-etl-project","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ChukwuemekaAham.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-09T00:53:43.000Z","updated_at":"2024-07-14T06:02:27.000Z","dependencies_parsed_at":"2024-07-14T06:37:40.578Z","dependency_job_id":null,"html_url":"https://github.com/ChukwuemekaAham/data-engineering-zoomcamp","commit_stats":{"total_commits":10,"total_committers":1,"mean_commits":10.0,"dds":0.0,"last_synced_commit":"c5b35d477467d3e6774c9bd2b16d08bd5f3ad140"},"previous_names":["chukwuemekaaham/data-engineering-zoomcamp"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ChukwuemekaAham%2Fdata-engineering-zoomcamp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ChukwuemekaAham%2Fdata-engineering-zoomcamp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ChukwuemekaAham%2Fdata-engineering-zoomcamp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ChukwuemekaAham%2Fdata-engineering-zoomcamp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ChukwuemekaAham","download_url":"https://codeload.github.com/ChukwuemekaAham/data-engineering-zoomcamp/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242250926,"owners_count":20096897,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bigquery","dbt","docker-compose","duckdb","gcp","gcp-cloud-storage","github-actions","jupyter-notebook","kafka","linux","looker-studio","mageai","pandas","postgresql","prefect","python","redpanda","risingwave","spark","terraform"],"created_at":"2024-09-28T07:01:03.244Z","updated_at":"2025-10-25T06:40:27.574Z","avatar_url":"https://github.com/ChukwuemekaAham.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Data Engineering Zoomcamp\n\n[Final Project Here \u003e\u003e](https://github.com/ChukwuemekaAham/uber-gcp-etl-project)\n\n![Datatalks-Club-data-engineering-zoomcamp](https://github.com/ChukwuemekaAham/data-engineering-zoomcamp/blob/main/Datatalks-Club-data-engineering-zoomcamp.png)\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://airtable.com/shr6oVXeQvSI5HuWD\"\u003e\u003cimg src=\"https://user-images.githubusercontent.com/875246/185755203-17945fd1-6b64-46f2-8377-1011dcb1a444.png\" height=\"50\" /\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n- Register in [DataTalks.Club's Slack](https://datatalks.club/slack.html)\n- Join the [`#course-data-engineering`](https://app.slack.com/client/T01ATQK62F8/C01FABYF2RG) channel\n- Join the [course Telegram channel with announcements](https://t.me/dezoomcamp)\n- The videos are published on [DataTalks.Club's YouTube channel](https://www.youtube.com/c/DataTalksClub) in [the course playlist](https://www.youtube.com/playlist?list=PL3MmuxUbc_hJed7dXYoJw8DoCuVHhGEQb)\n- [Frequently asked technical questions](https://docs.google.com/document/d/19bnYs80DwuUimHM65UV3sylsCn2j1vziPOwzBwQrebw/edit?usp=sharing)\n\nSyllabus\n\n* [Module 1: Containerization and Infrastructure as Code](#module-1-containerization-and-infrastructure-as-code)\n* [Module 2: Workflow Orchestration](#module-2-workflow-orchestration)\n* [Workshop 1: Data Ingestion](#workshop-1-data-ingestion)\n* [Module 3: Data Warehouse](#module-3-data-warehouse)\n* [Module 4: Analytics Engineering](#module-4-analytics-engineering)\n* [Module 5: Batch processing](#module-5-batch-processing)\n* [Module 6: Streaming](#module-6-streaming)\n* [Workshop 2: Stream Processing with SQL](#workshop-2-stream-processing-with-sql)\n* [Project](#project)\n\n## Taking the course\n\n### 2024 Cohort\n\n* **Start**: 15 January 2024 (Monday) at 17:00 CET\n* **Registration link**: https://airtable.com/shr6oVXeQvSI5HuWD\n* [Cohort folder](cohorts/2024/) with homeworks and deadlines\n* [Launch stream with course overview](https://www.youtube.com/live/AtRhA-NfS24?si=5JzA_E8BmJjiLi8l)\n\n\n### Self-paced mode\n\nAll the materials of the course are freely available, so that you\ncan take the course at your own pace\n\n* Follow the suggested syllabus (see below) week by week\n* You don't need to fill in the registration form. Just start watching the videos and join Slack\n* Check [FAQ](https://docs.google.com/document/d/19bnYs80DwuUimHM65UV3sylsCn2j1vziPOwzBwQrebw/edit?usp=sharing) if you have problems\n* If you can't find a solution to your problem in FAQ, ask for help in Slack\n\n\n## Syllabus\n\n\u003e **Note:** NYC TLC changed the format of the data we use to parquet.\n\u003e In the course we still use the CSV files accessible [here](https://github.com/DataTalksClub/nyc-tlc-data).\n\n\n### [Module 1: Containerization and Infrastructure as Code](01-docker-terraform/)\n\n* Course overview\n* Introduction to GCP\n* Docker and docker-compose\n* Running Postgres locally with Docker\n* Setting up infrastructure on GCP with Terraform\n* Preparing the environment for the course\n* Homework\n\n[More details](01-docker-terraform/)\n\n\n### [Module 2: Workflow Orchestration](02-workflow-orchestration/)\n\n* Data Lake\n* Workflow orchestration\n* Workflow orchestration with Mage\n* Homework\n\n[More details](02-workflow-orchestration/)\n\n\n### [Workshop 1: Data Ingestion](cohorts/2024/workshops/dlt.md)\n\n* Reading from apis\n* Building scalable pipelines\n* Normalising data\n* Incremental loading\n* Homework\n\n\n[More details](cohorts/2024/workshops/dlt.md)\n\n\n### [Module 3: Data Warehouse](03-data-warehouse/)\n\n* Data Warehouse\n* BigQuery\n* Partitioning and clustering\n* BigQuery best practices\n* Internals of BigQuery\n* BigQuery Machine Learning\n\n[More details](03-data-warehouse/)\n\n\n### [Module 4: Analytics engineering](04-analytics-engineering/)\n\n* Basics of analytics engineering\n* dbt (data build tool)\n* BigQuery and dbt\n* Postgres and dbt\n* dbt models\n* Testing and documenting\n* Deployment to the cloud and locally\n* Visualizing the data with google data studio and metabase\n\n\n[More details](04-analytics-engineering/)\n\n\n### [Module 5: Batch processing](05-batch/)\n\n* Batch processing\n* What is Spark\n* Spark Dataframes\n* Spark SQL\n* Internals: GroupBy and joins\n\n[More details](05-batch/)\n\n### [Module 6: Streaming](06-streaming/)\n\n* Introduction to Kafka\n* Schemas (avro)\n* Kafka Streams\n* Kafka Connect and KSQL\n\n[More details](06-streaming/)\n\n\n### [Workshop 2: Stream Processing with SQL](cohorts/2024/workshops/rising-wave.md)\n\n\n[More details](cohorts/2024/workshops/rising-wave.md)\n\n\n### [Project](projects)\n\nPutting everything we learned to practice\n\n* Week 1 and 2: working on your project\n* Week 3: reviewing your peers\n\n[More details](projects)\n\n## Overview\n\n\u003cimg src=\"images/architecture/arch_v3_workshops.jpg\" /\u003e\n\n### Prerequisites\n\nTo get the most out of this course, you should feel comfortable with coding and command line\nand know the basics of SQL. Prior experience with Python will be helpful, but you can pick\nPython relatively fast if you have experience with other programming languages.\n\nPrior experience with data engineering is not required.\n\n\n\n## Instructors\n\n- [Ankush Khanna](https://linkedin.com/in/ankushkhanna2)\n- [Victoria Perez Mola](https://www.linkedin.com/in/victoriaperezmola/)\n- [Alexey Grigorev](https://linkedin.com/in/agrigorev)\n- [Matt Palmer](https://www.linkedin.com/in/matt-palmer/)\n- [Luis Oliveira](https://www.linkedin.com/in/lgsoliveira/)\n- [Michael Shoemaker](https://www.linkedin.com/in/michaelshoemaker1/)\n\nPast instructors:\n\n- [Sejal Vaidya](https://www.linkedin.com/in/vaidyasejal/)\n- [Irem Erturk](https://www.linkedin.com/in/iremerturk/)\n\n\n## Asking for help in Slack\n\nThe best way to get support is to use [DataTalks.Club's Slack](https://datatalks.club/slack.html). Join the [`#course-data-engineering`](https://app.slack.com/client/T01ATQK62F8/C01FABYF2RG) channel.\n\nTo make discussions in Slack more organized:\n\n* Follow [these recommendations](asking-questions.md) when asking for help\n* Read the [DataTalks.Club community guidelines](https://datatalks.club/slack/guidelines.html)\n\n\n\n## Supporters and partners\n\nThanks to the course sponsors for making it possible to run this course\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://mage.ai/\"\u003e\n    \u003cimg height=\"120\" src=\"images/mage.svg\"\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://dlthub.com/\"\u003e\n    \u003cimg height=\"90\" src=\"images/dlthub.png\"\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://risingwave.com/\"\u003e\n    \u003cimg height=\"90\" src=\"images/rising-wave.png\"\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\nDo you want to support our course and our community? Please reach out to [alexey@datatalks.club](alexey@datatalks.club)\n\n## Star History\n\n[![Star History Chart](https://api.star-history.com/svg?repos=DataTalksClub/data-engineering-zoomcamp\u0026type=Date)](https://star-history.com/#DataTalksClub/data-engineering-zoomcamp\u0026Date)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchukwuemekaaham%2Fdata-engineering-zoomcamp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchukwuemekaaham%2Fdata-engineering-zoomcamp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchukwuemekaaham%2Fdata-engineering-zoomcamp/lists"}