https://github.com/DataTalksClub/data-engineering-zoomcamp
Data Engineering Zoomcamp is a free nine-week course that covers the fundamentals of data engineering.
https://github.com/DataTalksClub/data-engineering-zoomcamp
data-engineering dbt docker kafka kestra spark
Last synced: about 1 month ago
JSON representation
Data Engineering Zoomcamp is a free nine-week course that covers the fundamentals of data engineering.
- Host: GitHub
- URL: https://github.com/DataTalksClub/data-engineering-zoomcamp
- Owner: DataTalksClub
- Created: 2021-10-21T09:32:50.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2025-02-12T12:32:19.000Z (2 months ago)
- Last Synced: 2025-02-15T11:37:03.172Z (about 2 months ago)
- Topics: data-engineering, dbt, docker, kafka, kestra, spark
- Language: Jupyter Notebook
- Homepage:
- Size: 7.98 MB
- Stars: 28,843
- Watchers: 492
- Forks: 6,116
- Open Issues: 4
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
- awesome-dbt - Data Engineering Zoomcamp - Data engineering course on cutting edge tools including dbt. (Get Started)
- jimsghstars - DataTalksClub/data-engineering-zoomcamp - Free Data Engineering course! (Jupyter Notebook)
- StarryDivineSky - DataTalksClub/data-engineering-zoomcamp
- awesome-ai-data-github-repos - Data Engineering Zoomcamp
- awesome-ai-data-github-repos - Data Engineering Zoomcamp
- AiTreasureBox - DataTalksClub/data-engineering-zoomcamp - 04-07_29885_16](https://img.shields.io/github/stars/DataTalksClub/data-engineering-zoomcamp.svg)|Free Data Engineering course!| (Repos)
README
Data Engineering Zoomcamp: A Free 9-Week Course on Data Engineering Fundamentals
Master the fundamentals of data engineering by building an end-to-end data pipeline from scratch. Gain hands-on experience with industry-standard tools and best practices.
Join Slack •
#course-data-engineering Channel •
Telegram Announcements •
Course Playlist •
FAQ
## How to Enroll
### 2025 Cohort
- **Start Date**: January 13, 2025
- **Register Here**: [Sign up](https://airtable.com/shr6oVXeQvSI5HuWD)
- **Access Cohort Materials**: [2025 Cohort Folder](cohorts/2025/)
### Self-Paced Learning
All course materials are freely available for independent study. Follow these steps:
1. Watch the course videos.
2. Join the [Slack community](https://datatalks.club/slack.html).
3. Refer to the [FAQ document](https://docs.google.com/document/d/19bnYs80DwuUimHM65UV3sylsCn2j1vziPOwzBwQrebw/edit?usp=sharing) for guidance.
## Syllabus Overview
The course consists of structured modules, hands-on workshops, and a final project to reinforce your learning.
### **Prerequisites**
To get the most out of this course, you should have:
- Basic coding experience
- Familiarity with SQL
- Experience with Python (helpful but not required)
No prior data engineering experience is necessary.
### **Modules**
#### [Module 1: Containerization and Infrastructure as Code](01-docker-terraform/)
- Introduction to GCP
- Docker and Docker Compose
- Running PostgreSQL with Docker
- Infrastructure setup with Terraform
- Homework
#### [Module 2: Workflow Orchestration](02-workflow-orchestration/)
- Data Lakes and Workflow Orchestration
- Workflow orchestration with Kestra
- Homework
#### [Workshop 1: Data Ingestion](cohorts/2025/workshops/dlt/README.md)
- API reading and pipeline scalability
- Data normalization and incremental loading
- Homework
#### [Module 3: Data Warehousing](03-data-warehouse/)
- Introduction to BigQuery
- Partitioning, clustering, and best practices
- Machine learning in BigQuery
#### [Module 4: Analytics Engineering](04-analytics-engineering/)
- dbt (data build tool) with PostgreSQL & BigQuery
- Testing, documentation, and deployment
- Data visualization with Metabase
#### [Module 5: Batch Processing](05-batch/)
- Introduction to Apache Spark
- DataFrames and SQL
- Internals of GroupBy and Joins
#### [Module 6: Streaming](06-streaming/)
- Introduction to Kafka
- Kafka Streams and KSQL
- Schema management with Avro
#### [Final Project](projects/)
- Apply all concepts learned in a real-world scenario
- Peer review and feedback process
## Community & Support
### **Getting Help on Slack**
Join the [`#course-data-engineering`](https://app.slack.com/client/T01ATQK62F8/C01FABYF2RG) channel on [DataTalks.Club Slack](https://datatalks.club/slack.html) for discussions, troubleshooting, and networking.
To keep discussions organized:
- Follow [our guidelines](asking-questions.md) when posting questions.
- Review the [community guidelines](https://datatalks.club/slack/guidelines.html).
## Meet the Instructors
- [Victoria Perez Mola](https://www.linkedin.com/in/victoriaperezmola/)
- [Alexey Grigorev](https://linkedin.com/in/agrigorev)
- [Michael Shoemaker](https://www.linkedin.com/in/michaelshoemaker1/)
- [Zach Wilson](https://www.linkedin.com/in/eczachly)
- [Will Russell](https://www.linkedin.com/in/wrussell1999/)
- [Anna Geller](https://www.linkedin.com/in/anna-geller-12a86811a/)
Past instructors:
- [Ankush Khanna](https://linkedin.com/in/ankushkhanna2)
- [Sejal Vaidya](https://www.linkedin.com/in/vaidyasejal/)
- [Irem Erturk](https://www.linkedin.com/in/iremerturk/)
- [Luis Oliveira](https://www.linkedin.com/in/lgsoliveira/)
## Sponsors & Supporters
A special thanks to our course sponsors for making this initiative possible!
Interested in supporting our community? Reach out to [[email protected]](mailto:[email protected]).
## About DataTalks.Club
DataTalks.Club is a global online community of data enthusiasts. It's a place to discuss data, learn, share knowledge, ask and answer questions, and support each other.
Website •
Join Slack Community •
Newsletter •
Upcoming Events •
Google Calendar •
YouTube •
GitHub •
LinkedIn •
Twitter
All the activity at DataTalks.Club mainly happens on [Slack](https://datatalks.club/slack.html). We post updates there and discuss different aspects of data, career questions, and more.
At DataTalksClub, we organize online events, community activities, and free courses. You can learn more about what we do at [DataTalksClub Community Navigation](https://www.notion.so/DataTalksClub-Community-Navigation-bf070ad27ba44bf6bbc9222082f0e5a8?pvs=21).