https://github.com/jasontanx/prefect-learning
Prefect - Data orchestration tool practice & learning
https://github.com/jasontanx/prefect-learning
data-engineer data-orchestration prefect workflow-management
Last synced: 3 months ago
JSON representation
Prefect - Data orchestration tool practice & learning
- Host: GitHub
- URL: https://github.com/jasontanx/prefect-learning
- Owner: jasontanx
- Created: 2023-02-18T15:17:50.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2023-03-12T05:37:13.000Z (over 2 years ago)
- Last Synced: 2025-02-01T02:19:53.616Z (5 months ago)
- Topics: data-engineer, data-orchestration, prefect, workflow-management
- Language: Python
- Homepage:
- Size: 314 KB
- Stars: 1
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# prefect-learning
**What is Prefect 2.0?**
Prefect 2.0 is an open-source workflow management system designed to help data engineers and data scientists in automating data workflows. It features a modern API, a user-friendly interface, and advanced monitoring and debugging tools. Users can easily create, deploy, and manage complex data workflows across a variety of environments, including cloud and on-premises systems.
**Part 1: Concept of The Decorators → Flow & Task**
✨ Flows and tasks are the building blocks of workflows in Prefect 2.0
✨ A flow is a collection of tasks that represent a sequence of actions to be performed
✨ Tasks are individual units of work that can be composed into a flow
✨ Flows and tasks can be defined using Python code, making it easy to create complex workflows that integrate with a wide range of data sources and systems
**Prefect Documentation**
--> https://docs.prefect.io/tutorials/first-steps/
No | Date | Update
--- | --- | ---
1 | 19/02/2023 | Update Prefect basic folder - Flow and task concept
2 | 20/02/2023 | Update Prefect finance sample folder - Basic pipeline creation
3 | 22/02/2023 | Update Prefect Nike sample folder - Scheduling
4 | 24/02/2023 | Update Prefect GCP BQ query folder - Prefect 2.0 GCP Blocks and BigQuery
5 | 24/02/2023 | Update Prefect Slack notification folder - Prefect 2.0 + Slack Notification via webhook
6 | 25/02/2023 | Update Prefect Postgres data ingestion folder - Prefect 2.0 + ETL + Postgres SQL