{"id":27345207,"url":"https://github.com/markphamm/learn-airflow","last_synced_at":"2025-04-12T17:55:07.121Z","repository":{"id":282804315,"uuid":"949545374","full_name":"MarkPhamm/Learn-Airflow","owner":"MarkPhamm","description":"A hands-on repository for learning Apache Airflow, covering DAG creation, scheduling, task dependencies, and XComs. Includes practical examples, best practices, and real-world use cases to master workflow automation. 🚀","archived":false,"fork":false,"pushed_at":"2025-03-24T03:06:03.000Z","size":766,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-12T17:55:02.617Z","etag":null,"topics":["airflow","dags","dockers"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MarkPhamm.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-03-16T17:39:17.000Z","updated_at":"2025-03-24T03:06:08.000Z","dependencies_parsed_at":"2025-03-17T05:15:09.856Z","dependency_job_id":null,"html_url":"https://github.com/MarkPhamm/Learn-Airflow","commit_stats":null,"previous_names":["markphamm/learn-airflow"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MarkPhamm%2FLearn-Airflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MarkPhamm%2FLearn-Airflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MarkPhamm%2FLearn-Airflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MarkPhamm%2FLearn-Airflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MarkPhamm","download_url":"https://codeload.github.com/MarkPhamm/Learn-Airflow/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248610384,"owners_count":21132920,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","dags","dockers"],"created_at":"2025-04-12T17:55:06.476Z","updated_at":"2025-04-12T17:55:07.111Z","avatar_url":"https://github.com/MarkPhamm.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Learn-Airflow\r\nOpen-source platform that empowers data professional to efficiently create, schedule and monitor Tasks and Workflows\r\n* Pure Python \r\n* Robust Integration \r\n* Highly Scalable\r\n* Rich Interface\r\n* Open source/ Cost effective\r\n\r\nLesson 1: Intro to Airflow and first Airflow DAGs\r\n## DAG (Directed Acyclic Graph)\r\n* **Directed**: Dependencies have a specified direction\r\n* **Acyclic**: No cycles or Loops\r\n* **Graph**: Diagram that consist of Nodes and Edges\r\n\r\n## Tasks, Operators, Parameters, Depedencies, and Schedules\r\nDAG consist of one more muplitple tasks, each tasks is created using an operator\r\n* **Task**: A unit of work that is executed by the Airflow scheduler. Each task represents a single step in a workflow and can be defined using various operators.\r\n* **Operator**: A template/abstraction for defining a task in Airflow. Operators determine what kind of work is to be done, such as executing a Python function, running a Bash command, or interacting with external systems like databases or cloud services.\r\n* **Parameters**: Key-value pairs that are passed to operators to customize their behavior. Parameters allow users to define specific inputs for tasks, enabling dynamic workflows that can adapt to different data osr execution contexts.\r\n* **Dependencies**: Relationships between tasks that dictate the order in which they must be executed. Dependencies ensure that a task is not started until all of its upstream tasks have been successfully completed, thereby maintaining the integrity of the workflow.\r\n* **Schedules**: A schedule defines the timing and frequency at which tasks are executed within a DAG. It determines when a task should run, allowing users to specify intervals (e.g., hourly, daily) or specific times for execution. Schedules are crucial for automating workflows and ensuring that tasks are performed consistently and reliably over time.\r\n\r\n## Airflow Architecture\r\n* **Webserver**: The user interface for Airflow, allowing users to monitor and manage workflows, view logs, and trigger tasks manually.\r\n* **Database**: Stores metadata, task states, and configuration information necessary for the operation of the Airflow system.\r\n* **Scheduler**: Responsible for scheduling the execution of tasks based on their dependencies and defined schedules, ensuring that tasks are executed in the correct order.\r\n* **Executor**: The component that determines how and where tasks are executed. It can be configured to use different execution backends, such as LocalExecutor, CeleryExecutor, or KubernetesExecutor.\r\n* **Workers**: Compute layers that run the tasks, interacting with data and performing the actual work defined in the tasks. Workers can be distributed across multiple machines to scale the execution of workflows.\r\n\r\n## Docker\r\nA containerization platform that allows developers to package applications and their dependencies into containers, ensuring that they run consistently across different environments. We will use Docker to run Airflow\r\n* **Consistent**: Ensures that applications run the same way regardless of where they are deployed.\r\n* **Portable**: Containers can be easily moved between different environments, such as development, testing, and production.\r\n* **Repeatable**: Enables the same container image to be used multiple times, ensuring that deployments are predictable and reliable.\r\n\r\n# Lesson 2: Tasks, Operators, Parameters, Depedencies, Schedules, Providers \u0026 airflow.cfg\r\n\r\n## airflow.cfg\r\nThis is the configuration file for Apache Airflow, where various settings and parameters are defined to control the behavior of the Airflow environment. It includes configurations for the executor, database connection, logging, and other operational settings that dictate how tasks are scheduled and executed within the Airflow framework.\r\n\r\n## Tasks in Airflows\r\nA task is created by instantiating a specific operator and providing the necessary task-level parameters. Each task represents a single unit of work within a Directed Acyclic Graph (DAG) and can be defined using various operators, such as PythonOperator, BashOperator, and EmailOperator, among others. \r\n\r\n## Operators in Airflows\r\nOperators in Apache Airflow are the building blocks of tasks within a Directed Acyclic Graph (DAG). They define what kind of work is to be performed and can be categorized into several types based on their functionality:\r\n\r\n1. **Action Operators**: These operators execute a specific function or task. For instance:\r\n   - **PythonOperator**: This operator allows you to execute Python functions as tasks. It is commonly used for running custom Python code, making it versatile for various data processing tasks.\r\n     ```python\r\n     from airflow import DAG\r\n     from airflow.operators.python import PythonOperator\r\n     from airflow.utils.dates import days_ago\r\n\r\n     def my_python_function():\r\n         print(\"Hello from PythonOperator!\")\r\n\r\n     dag = DAG(\r\n         'example_dag',\r\n         default_args={'start_date': days_ago(1)},\r\n         schedule_interval='@daily',\r\n     )\r\n\r\n     python_task = PythonOperator(\r\n         task_id='run_my_python_function',\r\n         python_callable=my_python_function,\r\n         dag=dag,\r\n     )\r\n     ```\r\n   - **BashOperator**: This operator is used to execute bash commands or scripts. It is useful for tasks that require shell commands, such as file manipulation or executing scripts in a Unix-like environment.\r\n     ```python\r\n     from airflow import DAG\r\n     from airflow.operators.bash import BashOperator\r\n     from airflow.utils.dates import days_ago\r\n\r\n     dag = DAG(\r\n         'example_bash_dag',\r\n         default_args={'start_date': days_ago(1)},\r\n         schedule_interval='@daily',\r\n     )\r\n\r\n     bash_task = BashOperator(\r\n         task_id='run_bash_command',\r\n         bash_command='echo \"Hello from BashOperator!\"',\r\n         dag=dag,\r\n     )\r\n     ```\r\n   - **Azure DataFactory Run Pipeline Operator**: This operator is used to execute Azure Data Factory pipelines.\r\n     ```python\r\n     from airflow import DAG\r\n     from airflow.providers.microsoft.azure.operators.data_factory import AzureDataFactoryRunPipelineOperator\r\n     from airflow.utils.dates import days_ago\r\n\r\n     dag = DAG(\r\n         'example_azure_dag',\r\n         default_args={'start_date': days_ago(1)},\r\n         schedule_interval='@daily',\r\n     )\r\n\r\n     azure_pipeline_task = AzureDataFactoryRunPipelineOperator(\r\n         task_id='run_azure_pipeline',\r\n         pipeline_name='my_pipeline',\r\n         azure_data_factory_conn_id='my_azure_data_factory',\r\n         dag=dag,\r\n     )\r\n     ```\r\n\r\n2. **Transfer Operators**: These operators are used for moving data from one place to another. An excellent example of this is:\r\n   - **S3ToRedshiftOperator**: It moves data from Amazon S3 to Amazon Redshift.\r\n     ```python\r\n     from airflow import DAG\r\n     from airflow.providers.amazon.aws.transfers.s3_to_redshift import S3ToRedshiftOperator\r\n     from airflow.utils.dates import days_ago\r\n\r\n     dag = DAG(\r\n         's3_to_redshift_dag',\r\n         default_args={'start_date': days_ago(1)},\r\n         schedule_interval='@daily',\r\n     )\r\n\r\n     transfer_task = S3ToRedshiftOperator(\r\n         task_id='transfer_s3_to_redshift',\r\n         schema='public',\r\n         table='my_table',\r\n         s3_bucket='my_bucket',\r\n         s3_key='data/my_data.csv',\r\n         copy_options=['CSV'],\r\n         aws_conn_id='aws_default',\r\n         redshift_conn_id='redshift_default',\r\n         dag=dag,\r\n     )\r\n     ```\r\n\r\n3. **Sensor Operators**: These operators wait for a specific condition to be met before triggering the subsequent tasks in the workflow. Examples include:\r\n   - **S3KeySensor**: This sensor waits for one or more files to be created in an S3 bucket.\r\n     ```python\r\n     from airflow import DAG\r\n     from airflow.providers.amazon.aws.sensors.s3 import S3KeySensor\r\n     from airflow.utils.dates import days_ago\r\n\r\n     dag = DAG(\r\n         's3_sensor_dag',\r\n         default_args={'start_date': days_ago(1)},\r\n         schedule_interval='@daily',\r\n     )\r\n\r\n     wait_for_file = S3KeySensor(\r\n         task_id='wait_for_s3_file',\r\n         bucket_name='my_bucket',\r\n         bucket_key='data/my_data.csv',\r\n         aws_conn_id='aws_default',\r\n         poke_interval=10,\r\n         timeout=60 * 60 * 5,\r\n         soft_fail=True,\r\n         dag=dag,\r\n     )\r\n     ```\r\n   - **AWS Redshift Cluster Sensor**: This sensor waits for a Redshift cluster to reach a specific status.\r\n\r\n4. **DummyOperator**: This operator does nothing and is often used as a placeholder in a DAG. It can help in structuring workflows and managing dependencies without performing any actual work.\r\n   ```python\r\n   from airflow import DAG\r\n   from airflow.operators.dummy import DummyOperator\r\n   from airflow.utils.dates import days_ago\r\n\r\n   dag = DAG(\r\n       'dummy_operator_dag',\r\n       default_args={'start_date': days_ago(1)},\r\n       schedule_interval='@daily',\r\n   )\r\n\r\n   start_task = DummyOperator(\r\n       task_id='start',\r\n       dag=dag,\r\n   )\r\n   ```\r\n\r\n5. **BranchPythonOperator**: This operator allows for branching logic in workflows. It can be used to decide which path to take in a DAG based on certain conditions, enabling dynamic task execution.\r\n   ```python\r\n   from airflow import DAG\r\n   from airflow.operators.python import BranchPythonOperator\r\n   from airflow.utils.dates import days_ago\r\n\r\n   def choose_branch():\r\n       return 'branch_a'  # Logic to choose the branch\r\n\r\n   dag = DAG(\r\n       'branch_operator_dag',\r\n       default_args={'start_date': days_ago(1)},\r\n       schedule_interval='@daily',\r\n   )\r\n\r\n   branching_task = BranchPythonOperator(\r\n       task_id='branching_task',\r\n       python_callable=choose_branch,\r\n       dag=dag,\r\n   )\r\n   ```\r\n\r\n6. **SubDagOperator**: This operator allows you to define a sub-DAG within a parent DAG. It is useful for organizing complex workflows into smaller, manageable pieces.\r\n   ```python\r\n   from airflow import DAG\r\n   from airflow.operators.subdag import SubDagOperator\r\n   from airflow.utils.dates import days_ago\r\n\r\n   def subdag(parent_dag_name, child_dag_name, args):\r\n       dag_subdag = DAG(\r\n           dag_id=child_dag_name,\r\n           default_args=args,\r\n           schedule_interval='@daily',\r\n       )\r\n       # Define tasks for the subdag here\r\n       return dag_subdag\r\n\r\n   dag = DAG(\r\n       'subdag_operator_dag',\r\n       default_args={'start_date': days_ago(1)},\r\n       schedule_interval='@daily',\r\n   )\r\n\r\n   subdag_task = SubDagOperator(\r\n       task_id='subdag_task',\r\n       subdag=subdag('subdag_operator_dag', 'subdag_task', {'start_date': days_ago(1)}),\r\n       dag=dag,\r\n   )\r\n\r\n## Operators vs Tasks\r\nAn operator can be viewed as a blueprint or design template, while tasks are the concrete implementations derived from that blueprint. In terms of Python or object-oriented programming, an operator represents a class, and tasks are instances (or objects) created from that class.\r\n\r\n## Dependencies in Airflow\r\nIn Apache Airflow, managing task dependencies is crucial for ensuring that tasks are executed in the correct order. Here are some methods to define dependencies:\r\n\r\n1. **Bitwise Operators**: You can use bitwise operators (`\u0026` for \"and\", `|` for \"or\") to define dependencies between tasks in a more concise manner. For example, `task1 \u003e\u003e task2` can be expressed as `task1.set_downstream(task2)`. \r\n   - **Example**: \r\n     ```python\r\n     task1 \u003e\u003e task2  # Using bitwise operator\r\n     task1.set_downstream(task2)  # Using set_downstream method\r\n     ```\r\n\r\n2. **Set Upstream and Set Downstream Methods**: The `set_upstream()` and `set_downstream()` methods allow you to explicitly define the order of task execution. For instance, `task1.set_downstream(task2)` ensures that `task2` runs after `task1`.\r\n   - **Example**: \r\n     ```python\r\n     task2.set_upstream(task1)  # Ensures task1 runs before task2\r\n     ```\r\n\r\n3. **Chain Function**: The `chain()` function from the `airflow.utils.dag` module provides a convenient way to set multiple dependencies at once. For example, `chain(task1, task2, task3)` sets `task1` to run before `task2`, which in turn runs before `task3`.\r\n   - **Example**: \r\n     ```python\r\n     from airflow.utils.dag import chain\r\n     chain(task1, task2, task3)  # Sets task1 -\u003e task2 -\u003e task3\r\n     ```\r\n\r\n4. **TaskFlow API**: The TaskFlow API simplifies the creation of tasks and their dependencies using Python decorators. By using the `@dag` and `@task` decorators, you can define a DAG and its tasks in a more intuitive way, automatically managing dependencies based on the function calls.\r\n   - **Example**: \r\n     ```python\r\n     from airflow.decorators import dag, task\r\n\r\n     @dag(schedule_interval='@daily', start_date=days_ago(1))\r\n     def my_dag():\r\n         @task\r\n         def task1():\r\n             pass\r\n\r\n         @task\r\n         def task2():\r\n             pass\r\n\r\n         task1() \u003e\u003e task2()  # Automatically manages dependencies\r\n     ```\r\n\r\n# Variables\r\nVariables are small storage containers for values that can be reused throughout tasks. They are essentially key-value pairs, where the variable name holds a specific value.\r\n* **Regular Variable**: The value can be a standard string.\r\n* **JSON Variable**: The value is formatted as a JSON string.\r\nTo create variables, navigate to the admin panel and select \"Create Variables.\"\r\n\r\n## Advantages of Variables\r\n* **Ease of Change Implementation**: Modifications can be made easily without altering the code.\r\n* **Environment Specificity**: Variables can be tailored for different environments.\r\n* **Security**: Automatically detects and manages sensitive information.\r\n* **Dynamic DAGs**: Useful for creating DAGs that can adapt to specific dates or conditions.\r\n\r\n# Connection\r\nConnections in Apache Airflow are essential configuration objects that store the necessary details required to establish connections to external systems, such as databases, cloud services, and APIs. Each connection is defined by a set of parameters, including the connection ID, host, schema, login credentials, and any additional options specific to the service being connected to. \r\n\r\nTo create a new connection, navigate to the Admin panel, select the \"Connections\" option, and then click on \"Create.\" Here, you can define various types of connections, such as those for Amazon Redshift or Snowflake. You will need to enter the required information, including the connection ID, host address, username, password, and any other relevant parameters. After entering the details, it is crucial to test the connection to ensure that it can successfully communicate with the external system before integrating it into your DAG.\r\n\r\n## Benefits of Connections\r\n* **Reusability and Maintainability**: Connections can be reused across multiple DAGs and tasks, reducing redundancy and simplifying maintenance. If a connection's details change, you only need to update it in one place.\r\n* **Environment Specificity**: Connections can be tailored for different environments (e.g., development, testing, production), allowing for seamless transitions and configurations without modifying the DAG code.\r\n* **Ease of Implementing Changes**: Modifications to connection parameters can be made easily through the Airflow UI, enabling quick adjustments without the need to alter the underlying codebase.\r\n* **Security**: Connections help manage sensitive information, such as passwords and API keys, securely. Airflow provides mechanisms to encrypt these details, ensuring that they are not exposed in the code or logs.\r\n\r\n# Sensors\r\nAirflow sensors are a specialized type of operator designed to monitor specific conditions until they are met.\r\n\r\nThese sensors periodically check for the specified condition. Once the condition is satisfied, the associated task is marked as successful, allowing downstream tasks to proceed. Sensors enhance the event-driven nature of your DAGs, making them suitable for scenarios such as waiting for a file to be created, a database table to be updated, or an external API to become available.\r\n\r\n* **poke_interval:** The interval (in seconds) at which the sensor checks for the specified condition.\r\n* **mode:** Determines the execution mode of the sensor, such as \"poke\" or \"reschedule\".\r\n* **timeout:** The maximum time (in seconds) the sensor will wait for the condition to be met before failing.\r\n* **soft_fail:** If set to true, the task will not fail the DAG if the sensor times out.\r\n\r\nExample: The first task employs the S3KeySensor, which waits for a file to appear in an AWS S3 bucket. Once the file is detected, the subsequent task loads the file into a Snowflake table.\r\n\r\n# Deferrable Operators \u0026 Triggers\r\nBy default, sensors work in 'poke' mode, which worker slot continuously allocated, even if the task is inactive or in sleep mode. Schedule allow worker slot at fixed interval. More worker slot means higher cost\r\n\r\n**`Deferrable operators`** are designed to suspend their execution and completely free up the worker slot when they need to wait for a condition to be met. While in a suspended or deferred state, they do not occupy a worker slot, allowing for more efficient resource utilization. For instance, if you have 100 sensors, each would typically occupy a worker slot, leading to a one-to-one mapping. However, with deferrable operators, a single trigger can manage multiple sensors asynchronously, efficiently handling up to 100 sensors simultaneously without consuming individual worker slots for each one.\r\n\r\n\r\n```python\r\nfrom airflow import DAG\r\nfrom airflow.providers.common.sql.sensors.sql import SqlSensor\r\nfrom airflow.providers.common.sql.triggers.sql import SqlTrigger\r\nfrom airflow.providers.postgres.operators.postgres import PostgresOperator\r\nfrom airflow.providers.common.sql.sensors.sql import SqlSensorAsync\r\nfrom datetime import datetime, timedelta\r\n\r\n# Define default_args\r\ndefault_args = {\r\n    'owner': 'airflow',\r\n    'depends_on_past': False,\r\n    'start_date': datetime(2024, 3, 20),\r\n    'retries': 1,\r\n    'retry_delay': timedelta(minutes=5),\r\n}\r\n\r\n# Define DAG\r\ndag = DAG(\r\n    'deferrable_operator_example',\r\n    default_args=default_args,\r\n    schedule_interval=None,\r\n    catchup=False\r\n)\r\n\r\n# Normal SQL Sensor (Blocking Worker Slot)\r\nsql_sensor = SqlSensor(\r\n    task_id='blocking_sql_sensor',\r\n    conn_id='postgres_default',\r\n    sql='SELECT COUNT(*) FROM my_table WHERE status = \"ready\";',\r\n    mode='poke',  # Continuously runs, occupying a worker slot\r\n    poke_interval=10,  # Checks every 10 seconds\r\n    timeout=600,  # Timeout after 10 minutes\r\n    dag=dag\r\n)\r\n\r\n# Deferrable SQL Sensor (Non-blocking)\r\nasync_sql_sensor = SqlSensorAsync(\r\n    task_id='non_blocking_sql_sensor',\r\n    conn_id='postgres_default',\r\n    sql='SELECT COUNT(*) FROM my_table WHERE status = \"ready\";',\r\n    mode='reschedule',  # Frees worker slot while waiting\r\n    poke_interval=10,\r\n    timeout=600,\r\n    dag=dag\r\n)\r\n\r\n# Postgres Task (Executed after condition is met)\r\nprocess_data = PostgresOperator(\r\n    task_id='process_data',\r\n    postgres_conn_id='postgres_default',\r\n    sql=\"UPDATE my_table SET processed = TRUE WHERE status = 'ready';\",\r\n    dag=dag\r\n)\r\n\r\n# Define task dependencies\r\nasync_sql_sensor \u003e\u003e process_data\r\n```\r\n\r\n# Trigger vs Sensors\r\n* **Sensors** are a type of operator that continuously check for a certain condition to be met before allowing downstream tasks to proceed. They are typically used for scenarios like waiting for a file to appear in a specific location, a database record to be updated, or an external API to become available. Sensors operate in 'poke' mode by default, which means they periodically check the condition at defined intervals (poke_interval) until the condition is satisfied or a timeout occurs. This can lead to inefficient resource usage, as each sensor occupies a worker slot while it is active.\r\n\r\n* **Triggers**, on the other hand, are part of the deferrable operators in Airflow. They allow tasks to suspend their execution and free up the worker slot while waiting for a condition to be met. This is particularly useful for optimizing resource utilization, as it enables a single trigger to manage multiple conditions asynchronously without consuming individual worker slots for each one. When the condition is met, the trigger wakes up the task, allowing it to continue execution.\r\n\r\n# Xcom\r\n# XCom in Apache Airflow\r\n\r\n## Overview\r\n**XCom (Cross-Communication)** is a feature in Apache Airflow that allows tasks to exchange data with each other. This capability is crucial in workflows where the output of one task needs to be used as the input for another. XComs are stored in Airflow's metadata database and can be used to pass small pieces of data between tasks.\r\n\r\n## XCom Push\r\nThe **XCom push** mechanism is used to send data from one task to another. Data is pushed into the XCom table in Airflow's metadata database. You can push data using the `xcom_push` method or by returning a value from a Python task:\r\n\r\n#### Using `xcom_push` Method\r\n```python\r\nfrom airflow import DAG\r\nfrom airflow.operators.python import PythonOperator\r\nfrom datetime import datetime\r\n\r\ndef push_data(ti):\r\n    ti.xcom_push(key='message', value='Hello from XCom!')\r\n\r\ndag = DAG(\r\n    'xcom_example',\r\n    schedule_interval=None,\r\n    start_date=datetime(2024, 3, 20),\r\n    catchup=False\r\n)\r\n\r\npush_task = PythonOperator(\r\n    task_id='push_task',\r\n    python_callable=push_data,\r\n    provide_context=True,\r\n    dag=dag\r\n)\r\n```\r\n\r\n#### Using the Return Value of a Python Task\r\n```python\r\ndef push_data():\r\n    return \"Hello from XCom!\"\r\n\r\npush_task = PythonOperator(\r\n    task_id='push_task',\r\n    python_callable=push_data,\r\n    dag=dag\r\n)\r\n```\r\nIn this case, Airflow automatically pushes the returned value to XCom with the key `return`.\r\n\r\n## XCom Pull\r\nThe **XCom pull** mechanism retrieves data stored in XCom. This allows downstream tasks to access data pushed by upstream tasks. You can pull data using the `xcom_pull` method:\r\n\r\n#### Using `xcom_pull` in a Python Task\r\n```python\r\ndef pull_data(ti):\r\n    message = ti.xcom_pull(task_ids='push_task', key='message')\r\n    print(f\"Received message: {message}\")\r\n\r\npull_task = PythonOperator(\r\n    task_id='pull_task',\r\n    python_callable=pull_data,\r\n    provide_context=True,\r\n    dag=dag\r\n)\r\n```\r\n\r\n#### Using `xcom_pull` with Default Key (`return`)\r\nIf data was pushed using the return value of a Python function, it can be pulled as follows:\r\n```python\r\nmessage = ti.xcom_pull(task_ids='push_task', key='return')\r\n```\r\n\r\n## XCom Use Cases\r\n- **Data Sharing Between Tasks**: Tasks can exchange values without using external storage.\r\n- **Dynamic Task Execution**: The result of one task can determine the behavior of another.\r\n- **Triggering Conditional Logic**: Use XCom values to control the execution of downstream tasks.\r\n\r\n## Considerations\r\n- XCom is stored in the Airflow database and should be used for small data values.\r\n- For larger data, it's recommended to use cloud storage solutions or databases.\r\n- XCom values are serialized as JSON, so only serializable objects can be stored.\r\n\r\nFor example, if the first task generates a file, the second task can upload that file to an S3 bucket. In our current setup, we have the 'replace' parameter set to true, which means that each time we run this DAG, the new file overwrites the previous one in S3. However, if this DAG runs multiple times a day, the business may want to keep a history of the exchange rate files. Since the file name created in Task One is not automatically known to Task Two, we can use XCom to bridge this gap.\r\n\r\nIn Task One, we use the XCom push feature to store the generated file name in Airflow’s metadata database. Then, in Task Two, we can retrieve that file name using the XCom pull feature, ensuring the correct file is uploaded to S3.\r\n\r\nIt's important to note that XComs are intended for passing small amounts of data between tasks, such as file names, task metadata, dates, or single-value query results. For larger datasets, consider using a custom XCom backend or intermediary data storage solutions to manage the data more effectively.\r\n\r\n# Hooks  \r\nHooks are **pre-built Python classes** that simplify interactions with external systems and services.  \r\n\r\nFor example, the **S3Hook** (part of the AWS provider package) provides methods to interact with S3:  \r\n- `create_bucket()`: Creates an Amazon S3 bucket  \r\n- `load_string()`: Uploads a string as a file in S3  \r\n- `delete_objects()`: Deletes an S3 file  \r\n\r\n## Why Use Hooks?  \r\nHooks abstract low-level Python code, making it easier to interact with external services. Instead of writing complex logic, you can **import and use pre-built hooks** from Airflow’s provider packages.  \r\n\r\n## Hooks vs. Operators  \r\nBoth are Python classes, but:  \r\n- **Hooks** handle low-level interactions with external services.  \r\n- **Operators** wrap around Hooks and provide task-level abstraction.  \r\n\r\nFor example, an AWS operator may use the **S3Hook’s** `copy_object()` method internally.  \r\n\r\n## When to Use Hooks vs. Operators  \r\n- **Use an Operator** if one exists for your task.  \r\n- **Use a Hook** if no operator is available.  \r\n- **Write custom Python code** if neither a hook nor an operator exists.  \r\n\r\n## Using Hooks in DAGs  \r\nExample: Transform an S3 file and re-upload it.  \r\n1. Read the file using `read_key()` from S3Hook  \r\n2. Apply transformations  \r\n3. Upload it back using `load_string()`  \r\n\r\nWithin the DAG, call the transformation function with the bucket name, source file, and target file.\r\n\r\n# Airflow Datasets   \r\nDatasets enable **data-aware scheduling**, allowing DAGs to trigger based on the availability of files or datasets instead of time-based scheduling.  \r\n\r\nA **dataset** is a logical representation of underlying data. You can create a dataset in Airflow by instantiating the `Dataset` class, using either:  \r\n- A **complete URI** (e.g., `s3://bucket/exchange_rates.csv`)  \r\n- A **descriptive string** (e.g., `\"exchange_rates\"`)  \r\n\r\nAirflow does not access or validate the data—datasets serve as identifiers for establishing **producer-consumer DAG relationships**.  \r\n\r\n## Benefits of Datasets  \r\n- **Cross-DAG dependencies**: Connect DAGs that rely on shared data.  \r\n- **Flexible scheduling**: Trigger DAGs dynamically when data is available.  \r\n- **Reduced Airflow costs**: Unlike sensors, datasets do not use worker slots.  \r\n\r\n## Producer and Consumer DAGs  \r\nA **Producer DAG** updates a dataset using the `outlets` parameter.  \r\nA **Consumer DAG** is scheduled based on dataset availability.  \r\n\r\n### Example  \r\n- **Producer DAG**: Ingests exchange rate data into an S3 bucket.  \r\n- **Consumer DAG**: Transforms the data and loads it into Snowflake when new data is available.  \r\n\r\n## Viewing Datasets in Airflow  \r\n- **Datasets Tab**: Shows dataset triggers and history.  \r\n- **DAG Dependencies View**: Visualizes cross-DAG relationships.  \r\n\r\n## Triggering a DAG Based on Multiple Datasets  \r\nA DAG can be scheduled based on **multiple producer DAGs or datasets** by listing them as comma-separated values. The DAG runs once all producer tasks complete successfully. \r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarkphamm%2Flearn-airflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmarkphamm%2Flearn-airflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarkphamm%2Flearn-airflow/lists"}