{"id":20710190,"url":"https://github.com/oxylabs/building-scraping-pipeline-apache-airflow","last_synced_at":"2025-09-27T11:30:23.027Z","repository":{"id":134336550,"uuid":"526096840","full_name":"oxylabs/building-scraping-pipeline-apache-airflow","owner":"oxylabs","description":"Using Apache Airflow to Build a Pipeline for Scraped Data ","archived":false,"fork":false,"pushed_at":"2024-09-30T14:16:02.000Z","size":105,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-11-17T02:10:09.124Z","etag":null,"topics":["airflow-pipelines","airflow-python-api","apache-airflow","apache-airflow-tutorial","how-to-use-airflow","pipeline-for-scraped-data","python","sql"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oxylabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2022-08-18T07:07:03.000Z","updated_at":"2024-09-30T14:16:05.000Z","dependencies_parsed_at":"2024-04-04T14:54:29.479Z","dependency_job_id":"758376d7-f729-46e9-a843-a72abee2fa8a","html_url":"https://github.com/oxylabs/building-scraping-pipeline-apache-airflow","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fbuilding-scraping-pipeline-apache-airflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fbuilding-scraping-pipeline-apache-airflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fbuilding-scraping-pipeline-apache-airflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fbuilding-scraping-pipeline-apache-airflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oxylabs","download_url":"https://codeload.github.com/oxylabs/building-scraping-pipeline-apache-airflow/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":234429241,"owners_count":18831240,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow-pipelines","airflow-python-api","apache-airflow","apache-airflow-tutorial","how-to-use-airflow","pipeline-for-scraped-data","python","sql"],"created_at":"2024-11-17T02:10:26.900Z","updated_at":"2025-09-27T11:30:23.019Z","avatar_url":"https://github.com/oxylabs.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Using Apache Airflow to Build a Pipeline for Scraped Data\n\n[![Oxylabs promo code](https://raw.githubusercontent.com/oxylabs/product-integrations/refs/heads/master/Affiliate-Universal-1090x275.png)](https://oxylabs.io/pages/gitoxy?utm_source=877\u0026utm_medium=affiliate\u0026groupid=877\u0026utm_content=building-scraping-pipeline-apache-airflow-github\u0026transaction_id=102f49063ab94276ae8f116d224b67)\n\n\n[![](https://dcbadge.limes.pink/api/server/Pds3gBmKMH?style=for-the-badge\u0026theme=discord)](https://discord.gg/Pds3gBmKMH) [![YouTube](https://img.shields.io/badge/YouTube-Oxylabs-red?style=for-the-badge\u0026logo=youtube\u0026logoColor=white)](https://www.youtube.com/@oxylabs)\n\nUsing Oxylabs E-Commerce Scraper API (a part of Web Scraper API) has a wide variety of tools depending on your project and scraping goals. \n\nAlso, we recommend using the Push-Pull approach – it’s known as the most reliable data delivery method out there. \n\nTo use this approach effectively, you have to: \n\n1. Submit a URL of a website you want to scrape\n2. Check whether the URL has been scraped\n3. Fetch the content\n\nLet's start by building a class that will serve as a wrapper for the API.\n\n```python\nimport requests\n\n\nJOB_STATUS_DONE = \"done\"\n\nHTTP_NO_CONTENT = 204\n\nclass Client:\n    def __init__(self, username, password):\n        self.username = username\n        self.password = password\n\n    def create_jobs(self, urls):\n        payload = {\n            \"source\": \"universal\",\n            \"url\": urls,\n            \"parse\": True,\n            \"parsing_instructions\": {\n                \"title\": {\n                    \"_fns\": [\n                        {\"_fn\": \"css_one\", \"_args\": [\"h2\"]},\n                        {\"_fn\": \"element_text\"},\n                    ]\n                },\n                \"price\": {\n                    \"_fns\": [\n                        {\"_fn\": \"css_one\", \"_args\": [\".price\"]},\n                        {\"_fn\": \"element_text\"},\n                    ]\n                },\n                \"availability\": {\n                    \"_fns\": [\n                        {\"_fn\": \"css_one\", \"_args\": [\".availability\"]},\n                        {\"_fn\": \"element_text\"},\n                    ]\n                },\n            },\n        }\n\n        response = requests.request(\n            \"POST\",\n            \"https://data.oxylabs.io/v1/queries/batch\",\n            auth=(self.username, self.password),\n            json=payload,\n        )\n\n        return response.json()\n\n    def is_status_done(self, job_id):\n        job_status_response = requests.request(\n            method=\"GET\",\n            url=\"http://data.oxylabs.io/v1/queries/%s\" % job_id,\n            auth=(self.username, self.password),\n        )\n\n        job_status_data = job_status_response.json()\n\n        return job_status_data[\"status\"] == JOB_STATUS_DONE\n\n    def fetch_content_list(self, job_id):\n        job_result_response = requests.request(\n            method=\"GET\",\n            url=\"http://data.oxylabs.io/v1/queries/%s/results\" % job_id,\n            auth=(self.username, self.password),\n        )\n        if job_result_response.status_code == HTTP_NO_CONTENT:\n            return None\n\n        job_results_json = job_result_response.json()\n\n        return job_results_json[\"results\"]\n```\n\nThe client provides 3 methods:\n\n`create_jobs` uses the batch query to submit URLs for scraping. \n\n`is_status_done` checks the status of the previously submitted URL. \n\n`fetch_content_list` retrieves the content of the URL that has been scraped. \n\nKeep in mind that once we push the URL to the API, we’ll receive a job ID for fetching the content later. Hence, the job ID needs to be stored somewhere – we'll use PostgreSQL for it. \n\nLet's design a simple table that will represent queued jobs:\n\n```sql\ncreate sequence queue_seq;\n\ncreate table queue (\n  id int check (id \u003e 0) primary key default nextval ('queue_seq'),\n  created_at timestamp(0) not null DEFAULT CURRENT_TIMESTAMP,\n  updated_at timestamp(0) not null DEFAULT CURRENT_TIMESTAMP,\n  status varchar(255) not null DEFAULT 'pending',\n  job_id varchar(255)\n)\n```\n\nOur new table contains the following fields:\n\n`id`: a numerical value that uniquely identifies the record.\n\n`created_at`: a timestamp that shows when the record was created.\n\n`updated_at`: a timestamp that shows when the record was last updated.\n\n`job_id`: Oxylabs API job identifier.\n\n`status`: a value that describes what the current state of the job is.\n\n`pending` status means that the job is still processing.\n\n`completed` means that the job is already done.\n\n`deleted` means that we took too long to fetch the data, and the job has been deleted in Oxylabs API.\n\nNow let's create a `Queue` class for interacting with the database: \n\n```python\nimport atexit\n\nimport psycopg2.extras\n\nSTATUS_PENDING = 'pending'\nSTATUS_COMPLETE = 'complete'\nSTATUS_DELETED = 'deleted'\n\n\nclass Queue:\n    def __init__(self, connection):\n        self.connection = connection\n\n        atexit.register(self.cleanup)\n\n    def setup(self):\n        cursor = self.connection.cursor()\n\n        cursor.execute('''\n            select table_name\n              from information_schema.tables\n            where table_schema='public'\n              and table_type='BASE TABLE'\n        ''')\n        for cursor_result in cursor:\n            if cursor_result[0] == 'queue':\n                print('Table already exists')\n                return False\n\n        cursor.execute('''\n            create sequence queue_seq;\n\n            create table queue (\n              id int check (id \u003e 0) primary key default nextval ('queue_seq'),\n              created_at timestamp(0) not null DEFAULT CURRENT_TIMESTAMP,\n              updated_at timestamp(0) not null DEFAULT CURRENT_TIMESTAMP,\n              status varchar(255) not null DEFAULT 'pending',\n              job_id varchar(255)\n            )\n        ''')\n\n        return True\n\n    def push(self, job_id):\n        self.__execute_and_commit(\n            'insert into queue (job_id) values (%s)',\n            [job_id]\n        )\n\n    def pull(self):\n        cursor = self.connection.cursor(cursor_factory=psycopg2.extras.RealDictCursor)\n\n        cursor.execute('start transaction')\n        cursor.execute(\n            '''\n            select * from queue where status = %s and\n            updated_at \u003c now() - interval '10 second'\n            order by random()\n            limit 1\n            for update\n            ''',\n            [STATUS_PENDING]\n        )\n        return cursor.fetchone()\n\n    def delete(self, job_id):\n        self.__change_status(job_id, STATUS_DELETED)\n\n    def complete(self, job_id):\n        self.__change_status(job_id, STATUS_COMPLETE)\n\n    def touch(self, job_id):\n        self.__execute_and_commit(\n            'update queue set updated_at = now() where job_id = %s',\n            [job_id]\n        )\n\n    def __change_status(self, job_id, status):\n        self.__execute_and_commit(\n            'update queue set status = %s where job_id = %s',\n            [status, job_id]\n        )\n\n    def __execute_and_commit(self, sql, val):\n        cursor = self.connection.cursor()\n        cursor.execute(sql, val)\n\n        self.connection.commit()\n\n    def cleanup(self):\n        self.connection.commit()\n```\nThe most important methods of the `Queue` class are as follows:\n\n`setup`: asks the database whether the table `queue` already exists. If it doesn't, it creates the queue table.\n\n`push`: pushes the job to the database. The job id is retrieved from the Oxylabs Batch Query Endpoint.\n\n`pull`: fetches a single job that is ready to be checked for content.\n\nLet's focus on the pull part: \n\n```sql\nselect * from queue where status = 'pending' and\nupdated_at \u003c now() - interval '10 second'\norder by random()\nlimit 1\nfor update\n```\n\nWe fetch records that haven't been updated in the last 10 seconds so as not to spam the API with irrelevant requests.\n\nThe order by `random()` clause ensures no single record blocks our queue.\n\n`for update` locks the row and prevents other processes from picking it up in case parallelism is needed in the future.\n\nSince we use transactions to lock the row, we also register a commit method in `atexit.register` to perform a commit at the end of the script at all times.\n\n```python\n    def __init__(self, connection):\n        self.connection = connection\n\n        atexit.register(self.cleanup)\n    \n    # ...\n        \n    def cleanup(self):\n        self.connection.commit()\n```\n\nNow that we have  `Queue` and `Client` classes, we’re likely to use them in nearly all of our scripts. In addition, we need certain configuration options. \n\nFor that purpose, let's create the following bootstrap file:\n\n```python\nimport os\n\nimport psycopg2\n\nfrom messenger import Queue\nfrom oxylabs import Client\n\nDB_HOST = os.getenv('DB_HOST', 'postgres')\nDB_USER = os.getenv('DB_USER', 'airflow')\nDB_PASS = os.getenv('DB_PASS', 'airflow')\nDB_NAME = os.getenv('DB_NAME', 'scraper')\nOXYLABS_USERNAME = os.getenv('OXYLABS_USERNAME', 'your-oxylabs-username')\nOXYLABS_PASSWORD = os.getenv('OXYLABS_PASSWORD', 'your-oxylabs-password')\n\nconnection = psycopg2.connect(\n    host=DB_HOST,\n    user=DB_USER,\n    password=DB_PASS,\n    database=DB_NAME\n)\n\nqueue = Queue(\n    connection\n)\n\nclient = Client(\n    OXYLABS_USERNAME,\n    OXYLABS_PASSWORD,\n)\n```\n\nHere, we fetch the Oxylabs API and PostgreSQL configuration variables from the environment – it’s a standard industry practice encouraged by the twelve-factor app principles. Additionally, we create the `Queue` and `Client` classes, and their dependencies. \n\nNow that we have all the main classes initialized, let's create a script that makes the schema for our queue.\n\n```python\nfrom bootstrap import queue\n\nsuccess = queue.setup()\nif not success:\n    exit(1)\n```\n\n```python\nfrom bootstrap import queue\n\nsuccess = queue.setup()\nif not success:\n    exit(1)\n```\n\nThe `exit(1)` on failure is extremely important, as it signifies that the process has not completed successfully. Once the schema is created, we can **push** a collection of jobs in the Oxylabs Batch Query endpoint. \n\nThe `exit(1)` on failure is extremely important, as it signifies that the process has not completed successfully.\n\nOnce the schema is created, we can **push** a collection of jobs in the Oxylabs Batch Query endpoint. \n\n```python\nfrom bootstrap import queue, client\n\njobs = client.create_jobs([\n        \"https://sandbox.oxylabs.io/products/1\",\n        \"https://sandbox.oxylabs.io/products/2\",\n        \"https://sandbox.oxylabs.io/products/3\",\n        \"https://sandbox.oxylabs.io/products/4\",\n        \"https://sandbox.oxylabs.io/products/5\",\n])\n\nfor job in jobs['queries']:\n    queue.push(job['id'])\n    print('job id: %s' % job['id'])\n```\n\nThe script creates a bunch of jobs using the Oxylabs Client we created earlier. It then goes through each and every result and pushes it into the database using the Queue service.\n\n![](https://images.prismic.io/oxylabs-sm/a260d754-58ca-4ed0-9c9c-a706280d3c7d_4.png?auto=compress,format\u0026fm=webp\u0026dpr=2\u0026q=50)\n\nThe submitted jobs will soon be processed. In the meantime, we can use the job ID in our database to fetch the content – let’s create a file that will do exactly that.\n\n```python\nfrom pprint import pprint\nfrom bootstrap import queue, client\n\nqueue_item = queue.pull()\nif not queue_item:\n    print('No jobs left in the queue, exiting')\n    exit(0)\n\nif not client.is_status_done(queue_item['job_id']):\n    queue.touch(queue_item['job_id'])\n    print('Job is not yet finished, skipping')\n    exit(0)\n\ncontent_list = client.fetch_content_list(queue_item['job_id'])\nif content_list is None:\n    print('Job no longer exists in oxy')\n    queue.delete(queue_item['job_id'])\n    exit(0)\n\nqueue.complete(queue_item['job_id'])\n\nfor content in content_list:\n    pprint(content)\n```\n\nWe first use `queue.pull()` to fetch a single pending job and exit if none is found. \n\n```python\nqueue_item = queue.pull()\nif not queue_item:\n    print('No jobs left in the queue, exiting')\n    exit(0)\n```\n\nThen, we check the status. If the status says the URL is not yet scraped, we use the `touch` method to renew the `updated_at` field in the database. That way, the record will not be checked for at least 10 more seconds (to prevent spamming the API).\n\n```python\nif not client.is_status_done(queue_item['job_id']):\n    queue.touch(queue_item['job_id'])\n    print('Job is not yet finished, skipping')\n    exit(0)\n```\n\nOnce the status is `done`, we try to fetch the content. If no content is returned, it means we fetched an old record that has already been deleted.\n\n```python\ncontent_list = client.fetch_content_list(queue_item['job_id'])\nif content_list is None:\n    print('Job no longer exists in oxy')\n    queue.delete(queue_item['job_id'])\n    exit(0)\n```\n\nAnd finally, we go through the content and print it.\n\n```python\nqueue.complete(queue_item['job_id'])\n\nfor content in content_list:\n    pprint(content)\n```\n\nNote: in a real production application, you would likely save the content to files or a database, but this part is beyond the scope of this tutorial.\n\nHere's what we have so far:\n\n```yaml\n|-- src\n|   |-- bootstrap.py\n|   |-- messenger.py\n|   |-- oxylabs.py\n|   |-- puller.py\n|   |-- pusher.py\n|   |-- setup.py\n\n```\n\nNow that we're done with the coding part, it's time to run the scripts using Apache Airflow!\n\n## Setting up Apache Airflow\nApache Airflow is a platform created by the community to programmatically author, schedule, and monitor workflows. Let’s set it up following their official tutorial and using the official Docker Compose file. \n\nBefore you run docker-compose up, you need to expose the files we created. To do that, change the `docker-compose.yaml` file to include the `src` folder as a volume.\n\nHere’s how it looks before:\n\n```yaml\nvolumes:\n    - ./dags:/opt/airflow/dags\n    - ./logs:/opt/airflow/logs\n    - ./plugins:/opt/airflow/plugins\n```\n\nAnd after:\n\n```yaml\nvolumes:\n    - ./src:/opt/airflow/src\n    - ./dags:/opt/airflow/dags\n    - ./logs:/opt/airflow/logs\n    - ./plugins:/opt/airflow/plugins\n```\n\nOnce you set it up, a bunch of Airflow specific files and folders are created.\n\n```yaml\n|-- dags\n|-- docker-compose.yaml\n|-- .env\n|-- logs\n|-- plugins\n|-- src\n|   |-- bootstrap.py\n|   |-- messenger.py\n|   |-- oxylabs.py\n|   |-- puller.py\n|   |-- pusher.py\n|   |-- setup.py\n```\n\nNow visit `http://localhost:8080` and use the default credentials (`airflow:airflow`) to access the Airflow UI. \n\n![](https://images.prismic.io/oxylabs-sm/9565f060-60e3-4b7e-8e5d-ced99436ff0e_5.png?auto=compress,format\u0026fm=webp\u0026dpr=2\u0026q=50)\n\nAirflow uses a concept called DAG (Directed Acyclic Graph), which is a collection of all the tasks you want to run, organized in a way that reflects their relationships and dependencies.\n\n![](https://images.prismic.io/oxylabs-sm/adc49025-2478-4995-8d75-7d1980197e1f_dag.png?auto=compress,format\u0026fm=webp\u0026dpr=2\u0026q=50)\n\nA DAG is defined in a Python script, which represents the DAGs structure (tasks and their dependencies) as code.\n\nTo create a DAG file, we have to create a python script in the `dags` folder of the airflow project.\n\nLet's call it `scrape.py`. \n\nHere's what the final file structure looks like:\n\n```yaml\n|-- dags\n|   |-- scrape.py\n|-- docker-compose.yaml\n|-- .env\n|-- logs\n|-- plugins\n|-- src\n|   |-- bootstrap.py\n|   |-- messenger.py\n|   |-- oxylabs.py\n|   |-- puller.py\n|   |-- pusher.py\n|   |-- setup.py\n\n```\n\n## Creating a DAG\nLet's create a simple DAG that sets up the table by calling the `setup.py` script we created earlier.\n\n```python\nfrom datetime import timedelta\n\nimport pendulum\nfrom airflow import DAG\nfrom airflow.operators.bash import BashOperator\n\ndefault_args = {\n    'owner': 'airflow',\n    'depends_on_past': False,\n    'retries': 2,\n    'retry_delay': timedelta(hours=3),\n}\nwith DAG(\n        'setup',\n        default_args=default_args,\n        schedule_interval='@once',\n        description='Setup',\n        start_date=pendulum.datetime(2022, 5, 1, tz='UTC'),\n        dagrun_timeout=timedelta(minutes=1),\n        tags=['scrape', 'database'],\n        catchup=False\n) as dag:\n    setup_task = BashOperator(\n        task_id='setup',\n        bash_command='python /opt/airflow/src/setup.py',\n    )\n```\n\nEvery DAG has a bunch of arguments that allow you to configure execution.\n\n```python\ndefault_args = {\n    'owner': 'airflow',\n    'depends_on_past': False,\n    'retries': 2,\n    'retry_delay': timedelta(hours=3),\n}\n```\n\nThe `owner` property describes the system user that owns this DAG.\n\n`retries` determines how many additional attempts to run this script will be made if it fails. \n\n`retry_delay` tells us how often to retry (related to the previous parameter).\n\n`depends_on_past` is extremely important for multi-stage workflows – it determines whether the previous task needs to be successful to run our setup script.\n\nNext come the DAG parameters:\n\n```python\nwith DAG(\n        'setup',\n        default_args=default_args,\n        schedule_interval='@once',\n        description='Setup',\n        start_date=pendulum.datetime(2022, 5, 1, tz='UTC'),\n        dagrun_timeout=timedelta(minutes=1),\n        tags=['scrape', 'database'],\n        catchup=False\n) as dag:\n```\n\nHere are the most important parameters:\n\nThe first parameter (`setup`) always signifies the name of the DAG, which you will see in the Airflow UI.\n\n`schedule_interval` muses cron-like format to determine how often to run the task.\n\n`start_date` describes when the task has to be started. \n\n`catchup` determines whether Airflow needs to catch up to the current date by running the scripts for earlier dates\n\nNext comes the most important part: every DAG defines what tasks need to be done. \n\n```python\n    setup_task = BashOperator(\n        task_id='setup',\n        bash_command='python /opt/airflow/src/setup.py',\n    )\n```\n\nTasks are defined using Operators, which allow you to describe what needs to be done. For us, the easiest way is to simply run the `setup.py` file using a bash command.\n\nOur file is automatically registered and displayed in the Airflow UI.\n\n![](https://images.prismic.io/oxylabs-sm/d7de32a6-1ea3-441f-b1b9-b7e9838d63a4_6.png?auto=compress,format\u0026fm=webp\u0026dpr=2\u0026q=50)\n\n## Defining multiple tasks\n\nWe previously defined a DAG with a single task. While that is a great achievement, the main power of Airflow comes from managing multiple tasks. Let's leverage that by creating a push-pull workflow:\n\n```python\nfrom datetime import timedelta\n\nimport pendulum\nfrom airflow import DAG\nfrom airflow.operators.bash import BashOperator\n\ndefault_args = {\n    'owner': 'airflow',\n    'depends_on_past': True,\n    'retries': 2,\n    'retry_delay': timedelta(hours=3),\n}\nwith DAG(\n        'push_pull',\n        default_args=default_args,\n        schedule_interval='@daily',\n        description='Push-Pull workflow',\n        start_date=pendulum.datetime(2022, 5, 1, tz='UTC'),\n        dagrun_timeout=timedelta(minutes=1),\n        tags=['scrape', 'database'],\n        catchup=False\n) as dag:\n    task_push = BashOperator(\n        task_id='push',\n        bash_command='python /opt/airflow/src/pusher.py',\n    )\n\n    task_pull = BashOperator(\n        task_id='pull',\n        bash_command='python /opt/airflow/src/puller.py'\n    )\n\n    task_push.set_downstream(task_pull)\n```\n\nMost of the code remains the same. There is a major difference though: we now have two tasks: `task_pull` and `task_push`. Once we create them, we use the `set_downstream` to tell Airflow that `task_pull` needs to be executed after `task_push`.\n\n![](https://images.prismic.io/oxylabs-sm/464b4aa8-dcd9-41d8-aec9-7cb1d515a6cf_7.png?auto=compress,format\u0026fm=webp\u0026dpr=2\u0026q=50)\n\n## Using ShortCircuitOperator\n\nOur previous DAG has a glaring issue: we want to execute push once as it uses the batch endpoint. The pull task, however, needs to run multiple times. Unfortunately, tasks in the same dag cannot use different intervals.\n\n```python\n    task_push = BashOperator(\n        task_id='push',\n        bash_command='python /opt/airflow/src/pusher.py',\n        schedule_interval='daily',  # not allowed!\n    )\n\n    task_pull = BashOperator(\n        task_id='pull',\n        bash_command='python /opt/airflow/src/puller.py',\n        schedule_interval='@hourly', # not allowed!\n    )\n```\n\nWhile it is indeed possible to solve that by creating multiple DAGs, it ruins many advantages Airflow provides in managing the dependencies of the tasks. Let’s take a look at how this can be fixed. \n\nEnter `ShortCircuitOperator`. This powerful operator can skip tasks if conditions are not met.\n\n```python\nfrom datetime import timedelta\nimport pendulum\nfrom airflow import DAG\nfrom airflow.operators.bash import BashOperator\nfrom airflow.operators.python import ShortCircuitOperator\n\n\ndefault_args = {\n    'owner': 'airflow',\n    'depends_on_past': True,\n    'retries': 2,\n    'retry_delay': timedelta(hours=3),\n}\nwith DAG(\n        'push-pull-reworked',\n        default_args=default_args,\n        schedule_interval='* * * * *',\n        description='Scrape the website',\n        start_date=pendulum.datetime(2022, 5, 1, tz='UTC'),\n        dagrun_timeout=timedelta(minutes=1),\n        tags=['scrape', 'oxylabs', 'push', 'pull'],\n        catchup=False\n) as dag:\n    def is_midnight(logical_date):\n        return logical_date.hour == 0 and logical_date.minute == 0\n\n    trigger_once_per_day = ShortCircuitOperator(\n        task_id='once_per_day',\n        python_callable=is_midnight,\n        provide_context=True,\n        dag=dag\n    )\n\n    task_push = BashOperator(\n        task_id='push',\n        bash_command='python /opt/airflow/src/pusher.py',\n    )\n    trigger_once_per_day.set_downstream(task_push)\n\n    task_pull = BashOperator(\n        task_id='pull',\n        bash_command='python /opt/airflow/src/puller.py'\n    )\n```\n\nEven though the schedule interval is set to `* * * * *`, which means `execute every minute`, but we add an additional `once_per_day` task, that prevents the `push` task from running unless the current hour is '0'.\n\nThe main player here is the `is_midnight` function, that checks whether the current time is 00:00 (the only time when the push command is allowed to run!)\n\n![](https://images.prismic.io/oxylabs-sm/f3017157-b222-4c9d-b0fe-3ae5fa9af6b0_8.png?auto=compress,format\u0026fm=webp\u0026dpr=2\u0026q=50)\n\n## Combining all tasks\nYou might be curious whether combining all our tasks into a single workflow is possible. Luckily, it is – let’s look at how it’s done. \n\nAgain, by using `ShortCircuitOperator` we’re able to circumvent the task limitations.\n\n```python\nfrom datetime import timedelta\nimport pendulum\nfrom airflow import DAG\nfrom airflow.operators.bash import BashOperator\nfrom airflow.operators.python import ShortCircuitOperator\n\n\ndefault_args = {\n    'owner': 'airflow',\n    'depends_on_past': True,\n    'retries': 2,\n    'retry_delay': timedelta(hours=3),\n}\nwith DAG(\n        'scrape',\n        default_args=default_args,\n        schedule_interval='* * * * *',\n        description='Scrape the website',\n        start_date=pendulum.datetime(2022, 5, 1, tz='UTC'),\n        dagrun_timeout=timedelta(minutes=1),\n        tags=['scrape', 'oxylabs', 'push', 'pull'],\n        catchup=False\n) as dag:\n    trigger_always = ShortCircuitOperator(\n        task_id='always',\n        python_callable=lambda prev_start_date_success: prev_start_date_success is not None,\n        provide_context=True,\n        dag=dag\n    )\n\n    trigger_once = ShortCircuitOperator(\n        task_id='once',\n        python_callable=lambda prev_start_date_success: prev_start_date_success is None,\n        provide_context=True,\n        dag=dag\n    )\n\n    setup_task = BashOperator(\n        task_id='setup',\n        bash_command='python /opt/airflow/src/setup.py',\n    )\n\n    trigger_once.set_downstream(setup_task)\ndef is_midnight(logical_date):\n        return logical_date.hour == 0 and logical_date.minute == 0\n\n    trigger_once_per_day = ShortCircuitOperator(\n        task_id='once_per_day',\n        python_callable=is_midnight,\n        provide_context=True,\n        dag=dag\n    )\n\n    task_push = BashOperator(\n        task_id='push',\n        bash_command='python /opt/airflow/src/pusher.py',\n    )\n    trigger_once_per_day.set_downstream(task_push)\n\n    task_pull = BashOperator(\n        task_id='pull',\n        bash_command='python /opt/airflow/src/puller.py'\n    )\n\n    trigger_always.set_downstream(task_pull)\n    trigger_always.set_downstream(trigger_once_per_day)\n```\n\nHere we add three helper tasks:\n\n`always` - executed everytime\n\n`once` - executed one time\n\n`once_per_day` - executed daily\n\nThe implementation of the new tasks depends on a special variable called `prev_start_date_success`. It contains the start date from the previous run (if available). It allows us to determine whether any previous runs exist.\n\nHere's how our final workflow looks like:\n\n![](https://images.prismic.io/oxylabs-sm/77ffacd1-6175-42f5-b1da-7076000bdbe2_9.png?auto=compress,format\u0026fm=webp\u0026dpr=2\u0026q=50)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foxylabs%2Fbuilding-scraping-pipeline-apache-airflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foxylabs%2Fbuilding-scraping-pipeline-apache-airflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foxylabs%2Fbuilding-scraping-pipeline-apache-airflow/lists"}