{"id":21947634,"url":"https://github.com/vojay-dev/airflow-dynamic-task-mapping","last_synced_at":"2026-04-17T03:01:37.217Z","repository":{"id":264241230,"uuid":"892727095","full_name":"vojay-dev/airflow-dynamic-task-mapping","owner":"vojay-dev","description":"This project showcases how to use Dynamic Task Mapping in Apache Airflow","archived":false,"fork":false,"pushed_at":"2024-11-22T20:03:44.000Z","size":847,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-18T22:39:40.413Z","etag":null,"topics":["airflow","apache-airflow","data-engineering","duckdb","dynamic-task-mapping","modern-airflow","orchestration","python","taskflow","taskflow-api"],"latest_commit_sha":null,"homepage":"https://medium.com/data-engineer-things/stop-creating-multiple-airflow-dags-for-reloads-and-parallel-processing-3912974b5866","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vojay-dev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-11-22T16:55:24.000Z","updated_at":"2024-11-22T20:03:47.000Z","dependencies_parsed_at":"2024-11-22T21:29:36.329Z","dependency_job_id":null,"html_url":"https://github.com/vojay-dev/airflow-dynamic-task-mapping","commit_stats":null,"previous_names":["vojay-dev/airflow-dynamic-task-mapping"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/vojay-dev/airflow-dynamic-task-mapping","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vojay-dev%2Fairflow-dynamic-task-mapping","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vojay-dev%2Fairflow-dynamic-task-mapping/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vojay-dev%2Fairflow-dynamic-task-mapping/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vojay-dev%2Fairflow-dynamic-task-mapping/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vojay-dev","download_url":"https://codeload.github.com/vojay-dev/airflow-dynamic-task-mapping/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vojay-dev%2Fairflow-dynamic-task-mapping/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31913078,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-16T18:22:33.417Z","status":"online","status_checked_at":"2026-04-17T02:00:06.879Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","apache-airflow","data-engineering","duckdb","dynamic-task-mapping","modern-airflow","orchestration","python","taskflow","taskflow-api"],"created_at":"2024-11-29T05:08:51.316Z","updated_at":"2026-04-17T03:01:37.190Z","avatar_url":"https://github.com/vojay-dev.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Dynamic Task Mapping Demo\n\nThis project showcases how to use Dynamic Task Mapping in Apache Airflow.\n\n![Dynamic Task Mapping demo](doc/demo.gif)\n*Dynamic Task Mapping demo*\n\nFor details, please read my article: [Stop Creating Multiple Airflow DAGs for Reloads and Parallel Processing](https://medium.com/data-engineer-things/stop-creating-multiple-airflow-dags-for-reloads-and-parallel-processing-3912974b5866)\n\nIn this article, we’ll tackle a common challenge in Airflow development: the proliferation of nearly identical DAGs for\nhandling different data processing scenarios, especially those involving partitioned tables and historical reloads.\nYou’ll learn how to build a single, flexible DAG that leverages Dynamic Task Mapping to process partitions in parallel,\nhandling both daily operations and custom date range reloads with ease.\n\n![Dynamic Task Mapping DAG](doc/dag.png)\n*Dynamic Task Mapping DAG*\n\nThe DAG uses [DuckDB](https://duckdb.org/) to write partitioned [Parquet files](https://parquet.apache.org/),\nsimulating a typical data partitioning scenario. The data is partitioned by day. The DAG implements two modes:\n1. **Daily Mode:** Processes a single day's partition (regular scheduling scenario).\n2. **Custom Mode:** Reprocesses a specified date range in parallel using Dynamic Task Mapping.\n\nIn addition to Dynamic Task Mapping, the example showcases these modern Airflow features:\n* [TaskFlow API](https://airflow.apache.org/docs/apache-airflow/stable/tutorial/taskflow.html)\n* DAG parameters\n* Python Virtual Environment Operator\n\n## Setup\n\n1. Start Airflow on your local machine by running `astro dev start`.\n\nThis command will spin up 4 Docker containers on your machine, each for a different Airflow component:\n\n- Postgres: Airflow's Metadata Database\n- Webserver: The Airflow component responsible for rendering the Airflow UI\n- Scheduler: The Airflow component responsible for monitoring and triggering tasks\n- Triggerer: The Airflow component responsible for triggering deferred tasks\n\n2. Verify that all 4 Docker containers were created by running `docker ps`.\n\nNote: Running `astro dev start` will start your project with the Airflow Webserver exposed at port 8080 and Postgres\nexposed at port 5432.\n\n3. Access the Airflow UI for your local Airflow project. To do so, go to http://localhost:8080/ and log in with `admin` as user and password.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvojay-dev%2Fairflow-dynamic-task-mapping","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvojay-dev%2Fairflow-dynamic-task-mapping","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvojay-dev%2Fairflow-dynamic-task-mapping/lists"}