{"id":30957979,"url":"https://github.com/alvarocavalcante/airflow-custom-deferrable-dataflow-operator","last_synced_at":"2025-09-11T14:33:27.235Z","repository":{"id":280464508,"uuid":"937777549","full_name":"AlvaroCavalcante/airflow-custom-deferrable-dataflow-operator","owner":"AlvaroCavalcante","description":"Start your Dataflow jobs execution directly from the Triggerer without going to the Worker!","archived":false,"fork":false,"pushed_at":"2025-03-03T15:22:39.000Z","size":37,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-09-09T04:58:50.891Z","etag":null,"topics":["airflow","airflow-dags","airflow-operators","dags","data-engineering","dataflow","gcp","python","python3"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AlvaroCavalcante.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-02-23T21:33:23.000Z","updated_at":"2025-03-03T15:14:33.000Z","dependencies_parsed_at":"2025-03-03T15:45:31.655Z","dependency_job_id":"d96a51db-6fe9-4d9e-bc96-1e8685142504","html_url":"https://github.com/AlvaroCavalcante/airflow-custom-deferrable-dataflow-operator","commit_stats":null,"previous_names":["alvarocavalcante/airflow-custom-deferrable-dataflow-operator"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/AlvaroCavalcante/airflow-custom-deferrable-dataflow-operator","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AlvaroCavalcante%2Fairflow-custom-deferrable-dataflow-operator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AlvaroCavalcante%2Fairflow-custom-deferrable-dataflow-operator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AlvaroCavalcante%2Fairflow-custom-deferrable-dataflow-operator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AlvaroCavalcante%2Fairflow-custom-deferrable-dataflow-operator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AlvaroCavalcante","download_url":"https://codeload.github.com/AlvaroCavalcante/airflow-custom-deferrable-dataflow-operator/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AlvaroCavalcante%2Fairflow-custom-deferrable-dataflow-operator/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":274655587,"owners_count":25325756,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-11T02:00:13.660Z","response_time":74,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","airflow-dags","airflow-operators","dags","data-engineering","dataflow","gcp","python","python3"],"created_at":"2025-09-11T14:33:25.803Z","updated_at":"2025-09-11T14:33:27.209Z","avatar_url":"https://github.com/AlvaroCavalcante.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Airflow Custom Deferrable Dataflow Operator\n**Trigger Different: Cut Your AirFlow Costs By Starting From Triggerer!**\n\nUse this simple Airflow Operator to start your Dataflow jobs execution directly from the Triggerer without going to the Worker!\n\n# Contents\n\n- [How it works](#how)\n- [Installation](#installation)\n- [Usage](#usage)\n- [Contribute](#contribute)\n\n# How It Works \u003ca id=\"how\"\u003e\u003c/a\u003e\nThe main idea of this approach is to start the task instance execution [directly on the Triggerer](https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/deferring.html#triggering-deferral-from-task-start) component, bypassing the worker entirely.\n\nThis strategy is effective because, in this case, the only action the operator performs is making an HTTP request to start the external processing service and waiting for the job to complete.\n\nFor this reason, we can leverage the async design of the Triggerer for this execution, significantly reducing resource consumption. By using this architecture, the Airflow task execution proccess will be something like the following:\n\n![airflow_diagram](assets/airflow_diagram.png)\n\nTo know more about how the tool works, check out the Medium article.\n\n# Installation \u003ca id=\"installation\"\u003e\u003c/a\u003e\nThe installation process will depend on your cloud provider or how you have set up your environment. \n\nRegarding [Google Cloud Composer](https://cloud.google.com/composer?hl=en), for example, the **DAGs folder** is not synchronized with the Airflow Triggerer, as stated in the [documentation](https://cloud.google.com/composer/docs/composer-2/troubleshooting-triggerer#trigger_class_not_found). \n\nConsequently, just uploading your code to the DAGs folder will not work, and you'll likely face an error like this: ```ImportError: Module \"PACKAGE_NAME\" does not define a \"CLASS_NAME\" attribute/class```\n\nIn this case, it's necessary to [import the missing code from PyPI](https://cloud.google.com/composer/docs/composer-2/install-python-dependencies), meaning that you'll need to install the operator/trigger as a new library.\n\nTo do so, you can use the following command:\n\n```bash\npip install custom-deferrable-dataflow-operator\n```\n\n# Usage \u003ca id=\"usage\"\u003e\u003c/a\u003e\nAfter installing the library, you can successfully import and use the operator in your Airflow DAGs, as shown below:\n\n```python\nfrom deferrable_dataflow_operator import DeferrableDataflowOperator\n\ndataflow_triggerer_job = DeferrableDataflowOperator(\n    trigger_kwargs={\n        \"project_id\": GCP_PROJECT_ID,\n        \"region\": GCP_REGION,\n        \"body\": {\n            \"job_name\": MY_JOB_NAME,\n            \"parameters\": {\n                \"dataflow-parameters\": MY_PARAMETERS\n            },\n            \"environment\": {**dataflow_env_vars},\n            \"container_spec_gcs_path\": TEMPLATE_GCS_PATH,\n        }\n    },\n    start_from_trigger=True,\n    task_id=MY_TASK_ID\n)\n```\nIn the ```trigger_kwargs``` parameter, it's important to specify your GCP project ID and region. The ```body``` parameter, on the other hand, should contain all the relevant information for your Dataflow job, as stated in the [official documentation](https://cloud.google.com/python/docs/reference/dataflow/latest/google.cloud.dataflow_v1beta3.types.LaunchFlexTemplateParameter).\n\n# Contributing \u003ca id=\"contribute\"\u003e\u003c/a\u003e\nThis project is open to contributions! If you want to collaborate to improve the operator, please follow these steps:\n\n1.  Open a new issue to discuss the feature or bug you want to address.\n2.  Once approved, fork the repository and create a new branch.\n3.  Implement the changes.\n4.  Create a pull request with a detailed description of the changes.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falvarocavalcante%2Fairflow-custom-deferrable-dataflow-operator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falvarocavalcante%2Fairflow-custom-deferrable-dataflow-operator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falvarocavalcante%2Fairflow-custom-deferrable-dataflow-operator/lists"}