{"id":50176443,"url":"https://github.com/astronomer/apache-airflow-providers-transfers","last_synced_at":"2026-05-25T04:36:14.311Z","repository":{"id":148144596,"uuid":"618436068","full_name":"astronomer/apache-airflow-providers-transfers","owner":"astronomer","description":null,"archived":false,"fork":false,"pushed_at":"2025-09-16T15:59:11.000Z","size":955,"stargazers_count":11,"open_issues_count":28,"forks_count":3,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-09-16T18:18:38.963Z","etag":null,"topics":["airflow","airflow-operator","airflow-provider","apache-airflow","data-engineering","databricks","etl","fivetran","gcp","google-cloud","python","snowflake"],"latest_commit_sha":null,"homepage":"https://apache-airflow-provider-transfers.rtfd.io/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/astronomer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":null,"support":"docs/supported_databases.rst","governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2023-03-24T13:15:30.000Z","updated_at":"2025-09-16T15:59:14.000Z","dependencies_parsed_at":"2025-02-11T01:28:20.967Z","dependency_job_id":"f6758675-fca7-4f0f-92b3-5c26fa484b44","html_url":"https://github.com/astronomer/apache-airflow-providers-transfers","commit_stats":null,"previous_names":["astronomer/apache-airflow-providers-transfers","astronomer/apache-airflow-provider-transfers"],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/astronomer/apache-airflow-providers-transfers","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/astronomer%2Fapache-airflow-providers-transfers","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/astronomer%2Fapache-airflow-providers-transfers/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/astronomer%2Fapache-airflow-providers-transfers/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/astronomer%2Fapache-airflow-providers-transfers/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/astronomer","download_url":"https://codeload.github.com/astronomer/apache-airflow-providers-transfers/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/astronomer%2Fapache-airflow-providers-transfers/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33461090,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-25T02:24:28.008Z","status":"ssl_error","status_checked_at":"2026-05-25T02:23:23.339Z","response_time":57,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","airflow-operator","airflow-provider","apache-airflow","data-engineering","databricks","etl","fivetran","gcp","google-cloud","python","snowflake"],"created_at":"2026-05-25T04:36:12.507Z","updated_at":"2026-05-25T04:36:14.302Z","avatar_url":"https://github.com/astronomer.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## ⚠️ Discontinuation of project\n\u003e This project is no longer actively maintained by Astronomer, but we’d love to see it live on in the community. While Astronomer has paused development and is not accepting new contributions, bug fixes or releases, the code is still here for you to explore, fork and adapt under the terms of its license. \n\u003e Please note that it may not work with the latest dependencies or platforms, and it could contain security vulnerabilities. Astronomer can’t offer guarantees or warranties for its use.\n\u003e If you’re interested in adopting or stewarding this project, we’d be happy to chat, reach us at oss@astronomer.io. Thanks for being part of the open-source journey and helping keep great ideas alive!\n\n\u003ch1 align=\"center\"\u003e\n  Universal Transfer Operator\n\u003c/h1\u003e\n  \u003ch3 align=\"center\"\u003e\ntransfers made easy\u003cbr\u003e\u003cbr\u003e\n\u003c/h3\u003e\n\n\n[![CI](https://github.com/astronomer/apache-airflow-provider-transfers/actions/workflows/ci-uto.yaml/badge.svg)](https://github.com/astronomer/apache-airflow-provider-transfers)\n\nThe `UniversalTransferOperator` simplifies how users transfer data from a source to a destination using [Apache Airflow](https://airflow.apache.org/). It offers a consistent agnostic interface, improving the users' experience so they do not need to use explicitly specific providers or operators.\n\nAt the moment, it supports transferring data between [file locations](https://github.com/astronomer/apache-airflow-provider-transfers/blob/main/src/universal_transfer_operator/constants.py#L26-L32) and [databases](https://github.com/astronomer/apache-airflow-provider-transfers/blob/main/src/universal_transfer_operator/constants.py#L72-L74) (in both directions) and cross-database transfers.\n\nThis project is maintained by [Astronomer](https://astronomer.io).\n\n## Prerequisites\n\n- Apache Airflow \u003e= 2.4.0.\n\n## Install\n\nThe apache-airflow-provider-transfers is available at [PyPI](https://pypi.org/project/apache-airflow-provider-transfers/). Use the standard Python\n[installation tools](https://packaging.python.org/en/latest/tutorials/installing-packages/).\n\nTo install a cloud-agnostic version of the apache-airflow-provider-transfers, run:\n\n```shell\npip install apache-airflow-provider-transfers\n```\n\nYou can also install dependencies for using the `UniversalTransferOperator` with popular cloud providers:\n\n```shell\npip install apache-airflow-provider-transfers[amazon,google,snowflake]\n```\n\n## Quickstart\nUsers can get started quickly with following two approaches:\n1. Spinning up a local Airflow infrastructure using the open-source Astro CLI and Docker\n2. Using vanilla Airflow and Python\n\n### Run `UniversalTransferOperator` using [astro](https://docs.astronomer.io/astro/create-first-dag)\n* Create an [Astro project](https://docs.astronomer.io/astro/create-first-dag).\n  * Open your terminal or IDE\n  * Install [docker](https://docs.docker.com/engine/install/) and  [astro-cli](https://docs.astronomer.io/astro/cli/overview) using this [documentation](https://docs.astronomer.io/astro/cli/install-cli).\n  * Create a new directory for your Astro project:\n\n    ```shell\n    mkdir \u003cyour-astro-project-name\u003e\n     ```\n  * Open the directory:\n\n    ```shell\n    cd \u003cyour-astro-project-name\u003e\n    ```\n  * Run the following Astro CLI command to initialize an Astro project in the directory:\n    ```shell\n    astro dev init\n    ```\n  * This command generates the following files in the directory:\n\n\n        .\n        ├── .env # Local environment variables\n        ├── dags # Where your DAGs go\n        │   ├── example-dag-basic.py # Example DAG that showcases a simple ETL data pipeline\n        │   └── example-dag-advanced.py # Example DAG that showcases more advanced Airflow features, such as the TaskFlow API\n        ├── Dockerfile # For the Astro Runtime Docker image, environment variables, and overrides\n        ├── include # For any other files you'd like to include\n        ├── plugins # For any custom or community Airflow plugins\n        │   └── example-plugin.py\n        ├── tests # For any DAG unit test files to be run with pytest\n        │   └── test_dag_integrity.py # Test that checks for basic errors in your DAGs\n        ├── airflow_settings.yaml # For your Airflow connections, variables and pools (local only)\n        ├── packages.txt # For OS-level packages\n        └── requirements.txt # For Python packages (add apache-airflow-provider-transfers here)\n\n\n  * Add the following in requirements.txt\n\n    ```text\n    apache-airflow-provider-transfers[all]\n    ```\n\n  * Copy file named [example_transfer_and_return_files.py](./example_dags/example_transfer_and_return_files.py) and [example_snowflake_transfers.py](./example_dags/example_snowflake_transfers.py) and add it to the `dags` directory of your Airflow project:\n\n   https://github.com/astronomer/apache-airflow-provider-transfers/blob/04b53d780790eaa3b424458742bc89d6fbec2ccd/example_dags/example_transfer_and_return_files.py#L1-L45\n\n   https://github.com/astronomer/apache-airflow-provider-transfers/blob/a80dc84b7f33bb86ae244f79411b240f4f4c7e22/example_dags/example_snowflake_transfers.py#L1-L46\n\n   Alternatively, you can download `example_transfer_and_return_files.py` and `example_snowflake_transfers.py`.\n   ```shell\n    curl -O https://github.com/astronomer/apache-airflow-provider-transfers/blob/main/example_dags/example_transfer_and_return_files.py\n\n    curl -O https://github.com/astronomer/apache-airflow-provider-transfers/blob/main/example_dags/example_snowflake_transfers.py\n   ```\n\n  * Create the environment variable for AWS bucket, Snowflake and google cloud bucket for the transfer as per the following:\n\n    https://github.com/astronomer/apache-airflow-provider-transfers/blob/e711c41e841559542148c0b52f000e1159a818f7/example_dags/example_snowflake_transfers.py#L11-L12\n\n    https://github.com/astronomer/apache-airflow-provider-transfers/blob/e711c41e841559542148c0b52f000e1159a818f7/example_dags/example_transfer_and_return_files.py#L10-L14\n\n  * To start running the example DAGs in a local Airflow environment, run the following command from your project directory:\n\n    ```shell\n    astro dev start\n    ```\n  * After your project builds successfully, open the Airflow UI in your web browser at https://localhost:8080/. Find your DAGs in the dags directory in the Airflow UI.\n  * Create airflow connection for snowflake, google, SFTP and amazon using airflow UI and run the DAGs.\n  * Create airflow connection for snowflake, google and amazon using airflow UI as per documentation below.\n\n    | Cloud Provider | Connection name      |  Documentation link                                                                                                          |\n    |----------------|----------------------|------------------------------------------------------------------------------------------------------------------------------|\n    | Amazon         | aws_default          | [aws connection](https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/connections/aws.html)                |\n    | Google         | google_cloud_default | [google cloud connection](https://airflow.apache.org/docs/apache-airflow-providers-google/stable/connections/gcp.html)       |\n    | Snowflake      | snowflake_conn       | [snowflake connection](https://airflow.apache.org/docs/apache-airflow-providers-snowflake/stable/connections/snowflake.html) |\n\n  * Trigger the DAGs and validate the transfers.\n\n\n### Run `UniversalTransferOperator` using vanilla [airflow](https://airflow.apache.org/docs/apache-airflow/stable/installation/index.html) and python\n  * Install airflow and setup project following this [documentation](https://airflow.apache.org/docs/apache-airflow/stable/installation/index.html).\n  * Ensure that your Airflow environment is set up correctly by running the following commands:\n\n    ```shell\n    export AIRFLOW_HOME=`pwd`\n    airflow db init\n    ```\n\n  * Add the following in requirements.txt\n\n    ```text\n    apache-airflow-provider-transfers[all]\n    ```\n\n  * Copy file named [example_transfer_and_return_files.py](./example_dags/example_transfer_and_return_files.py) and [example_snowflake_transfers.py](./example_dags/example_snowflake_transfers.py) and add it to the `dags` directory of your Airflow project:\n\n   https://github.com/astronomer/apache-airflow-provider-transfers/blob/04b53d780790eaa3b424458742bc89d6fbec2ccd/example_dags/example_transfer_and_return_files.py#L1-L45\n\n   https://github.com/astronomer/apache-airflow-provider-transfers/blob/a80dc84b7f33bb86ae244f79411b240f4f4c7e22/example_dags/example_snowflake_transfers.py#L1-L46\n\n   Alternatively, you can download `example_transfer_and_return_files.py` and `example_snowflake_transfers.py`.\n   ```shell\n    curl -O https://github.com/astronomer/apache-airflow-provider-transfers/blob/main/example_dags/example_transfer_and_return_files.py\n\n    curl -O https://github.com/astronomer/apache-airflow-provider-transfers/blob/main/example_dags/example_snowflake_transfers.py\n   ```\n\n  * Create the environment variable for AWS bucket, Snowflake and google cloud bucket for the transfer as per the following:\n\n    https://github.com/astronomer/apache-airflow-provider-transfers/blob/e711c41e841559542148c0b52f000e1159a818f7/example_dags/example_snowflake_transfers.py#L11-L12\n\n    https://github.com/astronomer/apache-airflow-provider-transfers/blob/e711c41e841559542148c0b52f000e1159a818f7/example_dags/example_transfer_and_return_files.py#L10-L14\n\n  * Run your project in a local Airflow environment.\n  * After your project builds successfully, open the Airflow UI in your web browser at https://localhost:8080/. Find your DAGs in the dags directory in the Airflow UI.\n  * Create airflow connection for snowflake, google and amazon using airflow UI as per documentation below.\n\n    | Cloud Provider | Connection name      |  Documentation link                                                                                                          |\n    |----------------|----------------------|------------------------------------------------------------------------------------------------------------------------------|\n    | Amazon         | aws_default          | [aws connection](https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/connections/aws.html)                |\n    | Google         | google_cloud_default | [google cloud connection](https://airflow.apache.org/docs/apache-airflow-providers-google/stable/connections/gcp.html)       |\n    | Snowflake      | snowflake_conn       | [snowflake connection](https://airflow.apache.org/docs/apache-airflow-providers-snowflake/stable/connections/snowflake.html) |\n\n  * Trigger the DAGs and validate the transfers.\n\n## Example DAGs\n\nCheckout the [example_dags](./example_dags) folder for examples of how the `UniversalTransferOperator` can be used.\n\n\n## How Universal Transfer Operator Works\n\n![Approach](./docs/images/approach.png)\n\nWith `UniversalTransferOperator`, users can perform data transfers using the following transfer modes:\n\n1. Non-native\n2. Native\n3. Third-party\n\n\n### Non-native transfer\n\nNon-native transfers rely on transferring the data through the Airflow worker node. Chunking is applied where possible. This method can be suitable for datasets smaller than 2GB, depending on the source and target. The performance of this method is highly dependent upon the worker's memory, disk, processor and network configuration.\n\nInternally, the steps involved are:\n- Retrieve the dataset data in chunks from dataset storage to the worker node.\n- Send data to the cloud dataset from the worker node.\n\nFollowing is an example of non-native transfers between Google cloud storage and Sqlite:\n\nhttps://github.com/astronomer/apache-airflow-provider-transfers/blob/a80dc84b7f33bb86ae244f79411b240f4f4c7e22/example_dags/example_universal_transfer_operator.py#L68-L74\n\n### Improving bottlenecks by using native transfer\n\nAn alternative to using the Non-native transfer method is the native method. The native transfers rely on mechanisms and tools offered by the data source or data target providers. In the case of moving from object storage to a Snowflake database, for instance, a native transfer consists in using the built-in ``COPY INTO`` command. When loading data from S3 to BigQuery, the Universal Transfer Operator uses the GCP  Storage Transfer Service.\n\nThe benefit of native transfers is that they will likely perform better for larger datasets (2 GB) and do not rely on the Airflow worker node hardware configuration. With this approach, the Airflow worker nodes are used as orchestrators and do not perform the transfer. The speed depends exclusively on the service being used and the bandwidth between the source and destination.\n\nSteps:\n- Request destination dataset to ingest data from the source dataset.\n- Destination dataset requests source dataset for data.\n\n\u003e **_NOTE:_**\n The Native method implementation is in progress and will be available in future releases.\n\n\n### Transfer using a third-party tool\nThe `UniversalTransferOperator` can also offer an interface to generic third-party services that transfer data, similar to Fivetran.\n\nHere is an example of how to use Fivetran for transfers:\n\nhttps://github.com/astronomer/apache-airflow-provider-transfers/blob/7d5188c4af214d5cdaeb714654e9bdf5b48cb3fb/example_dags/example_dag_fivetran.py#L35-L56\n\n## Supported technologies\n\n- Databases supported:\n\n    https://github.com/astronomer/apache-airflow-provider-transfers/blob/513591afb967694062097d6b36170265883b77e3/src/universal_transfer_operator/constants.py#L72-L74\n\n- File store supported:\n\n    https://github.com/astronomer/apache-airflow-provider-transfers/blob/513591afb967694062097d6b36170265883b77e3/src/universal_transfer_operator/constants.py#L28-L31\n\n\n## Documentation\n\nThe documentation is a work in progress -- we aim to follow the [Diátaxis](https://diataxis.fr/) system.\n\n- **[Reference guide](https://apache-airflow-provider-transfers.readthedocs.io/)**: Commands, modules, classes and methods\n\n- **[Getting Started Tutorial](https://apache-airflow-provider-transfers.readthedocs.io/en/latest/getting-started/GETTING_STARTED.html)**: A hands-on introduction to the Universal Transfer Operator\n\n\n## Changelog\n\nThe **Universal Transfer Operator** follows semantic versioning for releases. Check the [changelog](/docs/CHANGELOG.md) for the latest changes.\n\n\n## Release management\n\nSee [Managing Releases](/docs/development/RELEASE.md) to learn more about our release philosophy and steps.\n\n\n## Contribution guidelines\n\nAll contributions, bug reports, bug fixes, documentation improvements, enhancements, and ideas are welcome.\n\nRead the [Contribution Guideline](/docs/development/CONTRIBUTING.md) for a detailed overview of how to contribute.\n\nContributors and maintainers should abide by the [Contributor Code of Conduct](CODE_OF_CONDUCT.md).\n\n\n## License\n\n[Apache Licence 2.0](LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fastronomer%2Fapache-airflow-providers-transfers","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fastronomer%2Fapache-airflow-providers-transfers","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fastronomer%2Fapache-airflow-providers-transfers/lists"}