{"id":20532079,"url":"https://github.com/prefecthq/prefect-airbyte","last_synced_at":"2025-10-11T23:30:50.510Z","repository":{"id":38222475,"uuid":"476824445","full_name":"PrefectHQ/prefect-airbyte","owner":"PrefectHQ","description":null,"archived":false,"fork":false,"pushed_at":"2024-12-03T20:01:24.000Z","size":1929,"stargazers_count":41,"open_issues_count":4,"forks_count":8,"subscribers_count":14,"default_branch":"main","last_synced_at":"2025-01-23T00:30:20.954Z","etag":null,"topics":["airbyte","prefect"],"latest_commit_sha":null,"homepage":"https://PrefectHQ.github.io/prefect-airbyte/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/PrefectHQ.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-04-01T18:11:57.000Z","updated_at":"2024-12-25T22:10:41.000Z","dependencies_parsed_at":"2023-01-31T09:01:17.279Z","dependency_job_id":"b9e327b7-eab6-40c0-adf2-cf3879952dbf","html_url":"https://github.com/PrefectHQ/prefect-airbyte","commit_stats":{"total_commits":128,"total_committers":6,"mean_commits":"21.333333333333332","dds":0.21875,"last_synced_commit":"9b2ba49ab775247d3f305b4f2f51445996daa387"},"previous_names":[],"tags_count":12,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PrefectHQ%2Fprefect-airbyte","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PrefectHQ%2Fprefect-airbyte/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PrefectHQ%2Fprefect-airbyte/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PrefectHQ%2Fprefect-airbyte/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/PrefectHQ","download_url":"https://codeload.github.com/PrefectHQ/prefect-airbyte/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":236136623,"owners_count":19100628,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airbyte","prefect"],"created_at":"2024-11-16T00:12:39.789Z","updated_at":"2025-10-11T23:30:45.186Z","avatar_url":"https://github.com/PrefectHQ.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003e [!IMPORTANT] This repo is no longer actively maintained by @PrefectHQ\n\u003e You can find integrations actively maintained by @PrefectHQ [here](https://docs.prefect.io/integrations/integrations)\n\n# prefect-airbyte\n\n\u003cp align=\"center\"\u003e\n    \u003ca href=\"https://pypi.python.org/pypi/prefect-airbyte/\" alt=\"PyPI version\"\u003e\n        \u003cimg alt=\"PyPI\" src=\"https://img.shields.io/pypi/v/prefect-airbyte?color=0052FF\u0026labelColor=090422\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://github.com/PrefectHQ/prefect-airbyte/\" alt=\"Stars\"\u003e\n        \u003cimg src=\"https://img.shields.io/github/stars/PrefectHQ/prefect-airbyte?color=0052FF\u0026labelColor=090422\" /\u003e\u003c/a\u003e\n    \u003ca href=\"https://pepy.tech/badge/prefect-airbyte/\" alt=\"Downloads\"\u003e\n        \u003cimg src=\"https://img.shields.io/pypi/dm/prefect-airbyte?color=0052FF\u0026labelColor=090422\" /\u003e\u003c/a\u003e\n    \u003ca href=\"https://github.com/PrefectHQ/prefect-airbyte/pulse\" alt=\"Activity\"\u003e\n        \u003cimg src=\"https://img.shields.io/github/commit-activity/m/PrefectHQ/prefect-airbyte?color=0052FF\u0026labelColor=090422\" /\u003e\u003c/a\u003e\n    \u003cbr\u003e\n    \u003ca href=\"https://prefect-community.slack.com\" alt=\"Slack\"\u003e\n        \u003cimg src=\"https://img.shields.io/badge/slack-join_community-red.svg?color=0052FF\u0026labelColor=090422\u0026logo=slack\" /\u003e\u003c/a\u003e\n    \u003ca href=\"https://discourse.prefect.io/\" alt=\"Discourse\"\u003e\n        \u003cimg src=\"https://img.shields.io/badge/discourse-browse_forum-red.svg?color=0052FF\u0026labelColor=090422\u0026logo=discourse\" /\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n## Welcome!\n\n`prefect-airbyte` is a collection of prebuilt Prefect tasks and flows that can be used to quickly construct Prefect flows to interact with [Airbyte](https://airbyte.io/).\n\n## Getting Started\n\n### Python setup\n\nRequires an installation of Python 3.7+\n\nWe recommend using a Python virtual environment manager such as pipenv, conda or virtualenv.\n\nThese tasks are designed to work with Prefect 2.0. For more information about how to use Prefect, please refer to the [Prefect documentation](https://orion-docs.prefect.io/).\n\n### Airbyte setup\nSee [the airbyte documention](https://docs.airbyte.com/deploying-airbyte) on how to get your own instance.\n\n### Installation\n\nInstall `prefect-airbyte`\n\n```bash\npip install prefect-airbyte\n```\n\nA list of available blocks in `prefect-airbyte` and their setup instructions can be found [here](https://PrefectHQ.github.io/prefect-airbyte/#blocks-catalog).\n\n### Examples\n#### Create an `AirbyteServer` block and save it\n```python\nfrom prefect_airbyte.server import AirbyteServer\n\n# running airbyte locally at http://localhost:8000 with default auth\nlocal_airbyte_server = AirbyteServer()\n\n# running airbyte remotely at http://\u003csomeIP\u003e:\u003csomePort\u003e as user `Marvin`\nremote_airbyte_server = AirbyteServer(\n    username=\"Marvin\",\n    password=\"DontPanic42\",\n    server_host=\"42.42.42.42\",\n    server_port=\"4242\"\n)\n\nlocal_airbyte_server.save(\"my-local-airbyte-server\")\n\nremote_airbyte_server.save(\"my-remote-airbyte-server\")\n\n```\n\n\n#### Trigger a defined connection sync\n```python\nfrom prefect import flow\nfrom prefect_airbyte.server import AirbyteServer\nfrom prefect_airbyte.connections import AirbyteConnection\nfrom prefect_airbyte.flows import run_connection_sync\n\nserver = AirbyteServer(server_host=\"localhost\", server_port=8000)\n\nconnection = AirbyteConnection(\n    airbyte_server=server,\n    connection_id=\"e1b2078f-882a-4f50-9942-cfe34b2d825b\",\n    status_updates=True,\n)\n\n@flow\ndef airbyte_syncs():\n    # do some setup\n\n    sync_result = run_connection_sync(\n        airbyte_connection=connection,\n    )\n\n    # do some other things, like trigger DBT based on number of records synced\n    print(f'Number of Records Synced: {sync_result.records_synced}')\n```\n\n```console\n\n❯ python airbyte_syncs.py\n03:46:03 | prefect.engine - Created flow run 'thick-seahorse' for flow 'example_trigger_sync_flow'\n03:46:03 | Flow run 'thick-seahorse' - Using task runner 'ConcurrentTaskRunner'\n03:46:03 | Flow run 'thick-seahorse' - Created task run 'trigger_sync-35f0e9c2-0' for task 'trigger_sync'\n03:46:03 | prefect - trigger airbyte connection: e1b2078f-882a-4f50-9942-cfe34b2d825b, poll interval 3 seconds\n03:46:03 | prefect - pending\n03:46:06 | prefect - running\n03:46:09 | prefect - running\n03:46:12 | prefect - running\n03:46:16 | prefect - running\n03:46:19 | prefect - running\n03:46:22 | prefect - Job 26 succeeded.\n03:46:22 | Task run 'trigger_sync-35f0e9c2-0' - Finished in state Completed(None)\n03:46:22 | Flow run 'thick-seahorse' - Finished in state Completed('All states completed.')\n```\n\n\n#### Export an Airbyte instance's configuration\n\n**NOTE**: The API endpoint corresponding to this task is no longer supported by open-source Airbyte versions as of v0.40.7. Check out the [Octavia CLI docs](https://github.com/airbytehq/airbyte/tree/master/octavia-cli) for more info.\n\n```python\nimport gzip\n\nfrom prefect import flow, task\nfrom prefect_airbyte.configuration import export_configuration\nfrom prefect_airbyte.server import AirbyteServer\n\n@task\ndef zip_and_write_somewhere(\n      airbyte_config: bytearray,\n      somewhere: str,\n):\n    with gzip.open(somewhere, 'wb') as f:\n        f.write(airbyte_config)\n\n@flow\ndef example_export_configuration_flow(filepath: str):\n\n    # Run other tasks and subflows here\n\n    airbyte_config = export_configuration(\n        airbyte_server=AirbyteServer.load(\"my-airbyte-server-block\")\n    )\n\n    zip_and_write_somewhere(\n        somewhere=filepath,\n        airbyte_config=airbyte_config\n    )\n\nif __name__ == \"__main__\":\n    example_export_configuration_flow('*://**/my_destination.gz')\n```\n#### Use `with_options` to customize options on any existing task or flow\n\n```python\nfrom prefect import flow\nfrom prefect_airbyte.connections import AirbyteConnection\nfrom prefect_airbyte.flows import run_connection_sync\n\ncustom_run_connection_sync = run_connection_sync.with_options(\n    name=\"Custom Airbyte Sync Flow\",\n    retries=2,\n    retry_delay_seconds=10,\n)\n \n @flow\n def some_airbyte_flow():\n    custom_run_connection_sync(\n        airbyte_connection=AirbyteConnection.load(\"my-airbyte-connection-block\")\n    )\n \n some_airbyte_flow()\n```\n\nFor more tips on how to use tasks and flows in a Collection, check out [Using Collections](https://orion-docs.prefect.io/collections/usage/)!\n\n\n## Resources\n\nIf you encounter and bugs while using `prefect-airbyte`, feel free to open an issue in the [prefect-airbyte](https://github.com/PrefectHQ/prefect-airbyte) repository.\n\nIf you have any questions or issues while using `prefect-airbyte`, you can find help in either the [Prefect Discourse forum](https://discourse.prefect.io/) or the [Prefect Slack community](https://prefect.io/slack)\n\nFeel free to star or watch [`prefect-airbyte`](https://github.com/PrefectHQ/prefect-airbyte) for updates too!\n\n## Contribute\n\nIf you'd like to help contribute to fix an issue or add a feature to `prefect-airbyte`, please [propose changes through a pull request from a fork of the repository](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request-from-a-fork).\n \n### Contribution Steps:\n1. [Fork the repository](https://docs.github.com/en/get-started/quickstart/fork-a-repo#forking-a-repository)\n2. [Clone the forked repository](https://docs.github.com/en/get-started/quickstart/fork-a-repo#cloning-your-forked-repository)\n3. Install the repository and its dependencies:\n```\npip install -e \".[dev]\"\n```\n4. Make desired changes.\n5. Add tests.\n6. Insert an entry to [CHANGELOG.md](https://github.com/PrefectHQ/prefect-airbyte/blob/main/CHANGELOG.md)\n7. Install `pre-commit` to perform quality checks prior to commit:\n```\n pre-commit install\n ```\n8. `git commit`, `git push`, and create a pull request.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprefecthq%2Fprefect-airbyte","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fprefecthq%2Fprefect-airbyte","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprefecthq%2Fprefect-airbyte/lists"}