{"id":18941155,"url":"https://github.com/couchbase-ecosystem/airflow-providers-couchbase","last_synced_at":"2026-02-21T02:02:12.072Z","repository":{"id":258079049,"uuid":"874155618","full_name":"Couchbase-Ecosystem/airflow-providers-couchbase","owner":"Couchbase-Ecosystem","description":"Apache airflow provider for couchbase","archived":false,"fork":false,"pushed_at":"2025-04-14T15:03:24.000Z","size":90192,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-12-22T01:56:08.541Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Couchbase-Ecosystem.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-10-17T10:51:11.000Z","updated_at":"2025-05-14T22:13:11.000Z","dependencies_parsed_at":"2025-04-15T22:37:13.781Z","dependency_job_id":null,"html_url":"https://github.com/Couchbase-Ecosystem/airflow-providers-couchbase","commit_stats":null,"previous_names":["couchbase-ecosystem/airflow-providers-couchbase"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/Couchbase-Ecosystem/airflow-providers-couchbase","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Couchbase-Ecosystem%2Fairflow-providers-couchbase","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Couchbase-Ecosystem%2Fairflow-providers-couchbase/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Couchbase-Ecosystem%2Fairflow-providers-couchbase/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Couchbase-Ecosystem%2Fairflow-providers-couchbase/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Couchbase-Ecosystem","download_url":"https://codeload.github.com/Couchbase-Ecosystem/airflow-providers-couchbase/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Couchbase-Ecosystem%2Fairflow-providers-couchbase/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29671513,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-21T00:11:43.526Z","status":"online","status_checked_at":"2026-02-21T02:00:07.432Z","response_time":107,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-08T12:26:20.923Z","updated_at":"2026-02-21T02:02:12.058Z","avatar_url":"https://github.com/Couchbase-Ecosystem.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003eairflow-providers-couchbase\u003c/h1\u003e\n\n\u003cp align=\"center\"\u003eA custom \u003ca href=\"https://airflow.apache.org/\"\u003e\u003ci\u003eApache Airflow\u003c/i\u003e\u003c/a\u003e provider for \u003ca href=\"https://www.couchbase.com\"\u003e\u003ci\u003eCouchbase\u003c/i\u003e\u003c/a\u003e\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://github.com/Couchbase-Ecosystem/airflow-providers-couchbase/actions?query=workflow%3Aci-integration-test\"\u003e\n    \u003cimg alt=\"ci\" src=\"https://github.com/Couchbase-Ecosystem/airflow-providers-couchbase/actions/workflows/ci-integration-test.yml/badge.svg\" /\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://pypi.org/project/airflow-providers-couchbase/\"\u003e\n    \u003cimg alt=\"pypi version\" src=\"https://img.shields.io/pypi/v/airflow-providers-couchbase.svg\" /\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://pypi.org/project/apache-airflow/\"\u003e\n    \u003cimg alt=\"airflow version\" src=\"https://img.shields.io/pypi/v/apache-airflow.svg?label=airflow\" /\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n---\n\n# Table of Contents\n\n- [Overview](#overview)\n- [Installation](#installation)\n- [Running a DAG in Docker](#running-a-dag-example-inside-an-airflow-docker-container)\n  - [Prerequisites](#prerequisites)\n  - [Connecting to a Local Couchbase Server](#connecting-to-a-local-couchbase-server)\n  - [Steps to Run the Example DAG](#steps-to-run-the-example-dag)\n  - [Example DAGs Explanation](#example-dags-explanation)\n  - [Clean Up](#clean-up)\n- [Additional Resources](#additional-resources)\n- [License](#license)\n\n## Overview\n\nThe `airflow-providers-couchbase` enables interactions with Couchbase clusters within Apache Airflow workflows. It provides custom Couchbase Hook that allow users to seamlessly interact with Couchbase databases, execute queries, manage documents, and more.\n\nFor those new to Apache Airflow, a [DAG (Directed Acyclic Graph)](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dags.html) represents a workflow as a collection of tasks with directional dependencies. Each task in our example DAGs performs specific operations with Couchbase, such as querying data or processing documents.\n\n## Installation\n\n```bash\npip install airflow-providers-couchbase\n```\n\n## Running a DAG Example Inside an Airflow Docker Container\n\n### Prerequisites\n\n- Docker and Docker Compose installed on your machine\n- A running Couchbase cluster accessible from your Docker container\n- The travel-sample bucket available in your Couchbase cluster\n\n### Connecting to a Local Couchbase Server\n\nIf you're running Couchbase Server locally (either installed natively or in Docker), you'll need to make some adjustments to connect from the Airflow containers:\n\n1. **For locally installed Couchbase Server**:\n   - Update the Couchbase connection host to use your machine's IP address instead of `localhost`\n   - Example: `couchbase://192.168.1.100` (replace with your actual IP)\n\n2. **For Couchbase Server running in Docker**:\n   - Add Couchbase service to your docker-compose.yaml:\n\n   ```yaml\n   services:\n     couchbase:\n       image: couchbase/server:latest\n       ports:\n         - \"8091-8096:8091-8096\"\n         - \"11210-11211:11210-11211\"\n       networks:\n         - airflow-couchbase\n\n     airflow-common:\n       networks:\n         - airflow-couchbase\n         \n   networks:\n     airflow-couchbase:\n       driver: bridge\n   ```\n\n   - Update the Couchbase connection host to use the service name: `couchbase://couchbase`\n\nRemember to initialize your Couchbase Server with:\n\n- Create a bucket named \"travel-sample\"\n- Import the travel-sample dataset\n- Create a user with appropriate permissions\n\n### Steps to Run the Example DAG\n\n1. **Navigate to the Docker Directory**:\n\n   ```bash\n   cd docker\n   ```\n\n2. **Build and Run the Docker Containers**:\n\n   ```bash\n   # Initialize the Airflow database and create the first user account\n   docker compose up airflow-init\n   \n   # Start all services defined in docker-compose.yml\n   docker compose up --build\n   ```\n\n3. **Access the Airflow Web UI**:\n   - Open your web browser: \u003chttp://localhost:8080\u003e\n   - Login credentials:\n     - Username: `airflow`\n     - Password: `airflow`\n\n4. **Configure a Couchbase Connection**:\n   - Go to Admin -\u003e Connections\n   - Click \"+\" to add a new connection\n   - Fill in the details:\n     - Connection Id: `couchbase_conn_id`\n     - Connection Type: Couchbase\n     - Host: `your_couchbase_host`\n     - Login: Your Couchbase username\n     - Password: Your Couchbase password\n     - Extra: Additional configuration parameters (JSON format)\n\n5. **Trigger the DAG**:\n   - Go to DAGs view\n   - Find \"airflow_test_couchbase_cluster\" DAG\n   - Click \"Play\" to trigger a manual run\n\n6. **Monitor the Execution**:\n   - Click on the DAG run to view progress\n   - View logs, duration, and status for each task\n   - For a visual guide of this process, check our step-by-step tutorial on [docs/videos/airflow](docs/videos/airflow.mp4)\n\n### Example DAGs Explanation\n\n#### 1. airflow_test_couchbase_cluster.py\n\n- Basic Couchbase interaction example\n- Connects using CouchbaseHook\n- Queries \"airline\" collection\n- Extracts document IDs\n- Converts data to CSV\n\n#### 2. airflow_test_couchbase_scope.py\n\n- Advanced example with direct scope/collection access\n- Retrieves document IDs and full documents\n- Includes error handling\n- Processes complete documents\n\nBoth DAGs follow ETL pattern:\n\n- **Extract**: \n  - Connect to Couchbase using CouchbaseHook\n  - Query the \"airline\" collection in \"travel-sample\" bucket\n  - Retrieve document IDs and data (airflow_test_couchbase_scope.py retrieves full documents, airflow_test_couchbase_cluster.py retrieves IDs only)\n\n- **Transform**: \n  - Convert Couchbase query results to pandas DataFrame\n  - Save DataFrame to a temporary CSV file using `tempfile.NamedTemporaryFile`\n\n- **Load**: \n  - Read the temporary CSV file to verify data integrity\n  - Clean up by removing the temporary file\n  - (Placeholder: In production, you would upload to S3 or similar storage)\n\n### Clean Up\n\n```bash\ndocker-compose down -v\n```\n\n## Additional Resources\n\n- [Apache Airflow Documentation](https://airflow.apache.org/docs/apache-airflow/stable/index.html)\n- [Airflow Installation Guide](https://airflow.apache.org/docs/apache-airflow/stable/installation/index.html)\n\n## License\n\n`airflow-providers-couchbase` is distributed under the terms of the [MIT](https://spdx.org/licenses/MIT.html) license.\n\n---\n\n## 📢 Support Policy\n\nWe truly appreciate your interest in this project!  \nThis project is **community-maintained**, which means it's **not officially supported** by our support team.\n\nIf you need help, have found a bug, or want to contribute improvements, the best place to do that is right here — by [opening a GitHub issue](https://github.com/Couchbase-Ecosystem/airflow-providers-couchbase/issues).  \nOur support portal is unable to assist with requests related to this project, so we kindly ask that all inquiries stay within GitHub.\n\nYour collaboration helps us all move forward together — thank you!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcouchbase-ecosystem%2Fairflow-providers-couchbase","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcouchbase-ecosystem%2Fairflow-providers-couchbase","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcouchbase-ecosystem%2Fairflow-providers-couchbase/lists"}