{"id":13904675,"url":"https://github.com/chandulal/airflow-testing","last_synced_at":"2025-07-18T02:31:55.464Z","repository":{"id":33155264,"uuid":"143761554","full_name":"chandulal/airflow-testing","owner":"chandulal","description":"Airflow Unit Tests and Integration Tests","archived":false,"fork":false,"pushed_at":"2022-11-16T05:35:05.000Z","size":476,"stargazers_count":252,"open_issues_count":3,"forks_count":45,"subscribers_count":10,"default_branch":"master","last_synced_at":"2024-08-07T23:39:38.172Z","etag":null,"topics":["airflow","airflow-dags","airflow-testing","testing"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/chandulal.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-08-06T17:32:56.000Z","updated_at":"2024-05-31T05:03:20.000Z","dependencies_parsed_at":"2023-01-14T23:42:40.881Z","dependency_job_id":null,"html_url":"https://github.com/chandulal/airflow-testing","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chandulal%2Fairflow-testing","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chandulal%2Fairflow-testing/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chandulal%2Fairflow-testing/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chandulal%2Fairflow-testing/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/chandulal","download_url":"https://codeload.github.com/chandulal/airflow-testing/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":226336358,"owners_count":17608820,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","airflow-dags","airflow-testing","testing"],"created_at":"2024-08-06T23:01:00.679Z","updated_at":"2024-11-25T13:30:34.058Z","avatar_url":"https://github.com/chandulal.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# Airflow Testing\nThis project contains different categories of tests with examples. \n \n## Five Categories of Tests\n1. DAG Validation Tests: To test the validity of the DAG, checking typos and cyclicity.\n2. DAG/Pipeline Definition Tests: To test the total number of tasks in the DAG, upstream and downstream dependencies of each task, etc.\n3. Unit Tests: To test the logic of custom Operators, custom Sensor, etc.\n4. Integration Tests: To test the communication between tasks. For example, task1 pass some information to task 2 using Xcoms.\n5. End to End Pipeline Tests: To test and verify the integration between each task. You can also assert the data on successful completion of the E2E pipeline.\n\nClone this repo to run these test in your local machine.\n\n## Unit Tests\n\nUnit tests cover all tests falls under teh first four categories.  \n\n#### How to run?\n1. Build the airflow image. Go to project root directory and run \u003cbr\u003e\n\n    ```docker build . -t airflow-test```\n\n2. Run the unit tests from the docker. Use your repository location for **{SourceDir}** (Eg. If you cloned your repo at `/User/username/airflow-testing/` then \nSourceDir is `/User/username`.)\n\n     ```docker run -ti  -v {SourceDir}/airflow-testing:/opt --entrypoint /mnt/entrypoint.sh airflow-test run_unit_tests```\n\n## End-to-End Tests\n\nEnd-to-End tests cover all tests of category five. To run these tests, \nwe need to set up airflow environment in minikube. Also, we need to set up\nall the component required by your DAGs. \n\n#### Minikube set up\n   \nPrerequisite:\n    \u003cpre\u003e\n    git clone https://github.com/chandulal/airflow-testing.git\n    brew cask install virtualbox (run if you don't have virtual box installed)\n    \u003c/pre\u003e\n\nInstall minikube\n    \u003cpre\u003e\n    brew cask install minikube\n    brew install kubernetes-cli\n    minikube start --cpus 4 --memory 8192\n    \u003c/pre\u003e\n\n#### Mount DAGs, Plugins, etc.\n\nMount all your DAGs,Plugins, etc. in minikube\n    \u003cpre\u003e \n    minikube mount {project dir}/src/main/python/:/data\n    \u003c/pre\u003e\n    \n#### Deploy Airflow in minikube\n\nOpen new terminal. Go to project root dir and run:\n    \u003cpre\u003e \n    kubectl apply -f airflow.kube.yaml\n    \u003c/pre\u003e\n\nwait for 3-4 min to start all airflow components.\n\nThis will set up following components: \u003cbr /\u003e\n   * Postgres (To store the metadata of airflow)\n   * Redis (Broker for celery executors)\n   * Airflow Scheduler\n   * Celery Workers\n   * Airflow Web Server\n   * Flower\n   \n#### Access Airflow   \nGet minikube ip by running ```minikube ip``` command\n\nUse minikube ip and access: \n\n    **Airflow UI:** {minikube-ip}:31317 \n    \n    **Flower:** {minikube-ip}:32081\n\n#### How Airflow works in minikube?\n\n![minkube_airflow_architecture](https://github.com/chandulal/airflow-testing/blob/master/how_minikube_work.png)\n\n#### How to run these tests?\n\n1. Install all required components to run your DAGs in minikube. To run integration tests,\navailable in this repo, we required MySQL and Presto on minikube.\n    \u003cpre\u003e\n    kubectl apply -f {SourceDir}/k8s/mysql/mysql.kube.yaml\n    kubectl apply -f {SourceDir}/k8s/presto/presto.kube.yaml\n    \u003c/pre\u003e \n\n2. Run the integration tests from the docker. Use absolute path of this repository in your machine for **{SourceDir}**\n\n     ```docker run -ti  -v {SourceDir}/airflow-testing:/opt --entrypoint /mnt/entrypoint.sh airflow-test run_integration_tests {minikube-ip} ```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchandulal%2Fairflow-testing","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchandulal%2Fairflow-testing","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchandulal%2Fairflow-testing/lists"}