{"id":16630698,"url":"https://github.com/marcosmarxm/airflow-testing-ci-workflow","last_synced_at":"2025-10-11T02:08:10.046Z","repository":{"id":54539858,"uuid":"297618913","full_name":"marcosmarxm/airflow-testing-ci-workflow","owner":"marcosmarxm","description":"(project \u0026 tutorial) dag pipeline tests + ci/cd setup","archived":false,"fork":false,"pushed_at":"2021-02-11T22:43:03.000Z","size":508,"stargazers_count":86,"open_issues_count":0,"forks_count":10,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-18T02:36:51.285Z","etag":null,"topics":["airflow","airflow-cicd","airflow-testing","data-engineering","data-pipeline","project","tdd","testing"],"latest_commit_sha":null,"homepage":"https://blog.magrathealabs.com/how-to-develop-data-pipeline-in-airflow-through-tdd-test-driven-development-c3333439f358","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/marcosmarxm.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-09-22T10:48:17.000Z","updated_at":"2024-12-04T14:07:52.000Z","dependencies_parsed_at":"2022-08-13T19:10:15.480Z","dependency_job_id":null,"html_url":"https://github.com/marcosmarxm/airflow-testing-ci-workflow","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marcosmarxm%2Fairflow-testing-ci-workflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marcosmarxm%2Fairflow-testing-ci-workflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marcosmarxm%2Fairflow-testing-ci-workflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marcosmarxm%2Fairflow-testing-ci-workflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/marcosmarxm","download_url":"https://codeload.github.com/marcosmarxm/airflow-testing-ci-workflow/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244822647,"owners_count":20516144,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","airflow-cicd","airflow-testing","data-engineering","data-pipeline","project","tdd","testing"],"created_at":"2024-10-12T04:48:34.617Z","updated_at":"2025-10-11T02:08:04.996Z","avatar_url":"https://github.com/marcosmarxm.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Airflow DAG development with tests + CI workflow\n[![CI](https://github.com/marcosmarxm/airflow-testing-ci-workflow/workflows/CI/badge.svg?branch=master)](https://github.com/marcosmarxm/airflow-testing-ci-workflow/actions?query=workflow%3ACI)\n[![MondayBuilding](https://github.com/marcosmarxm/airflow-testing-ci-workflow/workflows/MondayBuilding/badge.svg)](https://github.com/marcosmarxm/airflow-testing-ci-workflow/actions?query=workflow%3AMondayBuilding)\n\nThis code is complementar to the article [How to develop data pipeline in Airflow through TDD (test-driven development)](https://blog.magrathealabs.com/how-to-develop-data-pipeline-in-airflow-through-tdd-test-driven-development-c3333439f358).\nI suggest you to read to better understand the code and the way I think how to setup the project.\n\n[Step-by-step: How to develop a DAG using TDD (english version)](assets/how-to/create-dag-using-tdd.md)\u003cbr\u003e\n[Passo-a-passo: Como desenvolver uma DAG usando TDD (portuguese version)](assets/how-to/criar-dag-usando-tdd.md)\n\n## The project\n\nBelow is a summary of what will be accomplished in this project. We'll simulate the transfer of some fake transaction data from an ecommerce. A simple task transfering data from the `otlp-db` database to the `olap-db` database.\n\n![Diagram](./assets/images/diagram.png)\n\nTo help in the development we use a local development environment to build the pipeline with tests and also a Continuous Integration pipeline with Github Action to ensure that tests are applied at each change.\n\n**Containers**\n- **airflow**: container running local setup for development;\n- **oltp-db** and **olap-db**: container that simulate database in a production environment and receive fake data;\n\nIn this tutorial we won't developt the dashboard part only the pipeline.\n\n### Dependencies?\nDocker, docker-compose and makefile.\n\n### How to run?\n\nThe command below will setup the environment using docker-compose. Wait a few minutes (240s, yeah omg right?) to Airflow initialize its internal configuration, then the command will create credentials and connections.\n```bash\nmake setup\n```\nBy running the above command it is possible to access Airflow at `localhost: 8080`. \nA user of test user: admin / password: admin is created. At this stage you can develop your DAGs and test them as you modify them.\nAnd finally, the command that calls the `pytest` to perform tests.\n```bash\nmake testing\n```\n![Containers](./assets/images/workflow_with_tests.png)\n---\n\nSome resources about Airflow testing and DataOps:\n* [Pipelines on pipelines: Agile CI/CD workflows for Airflow DAGs @ Airflow Summit 2020](https://www.youtube.com/watch?v=tY4F9X5l6dg)\n* [Data Testing with Airflow](https://github.com/danielvdende/data-testing-with-airflow)\n* [Data's Inferno: 7 Circles of Data Testing Hell with Airflow](https://medium.com/wbaa/datas-inferno-7-circles-of-data-testing-hell-with-airflow-cef4adff58d8)\n* [Testing and Debugging in Apache Airflow by GoDataDriven](https://godatadriven.com/blog/testing-and-debugging-apache-airflow/)\n* [The Challenge of Testing Data Pipelines](https://medium.com/slalom-build/the-challenge-of-testing-data-pipelines-4450744a84f1)\n* [Automated Testing for Proceting Data Pipeliens from Undocumented Assumptions](https://www.youtube.com/watch?v=z-kPgEAJCrA\u0026ab_channel=Databricks)\n* [Why Great Data Engineering Needs Automated Testing](https://medium.com/weareservian/why-data-engineering-needs-automated-testing-a37a0844d7db)\n* [Testing in Airflow Part 1 - DAG validation tests, DAG definition tests and unit tests](https://blog.usejournal.com/testing-in-airflow-part-1-dag-validation-tests-dag-definition-tests-and-unit-tests-2aa94970570c)\n* [Testing in Airflow Part 2 - Integration Tests and e2e Pipeline Tests](https://medium.com/@chandukavar/testing-in-airflow-part-2-integration-tests-and-end-to-end-pipeline-tests-af0555cd1a82)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarcosmarxm%2Fairflow-testing-ci-workflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmarcosmarxm%2Fairflow-testing-ci-workflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarcosmarxm%2Fairflow-testing-ci-workflow/lists"}