https://github.com/michaelconan/personal-reporting-airflow

Infrastructure setup for personal Airflow server deployed on Azure
https://github.com/michaelconan/personal-reporting-airflow

Last synced: 6 days ago
JSON representation

Infrastructure setup for personal Airflow server deployed on Azure

• Host: GitHub
• URL: https://github.com/michaelconan/personal-reporting-airflow
• Owner: michaelconan
• Created: 2024-12-22T15:31:37.000Z (8 days ago)
• Default Branch: main
• Last Pushed: 2024-12-23T04:29:21.000Z (8 days ago)
• Last Synced: 2024-12-23T05:25:44.486Z (8 days ago)
• Language: Python
• Size: 2.93 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 1
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# personal-reporting

Airflow server for personal data integration and experimentation.

## Overview

Dev container, requirements and constraints files used for local development prior to deployment.

## Data Sources

1. Google Contacts

2. HubSpot

3. Notion

## Airflow Notes

- DAGs should contain a number of related steps (e.g., extract-transform-load)

- DAGs can be linked to one another via datasets to enable triggering and dependency graph

## Airflow Setup

1. Started with official [docker compose](https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html)

2. Switched to `LocalExecutor` 

3. Removed additional containers, combined webserver + scheduler on single image

## Azure Setup

I have used [this quickstart](https://github.com/savjani-zz/azure-quickstart-templates/tree/master/101-webapp-linux-airflow-postgresql) for reference and the idea to run the webserver and scheduler together in a single container with `LocalExecutor`. `CeleryExecutor` shouldn't be necessary unless scale increases and multiple workers are required.

1. Create container registry and enable [admin user access](https://learn.microsoft.com/en-us/azure/container-registry/container-registry-authentication?tabs=azure-cli#admin-account)

   `az acr update -n airflowreportingregistry --admin-enabled true`

2. Push container images to registry using Docker or automated pipeline (Azure DevOps)

3. Create App Service app with container and connected services (Postgres Database and Redis Cache)