{"id":18597571,"url":"https://github.com/lealre/etl-airflow","last_synced_at":"2025-04-10T17:31:03.427Z","repository":{"id":233041586,"uuid":"774595901","full_name":"lealre/etl-airflow","owner":"lealre","description":"An ETL Orchestration using Apache Airflow to extract CSV files from a Google Drive, validate, transform, and load into a PostgreSQL database.","archived":false,"fork":false,"pushed_at":"2024-06-30T16:04:03.000Z","size":471,"stargazers_count":24,"open_issues_count":0,"forks_count":7,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-25T01:29:34.663Z","etag":null,"topics":["airflow","astronomer","docker","pandas","pandas-datareader","pandera","pytest","sqlalchemy"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lealre.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-03-19T20:34:35.000Z","updated_at":"2025-03-12T13:05:42.000Z","dependencies_parsed_at":"2024-04-13T03:55:50.326Z","dependency_job_id":"b9661767-e711-4a4a-b9bd-78eddbb9ceeb","html_url":"https://github.com/lealre/etl-airflow","commit_stats":null,"previous_names":["lealre/etl-airflow"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lealre%2Fetl-airflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lealre%2Fetl-airflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lealre%2Fetl-airflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lealre%2Fetl-airflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lealre","download_url":"https://codeload.github.com/lealre/etl-airflow/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248261993,"owners_count":21074229,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","astronomer","docker","pandas","pandas-datareader","pandera","pytest","sqlalchemy"],"created_at":"2024-11-07T01:28:38.462Z","updated_at":"2025-04-10T17:31:03.169Z","avatar_url":"https://github.com/lealre.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Orchestrating an ETL with Airflow - From Google Drive to PostgreSQL\n\nThis project aims to orchestrate an ETL (Extract-Transform-Load) with Airflow, extracting CSV files from a folder in Google Drive, transforming values, and storing them in a PostgreSQL database.\n\nThe data is handled in a pandas DataFrame format, and all the data validation is performed using the [Pandera](https://pandera.readthedocs.io/en/stable/#) library, a Pydantic-based library to validate DataFrame schemas. By setting a specific data contract, validations occur in two phases: when extracted and when transformed.\n\nThe Airflow implementation was created using the Astro CLI, the command line interface for data orchestration from [Astronomer](https://docs.astronomer.io/).\n\nThis project also has a CI for every Pull Request made using GitHub Actions, where the schema contract is tested with the pytest library.\n\n![](pics/etl-diagram.png)\n\n## Table of Contents\n\n- [Context](#context)\n  - [Contract Schema](#contract-schema)\n- [How it works](#how-it-works)\n  - [Project Folder Structure](#project-folder-structure)\n- [How to run this project](#how-to-run-this-project)\n  - [Run without Airflow](#run-without-airflow)\n- [Further Tasks](#further-tasks)\n\n\n## Context\n\nThis project was built in a context where a folder inside Google Drive regularly receives different CSV files containing the operational revenue from various companies in three types of currencies: USD, EUR, and YEN. This data must be stored in a PostgreSQL database with the operational revenue converted to USD and the currency conversion rate, taking the last quotation of the month date that is in the CSV.\n\n### Contract Schema\n\nThe project uses the following contract schema to validate the data:\n\n* Schema-in: Used when extracting files from Google Drive in Task 02\n\n| Column               | Type                        | Constraints                                    |\n|----------------------|-----------------------------|------------------------------------------------|\n| company              | Series[str]                 |                                                |\n| currency             | Series[str]                 | in ['EUR', 'USD', 'YEN'], all values equal                     |\n| operational_revenue  | Series[float]               | greater than or equal to 0                    |\n| date                 | Series[DateTime]         | all values equal                                               |\n| file_id              | Optional[str]               |                                                |\n\n* Schema-out: Used when transforming data in Task 03\n\n| Column            | Type                | Constraints                     |\n|-------------------|---------------------|---------------------------------|\n| company           | Series[str]         |                                 |\n| currency          | Series[str]         | in ['EUR', 'USD', 'YEN'], all values equal      |\n| operational_revenue | Series[float]    | greater than or equal to 0     |\n| date              | Series[DateTime] | all values equal                             |\n| file_id           | Series[str]         |                                 |\n| convertion_rate   | Series[float]       | greater than or equal to 0     |\n| usd_converted     | Series[float]       | greater than or equal to 0     |\n\n## How it works\n\n**\u003cu\u003eTask 01:\u003c/u\u003e Connect with Google Drive API and extract CSV files**\n\nIn this initial task, the script connects to the Google Drive API by passing our credentials in JSON file format and specifying the parent folder name and the folder from which we want to extract the CSV files. Subsequently, it retrieves all file information from the designated folder, including file name, Google Drive file ID, and file type.\n\nGoogle Drive assigns a unique ID to each file uploaded to its folders, which is inserted into the database as a unique file identifier. This allows for filtering to determine if the file has already been uploaded to the database in subsequent Airflow triggers.\n\nTo facilitate this task, a GoogleDrive class was created to encapsulate all desired functionalities of the Google Drive API in `google_drive.py`. \n\nThe task output is a list of DataFrames, where each DataFrame represents a CSV file extracted from the Google Drive folder.\n\n**\u003cu\u003eTask 02:\u003c/u\u003e Validate extracted data**\n\nReceives the list of DataFrames extracted from Task 01 and validates it according to the contract schema using the Pandera library, raising an invalid schema error if applicable.\n\nThe output is a list of validated DataFrames.\n\n**\u003cu\u003eTask 03:\u003c/u\u003e Transform data and validate it**\n\nReceives a list of DataFrames and performs the transformations. It maps both the currency and the date from each DataFrame and includes the currency rate conversion and the USD amount converted columns using the rate conversion from that date. It also performs schema validation after this transformation using Pandera.\n\nThe conversion rate data is obtained from the FRED `pandas_datareader` library, from the links below:\n\n* [U.S. Dollars to Euro Spot Exchange Rate](https://fred.stlouisfed.org/series/DEXUSEU)\n* [Japanese Yen to U.S. Dollar Spot Exchange Rate](https://fred.stlouisfed.org/series/DEXJPUS)\n\nThe output is a list of transformed and validated DataFrames.\n\n**\u003cu\u003eTask 04:\u003c/u\u003e Load data in database**\n\nThis final task loads the data into a PostgreSQL database. \n\n### Project Folder Structure\n\n```\n.\n├── Dockerfile\n├── README.md\n├── airflow_settings.yaml\n├── dags\n│   └── dag_etl.py\n├── data\n│   ├── eur_revenue.csv\n│   ├── usd_revenue.csv\n│   ├── wrong_data.csv\n│   └── yen_revenue.csv\n├── dev-requirements.txt\n├── docker-compose.yml\n├── packages.txt\n├── pics\n│   └── etl-diagram.png\n├── pyproject.toml\n├── requirements.txt\n├── src\n│   ├── __init__.py\n│   ├── database.py\n│   ├── etl.py\n│   ├── google_drive.py\n│   ├── main.py\n│   ├── schema.py\n│   └── transform_utils.py\n└── tests\n    ├── dags\n    ├── test_schema_in.py\n    └── test_schema_out.py\n```\n\n\n## How to run this project\n\nAll the steps here were intended to a `bash` terminal.\n\nGoogle Drive API requires a JSON file to authenticate the connection, and although it's not correct, in this project it was uploaded in the root directory, where the file `dags/dag_etl.py` will search for the file. So, to run as it is, you need to upload the JSON file in the root directory with the name `service_account.json`.  Make sure that this file is included in `.gitignore`. Here are some resources on how to do it: [How to Upload Files Using the Google Drive API in Python](https://ragug.medium.com/how-to-upload-files-using-the-google-drive-api-in-python-ebefdfd63eab), [Using Google Drive API with Python and a Service Account](https://medium.com/@matheodaly/using-google-drive-api-with-python-and-a-service-account-d6ae1f6456c2), and a [video tutorial](https://www.youtube.com/watch?v=tamT_iGoZDQ).\n\n\nThe name of the parent folder in Google Drive is set to be `python_to_drive` and the folder from where it will extract the CSV files must be called `Operational Revenue`. You can change the names by just changing the variables `parent_folder_name` and `folder_to_extract_from` in [`dags/dag_etl.py`](dags/dag_etl.py).\n\nYou also need to connect the app to your own PostgreSQL database. You can do this by following the steps below:\n\n1.1 - Clone the repository locally:\n```bash\ngit https://github.com/lealre/etl-airflow.git\n```\n\n1.2 - Access the project folder:\n```bash\ncd etl-airflow\n```\n\n1.3 - Here we have two options to connect to PostgreSQL: locally with Docker (1.3.1) or to connect to an existing database using credentials passed in the `.env` file (1.3.2). By default, if there are no credentials provided, the program will attempt to automatically connect to Docker locally, using the credentials specified in [docker-compose.yml](docker-compose.yml).  \n    \n1.3.1 - Locally using Docker\n\nCreate the database and PGAdmin container with Docker:\n```bash\ndocker compose up -d\n```\n\nTo access the database:\n\nI. Go to the localhost link in your browser: [http://localhost:8888/](http://localhost:8888/)\n\nII. Access PGAdmin with the credentials specified in the docker-compose.yaml file:\n   - Username: `user-name@domain-name.com`\n   - Password: `pgadmin`\n\nIII. Set the master password (when accessing for the first time).\n\nIV. Right-click on the server to connect PGAdmin to the database.\n\nV. Connect to the database using the credentials defined in the `docker-compose.yaml` file:\n   - Host name: `db`\n   - Password: `postgres`\n\n\n**OR**\n    \n1.3.2 - Connecting with an existing database \n\nCreate the `.env` file in the root folder, passing the respective keys from your own PostgresSQL Database:\n```bash\necho \"POSTGRES_USER=\u003cyour-database-keys\u003e\" \u003e\u003e .env\necho \"POSTGRES_PASSWORD=\u003cyour-database-keys\u003e\" \u003e\u003e .env\necho \"POSTGRES_HOST=\u003cyour-database-keys\u003e\" \u003e\u003e .env\necho \"POSTGRES_PORT=\u003cyour-database-keys\u003e\" \u003e\u003e .env\necho \"POSTGRES_DB=\u003cyour-database-keys\u003e\" \u003e\u003e .env\n```\n\nMake sure `.env` file is included in `.gitignore`.\n\nFrom here, we use the Astro CLI to run Airflow with Docker.\n\n1.4 - Install Astro CLI:\n```bash\ncurl -sSL install.astronomer.io | sudo bash -s\n```\n\n1.5 - Run Airflow with Docker\n```bash\nastro dev start\n```\n\nNOTES: \n* Using the flag `--Wait=5m` to wait 5 minutes for the webserver to get healthy before timing out. The default is 1 minute for most machines.\n* `astro dev kill` force-stop and remove all running containers for your local Airflow environment.\n\nAfter these steps, it will automatically open the localhost link with the Airflow UI. The username and password to access are both `admin`.\n\n### Run without Airflow\n\nYou also can run this project without Airflow. To do it, you should follow all the steps until 1.3, and then do the following:\n\n2.1 - Install Python version 3.11.5:\n```bash\npyenv install 3.11.5\n```\n\n2.2 - Set the local version of Python to 3.11.5:\n```bash\npyenv local 3.11.5\n```\n\n2.3 - Create a virtual enviroment and activate it:\n```bash\npython -m venv .venv\nsource .venv/bin/activate\n```\n\n2.4 - Install all dependencies from the project:\n```bash\npip install -r dev-requirements.txt\n```\n\n2.5 - Run the project:\n```bash\ntask main\n```\n\n2.6 -  (Optional) Run the schema test:\n```bash\ntask test_schema_in # test schema in\ntask test_schema_out # test schema out\n```\n\n## Further Tasks\n\nThese are some areas where this project can improve:\n\n* The JSON file to connect with the Google Drive API should be passed in another way.\n* As we are interested only in the year and month, it can improve in validating the date in the schema-in, restricting the date to the YYYY-MM format.\n* When a file that doesn't match the schema is found, the script could move it from the folder where it is extracting the files to a new folder, designated for files in the same situation. This would make it easier to separate the correct files from the incorrect schema files.\n* The project can be extended to support more types of files than just CSV, such as PARQUET and XLSX files.\n* There are more tests to perform with pytest, including integration tests.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flealre%2Fetl-airflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flealre%2Fetl-airflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flealre%2Fetl-airflow/lists"}