{"id":15060672,"url":"https://github.com/alimarzouk/paris-aq","last_synced_at":"2026-02-06T21:32:03.020Z","repository":{"id":239663648,"uuid":"800192933","full_name":"AliMarzouk/Paris-AQ","owner":"AliMarzouk","description":"ELTL pipeline to monitor air quality in the Paris Île-de-France area","archived":false,"fork":false,"pushed_at":"2024-06-09T19:55:57.000Z","size":265,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-16T01:11:17.625Z","etag":null,"topics":["airflow","airquality","big-data","bigquery","dataengineering","gcs","spark"],"latest_commit_sha":null,"homepage":"https://alimarzouk.github.io/Paris-AQ/dashboard/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AliMarzouk.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-13T21:59:41.000Z","updated_at":"2024-06-09T19:56:01.000Z","dependencies_parsed_at":"2024-06-02T10:29:22.231Z","dependency_job_id":"bbd0dd77-4627-4bfc-b0c0-6ed1861dc367","html_url":"https://github.com/AliMarzouk/Paris-AQ","commit_stats":null,"previous_names":["alimarzouk/openaq-data-engineer"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/AliMarzouk/Paris-AQ","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AliMarzouk%2FParis-AQ","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AliMarzouk%2FParis-AQ/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AliMarzouk%2FParis-AQ/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AliMarzouk%2FParis-AQ/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AliMarzouk","download_url":"https://codeload.github.com/AliMarzouk/Paris-AQ/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AliMarzouk%2FParis-AQ/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":268167539,"owners_count":24206593,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-01T02:00:08.611Z","response_time":67,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","airquality","big-data","bigquery","dataengineering","gcs","spark"],"created_at":"2024-09-24T23:02:33.809Z","updated_at":"2026-02-06T21:32:02.991Z","avatar_url":"https://github.com/AliMarzouk.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003eWelcome to PARIS AQ 👋\u003c/h1\u003e\n\u003cp\u003e\n  \u003ca href=\"https://github.com/AliMarzouk/Paris-AQ#readme\" target=\"_blank\"\u003e\n    \u003cimg alt=\"Documentation\" src=\"https://img.shields.io/badge/documentation-yes-brightgreen.svg\" /\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://github.com/AliMarzouk/Paris-AQ/graphs/commit-activity\" target=\"_blank\"\u003e\n    \u003cimg alt=\"Maintenance\" src=\"https://img.shields.io/badge/Maintained%3F-yes-green.svg\" /\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://github.com/AliMarzouk/Paris-AQ/blob/master/LICENSE\" target=\"_blank\"\u003e\n    \u003cimg alt=\"License: MIT\" src=\"https://img.shields.io/github/license/bishkou/password-pwnd\" /\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n* **Air quality insight monitoring for the paris and Île de France Area**\n\n[![demo ](https://github.com/AliMarzouk/Paris-AQ/assets/47494225/b00172c6-3984-40a5-8963-d26ad3c15423)](https://alimarzouk.github.io/Paris-AQ/dashboard/)\n\n[**Try it**](https://alimarzouk.github.io/Paris-AQ/dashboard)\n\n[**Full article** ](https://www.linkedin.com/pulse/end-to-end-data-engineering-openaq-api-real-time-using-ali-marzouk-1enxe/?trackingId=M%2F50LUOST3%2BOp3lUDtDZig%3D%3D)\n\n## What is Paris AQ\n\n**Paris AQ** is a data engineering project that implements an *ELTL* pipeline \nto monitor air quality in the Paris Île-de-France area. \nThe project extracts air quality data from the **OpenAQ API**, \nstores it in a Google *Cloud Storage (GCS) data lake*,\nprocesses it with *Apache Spark*, and loads it into *Google BigQuery*.\nThe processed data is then visualized in an interactive web dashboard built \nwith *Flask* for the backend and *Dc.js* and *Crossfilter.js* for charts rendering.\n\n## How It Works\n\n**Apache Spark**\n\nApache spark is used in every aspect of the data processing in this project. \nThis includes the api requests, data transformation and storing.\n\nRefer to [this article for utilizing Spark for Rest API calls](https://medium.com/geekculture/how-to-execute-a-rest-api-call-on-apache-spark-the-right-way-in-python-4367f2740e78).\n\nThis project is mainly composed of two spark jobs:\n* spark_api_to_gcs: Retrieves yesterday's data from a Rest API and saves the data to a GCS data lake.\n* spark_gcs_to_big_query: Transforms the data in the data lake and loads it into BigQuery data warehouse.\n\nPS: Please note that the job files in this repo are in a Jupyter notebook format, so you should convert \nthem to a python script in order to use them.\nYou can use Jupyter's `` jupyter nbconvert --to script [YOUR_NOTEBOOK].ipynb `` command.\n\n**Orchestration**\n\nFor orchestrating our Spark jobs, we use Apache Airflow. A pipeline is run daily using the Airflow \nscheduler, submitting the two spark jobs sequentially.\n\n**Backend**\n\nFor the backend we used Flask to expose a Rest API. In order to avoid network bottleneck, \nwe opted for a long living connection that sends the data in chunks of 20000 records. \n\n**Fronted**\n\nIn order to monitor the data, we opted for an easy-to-access web interactive dashboard. \nUsed [Crossfilter.js](https://github.com/crossfilter/crossfilter/wiki/API-Reference), [DC.js](https://dc-js.github.io/dc.js/) and [D3.js](https://d3js.org/) to render the charts with cross filtering. For the map component,\nwe used the [Leaflet.js](https://leafletjs.com/) library.\n\n## Author\n\n👤 **Ali**\n\n* Github: [@AliMarzouk](https://github.com/AliMarzouk/)\n* LinkedIn: [@ali-marzouk](linkedin.com/in/ali-marzouk/)\n\n## 🤝 Contributing\n\nContributions, issues and feature requests are welcome!\u003cbr /\u003eFeel free to check [issues page](https://github.com/AliMarzouk/Paris-AQ/issues).\n\n## Show your support\n\nGive a [STAR](https://github.com/AliMarzouk/Paris-AQ/) if this project helped you!\n\n## 📝 License\n\n* Copyright © 2024 [Ali](https://github.com/AliMarzouk).\n* This project is [MIT](https://github.com/AliMarzouk/Paris-AQ/blob/master/LICENSE) licensed.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falimarzouk%2Fparis-aq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falimarzouk%2Fparis-aq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falimarzouk%2Fparis-aq/lists"}