{"id":21523305,"url":"https://github.com/santhin/air-pollution","last_synced_at":"2026-05-10T00:41:26.690Z","repository":{"id":110873458,"uuid":"363383646","full_name":"Santhin/air-pollution","owner":"Santhin","description":"Preprocessing air pollution data using ditrubuted computing and ML ","archived":false,"fork":false,"pushed_at":"2021-05-02T18:06:59.000Z","size":1303,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-01-24T05:08:16.218Z","etag":null,"topics":["air-pollution","coiled","dask","ml","optuna","pandas","plotly","python","xgboost"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Santhin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2021-05-01T10:35:17.000Z","updated_at":"2021-05-02T18:07:01.000Z","dependencies_parsed_at":"2023-05-09T14:32:37.723Z","dependency_job_id":null,"html_url":"https://github.com/Santhin/air-pollution","commit_stats":{"total_commits":5,"total_committers":2,"mean_commits":2.5,"dds":0.4,"last_synced_commit":"07f0a74d3171bd2b2b3c6c266e3d7909e03fa365"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Santhin%2Fair-pollution","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Santhin%2Fair-pollution/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Santhin%2Fair-pollution/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Santhin%2Fair-pollution/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Santhin","download_url":"https://codeload.github.com/Santhin/air-pollution/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244080566,"owners_count":20394940,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["air-pollution","coiled","dask","ml","optuna","pandas","plotly","python","xgboost"],"created_at":"2024-11-24T01:13:16.767Z","updated_at":"2026-05-10T00:41:26.649Z","avatar_url":"https://github.com/Santhin.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"\r\n---\r\n## 🧐 About \u003ca name = \"about\"\u003e\u003c/a\u003e\r\n\r\nThe project was created for academic purposes. Consists of unification archive measurement data from range 2000-2019 with existing database.\u003cbr\u003e\r\nMeasurement data were combined with meteorological data using Dask and distributed computing provided by Coiled\u003cbr\u003e\r\nThe model was trained using Xgboost and Optuna for hyperparameter tunning, which reached an RMSE of 6,932 [µg / m3].\r\n\r\n### Links to data:\r\nhttps://powietrze.gios.gov.pl/pjp/archives\r\nhttps://powietrze.gios.gov.pl/pjp/content/api \r\nhttps://danepubliczne.imgw.pl/data/\u003cbr\u003e\r\n\r\n\r\n## 🏁 Installing\r\n\r\n- Python 3.8.3\r\n\r\n```\r\ngit clone https://github.com/Santhin/air-pollution.git\r\n```\r\n\r\nInstalling dependencies:\r\n\r\n```\r\npip install -r requirements.txt\r\nor \r\npoetry install\r\n```\r\nRun jupyter notebook with:\r\n```\r\njupyter notebook\r\n```\r\nTo install coiled software environment:\r\n```\r\nimport coiled\r\n\r\ncoiled.create_software_environment(\r\n    name=\"my-software-env\",\r\n    conda=\"coiled-environment-py38.yml\",\r\n)\r\n```\r\n\r\n### Project structure\r\n```\r\n├── coiled-environment-py38.yml\r\n├── data\r\n│   ├── dictionaries\r\n│   │   ├── IndeksJakosciPowietrza.csv\r\n│   │   ├── Indeks\\ jako\\305\\233ci\\ powietrza\\ gio\\305\\233.xlsx\r\n│   │   ├── IndeksJakosciPowietrza.xlsx\r\n│   │   ├── Kody_stacji_pomiarowych.xlsx\r\n│   │   ├── Matching_stations\r\n│   │   │   ├── SmogoliczkaStacje.csv\r\n│   │   │   └── SynopStacje.csv\r\n│   │   ├── Metadane\\ -\\ stacje\\ i\\ stanowiska\\ pomiarowe.xlsx\r\n│   │   ├── Normy.pkl\r\n│   │   ├── PomiarySample.pkl\r\n│   │   ├── response_api_gios.json\r\n│   │   ├── RodzajeParametrow.csv\r\n│   │   ├── rodzaje_parametrow.pkl\r\n│   │   ├── SensoryPomiarowe.csv\r\n│   │   ├── SensoryPomiarowe.pkl\r\n│   │   ├── stacje_pom_api.json\r\n│   │   ├── StacjePomiarowe.xlsx\r\n│   │   └── stacjeSmogoliczka.csv\r\n│   ├── IndeksJakosciPowietrza.csv\r\n│   └── train_data.csv\r\n├── LICENSE\r\n├── notebooks\r\n│   ├── Air\\ Quality\\ Index\\ Gios.ipynb\r\n│   ├── eda\\ without\\ progress-Copy4.ipynb\r\n│   ├── Filtering\\ excel\\ files\\ and\\ picking\\ right\\ parameters.ipynb\r\n│   ├── Fixing\\ missing\\ lat\\ and\\ lon\\ in\\ stations\\ .ipynb\r\n│   ├── loader_sql.py\r\n│   ├── Matching\\ stations\\ synop\\ with\\ Smogoliczka\\ .ipynb\r\n│   ├── Matching\\ synop\\ data\\ with\\ smogoliczka.ipynb\r\n│   ├── Matching\\ Synop\\ with\\ Smogoliczka\\ final.ipynb\r\n│   ├── ML\\ PM2.5.ipynb\r\n│   ├── New\\ Strategy\\ script\\ for\\ excel\\ files.ipynb\r\n│   ├── __pycache__\r\n│   │   └── loader_sql.cpython-38.pyc\r\n│   ├── Repairing\\ stations\\ names\\ and\\ merging\\ into\\ one\\ .ipynb\r\n│   └── Smogoliczka\\ API\\ to\\ pomiary_pivot.ipynb\r\n├── poetry.lock\r\n├── pyproject.toml\r\n├── README.md\r\n└── requirements.txt\r\n```\r\n\r\n## ⛏️ Built Using \u003ca name = \"built_using\"\u003e\u003c/a\u003e\r\n\r\n- [MsSQL](https://www.microsoft.com/pl-pl/sql-server/sql-server-downloads) - Database\r\n- [S3Bucket](https://aws.amazon.com/s3/) - Cloud storage\r\n- [Coiled](https://coiled.io/) - Distributed computing\r\n- [Dask](https://dask.org/) - Preprocessing data\r\n- [Optuna](https://optuna.org/) - Hyperparameter optimization\r\n- [Xgboost](https://xgboost.readthedocs.io/en/latest/) - ML\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsanthin%2Fair-pollution","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsanthin%2Fair-pollution","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsanthin%2Fair-pollution/lists"}