https://github.com/vishvish/fetch-service

Rust app to download all the air quality data files from the European Union website. You can easily import this data into a database and map changes in the different pollutants over time across thousands of locations in Europe.
https://github.com/vishvish/fetch-service

air-quality climate-change climate-data environmental-monitoring eu europe european-union pollution-levels rust

Last synced: about 2 months ago
JSON representation

Rust app to download all the air quality data files from the European Union website. You can easily import this data into a database and map changes in the different pollutants over time across thousands of locations in Europe.

• Host: GitHub
• URL: https://github.com/vishvish/fetch-service
• Owner: vishvish
• Created: 2021-03-12T14:22:17.000Z (about 5 years ago)
• Default Branch: main
• Last Pushed: 2023-04-13T16:43:40.000Z (about 3 years ago)
• Last Synced: 2024-04-15T02:00:46.214Z (about 2 years ago)
• Topics: air-quality, climate-change, climate-data, environmental-monitoring, eu, europe, european-union, pollution-levels, rust
• Language: Rust
• Homepage:
• Size: 71.3 KB
• Stars: 0
• Watchers: 3
• Forks: 1
• Open Issues: 5
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

          
# Air Quality Data Pipeline

Imports European Air Quality data from the service here: https://discomap.eea.europa.eu/map/fme/AirQualityUTDExport.htm

Files are recreated every 30 minutes and contain 48 hours of data.

### Download the daily data files from the European repository

`cargo run`

### Importing the data after download

Install `csvkit` in order to get `csvstack`: `brew install csvkit`

Use `csvstack` to concatenate the downloaded files:

    csvstack data///
/**/*.csv -e iso-8859-1 > output.csv       

Import concatenated file into mongodb:


    mongoimport --type csv -d test -c airquality --headerline output.csv