Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/earthobservations/gribmagic

Generic weather forecast downloader
https://github.com/earthobservations/gribmagic

area-of-interest bbox deutscher-wetterdienst dwd grib grib2 numerical-weather-prediction nwp regrid regridding scraper transformer weather weather-data weather-forecast

Last synced: about 2 months ago
JSON representation

Generic weather forecast downloader

Host: GitHub
URL: https://github.com/earthobservations/gribmagic
Owner: earthobservations
License: mit
Created: 2020-12-18T10:22:39.000Z (about 4 years ago)
Default Branch: main
Last Pushed: 2021-10-24T20:11:02.000Z (about 3 years ago)
Last Synced: 2024-04-27T00:41:00.182Z (8 months ago)
Topics: area-of-interest, bbox, deutscher-wetterdienst, dwd, grib, grib2, numerical-weather-prediction, nwp, regrid, regridding, scraper, transformer, weather, weather-data, weather-forecast
Language: Python
Homepage: https://community.panodata.org/t/developing-gribmagic/233
Size: 404 KB
Stars: 20
Watchers: 4
Forks: 2
Open Issues: 6
Metadata Files:
- Readme: README.md
- Changelog: CHANGES.rst
- License: LICENSE

Awesome Lists containing this project

README

# GribMagic - generic weather forecast downloader

![CI](https://github.com/earthobservations/gribmagic/workflows/Tests/badge.svg?branch=main)
[![codecov](https://codecov.io/gh/earthobservations/gribmagic/branch/main/graph/badge.svg)](https://codecov.io/gh/earthobservations/gribmagic)
![python](https://img.shields.io/pypi/pyversions/gribmagic.svg)
![version](https://img.shields.io/pypi/v/gribmagic.svg)
![license](https://img.shields.io/pypi/l/gribmagic.svg)
![status](https://img.shields.io/pypi/status/gribmagic.svg)
![downloads](https://img.shields.io/pypi/dm/gribmagic.svg)

![image](https://user-images.githubusercontent.com/453543/102729922-fb641c80-4332-11eb-835d-b022fc5290d9.png)

_Simulations are believed by no one except those
who conducted them._

_Experimental results are believed by everyone except
those who conducted them._

ANONYMOUS

## About
The goal of the GribMagic project is to unify the download process of
public GRIB1/GRIB2 and netCDF data from numerical weather prediction
models originating from different organizations and data providers.

This is a work in progress, as such GribMagic is currently considered to
be beta software. As this is an early-stage project, contributions are
highly appreciated.

## Details
The software package includes different code bases. Both provide their features
by means of a `gribmagic` subcommand.

- GribMagic Unity program: `gribmagic unity --help`
- DWD GRIB Downloader program: `gribmagic dwd --help`

You can find corresponding invocation examples later in this document.

## Setup

Install the [ecCodes package by ECMWF](https://confluence.ecmwf.int/display/ECC).
```
# Debian Linux
apt-get install --yes libeccodes0 cdo

# macOS/Homebrew
brew install eccodes cdo
```

Install GribMagic Python package.
```
pip install gribmagic --upgrade
```

## Run GribMagic Unity program

### Ad hoc usage
```
# List labels of available models.
gribmagic unity list

# Acquire data.
gribmagic unity acquire --model=dwd-icon-eu --timestamp=2021-10-03T00:00:00Z --target=.gribmagic-data
```

### Configuration
To optionally adjust configuration settings, you can define the following environment variables:
```
GM_DATA_PATH={PATH_TO_PROJECT}/data

GM_MODEL_CONFIG={PATH_TO_PROJECT}/config/model_config.yml"
GM_MODEL_VARIABLES_MAPPING={PATH_TO_PROJECT}/config/model_variables_mapping.yml"
GM_MODEL_VARIABLES_LEVELS_MAPPING={PATH_TO_PROJECT}/config/model_variables_levels_mapping.yml"

ECCODES_DEFINITION_PATH=/usr/share/eccodes/definitions:/usr/local/opt/eccodes/share/eccodes/definitions
```

## Run DWD GRIB Downloader program
```
# Install DWD GRIB Downloader program.
gribmagic install dwd-grib-downloader

# Acquire wind-specific parameters from ICON-D2.
wget https://raw.githubusercontent.com/earthobservations/gribmagic/98da3fd4f/examples/dwd/recipe_d2_wind.py
gribmagic dwd acquire --recipe=recipe_d2_wind.py --timestamp="2021101800" --output=.gribmagic-data/raw

# Acquire assorted parameters from ICON-GLOBAL.
wget https://raw.githubusercontent.com/earthobservations/gribmagic/98da3fd4f/examples/dwd/recipe_global_assorted.py
gribmagic dwd acquire --recipe=recipe_global_assorted.py --timestamp="2021101800" --output=.gribmagic-data/raw
```

When omitting the `--timestamp` parameter, the most recent modelrun is automatically selected.
When omitting the `--output` parameter, it can be supplied using the `GM_DATA_PATH` environment variable.

In this manner, the most compact form to invoke `gribmagic dwd` would be something like:
```
export GM_DATA_PATH=.gribmagic-data/raw
gribmagic dwd acquire --recipe=recipe_d2_wind.py
```

## Run bbox tool
Extract area of interest from GRIB files using a bounding box.

Extract subset by coordinates, a space-separated list of `lat_min lat_max lon_min lon_max`.
```
gribmagic smith bbox \
".gribmagic-data/raw/icon-d2/**/*regular-lat-lon*.grib2" \
--output=.gribmagic-data/subgrid/icon-d2 \
--bbox=46.0 47.5 14.5 16.8
```

Extract subset by 2-letter country name.
```
gribmagic smith bbox \
".gribmagic-data/raw/icon-d2/**/*regular-lat-lon*.grib2" \
--output=.gribmagic-data/subgrid/icon-d2 \
--country=AT
```

For plotting the result on a map, install `Magics`:
```
# Linux
apt-get install --yes libmagplus3v5

# macOS
make magics-install
make magics-info
export MAGPLUS_HOME=/usr/local/opt/magics-4.9.3

# Install GribMagic with Magics bindings
pip install gribmagic[plotting]
```

and use the `--plot` option:
```
gribmagic smith bbox \
".gribmagic-data/raw/icon-d2/**/*regular-lat-lon*.grib2" \
--output=.gribmagic-data/subgrid \
--country=AT \
--plot
```

## Run regridding tool

Acquire grid information files.
```
gribmagic install dwd-grid-information
```

Process regridding on a single file. The default is to assume ICON GLOBAL files and the
regridding will take place with 0.250° resolution.
```
gribmagic smith regrid \
".gribmagic-data/raw/icon/**/*icon_global_icosahedral*.grib2" \
--output=.gribmagic-data/regridded/icon-global
```

When aiming to regrid with 0.125° resolution, please specify the `--resolution=0.125` argument, like:
```
gribmagic smith regrid \
".gribmagic-data/raw/icon/**/*icon_global_icosahedral*.grib2" \
--output=.gribmagic-data/regridded/icon-global \
--resolution=0.125
```

## Run program in Docker

To use GribMagic in a Docker container, you have to build the Docker image like
```
docker build --tag gribmagic .
```

and then invoke it like
```
docker run -it --volume=$PWD/.gribmagic-data:/var/spool/gribmagic gribmagic:latest gribmagic unity acquire --model=dwd-icon-eu --timestamp=2021-10-03T00:00:00Z
```

## Development

### Acquire source code
```
git clone https://github.com/earthobservations/gribmagic
cd gribmagic
```

### Run software tests
```
# Run all tests.
make test

# All tests, with code coverage report.
make test-coverage

# Run specific tests, with coverage report.
.venv/bin/pytest -vvv --cov-report=term-missing --cov=gribmagic.smith.regrid -k test_regrid

# Run tests in parallel.
make test-parallel
make test-coverage-parallel
```

---

#### Content attributions
The copyright of data sets, images and pictograms are held by their respective owners, unless otherwise noted.

##### Banner image
- 2 metre temperature from ECMWF, rendered using Magics.
- Source: MARS labelling or ensemble forecast data (2013-03-25)
- URL: http://download.ecmwf.int/test-data/magics/2m_temperature.grib