https://github.com/openclimatefix/satip

Satip contains the code necessary for retrieving, transforming and storing EUMETSAT data
https://github.com/openclimatefix/satip

eumetsat gis nowcasting python satellite-imagery

Last synced: about 1 year ago
JSON representation

Satip contains the code necessary for retrieving, transforming and storing EUMETSAT data

Host: GitHub
URL: https://github.com/openclimatefix/satip
Owner: openclimatefix
License: mit
Created: 2020-11-02T16:49:59.000Z (over 5 years ago)
Default Branch: main
Last Pushed: 2024-10-28T17:55:37.000Z (over 1 year ago)
Last Synced: 2024-10-29T22:32:36.770Z (over 1 year ago)
Topics: eumetsat, gis, nowcasting, python, satellite-imagery
Language: Jupyter Notebook
Homepage: https://satip.readthedocs.io/
Size: 467 MB
Stars: 41
Watchers: 2
Forks: 28
Open Issues: 49
Metadata Files:
- Readme: README.md
- License: LICENSE

Awesome Lists containing this project

README

# Satip

[![All Contributors](https://img.shields.io/badge/all_contributors-14-orange.svg?style=flat-square)](#contributors-)

[![PyPI version](https://badge.fury.io/py/satip.svg)](https://badge.fury.io/py/satip)

[![codecov](https://codecov.io/gh/openclimatefix/Satip/branch/main/graph/badge.svg?token=GTQDR2ZZ2S)](https://codecov.io/gh/openclimatefix/Satip)

> Satip is a library for satellite image processing, and provides all of the functionality necessary for retrieving, and storing EUMETSAT data

### Installation

To install the `satip` library please run:

```bash
pip install satip
```

Or if you're working in the development environment you can run the following from the directory root:

```bash
pip install -e .
```

#### Conda

Or, if you want to use `conda` from a cloned Satip repository:

```bash
conda env create -f environment.yml
conda activate satip
pip install -e .
```

If you plan to work on the development of Satip then also consider installing these development tools:

```bash
conda install pytest flake8 jedi mypy black pre-commit
pre-commit install
```

### Development Environment

In order to contribute:
- it's recommended that you use a Linux-based OS. This is currently used for all CI/CD testing, production, and development.
- At the time of writing (21-Dec-23), the Python version used is 3.11 with work being done to update to Python 3.12. This is subject to updates over time.

## Operation

### Getting your own API key

In order to contribute to development or just test-run some scripts, you will need your own Eumetsat-API-key. Please follow these steps:

1. Go to https://eoportal.eumetsat.int and register an account.
2. You can log in and go to https://data.eumetsat.int/ to check available data services. From there go to your profile and choose the option "API key" or go to https://api.eumetsat.int/api-key/ directly.
3. Please make sure that you added the key and secret to your user's environment variables.

### Downloading EUMETSAT Data

We have moved this to [here](https://github.com/openclimatefix/dagster-dags/blob/main/containers/sat/download_process_sat.py)

### Converting Native files to Zarr
`scripts/convert_native_to_zarr.py` converts EUMETSAT `.nat` files to Zarr datasets, using very mild lossy [JPEG-XL](https://en.wikipedia.org/wiki/JPEG_XL) compression. (JPEG-XL is the "new kid on the block" of image compression algorithms). JPEG-XL makes the files about a quarter the size of the equivalent `bz2` compressed files, whilst the images are visually indistinguishable. JPEG-XL cannot represent NaNs so NaNs. JPEG-XL understands float32 values in the range `[0, 1]`. NaNs are encoded as the value `0.025`. All "real" values are in the range `[0.075, 1]`. We leave a gap between "NaNs" and "real values" because there is very slight "ringing" around areas of constant value (see [this comment for more details](https://github.com/openclimatefix/Satip/issues/67#issuecomment-1036456502)). Use `satip.jpeg_xl_float_with_nans.JpegXlFloatWithNaNs` to decode the satellite data. This class will reconstruct the NaNs and rescale the data to the range `[0, 1]`.

## Running in Production

The live service uses `app.py` as the entrypoint for running the live data download for OCF's forecasting service, and has a few configuration options, configurable by command line argument or environment variable.

`--api-key` or `API_KEY` is the EUMETSAT API key

`--api-secret` or `API_SECRET` is the EUMETSAT API secret

`--save-dir` or `SAVE_DIR` is the top level directory to save the output files, a `latest` subfolder will be added to that directory to contain the latest data

`--history` or `HISTORY` is the amount of history timesteps to use in the `latest.zarr` files

`--db-url` or `DB_URL` is the URL to the database to save to when a run has finished

`--use-rescaler` or `USE_RESCALER` tells whether to rescale the satellite data to between 0 and 1 or not when saving to disk. Primarily used as backwards compatibility for the current production models, all new training and production Zarrs should use the rescaled data.

`--use-iodc` or `USE_IODC` is an option to get the IODC satellite data

## Testing

To run tests, simply run ```pytest .``` from the root of the repository. To generate the test plots, run ```python scripts/generate_test_plots.py```.

### Environmental Variables
Some tests require environmental variables to be set that would be passed in by command line argument when running the code in production. These are as follows:
- `EUMETSAT_USER_KEY`: the EUMETSAT API key
- `EUMETSAT_USER_SECRET`: the EUMETSAT API secret

These can be added using the `export` command in your shell environment. To add these permanently, the export statements can be added to the configuration file for the shell environment (e.g. "~/.bashrc" if using bash).

## Contributors ✨

Thanks goes to these wonderful people ([emoji key](https://allcontributors.org/docs/en/emoji-key)):

_{Jacob Bieker}
💻

_{Jack Kelly}
💻

_{Ayrton Bourn}
💻

_{Laurence Watson}
💻

_{Notger Heinz}
📖

_{Peter Dudfield}
📖

_{Azah Norbline}
💻

_{Tom Pughe}
💻

_{Zhenbang Feng}
💻

_jsbaasi
💻

_{Suleman Karigar}
💻

_Richa
💻

_{Nathan Simpson}
🐛

_peach280
🚧