https://github.com/rodekruis/nrt_rainfall_pipeline
Near Real-Time Rainfall
https://github.com/rodekruis/nrt_rainfall_pipeline
aa
Last synced: 5 months ago
JSON representation
Near Real-Time Rainfall
- Host: GitHub
- URL: https://github.com/rodekruis/nrt_rainfall_pipeline
- Owner: rodekruis
- Created: 2024-11-15T19:26:17.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2025-12-30T14:31:13.000Z (6 months ago)
- Last Synced: 2026-01-03T08:11:57.917Z (6 months ago)
- Topics: aa
- Language: Python
- Homepage:
- Size: 12.8 MB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Near real-time rainfall pipeline
Near real-time rainfall monitoring. Part of the project RIPOSTE for Cameroon Red Cross Society.
## Description
Data consumed:
- [Level 3 IMERG Late Run: PPS Near Real-time](https://gpm.nasa.gov/data/directory/imerg-late-run-pps-near-real-time-gis)
- Value: 1-day precipitation accumulation (mm)
- Data Download URL: https://jsimpsonhttps.pps.eosdis.nasa.gov/imerg/gis/
- Temporal range: most recent past days (e.g. last 14 days)
> [!IMPORTANT]
> IMERG Late Run data is updated daily, but the latest data is available only after about 14h.
> For example, the data of today will be available tomorrow at around 14:00 UTC+0. Read more about it [here](https://gpm.nasa.gov/data/imerg)
The pipeline roughly consists of three steps:
- Extract the 1-day data on rainfall measurement as above.
- Transform the data into pre-defined areas (health districts) and calculate average rainfall of the past days. Then determine which area has its average rainfall higher than pre-defined thresholds.
- Send this data as alert to the EspoCRM for the NS.
## Basic Usage
To run the pipeline locally
1. Fill in the secrets in .env.example and rename the file to .env; in this way, they will be loaded as environment variables
2. Install requirements
```
pip install poetry
poetry install --no-interaction
```
3. Edit the configuration in `config\config.yaml`
```
- name:
days-to-observe: # number of most recent days to observe rainfall e.g. 14
alert-on-threshold: # threshold to send to EspoCRM e.g. 50
shapefile-area: .geojson # shapefile of areas (. geojson) where the zonal stats bases on
espo-area: # entity storing areas code (and id)
entity:
field:
espo-destination: # entity to send alerts to
entity:
field:
```
4. Run the pipeline : `python nrt_rainfall_pipeline.py --extract --transform --send`
```
Usage: nrt_rainfall_pipeline.py [OPTIONS]
Options:
--country TEXT country ISO3
--extract extract NRT rainfall raster data
--transform calculate rainfall data in pre-defined administrative areas
--send send to EspoCRM
--dateend specify a customed latest date YYYY-mm-dd until which the data should be extracted, by default it is the date before today
--help Show this message and exit
```
__Note:__ Payload sent to EspoCRM
```
{
"status": "onhold",
"type": "heavyrainfall",
"source": "GPM",
"": "",
"": ""
}
```
Where:
- ``: is the value of `espo-area`'s `field` in the config
- ``: is to be automatically filled in the pipeline
- ``: is the value of `espo-destination`'s `field` in the config
- ``: is to be automatically filled in the pipeline
## Adding new country
1. Prepare shapefile
- Add a shapefile in `.geojson` format of the area (e.g. districts) in `data\admin_boundary`
- Rename header of the area code (e.g. district code) to `code`
2. Prepare EspoCRM entity for area
- Create an entity to store the area (e.g. District)
- In the entity, create a field to store area code
- Import the area (by exporting the shapefile to `.csv`) to this entity
- Make sure the area code in this entity exactly the same to those in the shapefile
3. Prepare EspoCRM entity for alert
- Create an entity to store alert (e.g. Climate Hazard)
- In the entity, create a ` float` field to store calculated rainfall value
- In the entity, create additional fields to match with the payload requirements:
| field name | type | value(s) |
| -----------| ---- | ---------|
| `status` | `enum` | `onhold` |
| `type` | `enum` | `heavyrainfall` |
| `source` | `enum` or `text` | `GPM` |
- Make sure the entity for area is linked with this one
4. Add the new country to the `config\config.yaml`, see Step 3 in __Basic usage__:
5. Test and adjust settings if needed.