https://github.com/openclimatefix/nowcasting-metrics

Repo to automatically run metrics on the nowcasting forecast
https://github.com/openclimatefix/nowcasting-metrics

Last synced: 5 months ago
JSON representation

Repo to automatically run metrics on the nowcasting forecast

Host: GitHub
URL: https://github.com/openclimatefix/nowcasting-metrics
Owner: openclimatefix
Created: 2022-08-25T05:50:27.000Z (almost 4 years ago)
Default Branch: main
Last Pushed: 2025-06-10T16:05:49.000Z (12 months ago)
Last Synced: 2025-10-05T18:43:57.079Z (8 months ago)
Language: Python
Size: 560 KB
Stars: 1
Watchers: 0
Forks: 4
Open Issues: 3
Metadata Files:
- Readme: README.md

Awesome Lists containing this project

README

# nowasting_metrics

[![All Contributors](https://img.shields.io/badge/all_contributors-4-orange.svg?style=flat-square)](#contributors-)

Repo to automatically run metrics on the nowcasting forecast

## Metrics

### DAILY

#### MAE

- The MAE is calculated for each GSP on the latest forecast
- We calculate MAE for for all the GSPs combined on the latest forecast
- We also calculate MAE for different forecast horizon, from 0 to 8 hours, for the national forecast.
- The MAE for PVLive initial and update estimate is also calculated for all GSPs

#### RMSE

- The RMSE is calculated for each GSP on the latest forecast
- We calculate RMSE for for all the GSPs combined on the latest forecast
- We also calculate RMSE for different forecast horizon, from 0 to 8 hours, for the national forecast.
- The RMSE for PVLive initial and update estimate is also calculated for all GSPs

### ME

- The ME is calculated for National only from the last week. It is grouped by `time of day` and `forecast_horizon`.

### Ramp Rate

- The MAE of the ramp rate is calculated for each model for the national forecast only.
We define ramp rate as
`((pred_{t+1 hour} - pred_{t}) - (true_{t+1 hour} - true_{t}))` for each individual forecast run.
We calculate the 0, 1 and 2 hour ramp rate. The 0 hour ramp rate looks at values at 0 and 1 hour from each forecast.

### Probabilistic

- We look at the average pinball loss and the average exceedance for plevels 10 and 90.
- This for `PVnet` and `National_xg`
- We look at different forecast horizons for this metric

## Tests
### Local pytest

To run local pytests you need to
1. add `src` to python path `export PYTHONPATH=$PYTHONPATH:./nowcasting_metrics`
3. run pytests: `pytest`

### Docker Tests

TO run tests use the following command
```bash
docker stop $(docker ps -a -q)
docker-compose -f test-docker-compose.yml build
docker-compose -f test-docker-compose.yml run nowcasting_metrics
```

## Running the app
### Environmental Variables
The environmental variables are
DB_URL: The database url you want to save the results to
N_GSPS: The number of gsps you want to pull
DATETIME_NOW: The datetime of when this app is ran. Default is None, and Now() is selected.
This is useful as the app calculates the daily metrics from yesterday
USE_PVNET_GSP_SUM: Option to use `pvnet_gsp_sum` or not. Default is false

These options can also be enter like this:

### Run Locally
First add 'nowcasting_metrics' to your python path:
```
export PYTHONPATH=$PYTHONPATH:./nowcasting_metrics
```
Then run the app.
```
python nowcasting_metrics/app.py --n-gsps=10
```
You will need to set 'DB_URL'

## Contributors ✨

Thanks goes to these wonderful people ([emoji key](https://allcontributors.org/docs/en/emoji-key)):

_Abhijeet
💻

_{Peter Dudfield}
💻

_{Ansh Roshan}
💻

_{Aarushi Chakraborty}
💻