{"id":20200946,"url":"https://github.com/openclimatefix/nowcasting-metrics","last_synced_at":"2026-01-11T03:02:03.249Z","repository":{"id":58725455,"uuid":"528712697","full_name":"openclimatefix/nowcasting-metrics","owner":"openclimatefix","description":"Repo to automatically run metrics on the nowcasting forecast","archived":false,"fork":false,"pushed_at":"2025-06-10T16:05:49.000Z","size":573,"stargazers_count":1,"open_issues_count":3,"forks_count":4,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-10-05T18:43:57.079Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openclimatefix.png","metadata":{"funding":{"github":["openclimatefix"],"patreon":null,"open_collective":"openclimatefix","ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"custom":null},"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2022-08-25T05:50:27.000Z","updated_at":"2025-06-10T16:05:53.000Z","dependencies_parsed_at":"2023-02-16T10:55:18.239Z","dependency_job_id":"4a7cc112-1656-4b8f-8c5e-57a6f577a0bc","html_url":"https://github.com/openclimatefix/nowcasting-metrics","commit_stats":{"total_commits":165,"total_committers":6,"mean_commits":27.5,"dds":"0.36969696969696975","last_synced_commit":"04f0d9b662433bc29e9e149c830d167133e84ce5"},"previous_names":["openclimatefix/nowcasting-metrics","openclimatefix/nowasting_metrics"],"tags_count":65,"template":false,"template_full_name":null,"purl":"pkg:github/openclimatefix/nowcasting-metrics","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2Fnowcasting-metrics","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2Fnowcasting-metrics/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2Fnowcasting-metrics/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2Fnowcasting-metrics/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openclimatefix","download_url":"https://codeload.github.com/openclimatefix/nowcasting-metrics/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2Fnowcasting-metrics/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28274223,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-11T02:08:32.518Z","status":"ssl_error","status_checked_at":"2026-01-11T02:08:32.093Z","response_time":60,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-14T04:46:18.140Z","updated_at":"2026-01-11T03:02:03.219Z","avatar_url":"https://github.com/openclimatefix.png","language":"Python","funding_links":["https://github.com/sponsors/openclimatefix","https://opencollective.com/openclimatefix"],"categories":[],"sub_categories":[],"readme":"# nowasting_metrics\n\u003c!-- ALL-CONTRIBUTORS-BADGE:START - Do not remove or modify this section --\u003e\n[![All Contributors](https://img.shields.io/badge/all_contributors-4-orange.svg?style=flat-square)](#contributors-)\n\u003c!-- ALL-CONTRIBUTORS-BADGE:END --\u003e\nRepo to automatically run metrics on the nowcasting forecast\n\n## Metrics\n\n### DAILY\n\n#### MAE\n\n- The MAE is calculated for each GSP on the latest forecast\n- We calculate MAE for for all the GSPs combined on the latest forecast\n- We also calculate MAE for different forecast horizon, from 0 to 8 hours, for the national forecast.\n- The MAE for PVLive initial and update estimate is also calculated for all GSPs\n\n#### RMSE\n\n- The RMSE is calculated for each GSP on the latest forecast\n- We calculate RMSE for for all the GSPs combined on the latest forecast\n- We also calculate RMSE for different forecast horizon, from 0 to 8 hours, for the national forecast.\n- The RMSE for PVLive initial and update estimate is also calculated for all GSPs\n\n### ME\n\n- The ME is calculated for National only from the last week. It is grouped by `time of day` and `forecast_horizon`. \n\n### Ramp Rate\n\n- The MAE of the ramp rate is calculated for each model for the national forecast only. \nWe define ramp rate as\n`((pred_{t+1 hour} - pred_{t}) - (true_{t+1 hour} - true_{t}))` for each individual forecast run. \nWe calculate the 0, 1 and 2 hour ramp rate. The 0 hour ramp rate looks at values at 0 and 1 hour from each forecast. \n\n### Probabilistic\n\n- We look at the average pinball loss and the average exceedance for plevels 10 and 90. \n- This for `PVnet` and `National_xg`\n- We look at different forecast horizons for this metric\n\n\n## Tests\n### Local pytest\n\nTo run local pytests you need to\n1. add `src` to python path `export PYTHONPATH=$PYTHONPATH:./nowcasting_metrics`\n3. run pytests: `pytest`\n\n\n### Docker Tests\n\nTO run tests use the following command\n```bash\ndocker stop $(docker ps -a -q)\ndocker-compose -f test-docker-compose.yml build\ndocker-compose -f test-docker-compose.yml run nowcasting_metrics\n```\n\n## Running the app\n### Environmental Variables\nThe environmental variables are\nDB_URL: The database url you want to save the results to\nN_GSPS: The number of gsps you want to pull\nDATETIME_NOW: The datetime of when this app is ran. Default is None, and Now() is selected.\nThis is useful as the app calculates the daily metrics from yesterday\nUSE_PVNET_GSP_SUM: Option to use `pvnet_gsp_sum` or not. Default is false\n\nThese options can also be enter like this:\n\n\n### Run Locally\nFirst add 'nowcasting_metrics' to your python path:\n```\nexport PYTHONPATH=$PYTHONPATH:./nowcasting_metrics\n```\nThen run the app.\n```\npython nowcasting_metrics/app.py --n-gsps=10\n```\nYou will need to set 'DB_URL'\n\n## Contributors ✨\n\nThanks goes to these wonderful people ([emoji key](https://allcontributors.org/docs/en/emoji-key)):\n\n\u003c!-- ALL-CONTRIBUTORS-LIST:START - Do not remove or modify this section --\u003e\n\u003c!-- prettier-ignore-start --\u003e\n\u003c!-- markdownlint-disable --\u003e\n\u003ctable\u003e\n  \u003ctbody\u003e\n    \u003ctr\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/abhijelly\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/75399048?v=4?s=100\" width=\"100px;\" alt=\"Abhijeet\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eAbhijeet\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/nowcasting-metrics/commits?author=abhijelly\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/peterdudfield\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/34686298?v=4?s=100\" width=\"100px;\" alt=\"Peter Dudfield\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003ePeter Dudfield\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/nowcasting-metrics/commits?author=peterdudfield\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/AnshRoshan\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/71877143?v=4?s=100\" width=\"100px;\" alt=\"Ansh Roshan\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eAnsh Roshan\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/nowcasting-metrics/commits?author=AnshRoshan\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/aarxshi\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/132636356?v=4?s=100\" width=\"100px;\" alt=\"Aarushi Chakraborty\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eAarushi Chakraborty\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/nowcasting-metrics/commits?author=aarxshi\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\n\u003c!-- markdownlint-restore --\u003e\n\u003c!-- prettier-ignore-end --\u003e\n\n\u003c!-- ALL-CONTRIBUTORS-LIST:END --\u003e\n\nThis project follows the [all-contributors](https://github.com/all-contributors/all-contributors) specification. Contributions of any kind welcome!","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenclimatefix%2Fnowcasting-metrics","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenclimatefix%2Fnowcasting-metrics","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenclimatefix%2Fnowcasting-metrics/lists"}