{"id":20201056,"url":"https://github.com/openclimatefix/pvnet","last_synced_at":"2025-04-10T11:22:43.519Z","repository":{"id":63180161,"uuid":"565799141","full_name":"openclimatefix/PVNet","owner":"openclimatefix","description":"PVnet main repo","archived":false,"fork":false,"pushed_at":"2025-04-09T13:06:49.000Z","size":353768,"stargazers_count":28,"open_issues_count":33,"forks_count":30,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-09T14:29:22.325Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openclimatefix.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null},"funding":{"github":["openclimatefix"],"patreon":null,"open_collective":"openclimatefix","ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"custom":null}},"created_at":"2022-11-14T10:55:59.000Z","updated_at":"2025-04-09T13:06:52.000Z","dependencies_parsed_at":"2023-09-27T22:55:52.741Z","dependency_job_id":"e5c16fb7-ba1d-4e19-b97f-8f53b6d7cdea","html_url":"https://github.com/openclimatefix/PVNet","commit_stats":{"total_commits":805,"total_committers":11,"mean_commits":73.18181818181819,"dds":0.6608695652173913,"last_synced_commit":"3e2f306bf9900c69f9152937cd0dc1f45b9dda5c"},"previous_names":[],"tags_count":166,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2FPVNet","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2FPVNet/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2FPVNet/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2FPVNet/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openclimatefix","download_url":"https://codeload.github.com/openclimatefix/PVNet/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248208605,"owners_count":21065203,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-14T04:46:44.637Z","updated_at":"2025-04-10T11:22:43.513Z","avatar_url":"https://github.com/openclimatefix.png","language":"Python","funding_links":["https://github.com/sponsors/openclimatefix","https://opencollective.com/openclimatefix"],"categories":[],"sub_categories":[],"readme":"# PVNet 2.1\n\u003c!-- ALL-CONTRIBUTORS-BADGE:START - Do not remove or modify this section --\u003e\n[![All Contributors](https://img.shields.io/badge/all_contributors-9-orange.svg?style=flat-square)](#contributors-)\n\u003c!-- ALL-CONTRIBUTORS-BADGE:END --\u003e\n\n [![Python Bump Version \u0026 release](https://github.com/openclimatefix/PVNet/actions/workflows/release.yml/badge.svg)](https://github.com/openclimatefix/PVNet/actions/workflows/release.yml) [![ease of contribution: hard](https://img.shields.io/badge/ease%20of%20contribution:%20hard-bb2629)](https://github.com/openclimatefix/ocf-meta-repo?tab=readme-ov-file#overview-of-ocfs-nowcasting-repositories)\n\n\nThis project is used for training PVNet and running PVNet on live data.\n\nPVNet2 is a multi-modal late-fusion model that largely inherits the same architecture from\n[PVNet1.0](https://github.com/openclimatefix/predict_pv_yield). The NWP (Numerical Weather Prediction) and\nsatellite data are sent through some neural network which encodes them down to\n1D intermediate representations. These are concatenated together with the GSP (Grid Supply Point)\noutput history, the calculated solar coordinates (azimuth and elevation) and the\nGSP ID which has been put through an embedding layer. This 1D concatenated\nfeature vector is put through an output network which outputs predictions of the\nfuture GSP yield. National forecasts are made by adding all the GSP forecasts\ntogether.\n\n\n## Experiments\n\nOur paper based on this repo was accepted into the Tackling Climate Change with Machine Learning workshop at ICLR 2024 and can be viewed [here](https://www.climatechange.ai/papers/iclr2024/46).\n\nSome slightly more structured notes on deliberate experiments we have performed with PVNet are [here](https://docs.google.com/document/d/1VumDwWd8YAfvXbOtJEv3ZJm_FHQDzrKXR0jU9vnvGQg).\n\nSome very rough, early working notes on this model are\n[here](https://docs.google.com/document/d/1fbkfkBzp16WbnCg7RDuRDvgzInA6XQu3xh4NCjV-WDA). These are now somewhat out of date.\n\n\n\n## Setup / Installation\n\n```bash\ngit clone https://github.com/openclimatefix/PVNet.git\ncd PVNet\npip install .\n```\n\nThe commit history is extensive. To save download time, use a depth of 1:\n```bash\ngit clone --depth 1 https://github.com/openclimatefix/PVNet.git\n```\nThis means only the latest commit and its associated files will be downloaded.\n\nNext, in the PVNet repo, install PVNet as an editable package:\n\n```bash\npip install -e .\n```\n\n### Additional development dependencies\n\n```bash\npip install \".[dev]\"\n```\n\n\n\n## Getting started with running PVNet\n\nBefore running any code in PVNet, copy the example configuration to a\nconfigs directory:\n\n```\ncp -r configs.example configs\n```\n\nYou will be making local amendments to these configs. See the README in\n`configs.example` for more info.\n\n### Datasets\n\nAs a minimum, in order to create batches of data/run PVNet, you will need to\nsupply paths to NWP and GSP data. PV data can also be used. We list some\nsuggested locations for downloading such datasets below:\n\n**GSP (Grid Supply Point)** - Regional PV generation data\\\nThe University of Sheffield provides API access to download this data:\nhttps://www.solar.sheffield.ac.uk/api/\n\nDocumentation for querying generation data aggregated by GSP region can be found\nhere:\nhttps://docs.google.com/document/d/e/2PACX-1vSDFb-6dJ2kIFZnsl-pBQvcH4inNQCA4lYL9cwo80bEHQeTK8fONLOgDf6Wm4ze_fxonqK3EVBVoAIz/pub#h.9d97iox3wzmd\n\n**NWP (Numerical weather predictions)**\\\nOCF maintains a Zarr formatted version of the German Weather Service's (DWD)\nICON-EU NWP model here:\nhttps://huggingface.co/datasets/openclimatefix/dwd-icon-eu which includes the UK\n\n**PV**\\\nOCF maintains a dataset of PV generation from 1311 private PV installations\nhere: https://huggingface.co/datasets/openclimatefix/uk_pv\n\n\n### Connecting with ocf-data-sampler for batch creation\n\nOutside the PVNet repo, clone the ocf-data-sampler repo and exit the conda env created for PVNet: https://github.com/openclimatefix/ocf-data-sampler\n```bash\ngit clone https://github.com/openclimatefix/ocf-data-sampler.git\nconda create -n ocf-data-sampler python=3.11\n```\n\nThen go inside the ocf-data-sampler repo to add packages\n\n```bash\npip install .\n```\n\nThen exit this environment, and enter back into the pvnet conda environment and install ocf-data-sampler in editable mode (-e). This means the package is directly linked to the source code in the ocf-data-sampler repo.\n\n```bash\npip install -e \u003cPATH-TO-ocf-data-sampler-REPO\u003e\n```\n\nIf you install the local version of `ocf-data-sampler` that is more recent than the version specified in PVNet, you might receive a warning. However, it should still function correctly.\n\n## Generating pre-made batches of data for training/validation of PVNet\n\nPVNet contains a script for generating batches of data suitable for training the PVNet models. To run the script you will need to make some modifications to the datamodule configuration.\n\nMake sure you have copied the example configs (as already stated above):\n```\ncp -r configs.example configs\n```\n\n### Set up and config example for batch creation\n\nWe will use the following example config file for creating batches: `/PVNet/configs/datamodule/configuration/example_configuration.yaml`. Ensure that the file paths are set to the correct locations in `example_configuration.yaml`: search for `PLACEHOLDER` to find where to input the location of the files. You will need to comment out or delete the parts of `example_configuration.yaml` pertaining to the data you are not using.\n\n\nWhen creating batches, an additional datamodule config located in `PVNet/configs/datamodule` is passed into the batch creation script: `streamed_batches.yaml`. Like before, a placeholder variable is used when specifying which configuration to use:\n\n```yaml\nconfiguration: \"PLACEHOLDER.yaml\"\n```\n\nThis should be given the whole path to the config on your local machine, for example:\n\n```yaml\nconfiguration: \"/FULL-PATH-TO-REPO/PVNet/configs/datamodule/configuration/example_configuration.yaml\"\n```\n\nWhere `FULL-PATH-TO-REPO` represent the whole path to the PVNet repo on your local machine.\n\nThis is also where you can update the train, val \u0026 test periods to cover the data you have access to.\n\n### Running the batch creation script\n\nRun the `save_samples.py` script to create batches with the parameters specified in the datamodule config (`streamed_batches.yaml` in this example):\n\n```bash\npython scripts/save_samples.py\n```\nPVNet uses\n[hydra](https://hydra.cc/) which enables us to pass variables via the command\nline that will override the configuration defined in the `./configs` directory, like this:\n\n```bash\npython scripts/save_samples.py datamodule=streamed_batches datamodule.sample_output_dir=\"./output\" datamodule.num_train_batches=10 datamodule.num_val_batches=5\n```\n\n`scripts/save_samples.py` needs a config under `PVNet/configs/datamodule`. You can adapt `streamed_batches.yaml` or create your own in the same folder.\n\nIf downloading private data from a GCP bucket make sure to authenticate gcloud (the public satellite data does not need authentication):\n\n```\ngcloud auth login\n```\n\nFiles stored in multiple locations can be added as a list. For example, in the `example_configuration.yaml` file we can supply a path to satellite data stored on a bucket:\n\n```yaml\nsatellite:\n    satellite_zarr_path: gs://solar-pv-nowcasting-data/satellite/EUMETSAT/SEVIRI_RSS/v4/2020_nonhrv.zarr\n```\n\nOr to satellite data hosted by Google:\n\n```yaml\nsatellite:\n    satellite_zarr_paths:\n      - \"gs://public-datasets-eumetsat-solar-forecasting/satellite/EUMETSAT/SEVIRI_RSS/v4/2020_nonhrv.zarr\"\n      - \"gs://public-datasets-eumetsat-solar-forecasting/satellite/EUMETSAT/SEVIRI_RSS/v4/2021_nonhrv.zarr\"\n```\n\nocf-data-sampler is currently set up to use 11 channels from the satellite data, the 12th of which is HRV and is not included in these.\n\n\n### Training PVNet\n\nHow PVNet is run is determined by the extensive configuration in the config\nfiles. The configs stored in `PVNet/configs.example` should work with batches created using the steps and batch creation config mentioned above.\n\nMake sure to update the following config files before training your model:\n\n1. In `configs/datamodule/local_premade_batches.yaml`:\n    - update `batch_dir` to point to the directory you stored your batches in during batch creation\n2. In `configs/model/local_multimodal.yaml`:\n    - update the list of encoders to reflect the data sources you are using. If you are using different NWP sources, the encoders for these should follow the same structure with two important updates:\n        - `in_channels`: number of variables your NWP source supplies\n        - `image_size_pixels`: spatial crop of your NWP data. It depends on the spatial resolution of your NWP; should match `image_size_pixels_height` and/or `image_size_pixels_width` in `datamodule/configuration/site_example_configuration.yaml` for the NWP, unless transformations such as coarsening was applied (e. g. as for ECMWF data)\n3. In `configs/local_trainer.yaml`:\n    - set `accelerator: 0` if running on a system without a supported GPU\n\nIf creating copies of the config files instead of modifying existing ones, update `defaults` in the main `./configs/config.yaml` file to use\nyour customised config files:\n\n```yaml\ndefaults:\n  - trainer: local_trainer.yaml\n  - model: local_multimodal.yaml\n  - datamodule: local_premade_batches.yaml\n  - callbacks: null\n  - logger: csv.yaml\n  - experiment: null\n  - hparams_search: null\n  - hydra: default.yaml\n```\n\nAssuming you ran the `save_samples.py` script to generate some premade train and\nval data batches, you can now train PVNet by running:\n\n```\npython run.py\n```\n\n## Backtest\n\nIf you have successfully trained a PVNet model and have a saved model checkpoint you can create a backtest using this, e.g. forecasts on historical data to evaluate forecast accuracy/skill. This can be done by running one of the scripts in this repo such as [the UK GSP backtest script](scripts/backtest_uk_gsp.py) or the [the pv site backtest script](scripts/backtest_sites.py), further info on how to run these are in each backtest file.\n\n\n## Testing\n\nYou can use `python -m pytest tests` to run tests\n\n## Contributors ✨\n\nThanks goes to these wonderful people ([emoji key](https://allcontributors.org/docs/en/emoji-key)):\n\n\u003c!-- ALL-CONTRIBUTORS-LIST:START - Do not remove or modify this section --\u003e\n\u003c!-- prettier-ignore-start --\u003e\n\u003c!-- markdownlint-disable --\u003e\n\u003ctable\u003e\n  \u003ctbody\u003e\n    \u003ctr\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/felix-e-h-p\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/137530077?v=4?s=100\" width=\"100px;\" alt=\"Felix\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eFelix\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/PVNet/commits?author=felix-e-h-p\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/Sukh-P\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/42407101?v=4?s=100\" width=\"100px;\" alt=\"Sukhil Patel\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eSukhil Patel\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/PVNet/commits?author=Sukh-P\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/dfulu\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/41546094?v=4?s=100\" width=\"100px;\" alt=\"James Fulton\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eJames Fulton\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/PVNet/commits?author=dfulu\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/AUdaltsova\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/43303448?v=4?s=100\" width=\"100px;\" alt=\"Alexandra Udaltsova\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eAlexandra Udaltsova\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/PVNet/commits?author=AUdaltsova\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/zakwatts\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/47150349?v=4?s=100\" width=\"100px;\" alt=\"Megawattz\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eMegawattz\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/PVNet/commits?author=zakwatts\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/peterdudfield\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/34686298?v=4?s=100\" width=\"100px;\" alt=\"Peter Dudfield\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003ePeter Dudfield\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/PVNet/commits?author=peterdudfield\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/mahdilamb\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/4696915?v=4?s=100\" width=\"100px;\" alt=\"Mahdi Lamb\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eMahdi Lamb\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"#infra-mahdilamb\" title=\"Infrastructure (Hosting, Build-Tools, etc)\"\u003e🚇\u003c/a\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://www.jacobbieker.com\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/7170359?v=4?s=100\" width=\"100px;\" alt=\"Jacob Prince-Bieker\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eJacob Prince-Bieker\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/PVNet/commits?author=jacobbieker\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/codderrrrr\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/149995852?v=4?s=100\" width=\"100px;\" alt=\"codderrrrr\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003ecodderrrrr\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/PVNet/commits?author=codderrrrr\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\n\u003c!-- markdownlint-restore --\u003e\n\u003c!-- prettier-ignore-end --\u003e\n\n\u003c!-- ALL-CONTRIBUTORS-LIST:END --\u003e\n\nThis project follows the [all-contributors](https://github.com/all-contributors/all-contributors) specification. Contributions of any kind welcome!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenclimatefix%2Fpvnet","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenclimatefix%2Fpvnet","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenclimatefix%2Fpvnet/lists"}