{"id":20200991,"url":"https://github.com/openclimatefix/power_perceiver","last_synced_at":"2025-05-07T06:31:10.035Z","repository":{"id":37469882,"uuid":"459717669","full_name":"openclimatefix/power_perceiver","owner":"openclimatefix","description":"Machine learning experiments using the Perceiver IO model to forecast the electricity system (starting with solar)","archived":true,"fork":false,"pushed_at":"2022-11-07T17:23:05.000Z","size":18924,"stargazers_count":7,"open_issues_count":76,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-18T03:47:57.012Z","etag":null,"topics":["deep-learning","machine-learning","pytorch"],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openclimatefix.png","metadata":{"funding":{"github":["openclimatefix"],"patreon":null,"open_collective":"openclimatefix","ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"custom":null},"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-02-15T19:18:57.000Z","updated_at":"2024-08-30T07:32:04.000Z","dependencies_parsed_at":"2023-01-20T21:18:33.789Z","dependency_job_id":null,"html_url":"https://github.com/openclimatefix/power_perceiver","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2Fpower_perceiver","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2Fpower_perceiver/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2Fpower_perceiver/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2Fpower_perceiver/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openclimatefix","download_url":"https://codeload.github.com/openclimatefix/power_perceiver/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252826748,"owners_count":21810177,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","machine-learning","pytorch"],"created_at":"2024-11-14T04:46:25.904Z","updated_at":"2025-05-07T06:31:08.932Z","avatar_url":"https://github.com/openclimatefix.png","language":"Jupyter Notebook","funding_links":["https://github.com/sponsors/openclimatefix","https://opencollective.com/openclimatefix"],"categories":[],"sub_categories":[],"readme":"# power_perceiver\nMachine learning experiments for forecasting the electricity system (starting with solar)\n\n\n# Installation\n\n## Installation with conda\nWe recommend installing [mamba](https://github.com/mamba-org/mamba) and using `mamba env create -f base_environment.yml` instead of `conda env create -f base_environment.yml`.\n\nIf installing on a platform without a GPU, then uncomment `- cpuonly` in `base_environment.yml`.\n\n```shell\nconda env create -f base_environment.yml\nconda activate power_perceiver\n\n# If training, then also install the dependencies listed in train_environment.yml:\n# See https://stackoverflow.com/a/43873901/732596\nconda env update --file train_environment.yml --prune\n\npip install -e .\npre-commit install\n```\n\nIf using `Ranger21` optimizer then please install [Ranger21 with my tiny little patch](https://github.com/JackKelly/Ranger21/tree/patch-1).\n\nTo prevent `mamba update --all` from trying to replace the GPU version of PyTorch with the CPU version,\nadd this to `~/miniconda3/envs/power_perceiver/conda-meta/pinned`:\n\n```\n# Prevent mamba update --all from trying to install CPU version of torch.\n# See: https://stackoverflow.com/a/70536292/732596\ncudatoolkit\u003c11.6\n```\n\n## Installation with pip only\nTo install the base config, use: `pip install -e .`\n\nTo install the code necessary to train, use: `pip install -e .[develop,train]`\n\n# Data pipelines\n\nThere are two different data pipelines:\n\n- `power_perceiver.load_prepared_batches`: Loads batches pre-prepared by `nowcasting_dataset`\n- `power_perceiver.load_raw`: Loads raw (well, intermediate) data\n\n## Data pipeline for data prepared by `nowcasting_dataset`\n\nThe data flows through several steps, in order:\n\n1. Every `PreparedDataSource` subclass loads a batch off disk and processes the `xr.Dataset` using the sequence of `transforms` passed into the `PreparedDataSource`'s constructor. The processed data for every `PreparedDataSource` goes into an `XarrayBatch`. The transforms live in `power_perceiver.transforms.\u003cdata source name\u003e.py`\n2. `PreparedDataset` then processes this `XarrayBatch` with its list of `xr_batch_processors`. The `xr_batch_processors` are processors which need to see across or touch multiple modalities at once while the data is still in an xarray Dataset.\n3. Each `XarrayBatch` is then converted to a `NumpyBatch` by that `PreparedDataSource`'s `to_numpy` method. The `to_numpy` method also normalises, converts units, etc.\n4. Finally, `PreparedDataset` passes the entire `NumpyBatch` through the sequence of `np_batch_processors`.\n\n# About the name \"power perceiver\"\n\nOriginally, when I started work on \"Power Perceiver\" 5 months ago, my intention was to use [DeepMind's Perceiver IO](https://www.deepmind.com/open-source/perceiver-io) at the core of the model. Right now, the model actually just uses a standard transformer encoder, not a Perceiver. But I plan to start using a Perceiver IO again within a month or two, when we start using more input elements than a standard transformer encoder can cope with!\n\n# Acknowledgements\n\nThank you to nvidia for their very generous support: nvidia gave us four RTX A6000 GPUs via the nvidia foundation, and a further two RTX A6000 GPUs via the nvidia hardware grant. Thank you nvidia!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenclimatefix%2Fpower_perceiver","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenclimatefix%2Fpower_perceiver","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenclimatefix%2Fpower_perceiver/lists"}