{"id":14970653,"url":"https://github.com/openclimatefix/skillful_nowcasting","last_synced_at":"2025-04-15T02:15:17.807Z","repository":{"id":37955577,"uuid":"402392585","full_name":"openclimatefix/skillful_nowcasting","owner":"openclimatefix","description":"Implementation of DeepMind's Deep Generative Model of Radar (DGMR)  https://arxiv.org/abs/2104.00954","archived":false,"fork":false,"pushed_at":"2025-04-14T17:18:15.000Z","size":223,"stargazers_count":244,"open_issues_count":24,"forks_count":61,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-04-15T02:15:11.829Z","etag":null,"topics":["gan","nowcasting","nowcasting-models","nowcasting-precipitation","pytorch","pytorch-implementation","pytorch-lightning"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openclimatefix.png","metadata":{"funding":{"github":["openclimatefix"],"patreon":null,"open_collective":"openclimatefix","ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"custom":null},"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2021-09-02T11:18:14.000Z","updated_at":"2025-04-10T11:41:55.000Z","dependencies_parsed_at":"2023-02-09T15:31:09.919Z","dependency_job_id":"715cb628-4d89-492a-b238-d47e3e24f083","html_url":"https://github.com/openclimatefix/skillful_nowcasting","commit_stats":{"total_commits":221,"total_committers":10,"mean_commits":22.1,"dds":0.4796380090497737,"last_synced_commit":"fc1c74850abb600717b65f1d1282e2139e09c140"},"previous_names":[],"tags_count":46,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2Fskillful_nowcasting","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2Fskillful_nowcasting/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2Fskillful_nowcasting/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2Fskillful_nowcasting/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openclimatefix","download_url":"https://codeload.github.com/openclimatefix/skillful_nowcasting/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248991559,"owners_count":21194894,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gan","nowcasting","nowcasting-models","nowcasting-precipitation","pytorch","pytorch-implementation","pytorch-lightning"],"created_at":"2024-09-24T13:43:56.648Z","updated_at":"2025-04-15T02:15:17.788Z","avatar_url":"https://github.com/openclimatefix.png","language":"Python","readme":"# Skillful Nowcasting with Deep Generative Model of Radar (DGMR)\n\u003c!-- ALL-CONTRIBUTORS-BADGE:START - Do not remove or modify this section --\u003e\n[![All Contributors](https://img.shields.io/badge/all_contributors-13-orange.svg?style=flat-square)](#contributors-)\n\u003c!-- ALL-CONTRIBUTORS-BADGE:END --\u003e\nImplementation of DeepMind's Skillful Nowcasting GAN Deep Generative Model of Radar (DGMR) (https://arxiv.org/abs/2104.00954) in PyTorch Lightning.\n\nThis implementation matches as much as possible the pseudocode released by DeepMind. Each of the components (Sampler, Context conditioning stack, Latent conditioning stack, Discriminator, and Generator) are normal PyTorch modules. As the model training is a bit complicated, the overall architecture is wrapped in PyTorch Lightning.\n\nThe default parameters match what is written in the paper.\n\n## Installation\n\nClone the repository, then run\n```shell\npip install -r requirements.txt\npip install -e .\n````\n\nAlternatively, you can also install through ```pip install dgmr```\n\n## Training Data\n\nThe open-sourced UK training dataset has been mirrored to [HuggingFace Datasets!](https://huggingface.co/datasets/openclimatefix/nimrod-uk-1km) This should enable training the original architecture on the original data for reproducing the results from the paper. The full dataset is roughly 1TB in size, and unfortunately, streaming the data from HF Datasets doesn't seem to work, so it has to be cached locally. We have added the sample dataset as well though, which can be directly streamed from GCP without costs.\n\nThe dataset can be loaded with\n\n```python\nfrom datasets import load_dataset\n\ndataset = load_dataset(\"openclimatefix/nimrod-uk-1km\")\n```\n\nFor now, only the sample dataset support streaming in, as its data files are hosted on GCP, not HF, so it can be used with:\n\n```python\nfrom datasets import load_dataset\n\ndataset = load_dataset(\"openclimatefix/nimrod-uk-1km\", \"sample\", streaming=True)\n```\n\nThe authors also used [MRMS](https://www.nssl.noaa.gov/projects/mrms/) US precipitation radar data as another comparison. While that dataset was not released, the MRMS data is publicly available, and we have made that data available on HuggingFace Datasets as well [here](https://huggingface.co/datasets/openclimatefix/mrms). This dataset is the raw 3500x7000 contiguous US MRMS data for 2016 through May 2022, is a few hundred GBs in size, with sporadic updates to more recent data planned. This dataset is in Zarr format, and can be streamed without caching locally through\n\n```python\nfrom datasets import load_dataset\n\ndataset = load_dataset(\"openclimatefix/mrms\", \"default_sequence\", streaming=True)\n```\n\nThis steams the data with 24 timesteps per example, just like the UK DGMR dataset. To get individual MRMS frames, instead of a sequence, this can be achieved through\n\n```python\nfrom datasets import load_dataset\n\ndataset = load_dataset(\"openclimatefix/mrms\", \"default\", streaming=True)\n```\n\n## Pretrained Weights\n\nPretrained weights are be available through [HuggingFace Hub](https://huggingface.co/openclimatefix), currently weights trained on the sample dataset. The whole DGMR model or different components can be loaded as the following:\n\n```python\nfrom dgmr import DGMR, Sampler, Generator, Discriminator, LatentConditioningStack, ContextConditioningStack\nmodel = DGMR.from_pretrained(\"openclimatefix/dgmr\")\nsampler = Sampler.from_pretrained(\"openclimatefix/dgmr-sampler\")\ndiscriminator = Discriminator.from_pretrained(\"openclimatefix/dgmr-discriminator\")\nlatent_stack = LatentConditioningStack.from_pretrained(\"openclimatefix/dgmr-latent-conditioning-stack\")\ncontext_stack = ContextConditioningStack.from_pretrained(\"openclimatefix/dgmr-context-conditioning-stack\")\ngenerator = Generator(conditioning_stack=context_stack, latent_stack=latent_stack, sampler=sampler)\n```\n\n## Example Usage\n\n```python\nfrom dgmr import DGMR\nimport torch.nn.functional as F\nimport torch\n\nmodel = DGMR(\n        forecast_steps=4,\n        input_channels=1,\n        output_shape=128,\n        latent_channels=384,\n        context_channels=192,\n        num_samples=3,\n    )\nx = torch.rand((2, 4, 1, 128, 128))\nout = model(x)\ny = torch.rand((2, 4, 1, 128, 128))\nloss = F.mse_loss(y, out)\nloss.backward()\n```\n\n## Citation\n```\n@article{ravuris2021skillful,\n  author={Suman Ravuri and Karel Lenc and Matthew Willson and Dmitry Kangin and Remi Lam and Piotr Mirowski and Megan Fitzsimons and Maria Athanassiadou and Sheleem Kashem and Sam Madge and Rachel Prudden Amol Mandhane and Aidan Clark and Andrew Brock and Karen Simonyan and Raia Hadsell and Niall Robinson Ellen Clancy and Alberto Arribas† and Shakir Mohamed},\n  title={Skillful Precipitation Nowcasting using Deep Generative Models of Radar},\n  journal={Nature},\n  volume={597},\n  pages={672--677},\n  year={2021}\n}\n```\n\n## Contributors ✨\n\nThanks goes to these wonderful people ([emoji key](https://allcontributors.org/docs/en/emoji-key)):\n\n\u003c!-- ALL-CONTRIBUTORS-LIST:START - Do not remove or modify this section --\u003e\n\u003c!-- prettier-ignore-start --\u003e\n\u003c!-- markdownlint-disable --\u003e\n\u003ctable\u003e\n  \u003ctbody\u003e\n    \u003ctr\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://www.jacobbieker.com\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/7170359?v=4?s=100\" width=\"100px;\" alt=\"Jacob Bieker\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eJacob Bieker\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/skillful_nowcasting/commits?author=jacobbieker\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"http://johmathe.name/\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/467643?v=4?s=100\" width=\"100px;\" alt=\"Johan Mathe\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eJohan Mathe\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/skillful_nowcasting/commits?author=johmathe\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/ZHANGZ1YUE\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/93907996?v=4?s=100\" width=\"100px;\" alt=\"Z1YUE\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eZ1YUE\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/skillful_nowcasting/issues?q=author%3AZHANGZ1YUE\" title=\"Bug reports\"\u003e🐛\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/GreenLimeSia\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/28706611?v=4?s=100\" width=\"100px;\" alt=\"Nan.Y\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eNan.Y\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"#question-GreenLimeSia\" title=\"Answering Questions\"\u003e💬\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/hedaobaishui\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/20534146?v=4?s=100\" width=\"100px;\" alt=\"Taisanai\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eTaisanai\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"#question-hedaobaishui\" title=\"Answering Questions\"\u003e💬\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/primeoc\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/75205487?v=4?s=100\" width=\"100px;\" alt=\"cameron\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003ecameron\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"#question-primeoc\" title=\"Answering Questions\"\u003e💬\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/zhrli\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/11074703?v=4?s=100\" width=\"100px;\" alt=\"zhrli\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003ezhrli\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"#question-zhrli\" title=\"Answering Questions\"\u003e💬\u003c/a\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/najeeb-kazmi\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/14131235?v=4?s=100\" width=\"100px;\" alt=\"Najeeb Kazmi\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eNajeeb Kazmi\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"#question-najeeb-kazmi\" title=\"Answering Questions\"\u003e💬\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/TQRTQ\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/29155385?v=4?s=100\" width=\"100px;\" alt=\"TQRTQ\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eTQRTQ\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"#question-TQRTQ\" title=\"Answering Questions\"\u003e💬\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://www.linkedin.com/in/viktor-bordiuzha-93b078211\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/43813476?v=4?s=100\" width=\"100px;\" alt=\"Viktor Bordiuzha\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eViktor Bordiuzha\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"#example-victor30608\" title=\"Examples\"\u003e💡\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/agijsberts\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/1579083?v=4?s=100\" width=\"100px;\" alt=\"agijsberts\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eagijsberts\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/skillful_nowcasting/commits?author=agijsberts\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/Mews\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/60406199?v=4?s=100\" width=\"100px;\" alt=\"Mews\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eMews\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/skillful_nowcasting/commits?author=Mews\" title=\"Tests\"\u003e⚠️\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/rutkovskii\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/70371908?v=4?s=100\" width=\"100px;\" alt=\"Aleksei Rutkovskii\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eAleksei Rutkovskii\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/skillful_nowcasting/commits?author=rutkovskii\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\n\u003c!-- markdownlint-restore --\u003e\n\u003c!-- prettier-ignore-end --\u003e\n\n\u003c!-- ALL-CONTRIBUTORS-LIST:END --\u003e\n\nThis project follows the [all-contributors](https://github.com/all-contributors/all-contributors) specification. Contributions of any kind welcome!\n","funding_links":["https://github.com/sponsors/openclimatefix","https://opencollective.com/openclimatefix"],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenclimatefix%2Fskillful_nowcasting","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenclimatefix%2Fskillful_nowcasting","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenclimatefix%2Fskillful_nowcasting/lists"}