{"id":13547062,"url":"https://github.com/FrontierDevelopmentLab/PyRain","last_synced_at":"2025-04-02T19:32:23.725Z","repository":{"id":54434497,"uuid":"287272839","full_name":"FrontierDevelopmentLab/PyRain","owner":"FrontierDevelopmentLab","description":null,"archived":false,"fork":false,"pushed_at":"2021-02-18T08:55:09.000Z","size":113,"stargazers_count":40,"open_issues_count":5,"forks_count":11,"subscribers_count":5,"default_branch":"master","last_synced_at":"2024-11-03T15:38:29.597Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/FrontierDevelopmentLab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-08-13T12:24:11.000Z","updated_at":"2024-05-16T08:37:05.000Z","dependencies_parsed_at":"2022-08-13T15:31:17.569Z","dependency_job_id":null,"html_url":"https://github.com/FrontierDevelopmentLab/PyRain","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FrontierDevelopmentLab%2FPyRain","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FrontierDevelopmentLab%2FPyRain/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FrontierDevelopmentLab%2FPyRain/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FrontierDevelopmentLab%2FPyRain/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/FrontierDevelopmentLab","download_url":"https://codeload.github.com/FrontierDevelopmentLab/PyRain/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246880187,"owners_count":20848822,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T12:00:50.407Z","updated_at":"2025-04-02T19:32:23.240Z","avatar_url":"https://github.com/FrontierDevelopmentLab.png","language":"Python","funding_links":[],"categories":["Weather"],"sub_categories":[],"readme":"# Update: Data now publicly available!\n\nWe are very happy to announce that the memmap datasets are now available publicly at:\nhttps://console.cloud.google.com/storage/browser/aaai_release\nYou require an ordinary Google account to access them.\n\nThe data comes in two different resolutions, `5.625` degrees, and `1.40625` degrees.\nTo see what variables (and timeranges) are contained in each dataset, simply download the associated .dill file and read out as follows (python3):\n\n```import dill\nimport pprint\nwith open(\"path-to-dill-file\", \"rb\") as f:\n    info = dill.load(f)\npprint.pprint(info)\n```\n\nPlease let us know if you have any questions/issues - for technical issues please use the github issues.\nMany thanks, and we hope you will find RainBench useful!\n\n# RainBench - Getting Started\n\n## Downloading the Dataset\nPlease register [here](https://forms.gle/3AdMJsKtuJ8M1E1Y8) to download the RainBench dataset.\n\nAfter downloading, you should update the data paths in config.yml.\n\n## Forecasting Precipitation from ERA\nSpecify `source` as {'simsat', 'simsat_era', 'era16_3'} to use data (*from 2016*) from Simsat alone, ERA5 alone, or both Simsat and ERA5, respectively. \n\nTo use all data available in ERA5 for training (*from 1971*), set `source` as 'era'.\n\nSet `inc_time` to concatenate inputs with hour, day, month.\n\nFor example, to train, run\n\n```\npython3 run_benchmark.py --sources simsat_era --inc_time --config_file config.yml\n```\n\n## Forecasting Precipitation from IMERG\nAgain, specify `source` as {'simsat', 'simsat_era', 'era16_3'} to use data (*from 2016*) from Simsat alone, ERA5 alone, or both Simsat and ERA5, respectively. \n\nTo use all data available in ERA5 for training (*from 2000*), set `source` as 'era'.\n\nFor predicting IMERG precipitation, we found empirically that removing the relu function at the end of the ConvLSTM works better.\n\nSet `inc_time` to concatenate inputs with hour, day, month.\n\n```\npython3 run_benchmark.py --sources simsat_era --no_relu --imerg --inc_time --config_file config.yml\n```\n\n## Evaluating trained models\n\nTo evaluate trained models on the test set, run the following.\n\n```\npython3 run_benchmark.py --test --phase test --load {MODEL_PATH}\n```\n\n\n# Visualizing Predictions\n\nTo visualize the predictions, run the following. \n\n```\npython3 -m src.benchmark.plot_outputs --load {MODEL_PATH} --nc_file {ANY_NC_FILE_PATH}\n```\n\nExample predictions for a randome test date (12 July 2019) is shown below:\n\n### Truth\n![](https://i.imgur.com/O1Fk0XS.gif)\n\n### Simsat\n![](https://i.imgur.com/uMvodFI.gif)\n\n### ERA\n![](https://i.imgur.com/UbOe0Ia.gif)\n\n### Simsat \u0026 ERA\n![](https://i.imgur.com/tX5pmLP.gif)\n\n# Advanced Topics\n\n## Going to higher spatial resolution\n\nRainBench contains memmap datasets at two different spatial resolutions: 5.625 degrees, and 1.46025 degrees. \nFortunately, the NetCDF--\u003eMemmap conversion scripts for 5.625 degrees that come with RainBench can be easily adjusted to NetCDF datasets at higher - or native - resolution. The main change needing to be done is to adjust the pixel width and height of the different variable channels. As the conversion scripts use use multiprocessing in order to saturate I/O during dataset conversion, even very high resolution datasets can be fine-grainedly converted to Memmaps.\n\n## Generating normalisation files\nUnder `src/benchmark/normalise.py`, you can generate your own normalisation files to be used for on-the-fly normalisation of training data. Simply insert your own sample configuration and partitioning setup into the section marked and run the file using python3. This will generate a pickled `dill` file, which contains a dictionary with normalisation entries (and indeed, packaged functions) for each variable field across each partition. Partition of type `repeat` are expressly supported. Just as data conversion, normalisation supports multiprocessing (and out-of-core computations), meaning even datasets at large resolutions can be handled. It is also easy to add new normalisation routines in the fields provided (also have a look at `src/benchmark/transforms.py` for patch-wise local normalisation techniques).\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FFrontierDevelopmentLab%2FPyRain","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FFrontierDevelopmentLab%2FPyRain","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FFrontierDevelopmentLab%2FPyRain/lists"}