{"id":20200790,"url":"https://github.com/openclimatefix/india-forecast-app","last_synced_at":"2025-04-10T11:21:21.636Z","repository":{"id":217689456,"uuid":"741956696","full_name":"openclimatefix/india-forecast-app","owner":"openclimatefix","description":"Runs wind and PV forecasts for India and saves to database","archived":false,"fork":false,"pushed_at":"2025-04-03T17:25:47.000Z","size":91787,"stargazers_count":5,"open_issues_count":28,"forks_count":13,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-03T17:26:14.774Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openclimatefix.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["openclimatefix"],"patreon":null,"open_collective":"openclimatefix","ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"custom":null}},"created_at":"2024-01-11T13:13:13.000Z","updated_at":"2025-03-30T11:15:29.000Z","dependencies_parsed_at":"2024-01-17T23:23:03.187Z","dependency_job_id":"7f792eb9-160f-4de4-844d-562169647769","html_url":"https://github.com/openclimatefix/india-forecast-app","commit_stats":null,"previous_names":["openclimatefix/india-forecast-app"],"tags_count":112,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2Findia-forecast-app","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2Findia-forecast-app/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2Findia-forecast-app/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2Findia-forecast-app/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openclimatefix","download_url":"https://codeload.github.com/openclimatefix/india-forecast-app/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248208603,"owners_count":21065203,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-14T04:45:41.344Z","updated_at":"2025-04-10T11:21:21.628Z","avatar_url":"https://github.com/openclimatefix.png","language":"Python","funding_links":["https://github.com/sponsors/openclimatefix","https://opencollective.com/openclimatefix"],"categories":[],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003eIndia-Forecast-App \u003c/h1\u003e\n\u003c!-- ALL-CONTRIBUTORS-BADGE:START - Do not remove or modify this section --\u003e\n[![All Contributors](https://img.shields.io/badge/all_contributors-4-orange.svg?style=flat-square)](#contributors-)\n\u003c!-- ALL-CONTRIBUTORS-BADGE:END --\u003e\n\n[![ease of contribution: hard](https://img.shields.io/badge/ease%20of%20contribution:%20hard-bb2629)](https://github.com/openclimatefix/ocf-meta-repo?tab=readme-ov-file#how-easy-is-it-to-get-involved)\n\nRuns wind and PV forecasts for India and saves to database\n\n## The model\n\nThe ML model is from [PVnet](https://github.com/openclimatefix/PVNet) and uses [ocf_datapipes](https://github.com/openclimatefix/ocf_datapipes) for the data processing\nFor both Wind and Solar we use ECMWF data and predict 48 hours into the future. \n\n### Wind\n\nThe latest change is to use a patch size of 84x84, an increase from 64x64. \nThis is to allow for more context in the image and to allow for the model to learn more about the wind patterns.\n\nThe validation error is ~ 7.15% (normalized using the maximum wind generation)\n\n- wandb [link](https://wandb.ai/openclimatefix/india/runs/xdlew7ib)\n- hugging face [link](https://huggingface.co/openclimatefix/windnet_india)\n\nThe weather variables are currently\n- t2m\n- u10\n- u100\n- u200\n- v10\n- v100\n- v200\n\nWe add some model smoothing\n- feathering [feathering](https://github.com/openclimatefix/india-forecast-app/blob/main/india_forecast_app/models/pvnet/model.py#L131) close to current generation: \n- [smoothing](https://github.com/openclimatefix/india-forecast-app/blob/main/india_forecast_app/models/pvnet/model.py#L188) over 1 hour rolling window\n\n\n### PV\n\nThe validation error is ~ 2.28% (normalized using the maximum solar generation)\n\n- wandb [link](https://wandb.ai/openclimatefix/pvnet_india2.1/runs/o4xpvzrc)\n- hugging face [link](https://huggingface.co/openclimatefix/pvnet_india)\n\nThe weather variables are\n- hcc\n- lcc\n- mcc\n- prate\n- sde\n- sr\n- t2m\n- tcc\n- u10\n- v10\n- dlwrf\n- dswrf\n\n### Adjuster\n\nThe Adjuster model improves forecast accuracy by learning from recent prediction errors. Here's how it works:\n\n1. For each forecast, it analyzes the Mean Error (ME) from forecasts made at the same hour over the past 7 days\n2. It calculates the average error for each forecast horizon (e.g., 1-hour ahead, 2-hours ahead, etc.)\n3. It then adjusts the current forecast by subtracting these systematic errors\n\n**Real-world example:**\nIf our ML model consistently under-predicts solar generation by 50kW during sunny mornings (positive ME), the Adjuster will add 50kW to future morning forecasts. Conversely, if it over-predicts evening wind generation by 30kW (negative ME), the Adjuster will subtract 30kW from future evening forecasts.\n\n**Key features:**\n- Time-specific: Adjustments depend on the time of day and forecast horizon\n- Safety limits: Adjustments are capped at 10% of site capacity to prevent extreme corrections\n- Special handling for solar: Ensures zero generation during nighttime\n\nThis approach significantly reduces systematic errors and improves overall forecast accuracy.\n\n| Without Adjuster | With Adjuster |\n|------------------|---------------|\n| Systematic errors persist | Learns from recent patterns |\n| Fixed model behavior | Adapts to changing conditions |\n| Higher overall error | Reduced forecast error |\n\n## Install dependencies (requires [poetry](https://python-poetry.org/))\n\n```\npoetry install\n```\n\n## Linting and formatting\n\nLint with:\n```\nmake lint\n```\n\nFormat code with:\n```\nmake format\n```\n\n## Running tests\n\n```\nmake test\n```\n\n⚠️ Note: one test for the AD model is skipped locally unless the HF token is set, this HF token can be found in AWS Secret Manager under {environment}/huggingface/token and then can be set via export HUGGINGFACE_TOKEN={token_value} in the repo to run the additional test. In CI tests this secret is set so the test will run there.\n\n## Running the app locally\nReplace `{DB_URL}` with a postgres DB connection string (see below for setting up a ephemeral local DB)\n\nIf testing on a local DB, you may use the following script to seed the the DB with a dummy user, site and site_group. \n```\nDB_URL={DB_URL} poetry run seeder\n```\n⚠️ Note this is a destructive script and will drop all tables before recreating them to ensure a clean slate. DO NOT RUN IN PRODUCTION ENVIRONMENTS\n\nThis example runs the application and writes the results to stdout\n```\nDB_URL={DB_URL} NWP_ZARR_PATH={NWP_ZARR_PATH} poetry run app\n```\n\nTo save batches, you need to set the `SAVE_BATCHES_DIR` environment variable to directory. \n```\n\n### Starting a local database using docker\n\n```bash\n    docker run \\\n        -it --rm \\\n        -e POSTGRES_USER=postgres \\\n        -e POSTGRES_PASSWORD=postgres \\\n        -p 54545:5432 postgres:14-alpine \\\n        postgres\n```\n\nThe corresponding `DB_URL` will be\n\n`postgresql://postgres:postgres@localhost:54545/postgres`\n\n## Building and running in [Docker](https://www.docker.com/)\n\nBuild the Docker image\n```\nmake docker.build\n```\n\nCreate a container from the image. This example runs the application and writes the results to stdout.\\\nReplace `{DB_URL}` with a postgres DB connection string.\\\n*N.B if the database host is `localhost` on the host machine, replace `localhost` with `host.docker.internal` so that docker can access the database from within the container*\n```\ndocker run -it --rm -e DB_URL={DB_URL} -e NWP_ZARR_PATH={NWP_ZARR_PATH} ocf/india-forecast-app\n```\n\n## Notes\n\nThis repo makes use of PyTorch (`torch` and `torchvision` packages) CPU-only version. In order to support installing PyTorch via poetry for various environments, we specify the exact wheels for each environment in the pyproject.toml file. Some background reading on why this is required can be found here: https://santiagovelez.substack.com/p/how-to-install-torch-cpu-in-poetry?utm_campaign=post\u0026utm_medium=web\u0026triedRedirect=true \n\n## Contributors ✨\n\nThanks goes to these wonderful people ([emoji key](https://allcontributors.org/docs/en/emoji-key)):\n\n\u003c!-- ALL-CONTRIBUTORS-LIST:START - Do not remove or modify this section --\u003e\n\u003c!-- prettier-ignore-start --\u003e\n\u003c!-- markdownlint-disable --\u003e\n\u003ctable\u003e\n  \u003ctbody\u003e\n    \u003ctr\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/priyanshubajaj\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/58442385?v=4?s=100\" width=\"100px;\" alt=\"priyanshubajaj\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003epriyanshubajaj\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/india-forecast-app/commits?author=priyanshubajaj\" title=\"Tests\"\u003e⚠️\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/peterdudfield\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/34686298?v=4?s=100\" width=\"100px;\" alt=\"Peter Dudfield\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003ePeter Dudfield\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/india-forecast-app/commits?author=peterdudfield\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/Dakshbir\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/144359831?v=4?s=100\" width=\"100px;\" alt=\"Dakshbir\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eDakshbir\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/india-forecast-app/commits?author=Dakshbir\" title=\"Documentation\"\u003e📖\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/MAYANK12SHARMA\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/145884197?v=4?s=100\" width=\"100px;\" alt=\"MAYANK SHARMA\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eMAYANK SHARMA\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/india-forecast-app/commits?author=MAYANK12SHARMA\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\n\u003c!-- markdownlint-restore --\u003e\n\u003c!-- prettier-ignore-end --\u003e\n\n\u003c!-- ALL-CONTRIBUTORS-LIST:END --\u003e\n\nThis project follows the [all-contributors](https://github.com/all-contributors/all-contributors) specification. Contributions of any kind welcome!\n\n## Troubleshooting\n\n### Poetry Installation Issues\n\n**Problem**: `poetry install` fails with dependency conflicts\n**Solution**: Try updating Poetry first with `pip install --upgrade poetry`, then run `poetry update` followed by `poetry install`\n\n**Problem**: Package installation errors\n**Solution**: Check your Python version matches the one specified in `pyproject.toml`. You can use `poetry env use python3.x` to set the correct version.\n\n### Docker Database Connection Issues\n\n**Problem**: Container can't connect to local database with \"connection refused\" error\n**Solution**: If using localhost in your DB_URL, replace it with `host.docker.internal` when running in Docker\n\n**Problem**: Database authentication failures\n**Solution**: Verify your DB_URL format is correct: `postgresql://username:password@hostname:port/database`\n\n### Model Loading Issues\n\n**Problem**: \"Failed to load model\" errors\n**Solution**: Ensure your HUGGINGFACE_TOKEN environment variable is set correctly. The token can be found in AWS Secret Manager under {environment}/huggingface/token.\n\n**Problem**: Out of memory errors when loading models\n**Solution**: Ensure your system has sufficient RAM, or consider using a smaller model variant.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenclimatefix%2Findia-forecast-app","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenclimatefix%2Findia-forecast-app","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenclimatefix%2Findia-forecast-app/lists"}