{"id":20200891,"url":"https://github.com/openclimatefix/nwp-consumer","last_synced_at":"2025-04-10T11:22:03.825Z","repository":{"id":148760276,"uuid":"619894232","full_name":"openclimatefix/nwp-consumer","owner":"openclimatefix","description":"Microservice for consuming NWP data.","archived":false,"fork":false,"pushed_at":"2025-04-02T11:12:28.000Z","size":242934,"stargazers_count":11,"open_issues_count":20,"forks_count":7,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-02T11:22:33.837Z","etag":null,"topics":["containerization","hexagonal-architecture","pyproject-toml","python-package","src-layout"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openclimatefix.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["openclimatefix"],"patreon":null,"open_collective":"openclimatefix","ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"custom":null}},"created_at":"2023-03-27T16:12:18.000Z","updated_at":"2025-03-27T13:31:14.000Z","dependencies_parsed_at":null,"dependency_job_id":"f324fa92-9d5a-409b-9e0d-7137e1badb44","html_url":"https://github.com/openclimatefix/nwp-consumer","commit_stats":{"total_commits":113,"total_committers":5,"mean_commits":22.6,"dds":0.07964601769911506,"last_synced_commit":"4e5fb5edce1a93da6aaa23a29132ecaee183e2b4"},"previous_names":[],"tags_count":116,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2Fnwp-consumer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2Fnwp-consumer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2Fnwp-consumer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2Fnwp-consumer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openclimatefix","download_url":"https://codeload.github.com/openclimatefix/nwp-consumer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248208604,"owners_count":21065203,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["containerization","hexagonal-architecture","pyproject-toml","python-package","src-layout"],"created_at":"2024-11-14T04:46:02.277Z","updated_at":"2025-04-10T11:22:03.816Z","avatar_url":"https://github.com/openclimatefix.png","language":"Python","funding_links":["https://github.com/sponsors/openclimatefix","https://opencollective.com/openclimatefix"],"categories":[],"sub_categories":[],"readme":"# NWP Consumer\n\n**Download and convert weather data for use in ML pipelines**\n\n[![tags badge](https://img.shields.io/github/v/tag/openclimatefix/nwp-consumer?include_prereleases\u0026sort=semver\u0026color=7BCDF3)](https://github.com/openclimatefix/nwp-consumer/tags)\n[![pypi badge](https://img.shields.io/pypi/v/nwp-consumer?\u0026color=086788)](https://pypi.org/project/nwp-consumer)\n[![documentation badge](https://img.shields.io/badge/docs-latest-333333)](https://openclimatefix.github.io/nwp-consumer/)\n[![contributors badge](https://img.shields.io/github/contributors/openclimatefix/nwp-consumer?color=FFFFFF)](https://github.com/openclimatefix/nwp-consumer/graphs/contributors)\n[![workflows badge](https://img.shields.io/github/actions/workflow/status/openclimatefix/nwp-consumer/branch_ci.yml?branch=main\u0026color=FFD053)](https://github.com/openclimatefix/nwp-consumer/actions/workflows/branch_ci.yml)\n[![ease of contribution: easy](https://img.shields.io/badge/ease%20of%20contribution:%20easy-32bd50)](https://github.com/openclimatefix/ocf-meta-repo?tab=readme-ov-file#overview-of-ocfs-nowcasting-repositories)\n\nSome renewables, such as solar and wind, generate power according to the weather conditions.\nAny forecasting therefore requires predictions of how these conditions will change.\nMany meteorological organisations provide Numerical Weather Prediction (NWP) data,\nwhich can then used for model training and inference. \n\nThis data is often very large and can come in various formats.\nFurthermore, these formats are not necessarily suitable for training,\nso may require preprocessing and conversion. \n\nThis package aims to streamline the collection and processing of this NWP data.\n\n\u003e [!Note]\n\u003e This is *not* built to replace tools such as [Herbie](https://github.com/blaylockbk/Herbie). \n\u003e It is built to produce data specific to the needs of Open Climate Fix's models,\n\u003e so things like the output format and the variable selection are hard-coded.\n\u003e If you need a more configurable cli-driven tool, consider using herbie instead.\n\n## Installation\n\nInstall from PyPi using pip:\n\n```bash\n$ pip install nwp-consumer\n```\n\nOr use the container image:\n\n```bash\n$ docker pull ghcr.io/openclimatefix/nwp-consumer\n```\n\n## Example usage\n\n**To download the latest available day of GFS data:***\n\n```bash\n$ nwp-consumer consume\n```\n\n**To create an archive of a month of GFS data:**\n\n\u003e [!Note]\n\u003e This will download several gigabytes of data to your home partition.\n\u003e Make sure you have plenty of free space (and time!)\n\n```bash\n$ nwp-consumer archive --year 2024 --month 1\n```\n\n## Documentation\n\nDocumentation is generated via [pdoc](https://pdoc.dev/docs/pdoc.html).\nTo build the documentation, run the following command in the repository root:\n\n```bash\n$ PDOC_ALLOW_EXEC=1 python -m pdoc -o docs --docformat=google src/nwp_consumer\n```\n\n\u003e [!Note]\n\u003e The `PDOC_ALLOW_EXEC=1` environment variable is required due to a facet\n\u003e of the `ocf_blosc2` library, which imports itself automatically and hence\n\u003e necessitates execution to be enabled.\n\n## FAQ\n\n### How do I authenticate with model repositories that require accounts?\n\nAuthentication, and model repository selection, is handled via environment variables. \nChoose a repository via the `MODEL_REPOSITORY` environment variable. Required environment\nvariables can be found in the repository's metadata function. Missing variables will be\nwarned about at runtime.\n\n### How do I use an S3 bucket for created stores?\n\nThe `ZARRDIR` environment variable can be set to an S3 url\n(ex: `s3://some-bucket-name/some-prefix`). Valid credentials for accessing the bucket\nmust be discoverable in the environment as per\n[Botocore's documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html)\n\n### How do I change what variables are pulled?\n\nWith difficulty! This package pulls data specifically tailored to Open Climate Fix's needs,\nand as such, the data it pulls (and the schema that data is surfaced with)\nis a fixed part of the package. A large part of the value proposition of this consumer is\nthat the data it produces is consistent and comparable between different sources, so pull\nrequests to the effect of adding or changing this for a specific model are unlikely to be\napproved.\n\nHowever, desired changes can be made via cloning the repo and making the relevant\nparameter modifications to the model's expected coordinates in it's metadata for the desired model\nrepository. \n\n## Development\n\n### Linting and static type checking\n \nThis project uses [MyPy](https://mypy.readthedocs.io/en/stable/) for static type checking\nand [Ruff](https://docs.astral.sh/ruff/) for linting.\nInstalling the development dependencies makes them available in your virtual environment.\n\nUse them via:\n\n```bash\n$ python -m mypy .\n$ python -m ruff check .\n```\n\nBe sure to do this periodically while developing to catch any errors early\nand prevent headaches with the CI pipeline. It may seem like a hassle at first,\nbut it prevents accidental creation of a whole suite of bugs.\n\n### Running the test suite\n\nRun the unittests with:\n\n```bash\n$ python -m unittest discover -s src/nwp_consumer -p \"test_*.py\"\n```\n\n## Further reading\n\nOn packaging a python project using setuptools and pyproject.toml:\n- The official [PyPA packaging guide](https://packaging.python.org/en/latest/tutorials/packaging-projects/).\n- A [step-by-step practical guide](https://godatadriven.com/blog/a-practical-guide-to-setuptools-and-pyproject-toml/)\non the *godatadriven* blog.\n- The pyproject.toml\n[metadata specification](https://packaging.python.org/en/latest/specifications/declaring-project-metadata).\n\nOn hexagonal architecture:\n- A [concrete example](https://medium.com/towards-data-engineering/a-concrete-example-of-the-hexagonal-architecture-in-python-d821213c6fb9)\nusing Python.\n- An [overview of the fundamentals](https://medium.com/ssense-tech/hexagonal-architecture-there-are-always-two-sides-to-every-story-bc0780ed7d9c) \nincorporating Typescript \n- Another [example](https://medium.com/@matiasvarela/hexagonal-architecture-in-go-cfd4e436faa3) using Go.\n\nOn the directory structure:\n- The official [PyPA discussion](https://packaging.python.org/en/latest/discussions/src-layout-vs-flat-layout/) on \nsrc and flat layouts.\n\n\n---\n\n## Contributing and community\n\n[![issues badge](https://img.shields.io/github/issues/openclimatefix/nwp-consumer?color=FFAC5F)](https://github.com/openclimatefix/nwp-consumer/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc)\n\n- PR's are welcome! See the [Organisation Profile](https://github.com/openclimatefix) for details on contributing\n- Find out about our other projects in the [OCF Meta Repo](https://github.com/openclimatefix/ocf-meta-repo)\n- Check out the [OCF blog](https://openclimatefix.org/blog) for updates\n- Follow OCF on [LinkedIn](https://uk.linkedin.com/company/open-climate-fix)\n\n\u003c!-- ALL-CONTRIBUTORS-LIST:START - Do not remove or modify this section --\u003e\n\u003c!-- prettier-ignore-start --\u003e\n\u003c!-- markdownlint-disable --\u003e\n\n\u003c!-- markdownlint-restore --\u003e\n\u003c!-- prettier-ignore-end --\u003e\n\n\u003c!-- ALL-CONTRIBUTORS-LIST:END --\u003e\n\n---\n\n*Part of the [Open Climate Fix](https://github.com/orgs/openclimatefix/people) community.*\n\n[![OCF Logo](https://cdn.prod.website-files.com/62d92550f6774db58d441cca/6324a2038936ecda71599a8b_OCF_Logo_black_trans.png)](https://openclimatefix.org)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenclimatefix%2Fnwp-consumer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenclimatefix%2Fnwp-consumer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenclimatefix%2Fnwp-consumer/lists"}