{"id":20200912,"url":"https://github.com/openclimatefix/ocf-data-sampler","last_synced_at":"2025-04-10T11:22:14.313Z","repository":{"id":252970992,"uuid":"835758806","full_name":"openclimatefix/ocf-data-sampler","owner":"openclimatefix","description":"A repo for sampling from weather data for renewable energy prediction","archived":false,"fork":false,"pushed_at":"2024-12-23T19:16:35.000Z","size":3677,"stargazers_count":4,"open_issues_count":39,"forks_count":8,"subscribers_count":5,"default_branch":"main","last_synced_at":"2024-12-24T09:17:49.163Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openclimatefix.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["openclimatefix"],"patreon":null,"open_collective":"openclimatefix","ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"custom":null}},"created_at":"2024-07-30T13:27:28.000Z","updated_at":"2024-12-23T09:24:57.000Z","dependencies_parsed_at":"2025-01-02T17:33:07.225Z","dependency_job_id":"8d16e994-115f-4783-b02d-9f6907785c3f","html_url":"https://github.com/openclimatefix/ocf-data-sampler","commit_stats":null,"previous_names":["openclimatefix/ocf-data-sampler"],"tags_count":36,"template":false,"template_full_name":"openclimatefix/ocf-template","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2Focf-data-sampler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2Focf-data-sampler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2Focf-data-sampler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2Focf-data-sampler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openclimatefix","download_url":"https://codeload.github.com/openclimatefix/ocf-data-sampler/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247805061,"owners_count":20999099,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-14T04:46:09.911Z","updated_at":"2025-04-10T11:22:14.306Z","avatar_url":"https://github.com/openclimatefix.png","language":"Python","funding_links":["https://github.com/sponsors/openclimatefix","https://opencollective.com/openclimatefix"],"categories":[],"sub_categories":[],"readme":"# ocf-data-sampler\n \n\u003c!-- ALL-CONTRIBUTORS-BADGE:START - Do not remove or modify this section --\u003e\n[![All Contributors](https://img.shields.io/badge/all_contributors-12-orange.svg?style=flat-square)](#contributors-)\n\u003c!-- ALL-CONTRIBUTORS-BADGE:END --\u003e\n\n[![tags badge](https://img.shields.io/github/v/tag/openclimatefix/ocf-data-sampler?include_prereleases\u0026sort=semver\u0026color=FFAC5F)](https://github.com/openclimatefix/ocf-data-sampler/tags)\n[![ease of contribution: easy](https://img.shields.io/badge/ease%20of%20contribution:%20easy-32bd50)](https://github.com/openclimatefix#how-easy-is-it-to-get-involved) \n\n**ocf-data-sampler** contains all the tools needed to create samples and feed them to our models, such as [PVNet](https://github.com/openclimatefix/PVNet/). The data we work with—typically energy data, satellite imagery, and numerical weather predictions (NWPs)—is usually too heavy to do this on the fly, so that's where this repo comes in: handling steps like opening the data, selecting the right samples, normalising and reshaping, and saving to and reading from disk.\n\nWe are currently migrating to this repo from [ocf_datapipes](https://github.com/openclimatefix/ocf_datapipes/), which performs the same functions but is built around `PyTorch DataPipes`, which are quite cumbersome to work with and are no longer maintained by PyTorch. **ocf-data-sampler** uses `PyTorch Datasets`, and we've taken the opportunity to make the code much cleaner and more manageable.\n\n\u003e [!Note]\n\u003e This repository is still in early development development and large changes to the user facing functions may still occur.\n\n## Documentation\n\n**ocf-data-sampler** doesn't have external documentation _yet_; you can read a bit about how our torch datasets work in the README [here](ocf_data_sampler/torch_datasets/README.md).\n\n## FAQ\n\nIf you have any questions about this or any other of our repos, don't hesitate to hop to our [Discussions Page](https://github.com/orgs/openclimatefix/discussions)!\n\n### How does ocf-data-sampler deal with data sources that use different projections (e.g. some are in latitude-longitude, and some in OSGB)?\n\nWhen creating samples, we make a spatial crop of a preset size centred around a point of interest (POI, usually a solar or wind farm). The size of the crop is set not in miles or kilometres, but in 'pixels', which would be different for different data sources, depending on their spatial resolution, projections they use, and where the POI is. For example, a latitude-longitude source with a 1° resolution will have pixel sizes corresponding to very different 'surface' distances (that you might measure in, e.g., kilometres) from a source with 0.1° resolution. The pixel size will even be different for the same source depending on how close the POI is to the equator!\n\nInstead of trying to accommodate for all these differences and make all the sources use the same spatial grid, we translate the POI's position into the corresponding coordinate system and select the crop using the source's original grid. This 'snapshot' is then passed to the model with no additional information on what specific coordinates it represents; instead, since the size is always the same and the POI is always in the centre, the model gets consistent information on the measurements at a location near the POI and how it affects the target, without any explicit knowledge of where that location is in coordinate system terms.\n\n## Development\n\nYou can install **ocf-data-sampler** for development as follows:\n\n``` \npip install git+https://github.com/openclimatefix/ocf-data-sampler.git\n```\n\n### Running the test suite\n\nThe tests in this project use `pytest`. Once you have it installed, you can run it from the project's directory:\n\n```\ncd ocf-data-sampler\npytest\n``` \n\n## Contributing and community\n\n[![issues badge](https://img.shields.io/github/issues/openclimatefix/ocf-data-sampler?color=FFAC5F)](https://github.com/openclimatefix/ocf-data-sampler/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc)\n\n- PR's are welcome! See the [Organisation Profile](https://github.com/openclimatefix) for details on contributing\n- Find out about our other projects in the [OCF Meta Repo](https://github.com/openclimatefix/ocf-meta-repo)\n- Check out the [OCF blog](https://openclimatefix.org/blog) for updates\n- Follow OCF on [LinkedIn](https://uk.linkedin.com/company/open-climate-fix)\n\n\n## Contributors\n\nThanks goes to these wonderful people ([emoji key](https://allcontributors.org/docs/en/emoji-key)):\n\n\u003c!-- ALL-CONTRIBUTORS-LIST:START - Do not remove or modify this section --\u003e\n\u003c!-- prettier-ignore-start --\u003e\n\u003c!-- markdownlint-disable --\u003e\n\u003ctable\u003e\n  \u003ctbody\u003e\n    \u003ctr\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/dfulu\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/41546094?v=4?s=100\" width=\"100px;\" alt=\"James Fulton\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eJames Fulton\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/ocf-data-sampler/commits?author=dfulu\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/AUdaltsova\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/43303448?v=4?s=100\" width=\"100px;\" alt=\"Alexandra Udaltsova\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eAlexandra Udaltsova\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/ocf-data-sampler/commits?author=AUdaltsova\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/Sukh-P\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/42407101?v=4?s=100\" width=\"100px;\" alt=\"Sukhil Patel\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eSukhil Patel\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/ocf-data-sampler/commits?author=Sukh-P\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/peterdudfield\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/34686298?v=4?s=100\" width=\"100px;\" alt=\"Peter Dudfield\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003ePeter Dudfield\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/ocf-data-sampler/commits?author=peterdudfield\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/VikramsDataScience\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/45002417?v=4?s=100\" width=\"100px;\" alt=\"Vikram Pande\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eVikram Pande\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/ocf-data-sampler/commits?author=VikramsDataScience\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/SophiaLi20\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/163532536?v=4?s=100\" width=\"100px;\" alt=\"Unnati Bhardwaj\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eUnnati Bhardwaj\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/ocf-data-sampler/commits?author=SophiaLi20\" title=\"Documentation\"\u003e📖\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/alirashidAR\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/110668489?v=4?s=100\" width=\"100px;\" alt=\"Ali Rashid\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eAli Rashid\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/ocf-data-sampler/commits?author=alirashidAR\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/felix-e-h-p\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/137530077?v=4?s=100\" width=\"100px;\" alt=\"Felix\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eFelix\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/ocf-data-sampler/commits?author=felix-e-h-p\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://timothyajaniportfolio-b6v3zq29k-timthegreat.vercel.app/\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/60073728?v=4?s=100\" width=\"100px;\" alt=\"Ajani Timothy\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eAjani Timothy\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/ocf-data-sampler/commits?author=Tim1119\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://rupeshmangalam.vercel.app/\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/91172425?v=4?s=100\" width=\"100px;\" alt=\"Rupesh Mangalam\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eRupesh Mangalam\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/ocf-data-sampler/commits?author=RupeshMangalam21\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"http://siddharth7113.github.io\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/114160268?v=4?s=100\" width=\"100px;\" alt=\"Siddharth\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eSiddharth\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/ocf-data-sampler/commits?author=siddharth7113\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/Sachin-G13\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/190184500?v=4?s=100\" width=\"100px;\" alt=\"Sachin-G13\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eSachin-G13\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/ocf-data-sampler/commits?author=Sachin-G13\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\n\u003c!-- markdownlint-restore --\u003e\n\u003c!-- prettier-ignore-end --\u003e\n\n\u003c!-- ALL-CONTRIBUTORS-LIST:END --\u003e\n\nThis project follows the [all-contributors](https://github.com/all-contributors/all-contributors) specification. Contributions of any kind welcome!\n\n---\n\n*Part of the [Open Climate Fix](https://github.com/orgs/openclimatefix/people) community.*\n\n[![OCF Logo](https://cdn.prod.website-files.com/62d92550f6774db58d441cca/6324a2038936ecda71599a8b_OCF_Logo_black_trans.png)](https://openclimatefix.org)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenclimatefix%2Focf-data-sampler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenclimatefix%2Focf-data-sampler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenclimatefix%2Focf-data-sampler/lists"}