{"id":20201057,"url":"https://github.com/openclimatefix/nowcasting_datamodel","last_synced_at":"2026-03-02T15:32:51.340Z","repository":{"id":37246828,"uuid":"456496591","full_name":"openclimatefix/nowcasting_datamodel","owner":"openclimatefix","description":"Datamodel for the nowcasting project","archived":false,"fork":false,"pushed_at":"2024-04-12T10:31:55.000Z","size":1326,"stargazers_count":6,"open_issues_count":20,"forks_count":5,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-04-14T06:46:56.947Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openclimatefix.png","metadata":{"funding":{"github":["openclimatefix"],"patreon":null,"open_collective":"openclimatefix","ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"custom":null},"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2022-02-07T12:29:36.000Z","updated_at":"2024-07-01T14:00:30.167Z","dependencies_parsed_at":"2024-01-13T15:17:35.267Z","dependency_job_id":"1da4a596-9c00-4361-a0fb-864eb6872c4d","html_url":"https://github.com/openclimatefix/nowcasting_datamodel","commit_stats":{"total_commits":875,"total_committers":13,"mean_commits":67.3076923076923,"dds":"0.48571428571428577","last_synced_commit":"13ae0d0fd8895a4e54a431bf069661bf6f5e4bee"},"previous_names":[],"tags_count":256,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2Fnowcasting_datamodel","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2Fnowcasting_datamodel/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2Fnowcasting_datamodel/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2Fnowcasting_datamodel/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openclimatefix","download_url":"https://codeload.github.com/openclimatefix/nowcasting_datamodel/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248208605,"owners_count":21065203,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-14T04:46:45.402Z","updated_at":"2026-03-02T15:32:51.294Z","avatar_url":"https://github.com/openclimatefix.png","language":"Python","funding_links":["https://github.com/sponsors/openclimatefix","https://opencollective.com/openclimatefix"],"categories":[],"sub_categories":[],"readme":"# nowcasting_datamodel\n\u003c!-- ALL-CONTRIBUTORS-BADGE:START - Do not remove or modify this section --\u003e\n[![All Contributors](https://img.shields.io/badge/all_contributors-9-orange.svg?style=flat-square)](#contributors-)\n\u003c!-- ALL-CONTRIBUTORS-BADGE:END --\u003e\nDatamodel for the nowcasting project\n\n**:warning: Note this repo will soon be deprecated in favour of a new Data Platform**\n\nThe data model has been made using `sqlalchemy` with a mirrored model in `pydantic`.\n\n⚠️ Database tables are currently made automatically,\nbut in the future there should be a migration process\n\nFuture: The data model could be moved, to be a more modular solution.\n\n## nowcasting_datamodel\n\n### models.py\nAll models are in `nowcasting_datamodel.models.py`.\nThe diagram below shows how the different tables are connected.\n\n![Models](https://raw.githubusercontent.com/openclimatefix/nowcasting_datamodel/main/diagram.png)\n![Models](https://raw.githubusercontent.com/openclimatefix/nowcasting_datamodel/main/diagram_pv.png)\n\n\n### connection.py\n\n`nowcasting_datamodel.connection.py` contains a connection class which can be used to make a sqlalchemy session.\n```python\nfrom nowcasting_datamodel.connection import DatabaseConnection\n\n# make connection object\ndb_connection = DatabaseConnection(url='sqlite:///test.db')\n\n# make sessions\nwith db_connection.get_session() as session:\n\n    # do something with the database\n    pass\n```\n\n### 👓 read.py\n\n`nowcasting_datamodel.read.py` contains functions to read the database.\nThe idea is that these are easy to use functions that query the database in an efficient and easy way.\n\n - get_latest_forecast: Get the latest `Forecast` for a specific GSP.\n - get_all_gsp_ids_latest_forecast: Get the latest `Forecast` for all GSPs.\n - get_forecast_values: Gets the latest `ForecastValue` for a specific GSP\n - get_latest_national_forecast: Returns the latest national forecast\n - get_location: Gets a `Location` object\n\n```python\nfrom nowcasting_datamodel.connection import DatabaseConnection\nfrom nowcasting_datamodel.read import get_latest_forecast\n\n# make connection object\ndb_connection = DatabaseConnection(url='sqlite:///test.db')\n\n# make sessions\nwith db_connection.get_session() as session:\n    f = get_latest_forecast(session=session, gsp_id=1)\n```\n\n### 💾 save.py\n`nowcasting_datamodel.save.py` has one functions to save a list of `Forecast` to the database\n\n### 🇬🇧 national.py\n`nowcasting_datamodel.fake.py` has a useful function for adding up forecasts for all GSPs into a national Forecast.\n\n### fake.py\n`nowcasting_datamodel.fake.py`\n\nFunctions used to make fake model data.\n\n\n## 🩺 Testing\n\nTests are run by using the following command\n```bash\ndocker stop $(docker ps -a -q)\ndocker-compose -f test-docker-compose.yml build\ndocker-compose -f test-docker-compose.yml run tests\n```\n\nThese sets up `postgres` in a docker container and runs the tests in another docker container.\nThis slightly more complicated testing framework is needed (compared to running `pytest`)\nas some queries can not be fully tested on a `sqlite` database\n\n### Mac M1 users\nAn upstream builds issue of libgp may cause the following error:\n\n`sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) SCRAM authentication requires libpq version 10 or above`\n\nAs suggested in this [thread](https://stackoverflow.com/questions/62807717/how-can-i-solve-postgresql-scram-authentication-problem), a temporary fix is to set the env variable `DOCKER_DEFAULT_PLATFORM=linux/amd64` prior to building the test images - although this reportedly comes with performance penalties.\n\n## 🛠️ infrastructure\n\n`.github/workflows` contains a number of CI actions\n1. linters.yaml: Runs linting checks on the code\n2. release.yaml: Make and pushes docker files on a new code release\n3. test-docker.yaml': Runs tests on every push\n\nThe docker file is in the folder `infrastructure/docker/`\n\nThe version is bumped automatically for any push to `main`.\n\n## Environmental Variables\n\n- DB_URL: The database url which the forecasts will be saved too\n\n## Contributors ✨\n\nThanks goes to these wonderful people ([emoji key](https://allcontributors.org/docs/en/emoji-key)):\n\n\u003c!-- ALL-CONTRIBUTORS-LIST:START - Do not remove or modify this section --\u003e\n\u003c!-- prettier-ignore-start --\u003e\n\u003c!-- markdownlint-disable --\u003e\n\u003ctable\u003e\n  \u003ctbody\u003e\n    \u003ctr\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"http://lostcoding.com\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/20285369?v=4?s=100\" width=\"100px;\" alt=\"Brandon Ly\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eBrandon Ly\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/nowcasting_datamodel/commits?author=branberry\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/lucasc896\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/1273006?v=4?s=100\" width=\"100px;\" alt=\"Chris Lucas\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eChris Lucas\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/nowcasting_datamodel/commits?author=lucasc896\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/dfulu\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/41546094?v=4?s=100\" width=\"100px;\" alt=\"James Fulton\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eJames Fulton\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/nowcasting_datamodel/commits?author=dfulu\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/roshnaeem\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/47316899?v=4?s=100\" width=\"100px;\" alt=\"Rosheen Naeem\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eRosheen Naeem\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/nowcasting_datamodel/commits?author=roshnaeem\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/Bvr4\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/48734689?v=4?s=100\" width=\"100px;\" alt=\"Henri Dewilde\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eHenri Dewilde\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/nowcasting_datamodel/commits?author=Bvr4\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/Sahil-Chhoker\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/155215308?v=4?s=100\" width=\"100px;\" alt=\"Sahil Chhoker\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eSahil Chhoker\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/nowcasting_datamodel/commits?author=Sahil-Chhoker\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/abdalahsalah\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/139065821?v=4?s=100\" width=\"100px;\" alt=\"Abdallah salah\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eAbdallah salah\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/nowcasting_datamodel/commits?author=abdalahsalah\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/tmi\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/147159?v=4?s=100\" width=\"100px;\" alt=\"tmi\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003etmi\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/nowcasting_datamodel/commits?author=tmi\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/Averagenormaljoe\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/99669149?v=4?s=100\" width=\"100px;\" alt=\"Database Missing no1\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eDatabase Missing no1\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/nowcasting_datamodel/commits?author=Averagenormaljoe\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\n\u003c!-- markdownlint-restore --\u003e\n\u003c!-- prettier-ignore-end --\u003e\n\n\u003c!-- ALL-CONTRIBUTORS-LIST:END --\u003e\n\nThis project follows the [all-contributors](https://github.com/all-contributors/all-contributors) specification. Contributions of any kind welcome!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenclimatefix%2Fnowcasting_datamodel","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenclimatefix%2Fnowcasting_datamodel","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenclimatefix%2Fnowcasting_datamodel/lists"}