{"id":24446398,"url":"https://github.com/openclimatefix/solar-consumer","last_synced_at":"2025-10-06T14:32:08.639Z","repository":{"id":265895369,"uuid":"895658029","full_name":"openclimatefix/neso-solar-consumer","owner":"openclimatefix","description":"Pull and save data solar forecasts from NESO","archived":false,"fork":false,"pushed_at":"2025-01-16T20:01:40.000Z","size":74,"stargazers_count":1,"open_issues_count":5,"forks_count":7,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-01-20T23:58:51.143Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openclimatefix.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["openclimatefix"],"patreon":null,"open_collective":"openclimatefix","ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"custom":null}},"created_at":"2024-11-28T16:12:39.000Z","updated_at":"2025-01-16T20:01:42.000Z","dependencies_parsed_at":null,"dependency_job_id":"30a6ac4e-12ee-4ab6-83c3-9a152d243b61","html_url":"https://github.com/openclimatefix/neso-solar-consumer","commit_stats":null,"previous_names":["openclimatefix/neso-solar-consumer"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2Fneso-solar-consumer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2Fneso-solar-consumer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2Fneso-solar-consumer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2Fneso-solar-consumer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openclimatefix","download_url":"https://codeload.github.com/openclimatefix/neso-solar-consumer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":235531147,"owners_count":19004907,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-01-20T23:58:51.961Z","updated_at":"2025-10-06T14:32:08.633Z","avatar_url":"https://github.com/openclimatefix.png","language":"Python","funding_links":["https://github.com/sponsors/openclimatefix","https://opencollective.com/openclimatefix"],"categories":["Renewable Energy"],"sub_categories":["Photovoltaics and Solar Energy"],"readme":"# Solar Consumer\n\u003c!-- ALL-CONTRIBUTORS-BADGE:START - Do not remove or modify this section --\u003e\n[![All Contributors](https://img.shields.io/badge/all_contributors-14-orange.svg?style=flat-square)](#contributors-)\n\u003c!-- ALL-CONTRIBUTORS-BADGE:END --\u003e\n\n[![ease of contribution: easy](https://img.shields.io/badge/ease%20of%20contribution:%20easy-32bd50)](https://github.com/openclimatefix#how-easy-is-it-to-get-involved)\n\nThis code can be used to download solar forecasts and save them to a PostgreSQL database. It fetches solar generation estimates for embedded solar farms and processes the data for analysis.\nForecast can be retreived from NESO, generation values can be retreived from Ned NL. \n\n## Requirements\n\n- Docker\n- Docker Compose\n\n## Installation \u0026 Running\n\n1. Clone the repository:\n```bash\ngit clone https://github.com/openclimatefix/neso-solar-consumer.git\ncd neso-solar-consumer\n```\n\n2. Copy the example environment file:\n```bash\ncp .example.env .env\n```\n\n3. Start the application:\n```bash\ndocker compose up -d\n```\n\nThe above command will:\n- Start a PostgreSQL database container\n- Build and start the NESO Solar Consumer application\n- Configure all necessary networking between containers\n\nTo stop the application:\n```bash\ndocker compose down\n```\n\nTo view logs:\n```bash\ndocker compose logs -f\n```\n\n\u003e **Note**: The PostgreSQL data is persisted in a Docker volume. To completely reset the database, use:\n\u003e ```bash\n\u003e docker compose down -v\n\u003e ```\n\n## Documentation\n\nThe package provides three main functionalities:\n\n1. **Data Fetching**: Retrieves solar forecast data from the NESO API\n2. **Data Formatting**: Processes the data into standardized forecast objects\n3. **Data Storage**: Saves the formatted forecasts to a PostgreSQL database\n\n### Key Components:\n\n- `fetch_data.py`: Handles API data retrieval\n- `format_forecast.py`: Converts raw data into forecast objects\n- `save_forecast.py`: Manages database operations\n- `app.py`: Orchestrates the entire pipeline\n\n### Environment Variables: (Can be found in the .example.env / .env file)\n\n- `DB_URL=postgresql://postgres:postgres@localhost:5432/neso_solar` : Database Configuration\n- `COUNTRY=\"gb\"` : Country code for fetching data. Currently, other options are [\"nl\"] \n- `SAVE_METHOD=\"db\"`: Ways to store the data. Currently other options are [\"csv\", \"site-db\"]\n- `CSV_DIR=None` : Directory to save CSV files if `SAVE_METHOD=\"csv\"`.\n\n## Development\n\n1. Set up the development environment:\n```bash\npip install \".[dev]\"\n```\n\n2. Run tests:\n```bash\npytest\n```\n\n3. Format code:\n```bash\nblack .\n```\n\n4. Run linter:\n```bash\nruff check .\n```\n\n### Running the Test Suite\n\nThe test suite includes unit tests and integration tests:\n\n```bash\n# Run all tests\npytest\n\n# Run specific test file\npytest tests/test_fetch_data.py\n\n# Run with coverage\npytest --cov=neso_solar_consumer\n```\n\n### Continuous Integration (CI)\n\nThis reposistory has 2 main CI workflows - `branch-ci` and `merged-ci`. \n\n- `branch-ci` is triggered on all pushes to any branch except `main`, and on any pull request that is opened, reopened or updated. It runs the tests suite, lints the project, and builds and pushes a dev image.\n- `merged-ci` is triggered on any pull request merged into `main`. It bumps the git tag, and builds and pushes a container with that tag.\n\n## FAQ\n\n**Q: What format is the data stored in?**\nA: The data is stored in PostgreSQL using SQLAlchemy models, with timestamps in UTC and power values in megawatts.\n\n**Q: How often should I run the consumer?**\nA: This depends on your use case and the NESO API update frequency. The consumer can be scheduled using cron jobs or other scheduling tools.\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n## Contributing and community\n\n[![issues badge](https://img.shields.io/github/issues/openclimatefix/neso-solar-consumer?color=FFAC5F)](https://github.com/openclimatefix/neso-solar-consumer/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc)\n\n- PR's are welcome! See the [Organisation Profile](https://github.com/openclimatefix) for details on contributing\n- Find out about our other projects in the [OCF Meta Repo](https://github.com/openclimatefix/ocf-meta-repo)\n- Check out the [OCF blog](https://openclimatefix.org/blog) for updates\n- Follow OCF on [LinkedIn](https://uk.linkedin.com/company/open-climate-fix)\n\n\n## Contributors\n\n\n\u003c!-- ALL-CONTRIBUTORS-LIST:START - Do not remove or modify this section --\u003e\n\u003c!-- prettier-ignore-start --\u003e\n\u003c!-- markdownlint-disable --\u003e\n\u003ctable\u003e\n  \u003ctbody\u003e\n    \u003ctr\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/peterdudfield\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/34686298?v=4?s=100\" width=\"100px;\" alt=\"Peter Dudfield\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003ePeter Dudfield\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"#ideas-peterdudfield\" title=\"Ideas, Planning, \u0026 Feedback\"\u003e🤔\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/Seao7\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/100257888?v=4?s=100\" width=\"100px;\" alt=\"Seao7\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eSeao7\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/solar-consumer/commits?author=Seao7\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"http://siddharth7113.github.io\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/114160268?v=4?s=100\" width=\"100px;\" alt=\"Siddharth\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eSiddharth\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/solar-consumer/pulls?q=is%3Apr+reviewed-by%3Asiddharth7113\" title=\"Reviewed Pull Requests\"\u003e👀\u003c/a\u003e \u003ca href=\"#infra-siddharth7113\" title=\"Infrastructure (Hosting, Build-Tools, etc)\"\u003e🚇\u003c/a\u003e \u003ca href=\"https://github.com/openclimatefix/solar-consumer/commits?author=siddharth7113\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/Conor0Callaghan\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/4090256?v=4?s=100\" width=\"100px;\" alt=\"Conor O Callaghan\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eConor O Callaghan\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/solar-consumer/commits?author=Conor0Callaghan\" title=\"Documentation\"\u003e📖\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/alirashidAR\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/110668489?v=4?s=100\" width=\"100px;\" alt=\"Ali Rashid\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eAli Rashid\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/solar-consumer/commits?author=alirashidAR\" title=\"Tests\"\u003e⚠️\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/ManzoorAhmedShaikh\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/110716002?v=4?s=100\" width=\"100px;\" alt=\"Manzoor Ahmed Shaikh\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eManzoor Ahmed Shaikh\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/solar-consumer/commits?author=ManzoorAhmedShaikh\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"http://anaskhan.me\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/83116240?v=4?s=100\" width=\"100px;\" alt=\"Anas Khan\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eAnas Khan\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/solar-consumer/commits?author=anxkhn\" title=\"Documentation\"\u003e📖\u003c/a\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/pjireland\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/16693035?v=4?s=100\" width=\"100px;\" alt=\"Peter Ireland\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003ePeter Ireland\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/solar-consumer/commits?author=pjireland\" title=\"Documentation\"\u003e📖\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/vashisthrahul13\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/182660137?v=4?s=100\" width=\"100px;\" alt=\"vashisthrahul13\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003evashisthrahul13\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/solar-consumer/commits?author=vashisthrahul13\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/rahul-ahuja\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/21355015?v=4?s=100\" width=\"100px;\" alt=\"rahul-ahuja\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003erahul-ahuja\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/solar-consumer/commits?author=rahul-ahuja\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://rahul-joon.github.io/My-Website/\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/61495262?v=4?s=100\" width=\"100px;\" alt=\"Rahul Joon\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eRahul Joon\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/solar-consumer/commits?author=Rahul-JOON\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/michael-gendy\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/64384201?v=4?s=100\" width=\"100px;\" alt=\"michael-gendy\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003emichael-gendy\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"#infra-michael-gendy\" title=\"Infrastructure (Hosting, Build-Tools, etc)\"\u003e🚇\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/Shohail-Ismail\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/149825575?v=4?s=100\" width=\"100px;\" alt=\"Shohail Ismail\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eShohail Ismail\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/solar-consumer/commits?author=Shohail-Ismail\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/Prafful-Vyas\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/118352579?v=4?s=100\" width=\"100px;\" alt=\"Prafful Vyas\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003ePrafful Vyas\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/solar-consumer/commits?author=Prafful-Vyas\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\n\u003c!-- markdownlint-restore --\u003e\n\u003c!-- prettier-ignore-end --\u003e\n\n\u003c!-- ALL-CONTRIBUTORS-LIST:END --\u003e\n*Part of the [Open Climate Fix](https://github.com/orgs/openclimatefix/people) community.*\n\n[![OCF Logo](https://cdn.prod.website-files.com/62d92550f6774db58d441cca/6324a2038936ecda71599a8b_OCF_Logo_black_trans.png)](https://openclimatefix.org)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenclimatefix%2Fsolar-consumer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenclimatefix%2Fsolar-consumer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenclimatefix%2Fsolar-consumer/lists"}