https://github.com/openclimatefix/solar-consumer
Pull and save data solar forecasts from NESO
https://github.com/openclimatefix/solar-consumer
Last synced: 8 months ago
JSON representation
Pull and save data solar forecasts from NESO
- Host: GitHub
- URL: https://github.com/openclimatefix/solar-consumer
- Owner: openclimatefix
- License: mit
- Created: 2024-11-28T16:12:39.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2025-01-16T20:01:40.000Z (over 1 year ago)
- Last Synced: 2025-01-20T23:58:51.143Z (over 1 year ago)
- Language: Python
- Size: 72.3 KB
- Stars: 1
- Watchers: 2
- Forks: 7
- Open Issues: 5
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- open-sustainable-technology - Solar Consumer - This code can be used to download solar forecasts and save them to a PostgreSQL database by retrieving solar generation estimates for embedded solar farms and preparing the data for analysis. (Renewable Energy / Photovoltaics and Solar Energy)
README
# Solar Consumer
[](#contributors-)
[](https://github.com/openclimatefix#how-easy-is-it-to-get-involved)
This code can be used to download solar forecasts and save them to a PostgreSQL database. It fetches solar generation estimates for embedded solar farms and processes the data for analysis.
Forecast can be retreived from NESO, generation values can be retreived from Ned NL.
## Requirements
- Docker
- Docker Compose
## Installation & Running
1. Clone the repository:
```bash
git clone https://github.com/openclimatefix/neso-solar-consumer.git
cd neso-solar-consumer
```
2. Copy the example environment file:
```bash
cp .example.env .env
```
3. Start the application:
```bash
docker compose up -d
```
The above command will:
- Start a PostgreSQL database container
- Build and start the NESO Solar Consumer application
- Configure all necessary networking between containers
To stop the application:
```bash
docker compose down
```
To view logs:
```bash
docker compose logs -f
```
> **Note**: The PostgreSQL data is persisted in a Docker volume. To completely reset the database, use:
> ```bash
> docker compose down -v
> ```
## Documentation
The package provides three main functionalities:
1. **Data Fetching**: Retrieves solar forecast data from the NESO API
2. **Data Formatting**: Processes the data into standardized forecast objects
3. **Data Storage**: Saves the formatted forecasts to a PostgreSQL database
### Key Components:
- `fetch_data.py`: Handles API data retrieval
- `format_forecast.py`: Converts raw data into forecast objects
- `save_forecast.py`: Manages database operations
- `app.py`: Orchestrates the entire pipeline
### Environment Variables: (Can be found in the .example.env / .env file)
- `DB_URL=postgresql://postgres:postgres@localhost:5432/neso_solar` : Database Configuration
- `COUNTRY="gb"` : Country code for fetching data. Currently, other options are ["nl"]
- `SAVE_METHOD="db"`: Ways to store the data. Currently other options are ["csv", "site-db"]
- `CSV_DIR=None` : Directory to save CSV files if `SAVE_METHOD="csv"`.
## Development
1. Set up the development environment:
```bash
pip install ".[dev]"
```
2. Run tests:
```bash
pytest
```
3. Format code:
```bash
black .
```
4. Run linter:
```bash
ruff check .
```
### Running the Test Suite
The test suite includes unit tests and integration tests:
```bash
# Run all tests
pytest
# Run specific test file
pytest tests/test_fetch_data.py
# Run with coverage
pytest --cov=neso_solar_consumer
```
### Continuous Integration (CI)
This reposistory has 2 main CI workflows - `branch-ci` and `merged-ci`.
- `branch-ci` is triggered on all pushes to any branch except `main`, and on any pull request that is opened, reopened or updated. It runs the tests suite, lints the project, and builds and pushes a dev image.
- `merged-ci` is triggered on any pull request merged into `main`. It bumps the git tag, and builds and pushes a container with that tag.
## FAQ
**Q: What format is the data stored in?**
A: The data is stored in PostgreSQL using SQLAlchemy models, with timestamps in UTC and power values in megawatts.
**Q: How often should I run the consumer?**
A: This depends on your use case and the NESO API update frequency. The consumer can be scheduled using cron jobs or other scheduling tools.
## License
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
## Contributing and community
[](https://github.com/openclimatefix/neso-solar-consumer/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc)
- PR's are welcome! See the [Organisation Profile](https://github.com/openclimatefix) for details on contributing
- Find out about our other projects in the [OCF Meta Repo](https://github.com/openclimatefix/ocf-meta-repo)
- Check out the [OCF blog](https://openclimatefix.org/blog) for updates
- Follow OCF on [LinkedIn](https://uk.linkedin.com/company/open-climate-fix)
## Contributors
Peter Dudfield
🤔
Seao7
💻
Siddharth
👀 🚇 💻
Conor O Callaghan
📖
Ali Rashid
⚠️
Manzoor Ahmed Shaikh
💻
Anas Khan
📖
Peter Ireland
📖
vashisthrahul13
💻
rahul-ahuja
💻
Rahul Joon
💻
michael-gendy
🚇
Shohail Ismail
💻
Prafful Vyas
💻
*Part of the [Open Climate Fix](https://github.com/orgs/openclimatefix/people) community.*
[](https://openclimatefix.org)