Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/michael-simons/pv
Some scripts to work with PV measurements and tooling to record measurements.
https://github.com/michael-simons/pv
duckdb energy olap photovoltaics pv sql
Last synced: 9 days ago
JSON representation
Some scripts to work with PV measurements and tooling to record measurements.
- Host: GitHub
- URL: https://github.com/michael-simons/pv
- Owner: michael-simons
- License: apache-2.0
- Created: 2023-05-09T19:26:00.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-10-31T07:20:54.000Z (19 days ago)
- Last Synced: 2024-11-02T12:52:13.777Z (16 days ago)
- Topics: duckdb, energy, olap, photovoltaics, pv, sql
- Language: Jupyter Notebook
- Homepage:
- Size: 1.29 MB
- Stars: 5
- Watchers: 4
- Forks: 2
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE.txt
Awesome Lists containing this project
README
# pv
⚠️ Programs and scripts in this repository are not meant to be used and may damage your equipment (Of course I use them myself but you never know with expensive equiptment).
## Requirements
### For the power logger
The logger is based on the [Energy systems reading toolkit](https://energy.basjes.nl). Thanks a ton for your work, [Niels Basjes](https://mastodon.basjes.nl/@niels).
Build the power logger with
```bash
mvn -f logger/pom.xml clean package
```
### For the database
[DuckDB](https://duckdb.org) >= 1.0.0.
The schema is split into base tables, shared views and eventually, the API, consisting of several views.
It can be applied as follows:
```bash
./bin/create_or_update_database.sh pv.db
```
Measurements are stored per quarterly hour, as local date times (local timezone is assumed).
For dealing with specifics to your area, i.e. changes during summer / winter time observations, scripts needs adjustment.
All views - think of them as public API to this database - start with an `v_`.
### For the Jupyter notebook
I have added a `requirements.txt` usable with `pip` like this:
```bash
pip install -r notebooks/requirements.txt
```
## Usage
### Logger
Run the power logger with:
```bash
./logger/target/assembly/bin/log-power-output
```
Again, this might damage your inverter, burn down the house and what not. Use at your own risk. The logger puts out 1 minute measurements in watt (W) by default which can be imported after creating a database file as described above like this:
```bash
more logger.csv | duckdb pv.db -c ".read attic/import_logger.sql"
```
### Jupyter Notebook
Run the notebook with:
```bash
jupyter notebook notebooks/Photovoltaik\ \|\ Familie\ Simons,\ Aachen.ipynb
```
Produce HTML without input boxes and code:
```bash
jupyter nbconvert --execute --to html --output index.html --no-input notebooks/Photovoltaik\ \|\ Familie\ Simons,\ Aachen.ipynb
```
Clear existing output with:
```bash
jupyter nbconvert --ClearOutputPreprocessor.enabled=True --inplace notebooks/Photovoltaik\ \|\ Familie\ Simons,\ Aachen.ipynb
```
I have a rendered version with my current dataset at [simons.ac/pv](http://simons.ac/pv).
### Database
#### Examples
You can get a list of all views to explorer directly from the information schema like this:
```bash
duckdb --readonly pv.db "SELECT table_name FROM information_schema.tables WHERE table_type = 'VIEW' ORDER BY table_name ASC"
```
Their titles should hopefully be self-explanatory, and I only want to show two highlights here.
The `bar` function and the pivot support. The following adds a bit to the view named `v_average_production_per_month` and produces a quick chart in the terminal, showing the average energy produced per hour:
```bash
duckdb --readonly notebooks/pv.db <<-SQL
WITH max_kWh AS (SELECT max(production) AS value FROM v_average_production_per_hour)
SELECT hour AS Hour, bar(production, 0, max_kWh.value) AS 'Average energy produced (kWh)'
FROM v_average_production_per_hour, max_kWH
ORDER BY hour ASC;
SQL
```
Should look something like this, which I totally love:
Of course this is also possible per month:
```bash
duckdb --readonly notebooks/pv.db <<-SQL
WITH max_kWH AS (SELECT max(production) AS value FROM v_average_production_per_month)
SELECT month AS Month,
bar(production, 0, max_kWH.value) AS 'Average energy produced (kWh)'
FROM v_average_production_per_month, max_kWH
ORDER BY month ASC;
SQL
```
Then there is the `PIVOT` statement being used in `v_average_production_per_month_and_hour` view to compute the average per
hour _and_ month. Originally I displayed the month names as headers, but that is not that useful as an API for the database when you want todo the translation later:
```bash
duckdb --readonly notebooks/pv.db "SELECT * FROM v_average_production_per_month_and_hour"
```
#### Creating backups
The whole database can be exported either as CSV files like this
```sql
EXPORT DATABASE 'target_folder';
```
Or if you prefer [Parquet](https://parquet.apache.org), use the following:
```sql
EXPORT DATABASE 'target_folder' (FORMAT PARQUET, COMPRESSION ZSTD);
```