{"id":15706776,"url":"https://github.com/michael-simons/pv","last_synced_at":"2025-05-12T14:27:26.848Z","repository":{"id":163303195,"uuid":"638638609","full_name":"michael-simons/pv","owner":"michael-simons","description":"Some scripts to work with PV measurements and tooling to record measurements.","archived":false,"fork":false,"pushed_at":"2025-03-25T21:30:42.000Z","size":1412,"stargazers_count":5,"open_issues_count":0,"forks_count":2,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-14T06:38:51.433Z","etag":null,"topics":["duckdb","energy","olap","photovoltaics","pv","sql"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/michael-simons.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-05-09T19:26:00.000Z","updated_at":"2025-03-25T21:30:46.000Z","dependencies_parsed_at":"2023-12-27T03:42:23.633Z","dependency_job_id":"94eb764c-7c17-4019-b7d5-4b6c39efc87a","html_url":"https://github.com/michael-simons/pv","commit_stats":{"total_commits":130,"total_committers":1,"mean_commits":130.0,"dds":0.0,"last_synced_commit":"dcb7b4dc734c85769a08f6700762412494c490a2"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michael-simons%2Fpv","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michael-simons%2Fpv/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michael-simons%2Fpv/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michael-simons%2Fpv/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/michael-simons","download_url":"https://codeload.github.com/michael-simons/pv/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249902369,"owners_count":21342826,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["duckdb","energy","olap","photovoltaics","pv","sql"],"created_at":"2024-10-03T20:28:14.257Z","updated_at":"2025-04-20T13:31:02.697Z","avatar_url":"https://github.com/michael-simons.png","language":"Jupyter Notebook","readme":"# pv\n\n⚠️ Programs and scripts in this repository are not meant to be used and may damage your equipment (Of course I use them myself but you never know with expensive equiptment).\n\n## Requirements\n\n### For the power logger\n\nThe logger is based on the [Energy systems reading toolkit](https://energy.basjes.nl). Thanks a ton for your work, [Niels Basjes](https://mastodon.basjes.nl/@niels). \n\nBuild the power logger with\n\n```bash\nmvn -f logger/pom.xml clean package\n```\n\n### For the database\n\n[DuckDB](https://duckdb.org) \u003e= 1.0.0.\n\nThe schema is split into base tables, shared views and eventually, the API, consisting of several views.\nIt can be applied as follows:\n\n```bash\n./bin/create_or_update_database.sh pv.db\n```\n\nMeasurements are stored per quarterly hour, as local date times (local timezone is assumed). \nFor dealing with specifics to your area, i.e. changes during summer / winter time observations, scripts needs adjustment.\nAll views - think of them as public API to this database - start with an `v_`.\n\n### For the Jupyter notebook\n\nI have added a `requirements.txt` usable with `pip` like this:\n\n```bash\npip install -r notebooks/requirements.txt\n```\n\n## Usage\n\n### Logger\n\nRun the power logger with:\n\n```bash\n./logger/target/assembly/bin/log-power-output\n```\n\nAgain, this might damage your inverter, burn down the house and what not. Use at your own risk. The logger puts out 1 minute measurements in watt (W) by default which can be imported after creating a database file as described above like this:\n\n```bash\nmore logger.csv | duckdb pv.db -c \".read attic/import_logger.sql\"\n```\n\n### Jupyter Notebook\n\nRun the notebook with:\n\n```bash\njupyter notebook notebooks/Photovoltaik\\ \\|\\ Familie\\ Simons,\\ Aachen.ipynb\n```\n\nProduce HTML without input boxes and code:\n\n```bash\njupyter nbconvert --execute --to html --output index.html --no-input notebooks/Photovoltaik\\ \\|\\ Familie\\ Simons,\\ Aachen.ipynb\n```\n\nClear existing output with:\n\n```bash\njupyter nbconvert --ClearOutputPreprocessor.enabled=True --inplace notebooks/Photovoltaik\\ \\|\\ Familie\\ Simons,\\ Aachen.ipynb\n```\n\nI have a rendered version with my current dataset at [simons.ac/pv](http://simons.ac/pv).\n\n### Database\n\n#### Examples\n\nYou can get a list of all views to explorer directly from the information schema like this:\n\n```bash\nduckdb --readonly pv.db \"SELECT table_name FROM information_schema.tables WHERE table_type = 'VIEW' ORDER BY table_name ASC\"\n```\n\nTheir titles should hopefully be self-explanatory, and I only want to show two highlights here. \nThe `bar` function and the pivot support. The following adds a bit to the view named `v_average_production_per_month` and produces a quick chart in the terminal, showing the average energy produced per hour:\n\n```bash\nduckdb --readonly notebooks/pv.db \u003c\u003c-SQL\nWITH max_kWh AS (SELECT max(production) AS value FROM v_average_production_per_hour)\nSELECT hour AS Hour, bar(production, 0, max_kWh.value) AS 'Average energy produced (kWh)'\nFROM v_average_production_per_hour, max_kWH\nORDER BY hour ASC;\nSQL\n```\n\nShould look something like this, which I totally love:\n\n![stats_avg_per_month_example](media/stats_avg_per_hour.png)\n\nOf course this is also possible per month:\n\n```bash\nduckdb --readonly notebooks/pv.db \u003c\u003c-SQL\nWITH max_kWH AS (SELECT max(production) AS value FROM v_average_production_per_month)\nSELECT month AS Month, \n     bar(production, 0, max_kWH.value) AS 'Average energy produced (kWh)'\nFROM v_average_production_per_month, max_kWH\nORDER BY month ASC;\nSQL\n```\n\nThen there is the `PIVOT` statement being used in `v_average_production_per_month_and_hour` view to compute the average per\nhour _and_ month. Originally I displayed the month names as headers, but that is not that useful as an API for the database when you want todo the translation later:\n\n```bash\nduckdb --readonly notebooks/pv.db \"SELECT * FROM v_average_production_per_month_and_hour\"\n```\n![stats_avg_per_hour_and_month](media/stats_avg_per_hour_and_month.png)\n\n#### Creating backups\n\nThe whole database can be exported either as CSV files like this\n\n```sql\nEXPORT DATABASE 'target_folder';\n```\n\nOr if you prefer [Parquet](https://parquet.apache.org), use the following:\n\n```sql\nEXPORT DATABASE 'target_folder' (FORMAT PARQUET, COMPRESSION ZSTD);\n```\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmichael-simons%2Fpv","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmichael-simons%2Fpv","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmichael-simons%2Fpv/lists"}