Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/pommes-public/pommesdata

A full-featured transparent data preparation routine from raw data to POMMES model inputs
https://github.com/pommes-public/pommesdata

data opensource power raw-data transparent

Last synced: 3 months ago
JSON representation

A full-featured transparent data preparation routine from raw data to POMMES model inputs

Awesome Lists containing this project

README 

          
# pommesdata



**A full-featured transparent data preparation routine from raw data to POMMES model inputs**



This is the **data preparation routine** of the fundamental power market model *POMMES* (**PO**wer **M**arket **M**odel of **E**nergy and re**S**ources).


Please navigate to the section of interest to find out more.



## Contents

* [Introduction](#introduction)

* [Documentation](#documentation)

* [Installation and usage](#installation-and-usage)

* [Contributing](#contributing)

* [Citing](#citing)

* [License](#license)



## Introduction

*POMMES* itself is a cosmos consisting of a **dispatch model**, a **data preparation routine** (stored in this repository and described here) and an **investment model** for the German wholesale power market. The model was originally developed by a group of researchers and students at the [chair of Energy and Resources Management of TU Berlin](https://www.er.tu-berlin.de/menue/home/) and is now maintained by a group of alumni and open for other contributions.



If you are interested in the actual dispatch or investment model, please find more information here:

- [pommesdispatch](https://github.com/pommes-public/pommesdispatch): A bottom-up fundamental power market model for the German electricity sector

- pommesinvest: A multi-period integrated investment and dispatch model for the German power sector (upcoming).



## Documentation

The data preparation is mainly carried out in this **[jupyter notebook](https://github.com/pommes-public/pommesdata/blob/dev/pommesdata/data_preparation.ipynb)**.

The data sources used as well as the calculation and transformation steps applied are described in a transparent manner.

In addition to that, there is a **[documentation of pommesdata](https://pommesdata.readthedocs.io/)** on readthedocs.

This in turn contains a documentation of the functions and classes used for data preparation. 



## Installation and usage

There are **two use cases** for using `pommesdata`:

1. Using readily prepared output data sets as `pommesdispatch` or `pommesinvest` inputs

2. Understanding and manipulating the data prep process (inspecting / developing)



If you are only interested in the readily prepared data sets (option 1), you can obtain

them from zenodo and download it here: [https://zenodo.org/](https://zenodo.org/)



If you are interested in understanding the data preparation process itself or

if you wish to include own additions, changes or assumptions, you can

[fork](https://docs.github.com/en/get-started/quickstart/fork-a-repo)

 and then clone the repository, in order to copy the files locally by typing



```

git clone https://github.com/pommes-public/pommesdata.git

```



After cloning the repository, you have to install the required dependencies.

Make sure you have conda installed as a package manager.

If not, you can download it [here](https://www.anaconda.com/).

Open a command shell and navigate to the folder where you copied the environment to. Use the following command to install dependencies



```

conda env create -f pommesdata_explicit.yml

```

Activate your environment by typing

```

conda activate pommesdata_explicit

```



*Note: Dependencies have not been regularly updated. Thus, use the listed explicit

dependencies from `pommesdata_explicit.yml` for now and not the `environment.yml` file.*



## Contributing

Every kind of contribution or feedback is warmly welcome.


We use the GitHub issue management as well as pull requests for collaboration. 



We try to stick to the PEP8 coding standards.



The jupyter notebook for the data preparation does not (necessarily have to) 

meet PEP8 standards, though readability should be made sure.



### Authors

* Authors of `pommesdata` are Johannes Kochems and Yannick Werner. It is maintained by Johannes Kochems.

* Florian Maurer contributed to the source code by providing a bug fix.

* All people mentioned below contributed to early-stage versions or predecessors of POMMES or ideally supported it.



### List of contributors to POMMES

The following people have contributed to *POMMES*.

Most of these contributions belong to early-stage versions and are not part

of the actual source code. Nonetheless, all contributions shall be acknowledged and the full list is provided for transparency reasons.



The main contributors are stated on top, the remainder

is listed in alphabetical order.



| Name                                       | Contribution                                                                                                                                                                                                                                                                                         |

|--------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|

| Johannes Kochems                           | major development & conceptualization
conceptualization, development of all investment-related parts; development of main data preparation routines (esp. future projection for all components, RES tender data and LCOE estimates, documentation), architecture, publishing process, maintenance |

| Yannick Werner                             | major development & conceptualization
conceptualization, development of main data preparation routines (status quo data for all components, detailed RES, interconnector and hydro data), architecture                                                                                            |

| Benjamin Grosse                            | data collection for conventional power plants in early development stage, ideal support and conceptionel counseling                                                                                                                                                                                  |

| Carla Spiller                              | data collection for conventional power plants in early stage development as an input to *pommesdata*; co-development of rolling horizon dispatch modelling in predecessor of *pommesdispatch*                                                                                                        |

| Christian Fraatz                           | data collection for conventional power plants in early stage development as an input to *pommesdata*                                                                                                                                                                                                 |

| Conrad Nicklisch                           | data collection for RES in early stage development as an input to *pommesdata*                                                                                                                                                                                                                       |

| Daniel Peschel                             | data collection on CHP power plants as an input to *pommesdata*                                                                                                                                                                                                                                      |

| Dr. Johannes Giehl                         | conceptionel support and research of data licensing; conceptionel support for investment modelling in *pommesinvest*                                                                                                                                                                                 |

| Dr. Paul Verwiebe                          | development of small test models as a predecessor of POMMES                                                                                                                                                                                                                                          |

| Fabian Büllesbach                          | development of a predecessor of the rolling horizon modeling approach in *pommesdispatch*                                                                                                                                                                                                            |

| Flora von Mikulicz-Radecki                 | extensive code and functionality testing in an early development stage for predecessors of *pommesdispatch* and *pommesinvest*                                                                                                                                                                       |

| Florian Maurer                             | support with / fix for python dependencies                                                                                                                                                                                                                                                           |

| Hannes Kachel                              | development and analysis of approaches for complexity reduction in a predecessor of *pommesinvest*                                                                                                                                                                                                   |

| Julian Endres                              | data collection for costs and conventional power plants in early stage development                                                                                                                                                                                                                   |

| Julien Faist                               | data collection for original coal power plant shutdown and planned installation of new power plants for *pommesdata*; co-development of a predecessor of *pommesinvest*                                                                                                                              |

| Leticia Encinas Rosa                       | ata collection for conventional power plants in early stage development as an input to *pommesdata*                                                                                                                                                                                                  |

| Prof. Dr.-Ing. Joachim Müller-Kirchenbauer | funding, enabling and conceptual support                                                                                                                                                                                                                                                             |

| Robin Claus                                | data collection for RES in early stage development as an input to *pommesdata*                                                                                                                                                                                                                       |

| Sophie Westphal                            | data collection for costs and conventional power plants in early stage development as an input for *pommesdata*                                                                                                                                                                                      |

| Timona Ghosh                               | data collection for interconnector data as an input to *pommesdata*                                                                                                                                                                                                                                  |



## Citing

Data sets created with `pommesdata` are shared at [zenodo](https://zenodo.org).

If you use these, please refer to the citation information given at zenodo.



If you are using `pommesdata` for your own analyses, we recommend citing as:


*Kochems, J. & Wener, Y. (2024): pommesdata. A full-featured transparent data preparation routine from raw data to POMMES model inputs. https://github.com/pommes-public/pommesdata, accessed YYYY-MM-DD.*



We furthermore recommend naming the version tag or the commit hash used for the sake of transparency and reproducibility.



Also see CITATION.cff for citation information. Licensing information stated

in the CITATION.cff is only applicable for the code itself, see 

[license](#License).



## License

Licensing for the code - in the following referred to as software - 

and the input data used differs. For the licensing of the data, 

please see the detailed list of data sets below.



### Software (code)



Copyright 2024 pommes developer group



Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:



The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.



THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.



### Data (input data)

The following table contains the primary data sources used to create data sets used for *POMMES* models.

The licensing of the different sources differs and the table should provide an overview over the licences used.

Thus, we cannot publish all the data under an open license, such as a Creative Commons Attribution license. Please

be aware that some data might be subject to copyright.



| institution | data set | license | download link |

| ---- | ---- | ---- | ---- |

| OPSD | data package conventional power plants | MIT License for software; for dataset-specific license see hyperlink | https://doi.org/10.25832/conventional_power_plants/2018-12-20 

| ÜNB / BNetzA | power plant list | free to use, license-free according to §5 Abs. 1 UrhG | https://www.netzentwicklungsplan.de/sites/default/files/paragraphs-files/Kraftwerksliste_%C3%9CNB_Entwurf_Szenariorahmen_2030_V2019_2_0_0.pdf

| FZJ / KIT / FIAS | FRESNA (PyPSA-EUR) PP matching | GPLv3 for software, for dataset-specific license see hyperlink | https://doi.org/10.5281/zenodo.3358985

| tmrowco | bidding zone geometries | MIT License | https://github.com/tmrowco/electricitymap-contrib/pull/1383

| UBA | new-built power plants | usage of data accordant to § 12a EGovG permitted | https://www.umweltbundesamt.de/sites/default/files/medien/384/bilder/dateien/4_tab_genehmigte-in_genehmigung-kraftwerksprojekte_2019-04-04.pdf

| BDEW | new-built power plants | All rights reserved | https://www.bdew.de/media/documents/PI_20190401_BDEW-Kraftwerksliste.pdf

| BNetzA | new-built & decommissioned power plants | free to use, license-free according to §5 Abs. 1 UrhG | https://www.bundesnetzagentur.de/DE/Sachgebiete/ElektrizitaetundGas/Unternehmen_Institutionen/Versorgungssicherheit/Erzeugungskapazitaeten/Kraftwerksliste/kraftwerksliste-node.html

| Energie SaarLorLux | new-built power plant | All rights reserved | https://www.energie-saarlorlux.com/unternehmen/mehr-gutes-klima/unsere-co2-projekte/

| ENTSOE | new-built power plants | CC BY 4.0 | https://tyndp.entsoe.eu/maps-data

| BNetzA | threshold for new-built power plants | free to use, license-free according to §5 Abs. 1 UrhG | https://www.bundesnetzagentur.de/SharedDocs/Downloads/DE/Sachgebiete/Energie/Unternehmen_Institutionen/Versorgungssicherheit/Berichte_Fallanalysen/BNetzA_Netzstabilitaetsanlagen13k.pdf?__blob=publicationFile&v=3

| DIW | efficiency estimates for power plants | All rights reserved | https://www.diw.de/documents/publikationen/73/diw_01.c.440963.de/diw_datadoc_2014-072.pdf

| BNetzA | power plants shutdown | free to use, license-free according to §5 Abs. 1 UrhG | https://www.bundesnetzagentur.de/DE/Sachgebiete/ElektrizitaetundGas/Unternehmen_Institutionen/Versorgungssicherheit/Erzeugungskapazitaeten/KWSAL/KWSAL

| juris | nuclear power plants shutdown | free to use, license-free according to §5 Abs. 1 UrhG | https://www.gesetze-im-internet.de/atg/

| juris | coal power plants shutdown | free to use, license-free according to §5 Abs. 1 UrhG | https://www.gesetze-im-internet.de/kvbg/index.html

| KWSB | coal power plants shutdown | CC BY-ND 3.0 DE | https://www.bmwi.de/Redaktion/DE/Downloads/A/abschlussbericht-kommission-wachstum-strukturwandel-und-beschaeftigung.pdf?__blob=publicationFile

| ENTSOE | Actual Generation per Generation Unit | Use pursuant to Article 5 of the Terms & Conditions of ENTSO-E; data owned by the specific TSOs |  https://transparency.entsoe.eu/generation/r2/actualGenerationPerGenerationUnit/show

| ENTSOE | Water Reservoirs and Hydro Storage Plants | Use pursuant to Article 5 of the Terms & Conditions of ENTSO-E; data owned by the specific TSOs | https://transparency.entsoe.eu/generation/r2/waterReservoirsAndHydroStoragePlants/show

| ENTSOE | Actual Generation per Production Type | Use pursuant to Article 5 of the Terms & Conditions of ENTSO-E; data owned by the specific TSOs | https://transparency.entsoe.eu/generation/r2/actualGenerationPerGenerationUnit/show

| UBA | specific emission factors | Use pursuant to § 12a EGovG for pre-calculations | https://www.umweltbundesamt.de/publikationen/entwicklung-der-spezifischen-kohlendioxid-6

| OPSD | time series data | MIT License for software; for dataset-specific license see hyperlink | https://data.open-power-system-data.org/time_series/2020-10-06

| ÜNB | Anlagenstammdaten | data owned by the German TSO | https://www.netztransparenz.de/EEG/Anlagenstammdaten

| ÜNB | EEG-Bewegungsdaten zur Jahresabrechnung 2017 | data owned by the German TSO | https://www.netztransparenz.de/EEG/Jahresabrechnungen

| IRENA | installed RES capacities | All rights reserved, data used for pre-calculations | https://www.irena.org/Statistics/Download-Data

| ENTSO-E | Installed Capacity per Production Type | Use pursuant to Article 5 of the Terms & Conditions of ENTSO-E; data owned by the specific TSOs | https://transparency.entsoe.eu/generation/r2/installedGenerationCapacityAggregation/show

| Prognos et al. | study on RES capacities for DE | All rights reserved, data used for pre calculations | https://www.agora-energiewende.de/veroeffentlichungen/klimaneutrales-deutschland/

| BNetzA | RES tender results solarPV | free to use, license-free according to §5 Abs. 1 UrhG | https://www.bundesnetzagentur.de/DE/Sachgebiete/ElektrizitaetundGas/Unternehmen_Institutionen/Versorgungssicherheit/Erzeugungskapazitaeten/Kraftwerksliste/kraftwerksliste-node.html

| BNetzA | RES tender results wind onshore | free to use, license-free according to §5 Abs. 1 UrhG | https://www.bundesnetzagentur.de/DE/Sachgebiete/ElektrizitaetundGas/Unternehmen_Institutionen/Ausschreibungen/Wind_Onshore/BeendeteAusschreibungen/BeendeteAusschreibungen_node.html

| BNetzA | RES tender results common tenders | free to use, license-free according to §5 Abs. 1 UrhG | https://www.bundesnetzagentur.de/DE/Sachgebiete/ElektrizitaetundGas/Unternehmen_Institutionen/Ausschreibungen/Wind_Onshore/BeendeteAusschreibungen/BeendeteAusschreibungen_node.html

| BNetzA | RES tender results offshore | free to use, license-free according to §5 Abs. 1 UrhG | https://www.bundesnetzagentur.de/DE/Service-Funktionen/Beschlusskammern/1_GZ/BK6-GZ/2017/BK6-17-001/Ergebnisse_erste_Ausschreibung.pdf?__blob=publicationFile&v=3

| BNetzA | RES tender results offshore | free to use, license-free according to §5 Abs. 1 UrhG | https://www.bundesnetzagentur.de/DE/Service-Funktionen/Beschlusskammern/1_GZ/BK6-GZ/2018/BK6-18-001/Ergebnisse_zweite_ausschreibung.pdf?__blob=publicationFile&v=3

| BNetzA | solarPV installations (and remuneration) | free to use, license-free according to §5 Abs. 1 UrhG | https://www.bundesnetzagentur.de/DE/Sachgebiete/ElektrizitaetundGas/Unternehmen_Institutionen/ErneuerbareEnergien/ZahlenDatenInformationen/EEG_Registerdaten/ArchivDatenMeldgn/ArchivDatenMeldgn_node.html

| ÜNB | capacity balance | All rights reserved, data used for pre-calculations | https://www.netztransparenz.de/portals/1/Bericht_zur_Leistungsbilanz_2019.pdf

| DIW | fuel costs uranium 2017 | All rights reserved | https://www.diw.de/documents/publikationen/73/diw_01.c.440963.de/diw_datadoc_2014-072.pdf

| DIW | operation costs | All rights reserved | https://www.diw.de/documents/publikationen/73/diw_01.c.440963.de/diw_datadoc_2014-072.pdf

| Öko Institut | fuel costs lignite 2017 | All rights reserved |  https://www.oeko.de/oekodoc/1995/2014-015-de.pdf

| Destatis | fuel costs hardcoal 2017 | CC BY 2.0 DE | https://www-genesis.destatis.de/genesis/online?&sequenz=tabelleErgebnis&selectionname=43511-0001#abreadcrumb

| BAFA | fuel costs natural gas 2017 | CC BY-ND 3.0 DE | https://www.bafa.de/SharedDocs/Downloads/DE/Energie/egas_aufkommen_export_1991.html

| BMWI | fuel costs heating oil 2017 | CC BY-ND 3.0 DE | https://www.bmwi.de/Redaktion/DE/Artikel/Energie/energiedaten-gesamtausgabe.html

| r2b | transport costs | CC BY-ND 3.0 DE | https://www.bmwi.de/Redaktion/DE/Publikationen/Studien/definition-und-monitoring-der-versorgungssicherheit-an-den-europaeischen-strommaerkten.pdf?__blob=publicationFile&v=18

| Fraunhofer ISI | operation costs | All rights reserved | https://www.ise.fraunhofer.de/content/dam/ise/de/documents/publications/studies/DE2018_ISE_Studie_Stromgestehungskosten_Erneuerbare_Energien.pdf



### Prepared data sets (data sets created with *pommesdata*)

The data is provided with no license. Please refer to the above licensing information.