{"id":31565757,"url":"https://github.com/correlaid/genesis-python","last_synced_at":"2025-10-05T07:17:23.471Z","repository":{"id":58566298,"uuid":"473957897","full_name":"CorrelAid/genesis-python","owner":"CorrelAid","description":null,"archived":false,"fork":false,"pushed_at":"2022-10-12T21:40:11.000Z","size":921,"stargazers_count":3,"open_issues_count":52,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2023-09-10T12:34:07.736Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CorrelAid.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-03-25T10:05:48.000Z","updated_at":"2022-10-12T21:40:16.000Z","dependencies_parsed_at":"2023-01-19T22:02:54.891Z","dependency_job_id":null,"html_url":"https://github.com/CorrelAid/genesis-python","commit_stats":null,"previous_names":[],"tags_count":0,"template":null,"template_full_name":null,"purl":"pkg:github/CorrelAid/genesis-python","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CorrelAid%2Fgenesis-python","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CorrelAid%2Fgenesis-python/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CorrelAid%2Fgenesis-python/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CorrelAid%2Fgenesis-python/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CorrelAid","download_url":"https://codeload.github.com/CorrelAid/genesis-python/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CorrelAid%2Fgenesis-python/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278421863,"owners_count":25984081,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-05T02:00:06.059Z","response_time":54,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-10-05T07:17:17.820Z","updated_at":"2025-10-05T07:17:23.464Z","avatar_url":"https://github.com/CorrelAid.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ``pystatis``\n\n```pystatis``` is a Python wrapper for the GENESIS web service interface (API). It simplifies accessing the data from the German statistical federal office.\n\nThe main features are:\n\n- **Simplified access** to the API. No more need to write cumbersome API calls.\n- **Credential management** removes need to manually add credentials.\n- **Integrated workflow** enables an end-to-end process from finding the relevant data to download it.\n- **Pandas support** instead of manually parsing results.\n- **Caching** to enable productive work despite strict query limits.\n- **Starting and handling background jobs** for datasets that are to big to be downloaded directly from GENESIS.\n\nTo learn more about GENESIS refer to the official documentation [here](https://www.destatis.de/EN/Service/OpenData/api-webservice.html).\n\n## Installation\n\nYou can install the package via\n\n```bash\n$ pip install pystatis\n```\n\nIf everything worked out correctly, you should be able to import ``pystatis`` like this\n\n```python\nimport pystatis as pystat\n\nprint(\"Version:\", pystat.__version__)\n```\n\n## Get started\n\nTo be able to use the web service/API of GENESIS-Online, you have to be a registered user. You can create your user [here](https://www-genesis.destatis.de/genesis/online?Menu=Anmeldung).\n\nOnce you have a registered user, you can use your username and password as credentials for authentication against the GENESIS-Online API.\n\nTo avoid entering your credentials each time you use ``pystatis``, your credentials will be stored locally with the `init_config()` helper function. This function accepts both a `username` and `password` argument and stores your credentials in a configuration file named `config.ini` that is stored under `\u003cuser home\u003e/.pystatis/config.ini` by default. You can change this path with the optional `config_dir` argument.\n\nSo before you can use ``pystatis`` you have to execute the following code **once**:\n\n```python\nfrom pystatis import init_config\n\ninit_config(username=\"myusername\", password=\"mypassword\")\n```\n\nAfter executing this code you should have a new `config.ini` file under the `\u003cuser home\u003e/.pystatis` directory.\n\nEach time ``pystatis`` is communicating with GENESIS-Online via the API, it is automatically using the stored credentials in this `config.ini`, so you don't have to specify them again. In case of updated credentials, you can either run `init_config()` again or update the values directly in the `config.ini` file.\n\nGENESIS-Online provides a `helloworld` endpoint that can be used to check your credentials:\n\n```python\nfrom pystatis import logincheck\n\nlogincheck()\n\u003e\u003e\u003e '{\"Status\":\"Sie wurden erfolgreich an- und abgemeldet!\",\"Username\":\"ASFJ582LJ\"}'\n```\n\nIf you can see a response like this, your setup is complete and you can start downloading data.\n\nFor more details, please study the provided sample notebook for [cache](./nb/cache.ipynb).\n\n## How to use\n\n### The GENESIS data model\n\nThe Genesis data structure consists of multiple elements as summarized in the image below.\n![Structure](assets/structure.png)\n\nThis package currently supports retrieving the following data types:\n\n- Cubes: Multi-dimensional data objects\n- Tables: Derivatives of cubes that are already packaged into logical units\n\n### Find the right data\n\n``pystatis`` offers the `Find` class to search for any piece of information with GENESIS. Behind the scene it's using the `find` endpoint.\n\nExample:\n\n```python\nfrom pystatis import Find\n\nresults = Find(\"Rohöl\") # Initiates object that contains all variables, statistics, tables and cubes\nresults.run() # Runs the query\nresults.tables.df # Results for tables\nresults.tables.get_code([1,2,3]) # Gets the table codes, e.g. for downloading the table\nresults.tables.get_metadata([1,2]) # Gets the metadata for the table\n```\n\nA complete overview of all use cases is provided in the [sample notebook.](/nb/find.py)\n\n### Download data\n\nData can be downloaded in to forms: as tables and as cubes. Both interfaces have been aligned to be as close as possible to each other.\n\nExample for downloading a Table:\n\n```python\nfrom pystatis import Table\n\nt = Table(name=\"21311-0001\")  # data is not yet downloaded\nt.get_data()  # Only now the data is either fetched from GENESIS or loaded from cache. If the data is downloaded from online, it will be also cached, so next time the data is loaded from cache.\nt.data  # a pandas data frame\n```\n\nExample for downloading a Cube:\n\n```python\nfrom pystatis import Cube\n\nc = Cube(name=\"22922KJ1141\")  # data is not yet downloaded\nc.get_data()  # Only now the data is either fetched from GENESIS or loaded from cache. If the data is downloaded from online, it will be also cached, so next time the data is loaded from cache.\nc.data  # a pandas data frame\n```\n\nFor more details, please study the provided sample notebook for [tables](./nb/table.ipynb) and [cubes](./nb/cube.ipynb).\n\n### Clear Cache\n\nWhen a cube or table is queried, it will be put into cache automatically. The cache can be cleared using the following function:\n\n```python\nfrom pystatis import clear_cache\n\nclear_cache(\"21311-0001\")  # only deletes the data for the object with the specified name\nclear_cache()  # deletes the complete cache\n```\n\n## License\n\nDistributed under the MIT License. See `LICENSE.txt` for more information.\n\n## Roadmap\n\nA few ideas we should implement in the maybe-near future:\n\n- Improve Table parsing. Right now, the parsing is really simple and we should align the cube and table format so that the data frame for tables is more convenient to use.\n- Create a source code documentation with Sphinx or similar tools.\n- Mechanism to download data that is newer than the cached version. Right now, once data is cached, it is always retrieved from cache no matter if there is a newer version online. However, this could be quite challenging as the GENESIS API is really bad in providing a good and consistent field for the last update datetime.\n- Improve Table and Cube metadata so the user can look up the variables contained in the dataset and for each variable the values that this variable can have.\n- Understand and support time series.\n\n## How to contribute?\n\nContributions to this project are highly appreciated! You can either contact the maintainers or directly create a pull request for your proposed changes:\n\n1. Fork the Project\n2. Create your Feature Branch (`git checkout -b feature/\u003cdescriptive-name\u003e`)\n3. Commit your changes (`git commit -m 'Added NewFeature'`)\n4. Push to remote (`git push origin feature/\u003cdescriptive-name\u003e`)\n5. Open a Pull Request to be merged with dev\n\n## Developer information\n\nTo contribute to this project, please follow these steps:\n\n1. Install [poetry](https://python-poetry.org/docs/). We recommend installing `poetry` via [pipx](https://pypa.github.io/pipx/) which gives you a global `poetry` command in an isolated virtual environment.\n2. Clone the repository via git.\n3. Change into the project root directory.\n4. Run `poetry install` to create the virtual environment within `poetry`'s cache folder (run `poetry env info` to see the details of this new virtual environment). `poetry` has installed all dependencies for you, as well as the package itself.\n5. Install pre-commit: `poetry run pre-commit install`. This will activate the pre-commit hooks that will run prior every commit to ensure code quality.\n6. Do your changes.\n7. Run `poetry run pytest` to see if all existing tests still run through. It is important to use `poetry run` to call `pytest` so that `poetry` uses the created virtual environment and not the system's default Python interpreter. Alternatively, you can run `poetry shell` to let `poetry` activate the virtual environment for the current session. Afterwards, you can run `pytest` as usual without any prefix. You can leave the poetry shell with the `exit` command.\n8. Add new tests depending on your changes.\n9. Run `poetry run pytest` again to make sure your tests are also passed.\n10. Commit and push your changes.\n11. Create a PR.\n\nTo learn more about `poetry`, see [Dependency Management With Python Poetry](https://realpython.com/dependency-management-python-poetry/#command-reference) by realpython.com.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcorrelaid%2Fgenesis-python","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcorrelaid%2Fgenesis-python","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcorrelaid%2Fgenesis-python/lists"}