{"id":43651765,"url":"https://github.com/nes-lab/shepherd-tools","last_synced_at":"2026-06-09T16:00:58.792Z","repository":{"id":38960708,"uuid":"489690827","full_name":"nes-lab/shepherd-tools","owner":"nes-lab","description":"Tooling for the HDF5-dataformat in \u0026 around the Shepherd Testbed","archived":false,"fork":false,"pushed_at":"2026-05-21T13:51:32.000Z","size":3767,"stargazers_count":3,"open_issues_count":11,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2026-05-29T07:00:10.680Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://nes-lab.github.io/shepherd/tools/data","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":"geissdoerfer/shepherd-datalib","license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nes-lab.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2022-05-07T14:08:10.000Z","updated_at":"2026-05-18T14:22:22.000Z","dependencies_parsed_at":"2025-12-18T04:08:28.496Z","dependency_job_id":null,"html_url":"https://github.com/nes-lab/shepherd-tools","commit_stats":{"total_commits":418,"total_committers":2,"mean_commits":209.0,"dds":0.009569377990430672,"last_synced_commit":"07189ca87078bce057c3912c4867bd4698f9a5f1"},"previous_names":["nes-lab/shepherd-tools"],"tags_count":80,"template":false,"template_full_name":null,"purl":"pkg:github/nes-lab/shepherd-tools","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nes-lab%2Fshepherd-tools","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nes-lab%2Fshepherd-tools/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nes-lab%2Fshepherd-tools/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nes-lab%2Fshepherd-tools/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nes-lab","download_url":"https://codeload.github.com/nes-lab/shepherd-tools/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nes-lab%2Fshepherd-tools/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34114437,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-09T02:00:06.510Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-02-04T20:08:40.692Z","updated_at":"2026-06-09T16:00:58.753Z","avatar_url":"https://github.com/nes-lab.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Shepherd - Tools\n\n[![PyPIVersion](https://img.shields.io/pypi/v/shepherd_data.svg)](https://pypi.org/project/shepherd_data)\n[![image](https://img.shields.io/pypi/pyversions/shepherd_data.svg)](https://pypi.python.org/pypi/shepherd-data)\n[![QA-Tests](https://github.com/nes-lab/shepherd-tools/actions/workflows/quality_assurance.yaml/badge.svg)](https://github.com/nes-lab/shepherd-tools/actions/workflows/quality_assurance.yaml)\n[![CodeStyle](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)\n\n**Data-Package**: \u003chttps://pypi.org/project/shepherd_data\u003e\n\n**Core-library**: \u003chttps://pypi.org/project/shepherd_core\u003e\n\n**Main Documentation**: \u003chttps://nes-lab.github.io/shepherd\u003e\n\n**Main Project**: \u003chttps://github.com/nes-lab/shepherd\u003e\n\n**Source Code**: \u003chttps://github.com/nes-lab/shepherd-tools\u003e\n\n---\n\nThe Repository contains python packages for the [shepherd](https://github.com/nes-lab/shepherd)-testbed\n\n- `/shepherd_core` bundles functionality that is used by multiple tools and the community\n- `/shepherd_data` holds the data-module that is designed for users of the testbed\n\nNavigate there to get an in depth view for the tools.\n\n## Development\n\n### Virtual Environment\n\nThe environment brings everything needed for dev-work, steps for installing are described below also as shell-commands (OS-independent).\n\n- clone repository\n- navigate shell into directory\n- install environment\n- activate shell (`uv` will tell you how)\n\n```Shell\ngit clone https://github.com/nes-lab/shepherd-tools\ncd .\\shepherd-tools\n\nuv venv\n# activate shell (`uv` will tell you how)\nuv lock -U\nuv sync -U\nuv pip install .\n```\n\n### Update dynamic Fixtures\n\nWhen external dependencies ([Targets-Repo](https://github.com/nes-lab/shepherd-targets/)) or core-models change, the fixtures should be also updated for the testbed.\n\n```shell\npython3 extra/gen_firmwares.py\npython3 extra/gen_energy_envs.py\npython3 extra/prime_database.py\n# commit the updated 'shepherd_core/shepherd_core/data_models/content/_external_fixtures.yaml'\n# delete (optional) 'extra/content'\n```\n\n### Running Testbench\n\n- run pytest in ``_core``- or ``_data``-subdirectory\n- alternative (bottom-cmd) is running from failed test to next fail (if any)\n\n```shell\npytest\npytest --stepwise\n```\n\n### Code Coverage (with pytest)\n\n- run coverage in ``_core``- or ``_data``-subdirectory\n- check results (in browser `./htmlcov/index.html`)\n\n```shell\ncoverage run -m pytest\n\ncoverage html\n# or simpler\ncoverage report\n```\n\n## Release-Procedure\n\n- if models were changed run all scripts in `/extra` to update pseudo-database\n- increase version number by executing ``bump2version`` (see cmds below)\n- install and run ``prek`` for QA-Checks (pre-commit plugin-replacement), see steps below\n- run unittests from both packages locally\n  - additionally every commit gets automatically tested by GitHub workflows\n- update changelog in ``CHANGELOG.md``\n- move code from dev-branch to main by PR\n- add tag to commit - reflecting current version number - i.e. ``v2023.9.0``\n  - GitHub automatically creates a release \u0026 pushes the release to PyPI\n- update release-text with latest Changelog (from `CHANGELOG.md`)\n- rebase dev-branch\n\n```shell\nbump2version --allow-dirty --new-version 2026.06.1 patch\n# ⤷ format: year.month.patch_release\n\nprek run --all-files\n\n# additional QA-Tests (currently with open issues)\nty check\n\n# inside sub-modules unittests\ncd shepherd_core\npytest --stepwise\ncd ../shepherd_data\npytest --stepwise\n# when developers add code they should make sure its covered by the testsuite\ncoverage run -m pytest\ncoverage html\n```\n\n## Open Tasks / TODO\n\n- remove db-specific fields\n- add map-generator\n- add tests for broken h5-files\n- divide h5-tests in valid and healthy\n- add multi-processing\n- divide core into sub-libs:\n  - shepherd_models: data_models + vsource\n  - shepherd_fw_tools\n  - shepherd_decoders\n  - shepherd_core:\n- allow combining measurements (data_0...data_#)\n  - either hostname, date or seq. number after data_\n  - shepherd-dataset must be explained more clearly\n  - common time-vector for all? or can hdf5 compress 5 copies of the same vector? TEST\n- clearer rules on how delta-time can be generated\n  - generally: t[1:] - t[:-1], but last sample is missing (fill with minimum or mean dt)\n  - more advanced integration methods (trapezoidal, polynomial, ..)\n  - resulting energy-vectors can be base for more precise upsampling-routines\n- more generalized hdf5-structure-explorer (yaml-export) to allow opening all variants\n  - currently it faults when expecting shepherd-conform structure\n  - additional features: show 1..10 of front and tail from dataset\n- monitors \u0026 recorder should move here\n  - add fn for validation, export, visualization\n  - also add basic IV-group as recorder\n\n- [click progressbar](https://click.palletsprojects.com/en/8.1.x/api/#click.progressbar) ⇾ could replace tqdm\n- implementations for this lib\n  - generalize up- and down-sampling, use out_sample_rate instead of ds-factor\n    - lib samplerate (tested) ⇾ promising, but designed for float32 and range of +-1.0\n    - lib resampy (tested) ⇾ could be problematic with slice-iterator\n    - https://stackoverflow.com/questions/29085268/resample-a-numpy-array\n    - scipy.signal.resample, https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.resample.html\n    - scipy.signal.decimate, https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.decimate.html\n    - scipy.signal.resample_poly, https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.resample_poly.html#scipy.signal.resample_poly\n    - timestamps could be regenerated with np.arange( tmin, tmax, 1e9/samplerate)\n  - generalize converters (currently in IVonne)\n    - isc\u0026voc \u003c-\u003e ivcurve\n    - ivcurve ⇾ ivsample\n  - plotting and downsampling for IVSurface\n  - plotting more generalized (power, cpu-util, ..., if IV then offer power as well)\n  - some metadata is calculated wrong (non-scalar datasets)\n  - unittests \u0026 codecoverage ⇾ 79% with v22.5.4, https://pytest-cov.readthedocs.io/en/latest/config.html\n    - test example: https://github.com/kvas-it/pytest-console-scripts\n    - use coverage to test some edge-cases\n  - sub-divide valid() into healthy()\n  - add gain/factor to time, with repair-code\n  - add https://pypi.org/project/nessie-recorder/#files\n- main shepherd-code\n  - proper validation first\n  - update commentary\n  - pin-description should be in yaml (and other descriptions for cpu, io, ...)\n  - datatype-hint in h5-file (`ivcurve`, `ivsample`, `isc_voc`), add mechanism to prevent misuse\n  - hostname for emulation\n  - full and minimal config into h5\n  - use shepherd-core as a base\n  - isc-voc-harvesting\n  - directly process isc-voc ⇾ resample to ivcurve?\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnes-lab%2Fshepherd-tools","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnes-lab%2Fshepherd-tools","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnes-lab%2Fshepherd-tools/lists"}