https://github.com/nes-lab/shepherd-tools
Tooling for the HDF5-dataformat in & around the Shepherd Testbed
https://github.com/nes-lab/shepherd-tools
Last synced: about 2 months ago
JSON representation
Tooling for the HDF5-dataformat in & around the Shepherd Testbed
- Host: GitHub
- URL: https://github.com/nes-lab/shepherd-tools
- Owner: nes-lab
- License: mit
- Created: 2022-05-07T14:08:10.000Z (about 4 years ago)
- Default Branch: main
- Last Pushed: 2026-05-21T13:51:32.000Z (2 months ago)
- Last Synced: 2026-05-29T07:00:10.680Z (about 2 months ago)
- Language: Python
- Homepage: https://nes-lab.github.io/shepherd/tools/data
- Size: 3.59 MB
- Stars: 3
- Watchers: 2
- Forks: 0
- Open Issues: 11
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
Awesome Lists containing this project
README
# Shepherd - Tools
[](https://pypi.org/project/shepherd_data)
[](https://pypi.python.org/pypi/shepherd-data)
[](https://github.com/nes-lab/shepherd-tools/actions/workflows/quality_assurance.yaml)
[](https://github.com/astral-sh/ruff)
**Data-Package**:
**Core-library**:
**Main Documentation**:
**Main Project**:
**Source Code**:
---
The Repository contains python packages for the [shepherd](https://github.com/nes-lab/shepherd)-testbed
- `/shepherd_core` bundles functionality that is used by multiple tools and the community
- `/shepherd_data` holds the data-module that is designed for users of the testbed
Navigate there to get an in depth view for the tools.
## Development
### Virtual Environment
The environment brings everything needed for dev-work, steps for installing are described below also as shell-commands (OS-independent).
- clone repository
- navigate shell into directory
- install environment
- activate shell (`uv` will tell you how)
```Shell
git clone https://github.com/nes-lab/shepherd-tools
cd .\shepherd-tools
uv venv
# activate shell (`uv` will tell you how)
uv lock -U
uv sync -U
uv pip install .
```
### Update dynamic Fixtures
When external dependencies ([Targets-Repo](https://github.com/nes-lab/shepherd-targets/)) or core-models change, the fixtures should be also updated for the testbed.
```shell
python3 extra/gen_firmwares.py
python3 extra/gen_energy_envs.py
python3 extra/prime_database.py
# commit the updated 'shepherd_core/shepherd_core/data_models/content/_external_fixtures.yaml'
# delete (optional) 'extra/content'
```
### Running Testbench
- run pytest in ``_core``- or ``_data``-subdirectory
- alternative (bottom-cmd) is running from failed test to next fail (if any)
```shell
pytest
pytest --stepwise
```
### Code Coverage (with pytest)
- run coverage in ``_core``- or ``_data``-subdirectory
- check results (in browser `./htmlcov/index.html`)
```shell
coverage run -m pytest
coverage html
# or simpler
coverage report
```
## Release-Procedure
- if models were changed run all scripts in `/extra` to update pseudo-database
- increase version number by executing ``bump2version`` (see cmds below)
- install and run ``prek`` for QA-Checks (pre-commit plugin-replacement), see steps below
- run unittests from both packages locally
- additionally every commit gets automatically tested by GitHub workflows
- update changelog in ``CHANGELOG.md``
- move code from dev-branch to main by PR
- add tag to commit - reflecting current version number - i.e. ``v2023.9.0``
- GitHub automatically creates a release & pushes the release to PyPI
- update release-text with latest Changelog (from `CHANGELOG.md`)
- rebase dev-branch
```shell
bump2version --allow-dirty --new-version 2026.06.1 patch
# ⤷ format: year.month.patch_release
prek run --all-files
# additional QA-Tests (currently with open issues)
ty check
# inside sub-modules unittests
cd shepherd_core
pytest --stepwise
cd ../shepherd_data
pytest --stepwise
# when developers add code they should make sure its covered by the testsuite
coverage run -m pytest
coverage html
```
## Open Tasks / TODO
- remove db-specific fields
- add map-generator
- add tests for broken h5-files
- divide h5-tests in valid and healthy
- add multi-processing
- divide core into sub-libs:
- shepherd_models: data_models + vsource
- shepherd_fw_tools
- shepherd_decoders
- shepherd_core:
- allow combining measurements (data_0...data_#)
- either hostname, date or seq. number after data_
- shepherd-dataset must be explained more clearly
- common time-vector for all? or can hdf5 compress 5 copies of the same vector? TEST
- clearer rules on how delta-time can be generated
- generally: t[1:] - t[:-1], but last sample is missing (fill with minimum or mean dt)
- more advanced integration methods (trapezoidal, polynomial, ..)
- resulting energy-vectors can be base for more precise upsampling-routines
- more generalized hdf5-structure-explorer (yaml-export) to allow opening all variants
- currently it faults when expecting shepherd-conform structure
- additional features: show 1..10 of front and tail from dataset
- monitors & recorder should move here
- add fn for validation, export, visualization
- also add basic IV-group as recorder
- [click progressbar](https://click.palletsprojects.com/en/8.1.x/api/#click.progressbar) ⇾ could replace tqdm
- implementations for this lib
- generalize up- and down-sampling, use out_sample_rate instead of ds-factor
- lib samplerate (tested) ⇾ promising, but designed for float32 and range of +-1.0
- lib resampy (tested) ⇾ could be problematic with slice-iterator
- https://stackoverflow.com/questions/29085268/resample-a-numpy-array
- scipy.signal.resample, https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.resample.html
- scipy.signal.decimate, https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.decimate.html
- scipy.signal.resample_poly, https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.resample_poly.html#scipy.signal.resample_poly
- timestamps could be regenerated with np.arange( tmin, tmax, 1e9/samplerate)
- generalize converters (currently in IVonne)
- isc&voc <-> ivcurve
- ivcurve ⇾ ivsample
- plotting and downsampling for IVSurface
- plotting more generalized (power, cpu-util, ..., if IV then offer power as well)
- some metadata is calculated wrong (non-scalar datasets)
- unittests & codecoverage ⇾ 79% with v22.5.4, https://pytest-cov.readthedocs.io/en/latest/config.html
- test example: https://github.com/kvas-it/pytest-console-scripts
- use coverage to test some edge-cases
- sub-divide valid() into healthy()
- add gain/factor to time, with repair-code
- add https://pypi.org/project/nessie-recorder/#files
- main shepherd-code
- proper validation first
- update commentary
- pin-description should be in yaml (and other descriptions for cpu, io, ...)
- datatype-hint in h5-file (`ivcurve`, `ivsample`, `isc_voc`), add mechanism to prevent misuse
- hostname for emulation
- full and minimal config into h5
- use shepherd-core as a base
- isc-voc-harvesting
- directly process isc-voc ⇾ resample to ivcurve?