Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/ma7555/kerasgen
A Keras/Tensorflow compatible image data generator for TripletLoss
https://github.com/ma7555/kerasgen
data-generation data-generator data-generators data-science keras keras-tensorflow tensorflow triplet triplet-loss triplet-neural-network
Last synced: about 6 hours ago
JSON representation
A Keras/Tensorflow compatible image data generator for TripletLoss
- Host: GitHub
- URL: https://github.com/ma7555/kerasgen
- Owner: ma7555
- License: mit
- Created: 2022-02-02T14:42:30.000Z (over 2 years ago)
- Default Branch: master
- Last Pushed: 2022-08-14T00:28:56.000Z (about 2 years ago)
- Last Synced: 2024-09-02T23:46:32.481Z (about 1 month ago)
- Topics: data-generation, data-generator, data-generators, data-science, keras, keras-tensorflow, tensorflow, triplet, triplet-loss, triplet-neural-network
- Language: Python
- Homepage:
- Size: 2.31 MB
- Stars: 14
- Watchers: 5
- Forks: 10
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Citation: CITATION.cff
Awesome Lists containing this project
README
kerasgen
========
[](https://pypi.python.org/pypi/kerasgen)
[](https://zenodo.org/badge/latestdoi/454812299)
A Keras/Tensorflow compatible image data generator for creating balanced batches.
This datagenerator is compatible with TripletLoss as it guarantees the existence of postive pairs in every batch.
Installation
------------
```bash
pip install kerasgen
```
Usage
-----
```python
from kerasgen.balanced_image_dataset import balanced_image_dataset_from_directory
train_ds = balanced_image_dataset_from_directory(
directory, num_classes_per_batch=2,
num_images_per_class=5, image_size=(256, 256),
validation_split=0.2, subset='training', seed=555,
safe_triplet=True)
val_ds = balanced_image_dataset_from_directory(
directory, num_classes_per_batch=2,
num_images_per_class=5, image_size=(256, 256),
validation_split=0.2, subset='validation', seed=555,
safe_triplet=True)
```
Generates a balanced per batch `tf.data.Dataset` from image files in a directory.
Your directory structure should be:
```
main_directory/
...class_a/
......a_image_1.jpg
......a_image_2.jpg
...class_b/
......b_image_1.jpg
......b_image_2.jpg
```
Behind the scenes, this module creates a different dataset for every class and by using weighted random sampling, some random classes `(num_classes_per_batch)` are drawn and a specific number of images is selected from every choosen class `(num_images_per_class)` as long as there are enough samples from this class.
If there is no enough samples remaining from the choosen class, it is skipped and another class is choosen (This behaviour can be disabled and we indefinitely repeat the classes datasets)
Setting `safe_triplet` to `False` (Default) makes sure that every image is seen exactly one time per epoch but it does not guarantee a fixed num_classes_per_batch or num_images_per_class in later batches.
Setting `safe_triplet` to `True` does not guarantee that every epoch will include all different samples from the dataset. But as sampling is weighted per class, every epoch will include a very high percentage of the dataset and should approach 100% as dataset size increases. This however guarantee that both num_classes_per_batch and num_images_per_class are fixed for all batches including later ones.
If you are to use this generator with TripletLoss, your should either:
* Set `safe_triplet` to `True`
* Keep `safe_triplet` default `False` value but be careful with choosing the batch_size so you do not end up with a last batch containing a single class (or a single sample)
Batch size is calculated by multiplying num_classes_per_batch and num_images_per_class.
Requirements
------------
* Tensorflow >= 2.9
* Numpy >= 1.19
Compatible with
-------------
* `tf.data.Dataset` API
* [TripletHardLoss](https://www.tensorflow.org/addons/api_docs/python/tfa/losses/TripletHardLoss)
* [TripletSemiHardLoss](https://www.tensorflow.org/addons/api_docs/python/tfa/losses/TripletSemiHardLoss)
Versions
--------
* v0.1.3: TF>= 2.9
* v0.1.2: TF>= 2.8
* v0.1.1: TF>= 2.7
Licence
-------
MIT
Authors
-------
`KerasGen` was written by [Mahmoud Bahaa](mailto:[email protected]?subject=[GitHub]%20KerasGen%20Support)
If you use this software, please cite it using the metadata from this `CITATION.cff`