{"id":17383812,"url":"https://github.com/numblr/datagenerator","last_synced_at":"2026-04-29T18:34:32.403Z","repository":{"id":79291321,"uuid":"150470422","full_name":"numblr/datagenerator","owner":"numblr","description":"Python library to generate batches of data to train Keras models ","archived":false,"fork":false,"pushed_at":"2019-04-12T14:45:19.000Z","size":39,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-27T21:15:01.266Z","etag":null,"topics":["batch","data-generator","generator","keras"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/numblr.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-09-26T18:16:52.000Z","updated_at":"2019-04-12T14:45:21.000Z","dependencies_parsed_at":null,"dependency_job_id":"f73d5a47-3757-4581-9c27-620bc70f76c2","html_url":"https://github.com/numblr/datagenerator","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/numblr/datagenerator","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/numblr%2Fdatagenerator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/numblr%2Fdatagenerator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/numblr%2Fdatagenerator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/numblr%2Fdatagenerator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/numblr","download_url":"https://codeload.github.com/numblr/datagenerator/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/numblr%2Fdatagenerator/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32439294,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-29T18:12:22.909Z","status":"ssl_error","status_checked_at":"2026-04-29T18:11:33.322Z","response_time":110,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["batch","data-generator","generator","keras"],"created_at":"2024-10-16T07:43:53.516Z","updated_at":"2026-04-29T18:34:32.378Z","avatar_url":"https://github.com/numblr.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Generator Data Set\n\nA library to provide data generators that can be used to train Keras models.\n\nThe main class of the library provides is the *GeneratorDataSet* that is based on an inventory of records and provides generators that create batches of data based on the records in the inventory.\n\nA typical inventory would consists of unique file names with an associated target label. During data generation for each file name entry the associated file is loaded and processed into a feature vector (or array). In addition, the target label associated with the entry would be encoded into some scalar or vector representation. The batches of featurized data and encoded targets are fed to the model for training via a python generator that iterates over the inventory in each epoch.\n\n## General concept\n\nThe general idea of the library is to operate on the inventory, i.e. a list of records, where a record contains data as key value pairs. The inventory is divided into batches, and each record in a batch is processed by a data encoder to load and encode the data associated with the record, as well as by a target encoder to load and encode the learning target for the record.\n\n![General concept](/doc/diagram.svg)\n\nOn top of this very general setup the library provides convenience classes to support the most common use cases and make the creation of data and target encoders as easy as possible.\n\nIn particular data encoder skeletons for inventories based on files and URLs  are provided, as well as skeletons to encode labels from the inventory based on standard SciPy data encoders, such as LabelEncoder or one hot encodings.\n\n## Usage\n\nThe following sections describe the detailed API that provides most flexibility. The library provides also a couple of convenience methods that cover common use cases:\n\n**[Data from files](#data-from-files)**\u003cbr\u003e\n**[Data from URLs](#data-from-urls)**\n\n### Inventory\n\nThe inventory used to create the *GeneratorDataSet* must be a [pandas DataFrame](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html).\n\n### Encoders\n\nEncoders provide methods to transform a record in the inventory into a feature/target vector representation. Commonly this transformation involves IO operations or data augmentation or both. Data and target encoders must implement one of the following two interfaces:\n\n    class BatchDataEncoder():\n        def fit(self, inventory):\n            pass\n\n        def transform_batch(self, records):\n            raise NotImplementedError()\n\n\n    class DataEncoder():\n        def __call__(self, record):\n            return self.transform(record)\n\n        def fit(self, inventory):\n            pass\n\n        def transform(self, record):\n            raise NotImplementedError()\n\n        def finalize_batch(self, records):\n            return records\n\nThe fit method is optional in both cases. In the second case it is sufficient for the encoder to be callable, all other methods are optional.\n\nThe library provides encoders for common use cases:\n\n#### File based data\n\nFor data that is loaded from files it provides a *FileDataEncoder* that will take care of basic file handling and only data transformation from the file handle needs to be implemented.\n\n#### URL based data\n\nFor data that is loaded from files it provides a *UrlDataEncoder* that will take care of basic resource loading and only data transformation from the the returned data needs to be implemented.\n\n### Target encoders\n\nAlso several target encoders are provided to make the transformation from e.g. labels in the inventory to e.g. integer or one-hot encoding as easy as possible. See the unit tests for examples.\n\n### Creation of a *GeneratorDataSet*\n\nThe library provides factory methods for the most common use cases.\n\n#### Data from files\n\nFor this case we assume that the data consists of a list of files contained in a data directory. Next to the data files there is an inventory *.csv* file that contains the association of each file to a target category. Then a *GeneratorDataSet* based on the inventory file can be created by\n\n    from numblr.datagenerator import generator_for_files, LabelEncoder\n\n    generator_for_files('inventory.csv', 'my/data/dir', MyDataEncoder(), LabelEncoder())\n\nIn the full form\n\n    generator_for_files(('inventory.csv', 'my/data/dir', MyDataEncoder(), LabelEncoder(),\n            id_mapper=lambda id: id + '.ext',\n            id='file_name',\n            target='label',\n            binary=False)\n\n#### Data from URLs\n\n    from numblr.datagenerator import generator_for_urls, LabelEncoder\n\n    generator_for_urls('inventory.csv', 'my/data/dir', MyDataEncoder(), LabelEncoder())\n\nIn the full form\n\n    generator_for_urls(('inventory.csv', 'my/data/dir', MyDataEncoder(), LabelEncoder(),\n        id_mapper=lambda id: id + '.ext',\n        id='file_name',\n        target='label',\n        binary=False)\n\n## Examples\n\nFor basic usage see the integration test in */test/test_integration.py* and the\nexamples in the */examples* directory.\n\nTo run the examples the top level directory of this repository must be added to the python path for execution. One way to do this is run the script from within the respective */examples/the_example* directory with the following command:\n\n    \u003e PYTHONPATH=\"..\" python the_example.py\n\n### MNIST example\n\nThe example creates an example file based data set based on the MNIST data set provided by *Keras*. The entries in the MNIST data set are exported a text file for each digit and an inventory of the files with the associated digit they represent is created. Based on the inventory a *GeneratorDataSet* is created with a file based data encoder and a one hot target encoder to train a simple DNN with the generators obtained from the *GeneratorDataSet*.\n\n## Tests\n\nRun the unit tests of the library with\n\n    pyhton -m unittest\n\nfrom the root directory of the repository.\n\n## References\n\nSee also *fit_generator* on [Keras models](https://keras.io/models/model/).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnumblr%2Fdatagenerator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnumblr%2Fdatagenerator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnumblr%2Fdatagenerator/lists"}