{"id":24568848,"url":"https://github.com/aeon0/mlpipe-trainer","last_synced_at":"2025-03-17T05:27:37.734Z","repository":{"id":57442371,"uuid":"186129077","full_name":"aeon0/MLPipe-Trainer","owner":"aeon0","description":"Manage training results, weights and data flow of your Tensorflow models","archived":false,"fork":false,"pushed_at":"2021-10-28T05:44:57.000Z","size":64,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-02-18T15:02:40.282Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aeon0.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-05-11T12:16:50.000Z","updated_at":"2021-10-28T05:44:59.000Z","dependencies_parsed_at":"2022-09-26T17:21:10.985Z","dependency_job_id":null,"html_url":"https://github.com/aeon0/MLPipe-Trainer","commit_stats":null,"previous_names":["j-o-d-o/mlpipe-trainer"],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aeon0%2FMLPipe-Trainer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aeon0%2FMLPipe-Trainer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aeon0%2FMLPipe-Trainer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aeon0%2FMLPipe-Trainer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aeon0","download_url":"https://codeload.github.com/aeon0/MLPipe-Trainer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243978812,"owners_count":20378105,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-01-23T14:55:33.021Z","updated_at":"2025-03-17T05:27:37.706Z","avatar_url":"https://github.com/aeon0.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cimg src=\"docs/assets/mlpipe_icon_full.png\" width=\"500\"\u003e\n\n# MLPipe-Trainer\n\nManage your Data Pipline and Tensorflow \u0026 Keras models with MLPipe. It is NOT another \"wrapper\" around Tensorflow, but rather adds utilities to setup an environment to control data flow and managed trained models (weights \u0026 results) with the help of MongoDB.\u003c/br\u003e\n\n```bash\n\u003e\u003e pip install mlpipe-trainer\n```\n\n## Setup - install MongoDB\nMongoDB database is used to store trained Models including their weights and results. Additionally there is also a data reader for MongoDB implemented (basically just a generator as you know and love from using keras). Currenlty that is the only implemented data reader working \"out of the box\".\u003c/br\u003e\nFollow the instructions on the MongoDB website for installation e.g. for Linux: https://docs.mongodb.com/manual/administration/install-on-linux/\n\n## Code Examples\n\n#### Config\n```python\n# The config is used to specify the localhost connections\n# for saving trained models to the mongoDB as well as fetching training data\nfrom mlpipe.utils import Config\nConfig.add_config('./path_to/config.ini')\n```\nEach Connection config consists of these fields in the .ini file\n```ini\n[example_mongo_db_connection]\ndb_type=MongoDB\nurl=localhost\nport=27017\nuser=read_write\npwd=rw\n```\n\n#### Data Pipline\n```python\nfrom mlpipe.processors.i_processor import IPreProcessor\nfrom mlpipe.data_reader.mongodb import MongoDBGenerator\n\nclass PreProcessData(IPreProcessor):\n    def process(self, raw_data, input_data, ground_truth, piped_params=None):\n        # Process raw_data to output input_data and ground_truth\n        # which will be the input for the model\n        ...\n        return raw_data, input_data, ground_truth, piped_params\n\ntrain_data = [...]  # consists of MongoDB ObjectIds that are used for training\nprocessors = [PreProcessData()]  # Chain of Processors (in our case its just one)\n# Generator that can be used e.g. with keras' fit_generator()\ntrain_gen = MongoDBGenerator(\n    (\"connection_name\", \"cifar10\", \"train\"),  # specify data source from a MongoDB\n    train_data,\n    batch_size=128,\n    processors=processors\n)\n```\nData generators inherit from `tf.keras.utils.Sequence`. Check out this [tensorflow docu](https://www.tensorflow.org/api_docs/python/tf/keras/utils/Sequence) to find out how you can write your custom generators (e.g. for other data sources than MongoDB).\n\n#### Model\nAs long as there is a keras (tensorflow.keras) model in the end, there are no restrictions on this step\n```python\nmodel = Sequential()\nmodel.add(Conv2D(32, (3, 3), padding='same', input_shape=(32, 32, 3)))\n...\nmodel.add(Dense(10, activation='softmax'))\n\nopt = optimizers.RMSprop(lr=0.0001, decay=1e-6)\nmodel.compile(optimizer=opt, loss='categorical_crossentropy', metrics=[\"accuracy\"])\n```\n\n#### Training and Callbacks\n```python\nfrom mlpipe.callbacks import SaveToMongoDB\n\nsave_to_mongodb_cb = SaveToMongoDB((\"localhost_mongo_db\", \"models\"), \"test\", model)\n\nmodel.fit_generator(\n    generator=train_gen,\n    validation_data=val_gen,\n    epochs=10,\n    verbose=1,\n    callbacks=[save_to_mongodb_cb],\n    initial_epoch=0,\n)\n```\n`SaveToMongoDB` is a custom keras callback class as described in the [tensorflow docu](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/Callback). Again, feel free to create custom callbacks for any specific needs.\u003c/br\u003e\nIf, instead of `fit_generator()`, each batch is trained one-by-one e.g. with a native tensorflow model, you can still loop over the generator. Just remember to call the callback methods at the specific steps e.g. `on_batch_end()`.\n\nA full Cifar10 example can be found in the example folder [here](https://github.com/j-o-d-o/MLPipe-Trainer/tree/master/examples/cifar10)\n\n## Road Map\n- Create and generat MkDocs documentation \u0026 host documentation\n- Add tests\n- Set Up CI\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faeon0%2Fmlpipe-trainer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faeon0%2Fmlpipe-trainer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faeon0%2Fmlpipe-trainer/lists"}