{"id":13564832,"url":"https://github.com/toodef/neural-pipeline","last_synced_at":"2025-04-03T21:31:58.272Z","repository":{"id":57445769,"uuid":"161836727","full_name":"toodef/neural-pipeline","owner":"toodef","description":"Neural networks training pipeline based on PyTorch","archived":true,"fork":false,"pushed_at":"2020-06-01T17:50:58.000Z","size":14297,"stargazers_count":312,"open_issues_count":29,"forks_count":24,"subscribers_count":7,"default_branch":"master","last_synced_at":"2024-10-14T06:38:21.179Z","etag":null,"topics":["deep-learning","image-classification","image-segmentation","neural-networks","object-detection","pipeline","pytorch","supervised-learning","training-pipeline"],"latest_commit_sha":null,"homepage":"https://neural-pipeline.readthedocs.io","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/toodef.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"docs/CONTRIBUTING.rst","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-12-14T20:24:38.000Z","updated_at":"2024-06-26T02:33:27.000Z","dependencies_parsed_at":"2022-09-13T03:03:06.539Z","dependency_job_id":null,"html_url":"https://github.com/toodef/neural-pipeline","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/toodef%2Fneural-pipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/toodef%2Fneural-pipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/toodef%2Fneural-pipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/toodef%2Fneural-pipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/toodef","download_url":"https://codeload.github.com/toodef/neural-pipeline/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223049508,"owners_count":17079534,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","image-classification","image-segmentation","neural-networks","object-detection","pipeline","pytorch","supervised-learning","training-pipeline"],"created_at":"2024-08-01T13:01:36.697Z","updated_at":"2024-11-04T18:30:48.423Z","avatar_url":"https://github.com/toodef.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# This project are closed. New versions and much more are there: [PiePline](https://github.com/PiePline)\n\n# Neural Pipeline\n\nNeural networks training pipeline based on PyTorch. Designed to standardize training process and accelerate experiments.\n\n[![Build Status](https://travis-ci.org/toodef/neural-pipeline.svg?branch=master)](https://travis-ci.org/toodef/neural-pipeline)\n[![Coverage Status](https://coveralls.io/repos/github/toodef/neural-pipeline/badge.svg?branch=master)](https://coveralls.io/github/toodef/neural-pipeline?branch=master)\n[![Maintainability](https://api.codeclimate.com/v1/badges/1feaafcc614adf27c30f/maintainability)](https://codeclimate.com/github/toodef/neural-pipeline/maintainability)\n[![Gitter chat](https://badges.gitter.im/neural-pipeline/gitter.png)](https://gitter.im/neural-pipeline/community)\n\n* Core is about 2K lines, covered by tests, that you don't need to write again\n* Flexible and customizable training process\n* Checkpoints management and train process resuming (source and target device independent)\n* Metrics processing and visualization by builtin ([tensorboard](https://www.tensorflow.org/guide/summaries_and_tensorboard), [Matplotlib](https://matplotlib.org)) or custom monitors\n* Training best practices (e.g. learning rate decaying and hard negative mining)\n* Metrics logging and comparison (DVC compatible)\n\n# Getting started:\n### Documentation\n[![Documentation Status](https://readthedocs.org/projects/neural-pipeline/badge/?version=master)](https://neural-pipeline.readthedocs.io/en/master/?badge=master)\n* [See the full documentation there](https://neural-pipeline.readthedocs.io/en/master/)\n* [Read getting started guide](https://neural-pipeline.readthedocs.io/en/master/getting_started/index.html)\n\n### See the examples\n* MNIST classification - [notebook](https://github.com/toodef/neural-pipeline/blob/master/examples/notebooks/img_classification.ipynb), [file](https://github.com/toodef/neural-pipeline/blob/master/examples/files/img_classification.py), [Kaggle kernel](https://www.kaggle.com/toodef/cnn-training-with-less-code)\n* Segmentation - [notebook](https://github.com/toodef/neural-pipeline/blob/master/examples/notebooks/img_segmentation.ipynb), [file](https://github.com/toodef/neural-pipeline/blob/master/examples/files/img_segmentation.py)\n* Resume training process - [file](https://github.com/toodef/neural-pipeline/blob/master/examples/files/resume_train.py)\n\n### Neural Pipeline short overview:\n```python\nimport torch\n\nfrom neural_pipeline.builtin.monitors.tensorboard import TensorboardMonitor\nfrom neural_pipeline.monitoring import LogMonitor\nfrom neural_pipeline import DataProducer, TrainConfig, TrainStage,\\\n    ValidationStage, Trainer, FileStructManager\n\nfrom somethig import MyNet, MyDataset\n\nfsm = FileStructManager(base_dir='data', is_continue=False)\nmodel = MyNet().cuda()\n\ntrain_dataset = DataProducer([MyDataset()], batch_size=4, num_workers=2)\nvalidation_dataset = DataProducer([MyDataset()], batch_size=4, num_workers=2)\n\ntrain_config = TrainConfig(model, [TrainStage(train_dataset),\n                                   ValidationStage(validation_dataset)], torch.nn.NLLLoss(),\n                           torch.optim.SGD(model.parameters(), lr=1e-4, momentum=0.5))\n\ntrainer = Trainer(train_config, fsm, torch.device('cuda:0')).set_epoch_num(50)\ntrainer.monitor_hub.add_monitor(TensorboardMonitor(fsm, is_continue=False))\\\n                   .add_monitor(LogMonitor(fsm))\ntrainer.train()\n```\nThis example of training MyNet on MyDataset with vizualisation in Tensorflow and with metrics logging for further experiments comparison.\n\n# Installation:\n[![PyPI version](https://badge.fury.io/py/neural-pipeline.svg)](https://badge.fury.io/py/neural-pipeline)\n[![PyPI Downloads/Month](https://pepy.tech/badge/neural-pipeline/month)](https://pepy.tech/project/neural-pipeline)\n[![PyPI Downloads](https://pepy.tech/badge/neural-pipeline)](https://pepy.tech/project/neural-pipeline)\n\n`pip install neural-pipeline`\n\n##### For `builtin` module using install:\n`pip install tensorboardX matplotlib`\n\n##### Install latest version before it's published on PyPi\n`pip install -U git+https://github.com/toodef/neural-pipeline`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftoodef%2Fneural-pipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftoodef%2Fneural-pipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftoodef%2Fneural-pipeline/lists"}