{"id":13935656,"url":"https://github.com/IBM/pytorchpipe","last_synced_at":"2025-07-19T20:33:37.607Z","repository":{"id":66065279,"uuid":"179741767","full_name":"IBM/pytorchpipe","owner":"IBM","description":"PyTorchPipe (PTP) is a component-oriented framework for rapid prototyping and training of computational pipelines combining vision and language","archived":true,"fork":false,"pushed_at":"2019-11-05T23:36:22.000Z","size":6152,"stargazers_count":225,"open_issues_count":16,"forks_count":33,"subscribers_count":22,"default_branch":"develop","last_synced_at":"2024-08-08T23:21:26.078Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/IBM.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2019-04-05T19:21:16.000Z","updated_at":"2024-07-18T20:03:09.000Z","dependencies_parsed_at":"2023-02-21T00:16:28.761Z","dependency_job_id":null,"html_url":"https://github.com/IBM/pytorchpipe","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IBM%2Fpytorchpipe","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IBM%2Fpytorchpipe/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IBM%2Fpytorchpipe/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IBM%2Fpytorchpipe/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/IBM","download_url":"https://codeload.github.com/IBM/pytorchpipe/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":226677185,"owners_count":17666014,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-07T23:01:57.750Z","updated_at":"2024-11-27T03:31:01.117Z","avatar_url":"https://github.com/IBM.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# PyTorchPipe\r\n\r\n![Language](https://img.shields.io/badge/language-Python-blue.svg)\r\n[![GitHub license](https://img.shields.io/github/license/IBM/pytorchpipe.svg)](https://github.com/IBM/pytorchpipe/blob/develop/LICENSE)\r\n[![GitHub version](https://badge.fury.io/gh/IBM%2Fpytorchpipe.svg)](https://badge.fury.io/gh/IBM%2Fpytorchpipe)\r\n\r\n[![Build Status](https://travis-ci.com/IBM/pytorchpipe.svg?branch=develop)](https://travis-ci.com/IBM/pytorchpipe)\r\n[![Language grade: Python](https://img.shields.io/lgtm/grade/python/g/IBM/pytorchpipe.svg?logo=lgtm\u0026logoWidth=18)](https://lgtm.com/projects/g/IBM/pytorchpipe/context:python)\r\n[![Total alerts](https://img.shields.io/lgtm/alerts/g/IBM/pytorchpipe.svg?logo=lgtm\u0026logoWidth=18)](https://lgtm.com/projects/g/IBM/pytorchpipe/alerts/)\r\n[![Coverage Status](https://coveralls.io/repos/github/IBM/pytorchpipe/badge.svg?branch=develop)](https://coveralls.io/github/IBM/pytorchpipe?branch=develop)\r\n[![Maintainability](https://api.codeclimate.com/v1/badges/e8d37123b856ee5bb10b/maintainability)](https://codeclimate.com/github/IBM/pytorchpipe/maintainability)\r\n\r\n## Description\r\n\r\nPyTorchPipe (PTP) is a component-oriented framework that facilitates development of computational _multi-modal pipelines_ and comparison of diverse neural network-based models.\r\n\r\nPTP frames training and testing procedures as _pipelines_ consisting of many components communicating through data streams.\r\nEach such a stream can consist of several components, including one task instance (providing batches of data), any number of trainable components (models) and additional components providing required transformations and computations.\r\n\r\n\r\n![Alt text](docs/source/img/data_flow_vqa_5_attention_gpu_loaders.png?raw=true \"Exemplary multi-modal data flow diagram\")\r\n\r\n\r\nAs a result, the training \u0026 testing procedures are no longer pinned to a specific task or model, and built-in mechanisms for compatibility checking (handshaking), configuration and global variables management \u0026 statistics collection facilitate rapid development of complex pipelines and running diverse experiments.\r\n\r\nIn its core, to _accelerate the computations_ on their own, PTP relies on PyTorch and extensively uses its mechanisms for distribution of computations on CPUs/GPUs, including multi-process data loaders and multi-GPU data parallelism.\r\nThe models are _agnostic_ to those operations and one indicates whether to use them in configuration files (data loaders) or by passing adequate argument (--gpu) at run-time.\r\n\r\nPlease refer to the  [tutorial presentation](https://zenodo.org/record/3269928) for more details.\r\n\r\n**Datasets:**\r\nPTP focuses on multi-modal reasoning combining vision and language. Currently it offers the following _Tasks_ from the following task, categorized into three domains:\r\n\r\n![Alt text](docs/source/img/components/ptp_tasks.png?raw=true)\r\n\r\nAside of providing batches of samples, the Task class will automatically download the files associated with a given dataset (as long as the dataset is publicly available).\r\nThe diversity of those tasks (and the associated models) proves the flexibility of the framework.\r\nWe are constantly working on incorporation of new Tasks into PTP.\r\n\r\n**Pipelines:**\r\nWhat people typically define as a _model_ in PTP is framed as a _pipeline_, consisting of many inter-connected components, with one or more _Models_ containing trainable elements.\r\nThose components are loosely coupled and care only about the _input streams_ they retrieve and _output streams_ they produce.\r\nThe framework offers full flexibility and it is up to the programmer to choose the _granularity_ of his/her components/models/pipelines.\r\nSuch a decomposition enables one to easily combine many components and models into pipelines, whereas the framework supports loading of pretrained models, freezing during training, saving them to checkpoints etc.\r\n\r\n**Model/Component Zoo:**\r\nPTP provides several ready to use, out of the box models and other, non-trainable (but parametrizable) components.\r\n\r\n\r\n![Alt text](docs/source/img/components/ptp_models.png?raw=true)\r\n\r\nThe model zoo includes several general usage components, such as:\r\n  * Feed Forward Network (variable number of Fully Connected layers with activation functions and dropout)\r\n  * Recurrent Neural Network (different cell types with activation functions and dropout, a single model can work both as encoder or decoder)\r\n\r\nIt also inludes few models specific for a given domain, but still quite general:\r\n  * Convnet Encoder (CNNs with ReLU and MaxPooling, can work with different sizes of images)\r\n  * General Image Encoder (wrapping several models from Torch Vision)\r\n  * Sentence Embeddings (encoding words using the embedding layer)\r\n\r\nThere are also some classical baselines both for vision like LeNet-5 or language domains, e.g. Seq2Seq (Sequence to Sequence model) or Attention Decoder (RNN-based decoder implementing Bahdanau-style attention).\r\nPTP also offers the several models useful for multi-modal fusion and reasoning.\r\n\r\n![Alt text](docs/source/img/components/ptp_components_others.png?raw=true)\r\n\r\nThe framework also offers components useful when working with language, vision or other types of streams (e.g. tensor transformations).\r\nThere are also several general-purpose components, from components calculating losses and statistics to publishers and viewers.\r\n\r\n**Workers:**\r\nPTP workers are python scripts that are _agnostic_ to the tasks/models/pipelines that they are supposed to work with.\r\nCurrently framework offers three workers:\r\n\r\n  * ptp-offline-trainer (a trainer relying on classical methodology interlacing training and validation at the end of every epoch, creates separate instances of training and validation tasks and trains the models by feeding the created pipeline with batches of data, relying on the notion of an _epoch_)\r\n\r\n  * ptp-online-trainer (a flexible trainer creating separate instances of training and validation tasks and training the models by feeding the created pipeline with batches of data, relying on the notion of an _episode_)\r\n\r\n  * ptp-processor (performing one pass over the all samples returned by a given task instance, useful for collecting scores on test set, answers for submissions to competitions etc.)\r\n\r\n\r\n## Installation\r\n\r\nPTP relies on [PyTorch](https://github.com/pytorch/pytorch), so you need to install it first.\r\nPlease refer to the official installation [guide](https://github.com/pytorch/pytorch#installation) for details.\r\nIt is easily installable via conda_, or you can compile it from source to optimize it for your machine.\r\n\r\nPTP is not (yet) available as a [pip](https://pip.pypa.io/en/stable/quickstart/) package, or on [conda](https://anaconda.org/pytorch/pytorch).\r\nHowever, we provide the `setup.py` script and recommend to use it for installation.\r\nFirst please clone the project repository:\r\n\r\n```console\r\ngit clone git@github.com:IBM/pytorchpipe.git\r\ncd pytorchpipe/\r\n```\r\n\r\nNext, install the dependencies by running:\r\n\r\n```console\r\n  python setup.py develop\r\n```\r\n\r\nThis command will install all dependencies via pip_, while still enabling you to change the code of the existing components/workers and running them by calling the associated ``ptp-*`` commands.\r\nMore in that subject can be found in the following blog post on [dev_mode](https://setuptools.readthedocs.io/en/latest/setuptools.html#development-mode).\r\n\r\n\r\n## Quick start: MNIST image classification with a simple ConvNet model\r\n\r\nPlease consider a simple ConvNet model consisting of two parts: \r\n  * few convolutional layers accepting the MNIST images and returning feature maps being, in general, a 4D tensor (first dimension being the batch size, a rule of thumb in PTP),\r\n  * one (or more) dense layers that accept the (flattened) feature maps and return predictions in the form of logarithm of probability distributions (LogSoftmax as last non-linearity).\r\n\r\n### Training the model\r\n\r\nAssume that we will use ```NLL Loss``` function, and, besides, want to monitor the ```Accuracy``` statistics.\r\nThe resulting pipeline is presented below.\r\nThe additional ```Answer Decoder``` component translates the predictions into class names, whereas ```Stream Viewer``` displays content of the indicated data streams for a single sample randomly picked from the batch.\r\n\r\n\r\n![Alt text](docs/source/img/1_tutorials/data_flow_tutorial_mnist_1_training.png?raw=true \"Trainining of a simple ConvNet model on MNIST dataset\")\r\n\r\n__Note__: The associated ```mnist_classification_convnet_softmax.yml``` configuration file can be found in ```configs/tutorials``` folder.\r\n\r\nWe will train the model with _ptp-offline-trainer_, a general _worker_ script that follows the classical training-validation, epoch-based methodology.\r\nThis means, that despite the presence of three sections (associated with training, validation and test splits of the MNIST dataset) the trainer will consider only the content of ``training`` and ```validation``` sections (plus ```pipeline```, containing the definition of the whole pipeline).\r\nLet's run the training by calling the following from the command line:\r\n\r\n```console\r\nptp-offline-trainer --c configs/tutorials/mnist_classification_convnet_softmax.yml\r\n```\r\n\r\n__Note__: Please call ```offline-trainer --h``` to learn more about the run-time arguments. In order to understand the structure of the main configuration file please look at the default configuration file of the trainer located in ```configs/default/workers``` folder.\r\n\r\nThe trainer will log on the console training and validation statistis, along with additional information logged by the components, e.g. contents of the streams:\r\n\r\n```console\r\n[2019-07-05 13:31:44] - INFO - OfflineTrainer \u003e\u003e\u003e episode 006000; epoch 06; loss 0.1968410313; accuracy 0.9219\r\n[2019-07-05 13:31:45] - INFO - OfflineTrainer \u003e\u003e\u003e End of epoch: 6\r\n================================================================================\r\n[2019-07-05 13:31:45] - INFO - OfflineTrainer \u003e\u003e\u003e episode 006019; episodes_aggregated 000860; epoch 06; loss 0.1799264401; loss_min 0.0302138925; loss_max 0.5467863679; loss_std 0.0761705562; accuracy 0.94593; accuracy_std 0.02871 [Full Training]\r\n[2019-07-05 13:31:45] - INFO - OfflineTrainer \u003e\u003e\u003e Validating over the entire validation set (5000 samples in 79 episodes)\r\n[2019-07-05 13:31:45] - INFO - stream_viewer \u003e\u003e\u003e Showing selected streams for sample 20 (index: 55358):\r\n 'labels': One\r\n 'targets': 1\r\n 'predictions': tensor([-1.1452e+01, -1.6804e-03, -1.1357e+01, -1.1923e+01, -6.6160e+00,\r\n        -1.4658e+01, -9.6191e+00, -8.6472e+00, -9.6082e+00, -1.3505e+01])\r\n 'predicted_answers': One\r\n```\r\n\r\nPlease note that whenever the validation loss goes down, the trainer automatically will save the pipeline to the checkpoint file:\r\n\r\n```console\r\n[2019-07-05 13:31:47] - INFO - OfflineTrainer \u003e\u003e\u003e episode 006019; episodes_aggregated 000079; epoch 06; loss 0.1563445479; loss_min 0.0299939774; loss_max 0.5055227876; loss_std 0.0854654983; accuracy 0.95740; accuracy_std 0.02495 [Full Validation]\r\n[2019-07-05 13:31:47] - INFO - mnist_classification_convnet_softmax \u003e\u003e\u003e Exporting pipeline 'mnist_classification_convnet_softmax' parameters to checkpoint:\r\n /users/tomaszkornuta/experiments/mnist/mnist_classification_convnet_softmax/20190705_132624/checkpoints/mnist_classification_convnet_softmax_best.pt\r\n  + Model 'image_encoder' [ConvNetEncoder] params saved\r\n  + Model 'classifier' [FeedForwardNetwork] params saved\r\n```\r\n\r\nAfter the training finsh the trainer will inform about the termination reason and indicate where the experiment files (model checkpoint, log files, statistics etc.) can be found:\r\n\r\n```console\r\n[2019-07-05 13:32:33] - INFO - mnist_classification_convnet_softmax \u003e\u003e\u003e Updated training status in checkpoint:\r\n /users/tomaszkornuta/experiments/mnist/mnist_classification_convnet_softmax/20190705_132624/checkpoints/mnist_classification_convnet_softmax_best.pt\r\n[2019-07-05 13:32:33] - INFO - OfflineTrainer \u003e\u003e\u003e\r\n================================================================================\r\n[2019-07-05 13:32:33] - INFO - OfflineTrainer \u003e\u003e\u003e Training finished because Converged (Full Validation Loss went below Loss Stop threshold of 0.15)\r\n[2019-07-05 13:32:33] - INFO - OfflineTrainer \u003e\u003e\u003e Experiment finished!\r\n[2019-07-05 13:32:33] - INFO - OfflineTrainer \u003e\u003e\u003e Experiment logged to: /users/tomaszkornuta/experiments/mnist/mnist_classification_convnet_softmax/20190705_132624/\r\n```\r\n\r\n\r\n### Testing the model\r\n\r\nIn order to test the model generalization we will use _ptp-processor_, yet another general _worker_ script that performs a single pass over the indicated set.\r\n\r\n\r\n![Alt text](docs/source/img/1_tutorials/data_flow_tutorial_mnist_2_test.png?raw=true \"Test of the pretrained model on test split of the MNIST dataset \")\r\n\r\n\r\n```console\r\nptp-processor --load /users/tomaszkornuta/experiments/mnist/mnist_classification_convnet_softmax/20190705_132624/checkpoints/mnist_classification_convnet_softmax_best.pt\r\n```\r\n\r\n__Note__: _ptp-processor_ uses the content of _test_ section as default, but it can be changed at run-time. Please call ```ptp-processor --h``` to learn about the available run-time arguments.\r\n\r\n\r\n```console\r\n[2019-07-05 13:34:41] - INFO - Processor \u003e\u003e\u003e episode 000313; episodes_aggregated 000157; loss 0.1464060694; loss_min 0.0352710858; loss_max 0.3801054060; loss_std 0.0669835582; accuracy 0.95770; accuracy_std 0.02471 [Full Set]\r\n[2019-07-05 13:34:41] - INFO - Processor \u003e\u003e\u003e Experiment logged to: /users/tomaszkornuta/experiments/mnist/mnist_classification_convnet_softmax/20190705_132624/test_20190705_133436/\r\n```\r\n\r\n__Note__: Please analyze the ```mnist_classification_convnet_softmax.yml``` configuration file (located in ```configs/tutorials``` directory). Keep in mind that:\r\n  * all components come with default configuration files, located in ```configs/default/components``` folders,\r\n  * all workers come with default configuration files, located in ```configs/default/workers``` folders.\r\n## Documentation\r\n\r\nCurrently PTP does not have an on-line documentation.\r\nHowever, there are high-quality comments in all source/configuration files, that will be used for automatic generation of documentation (Sphinx + ReadTheDocs).\r\nBesides, we have shared a [tutorial presentation](https://zenodo.org/record/3269928) explaining motivations and core concepts as well as providing hints how to use the tool and develop your own solutions.\r\n\r\n\r\n## Contributions\r\n\r\nPTP is open for external contributions.\r\nWe follow the [Git Branching Model](https://nvie.com/posts/a-successful-git-branching-model/), in short:\r\n  * ```develop``` branch is the main branch, ```master``` branch is for used for releases only\r\n  * all changes are integrated by merging pull requests from feat/fix/other branches\r\n  * PTP is integrated with several DevOps monitoring the quality of code/pull requests\r\n  * we strongly encourage unit testing and Test-Driven Development\r\n  * we use projects and kanban to monitor issues/progress/etc.\r\n\r\n\r\n## Maintainers\r\n\r\nA project of the Machine Intelligence team, IBM Research AI, Almaden Research Center.\r\n\r\n* Tomasz Kornuta (tkornut@us.ibm.com)\r\n\r\n[![HitCount](http://hits.dwyl.io/tkornut/tkornut/pytorchpipe.svg)](http://hits.dwyl.io/tkornut/tkornut/pytorchpipe)\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FIBM%2Fpytorchpipe","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FIBM%2Fpytorchpipe","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FIBM%2Fpytorchpipe/lists"}