{"id":16383734,"url":"https://github.com/kartik4949/tensorpipe","last_synced_at":"2025-10-26T21:36:49.175Z","repository":{"id":41974456,"uuid":"302370551","full_name":"kartik4949/TensorPipe","owner":"kartik4949","description":"High Performance Tensorflow Data Pipeline with State of Art Augmentations and low level optimizations.","archived":false,"fork":false,"pushed_at":"2022-04-22T20:09:31.000Z","size":187,"stargazers_count":86,"open_issues_count":2,"forks_count":20,"subscribers_count":9,"default_branch":"main","last_synced_at":"2024-10-12T04:09:30.120Z","etag":null,"topics":["computer-vision","datapipeline","tensorflow"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kartik4949.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-10-08T14:39:00.000Z","updated_at":"2024-01-04T16:51:05.000Z","dependencies_parsed_at":"2022-08-12T01:10:21.402Z","dependency_job_id":null,"html_url":"https://github.com/kartik4949/TensorPipe","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kartik4949%2FTensorPipe","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kartik4949%2FTensorPipe/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kartik4949%2FTensorPipe/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kartik4949%2FTensorPipe/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kartik4949","download_url":"https://codeload.github.com/kartik4949/TensorPipe/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221665791,"owners_count":16860315,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","datapipeline","tensorflow"],"created_at":"2024-10-11T04:09:32.960Z","updated_at":"2025-10-26T21:36:44.137Z","avatar_url":"https://github.com/kartik4949.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# TensorPipe\n[![Library][tensorflow-shield]][tensorflow-url]\n[![MIT License][license-shield]][license-url]\n[![LinkedIn][linkedin-shield]][linkedin-url]\n[![codecov](https://codecov.io/gh/kartik4949/TensorPipe/branch/main/graph/badge.svg?token=RWGTCX17C1)](https://codecov.io/gh/kartik4949/TensorPipe)\n\n![alt text](logo.png)\n\n\nHigh Performance Tensorflow Data Pipeline with State of Art Augmentations build as a wrapper aroung tensorflow datasets api, with low level optimizations.\n\n## Requirements\n\n* Python 3.8\n* Tensorflow 2.2\n* Tensorflow addons\n* Sklearn\n* typeguard\n\nInstall using:\n\n```\npip install tensorflow-addons==0.11.2\npip install tensorflow==2.2.0\npip install sklearn\npip install typeguard\n```\n\n## Features\n\n- [x] High Performance tf.data pipline\n- [x] Core tensorflow support for high performance\n- [x] Classification data support\n- [x] Bbox data support\n- [ ] Keypoints data support\n- [ ] Segmentation data support\n- [x] GridMask in core tf2.x\n- [x] Mosiac Augmentation in core tf2.x\n- [x] CutOut in core tf2.x\n- [x] Flexible and easy configuration\n- [x] Gin-config support\n- [x] Custom numpy function injection.\n## Advance Users Section: \n## Example Usage 1\n### Create a Data Pipeline for Training. (Categorical Data).\n```python\nfrom pipe import Funnel                                                         \nfrom bunch import Bunch                                                         \n\"\"\"                                                                             \nCreate a Funnel for the Pipeline!                                               \n\"\"\"                                                                             \n\n\n# Config for Funnel\nconfig = {                                                                      \n    \"batch_size\": 2,                                                            \n    \"image_size\": [512,512],                                                    \n    \"transformations\": {                                                        \n        \"flip_left_right\": None,                                                \n        \"gridmask\": None,                                                       \n        \"random_rotate\":None,                                                   \n    },                                                                          \n    \"categorical_encoding\":\"labelencoder\"                                       \n}                                                                               \nconfig = Bunch(config)                                                          \npipeline = Funnel(data_path=\"testdata\", config=config, datatype=\"categorical\")  \npipeline = pipeline.from_dataset(type=\"train\")                                       \n                                                                                \n# Pipline ready to use, iter over it to use.\n# Custom loop example.\nfor data in pipeline:\n    image_batch , label_batch = data[0], data[1]\n    # you can use _loss = loss(label_batch,model.predict(image_batch))\n    # calculate gradients on loss and optimize the model.\n    print(image_batch,label_batch)                                      \n\n```\n\n## Example Usage 2\n### Create a Data Pipeline for Bounding Boxes with tf records.\n\n```python\nimport numpy as np\nfrom tensorpipe.pipe import Funnel\n\n\"\"\"\nCreate a Funnel for the Pipeline!\n\"\"\"\n\n# Custom numpy code for injection.\ndef numpy_function(image, label):\n    \"\"\"normalize image\"\"\"\n    image = image / 255.0\n    return image, label\n\n\nconfig = {\n    \"batch_size\": 2,\n    \"image_size\": [512, 512],\n    \"transformations\": {\n        \"flip_left_right\": None,\n        \"gridmask\": None,\n        \"random_rotate\": None,\n    },\n    \"max_instances_per_image\": 100,\n    \"categorical_encoding\": \"labelencoder\",\n    \"numpy_function\": numpy_function,\n}\nfunnel = Funnel(data_path=\"tfrecorddata\", config=config, datatype=\"bbox\")\ndataset = funnel.from_tfrecords(type=\"train\")\n\nfor data in dataset:\n    print(data[1].shape)\n\n```\n# Object Detection Usage\n### Now build your custom bounding box funnel with subclassing BboxFunnel.\n\n```python\nfrom tensorpipe.funnels import funnels\nclass CustomObjectDetectionLoader(funnels.BboxFunnel):\n      def __init__(self, *args):\n          super().__init__(*args)\n\n      def encoder(self,args):\n          # encoder is overriden to give custom anchors to the model as per the need.\n\n          image_id, image, bbox, classes = args\n          # make custom anchors and encode the image and bboxes as per the model need.\n          return image, custom_anchors, classes\n      def decoder(self, args):\n          # override decoder if using custom tf records and decode your custom tfrecord in this method\n          return decoded_data\n          \nfunnel = CustomObjectDetectionLoader(data_path=\"tfrecorddata\", config=config, datatype=\"bbox\")\ndataset = funnel.from_tfrecords(type=\"train\")\n```\n\n# Steps to build tfrecords and custom input loader.\n- use funnels.create_records script to build tfrecord or build your own tfrecords using your custom script.\n- if used the create_records script to build records, we dont need to overide decoder, but if using custom\n  script overriding the decoder function is mandatory.\n- If using anchors in the models, please override encoder with custom anchor script.\n\n\n## Beginners Section.\n## Keras Compatiblity.\n### Very simple example to use pipeline with keras model.fit as iterable.\n```python\nimport tensorflow as tf\nfrom pipe import Funnel\n\n\"\"\"\nCreate a Funnel for the Pipeline!\n\"\"\"\n\nconfig = {\n    \"batch_size\": 2,\n    \"image_size\": [100, 100],\n    \"transformations\": {\n        \"flip_left_right\": None,\n        \"gridmask\": None,\n        \"random_rotate\": None,\n    },\n    \"categorical_encoding\": \"labelencoder\",\n}\npipeline = Funnel(data_path=\"testdata\", config=config, datatype=\"categorical\")\n# from dataset i.e normal dataset.\npipeline = pipeline.from_dataset(type=\"train\")\n\n# e.g from tfrecords i.e tfrecord dataset.\n# pipeline = pipeline.from_tfrecords(type=\"train\") # testdata/train/*.tfrecord\n\n# Create Keras model\nmodel = tf.keras.applications.VGG16(\n    include_top=True, weights=None,input_shape=(100,100,3),\n    pooling=None, classes=2, classifier_activation='sigmoid'\n)\n\n# compile\nmodel.compile(loss='mse', optimizer='adam')\n\n# pass pipeline as iterable\nmodel.fit(pipeline , batch_size=2,steps_per_epoch=5,verbose=1)\n```\n\n## Config.\n* **image_size** - Output Image Size for the pipeline.\n* **batch_size** - Batch size for the pipeline.\n* **transformations** - Dictionary of transformations to apply with respective keyword arguments.\n* **categorical_encoding** - Encoding for categorical data - ('labelencoder' , 'onehotencoder').\n\n## Augmentations:\n\n### GridMask\nCreates a gridmask on input image with rotation defined on range.\n* **params**:\n    * **ratio** - grid to space ratio\n    * **fill** - fill value\n    * **rotate** - rotation range in degrees\n\n### MixUp\nMixes two randomly sampled images and their respective labels with given alpha.\n* **params**:\n    * **alpha** - value for blend function.\n\n### RandomErase\nRandomly erases rectangular chunk with is sampled randomly on given image.\n* **params**:\n    * **prob** - probablity to randomerase on image.\n\n### CutMix\nOverlaps a resized randomly sample image on given image with complete overlay on subset portion of image.\n* **params**:\n    * **prob** - probablity to CutMix on image.\n\n### Mosaic\nCreates a mosaic of input 4 images into one single image.\n* **params**:\n    * **prob** - Probablity to mosaic.\n\n## CutMix, CutOut, MixUp\n\n![alt text](https://www.researchgate.net/publication/340296142/figure/fig1/AS:874996595429376@1585626853032/Comparison-of-our-proposed-Attentive-CutMix-with-Mixup-5-Cutout-1-and-CutMix-3.png)\n#### source (https://www.researchgate.net/publication/340296142/figure/fig1/AS:874996595429376@1585626853032/Comparison-of-our-proposed-Attentive-CutMix-with-Mixup-5-Cutout-1-and-CutMix-3.png)\n\n## Mosaic\n![alt-text](https://hoya012.github.io/assets/img/yolov4/8.PNG)\n#### source (https://hoya012.github.io/assets/img/yolov4/8.PNG)\n\n## Grid Mask\n![alt-text](https://storage.googleapis.com/groundai-web-prod/media/users/user_302546/project_404544/images/x1.png)\n#### source (https://storage.googleapis.com/groundai-web-prod/media/users/user_302546/project_404544/images/x1.png)\n\n\n## Release History\n* v1.0\n    * Bbox, Keypoints, Custom Py Functions Support.(WIP)\n* v1.0-beta\n    * Classification Support with gridmask and mosaic augmentations.\n\n## Meta\n\nKartik Sharma – [@linkedIn](https://www.linkedin.com/in/kartik-sharma-aaa021169/) – kartik4949@gmail.com\n\nDistributed under the Apache 2.0 license. See ``LICENSE`` for more information.\n\n\n\u003c!-- MARKDOWN LINKS \u0026 IMAGES --\u003e\n\u003c!-- https://www.markdownguide.org/basic-syntax/#reference-style-links --\u003e\n[tensorflow-shield]: https://img.shields.io/badge/Tensorflow-2.x-orange\n[tensorflow-url]: https://tensorflow.org\n[license-shield]: https://img.shields.io/badge/OpenSource-%E2%9D%A4%EF%B8%8F-blue\n[license-url]: LICENSE.txt\n[linkedin-shield]: https://img.shields.io/badge/-LinkedIn-black.svg?style=flat-square\u0026logo=linkedin\u0026colorB=555\n[linkedin-url]: https://www.linkedin.com/in/kartik-sharma-aaa021169/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkartik4949%2Ftensorpipe","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkartik4949%2Ftensorpipe","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkartik4949%2Ftensorpipe/lists"}