{"id":15065050,"url":"https://github.com/zfturbo/keras-inference-time-optimizer","last_synced_at":"2025-04-08T04:19:50.132Z","repository":{"id":62574296,"uuid":"134827232","full_name":"ZFTurbo/Keras-inference-time-optimizer","owner":"ZFTurbo","description":"Optimize layers structure of Keras model to reduce computation time","archived":false,"fork":false,"pushed_at":"2020-07-18T16:14:23.000Z","size":79,"stargazers_count":157,"open_issues_count":15,"forks_count":18,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-04-08T04:19:46.060Z","etag":null,"topics":["inference-optimization","keras"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ZFTurbo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null},"funding":{"custom":"https://paypal.me/zfturbo"}},"created_at":"2018-05-25T08:26:03.000Z","updated_at":"2025-03-29T10:22:45.000Z","dependencies_parsed_at":"2022-11-03T18:49:11.408Z","dependency_job_id":null,"html_url":"https://github.com/ZFTurbo/Keras-inference-time-optimizer","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ZFTurbo%2FKeras-inference-time-optimizer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ZFTurbo%2FKeras-inference-time-optimizer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ZFTurbo%2FKeras-inference-time-optimizer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ZFTurbo%2FKeras-inference-time-optimizer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ZFTurbo","download_url":"https://codeload.github.com/ZFTurbo/Keras-inference-time-optimizer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247774014,"owners_count":20993690,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["inference-optimization","keras"],"created_at":"2024-09-25T00:30:07.981Z","updated_at":"2025-04-08T04:19:50.098Z","avatar_url":"https://github.com/ZFTurbo.png","language":"Python","funding_links":["https://paypal.me/zfturbo"],"categories":[],"sub_categories":[],"readme":"# Keras inference time optimizer (KITO)\n\nThis code takes on input trained Keras model and optimize layer structure and weights in such a way that model became \nmuch faster (~10-30%), but works identically to initial model. It can be extremely useful in case you need to process large \namount of images with trained model. Reduce operation was tested on all Keras models zoo. See \ncomparison table below.\n\n## Installation\n\n```\npip install kito\n```\n\n## How it works?\n \nIn current version it only apply single type of optimization: It reduces Conv2D + BatchNormalization set of layers to \nsingle Conv2D layer. Since Conv2D + BatchNormalization is very common set of layers, optimization works well \nalmost on all modern CNNs for image processing.\n\nAlso supported:\n* DepthwiseConv2D + BatchNormalization =\u003e DepthwiseConv2D \n* SeparableConv2D + BatchNormalization =\u003e SeparableConv2D\n* Conv2DTranspose + BatchNormalization =\u003e Conv2DTranspose\n* Conv3D + BatchNormalization =\u003e Conv3D\n* Conv1D + BatchNormalization =\u003e Conv1D\n\n## How to use\n\nTypical code:\n\n```\nmodel.fit(...)\n...\nmodel.predict(...)\n```\n\nmust be replaced with following block:\n\n```\nfrom kito import reduce_keras_model\nmodel.fit(...)\n...\nmodel_reduced = reduce_keras_model(model)\nmodel_reduced.predict(...)\n```\n\nSo basically you need to insert 2 lines in your code to speed up operations. But note that it requires \nsome time to convert model. You can see usage example in [test_bench.py](https://github.com/ZFTurbo/Keras-inference-time-optimizer/blob/master/kito/test_bench.py)\n\n## Comparison table\n\n| Neural net | Input shape | Number of layers (Init) | Number of layers (Reduced) | Number of params (Init) | Number of params (Reduced) | Time to process 10000 images (Init) |  Time to process 10000 images (Reduced) | Conversion Time (sec) | Maximum diff on final layer | Average difference on final layer |  \n| --- | --- | --- |  --- |  --- |  --- |  --- |  --- |  --- |  --- |  --- |  \n| MobileNet (1.0) | (224, 224, 3) | 102 | 75 | 4,253,864| 4,221,032| **32.38** | **22.13** | 12.45 | 2.80e-06 | 4.41e-09 |\n| MobileNetV2 (1.4) | (224, 224, 3) | 152 | 100 | 6,156,712| 6,084,808| **52.53** | **37.71** | 87.00 | 3.99e-06 | 6.88e-09 |\n| ResNet50 | (224, 224, 3) | 177 | 124 | 25,636,712 | 25,530,472 | **58.87** | **35.81** | 45.28 | 5.06e-07 | 1.24e-09 |\n| Inception_v3 | (299, 299, 3) | 313 | 219 | 23,851,784 | 23,817,352 | **79.15** | **59.55** | 126.02 | 7.74e-07 | 1.26e-09 |\n| Inception_Resnet_v2 | (299, 299, 3) | 782 | 578 | 55,873,736 | 55,813,192 | **131.16** | **102.38** | 766.14 | 8.04e-07 | 9.26e-10 |\n| Xception | (299, 299, 3) | 134 | 94 | 22,910,480 | 22,828,688 | **115.56** | **76.17** | 28.15 | 3.65e-07 | 9.69e-10 |\n| DenseNet121 | (224, 224, 3) | 428 | 369 | 8,062,504 | 8,040,040 | **68.25** | **57.57** | 392.24 | 4.61e-07 | 8.69e-09 |\n| DenseNet169 | (224, 224, 3) | 596 | 513 | 14,307,880 | 14,276,200 | **80.56** | **68.74** | 772.54 | 2.14e-06 | 1.79e-09 |\n| DenseNet201 | (224, 224, 3) | 708 | 609 | 20,242,984 | 20,205,160 | **98.99** | **87.04** | 1120.88 | 7.00e-07 | 1.27e-09 |\n| NasNetMobile | (224, 224, 3) | 751 | 563 | 5,326,716 | 5,272,599 | **46.05** | **31.76** | 728.96 | 1.10e-06 | 1.60e-09 |\n| NasNetLarge | (331, 331, 3) | 1021 | 761 | 88,949,818 | 88,658,596 | **445.58** | **328.16** | 1402.61 | 1.43e-07 | 5.88e-10 |\n| [ZF_UNET_224](https://github.com/ZFTurbo/ZF_UNET_224_Pretrained_Model) | (224, 224, 3) | 85 | 63 | 31,466,753 | 31,442,689 | **96.76** | **69.17** | 9.93 | 4.72e-05 | 7.54e-09 |\n| [DeepLabV3+](https://github.com/bonlime/keras-deeplab-v3-plus) (mobile) | (512, 512, 3) | 162 | 108 | 2,146,645 | 2,097,013 | **583.63** | **432.71** | 48.00 | 4.72e-05 | 1.00e-05 |\n| [DeepLabV3+](https://github.com/bonlime/keras-deeplab-v3-plus) (xception) | (512, 512, 3) | 409 | 263 | 41,258,213 | 40,954,013 | **1000.36** | **699.24** | 333.1 | 8.63e-05 | 5.22e-06 |\n| [ResNet152](https://github.com/broadinstitute/keras-resnet) | (224, 224, 3) | 566 | 411 | 60,344,232 | 60,117,096 | **107.92** | **68.53** | 357.65 | 8.94e-07 | 1.27e-09 |\n\n**Config**: Single NVIDIA GTX 1080 8 GB. Timing obtained on Tensorflow 1.4 (+ CUDA 8.0) backend\n\n## Notes\n\n* It feels like conversion works very slow for no reason, but it should be much faster since all \nmanipulations with layers and weights are very fast. Probably I use some very slow Keras operations in process. \nFeel free to give advice on how to change code to make it faster.\n* You can check that both models work the same with function: compare_two_models_results(model, model_reduced, 10000)\n* Non-zero difference on final layer is accumulated because of large amount of floating point operations, which is not precise\n* Some non-standard layer or parameters (which is not used in any keras.applications CNN) can produce wrong results. \nMost likely code will just fail in these conditions and you will see layer which cause it in python error message.\n \n## Requirements\n\n* Code was tested on Keras 2.1.6 (TensorFlow 1.4 backend) and on Keras 2.2.0 (TensorFlow 1.8.0 backend)\n\n## Formulas\n\n![Base formulas](https://raw.githubusercontent.com/ZFTurbo/Keras-inference-time-optimizer/master/img/conv_bn_fusion.png)\n\n## Other implementations\n\n[PyTorch BN Fusion](https://github.com/MIPT-Oulu/pytorch_bn_fusion) - with support for VGG, ResNet, SeNet.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzfturbo%2Fkeras-inference-time-optimizer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzfturbo%2Fkeras-inference-time-optimizer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzfturbo%2Fkeras-inference-time-optimizer/lists"}