{"id":13575511,"url":"https://github.com/stratospark/keras-multiprocess-image-data-generator","last_synced_at":"2025-04-04T22:31:24.072Z","repository":{"id":137595326,"uuid":"80574991","full_name":"stratospark/keras-multiprocess-image-data-generator","owner":"stratospark","description":"Accelerating Deep Learning with Multiprocess Image Augmentation in Keras","archived":false,"fork":false,"pushed_at":"2018-06-19T22:14:54.000Z","size":13615,"stargazers_count":317,"open_issues_count":4,"forks_count":67,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-03-29T04:16:30.810Z","etag":null,"topics":["deep-learning","keras","multiprocessing","tensorflow"],"latest_commit_sha":null,"homepage":"http://blog.stratospark.com/multiprocess-image-augmentation-keras.html","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/stratospark.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2017-02-01T00:02:50.000Z","updated_at":"2024-01-04T16:11:01.000Z","dependencies_parsed_at":null,"dependency_job_id":"8701ec7a-9ee9-4c86-848f-164563250701","html_url":"https://github.com/stratospark/keras-multiprocess-image-data-generator","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stratospark%2Fkeras-multiprocess-image-data-generator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stratospark%2Fkeras-multiprocess-image-data-generator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stratospark%2Fkeras-multiprocess-image-data-generator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stratospark%2Fkeras-multiprocess-image-data-generator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/stratospark","download_url":"https://codeload.github.com/stratospark/keras-multiprocess-image-data-generator/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247260600,"owners_count":20910040,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","keras","multiprocessing","tensorflow"],"created_at":"2024-08-01T15:01:01.706Z","updated_at":"2025-04-04T22:31:19.053Z","avatar_url":"https://github.com/stratospark.png","language":"Jupyter Notebook","funding_links":[],"categories":["Jupyter Notebook","Image Generation \u0026 Editing"],"sub_categories":[],"readme":"\n# Accelerating Deep Learning with Multiprocess Image Augmentation in Keras\n\n![Benchmark Results](./results.png)\n\n**Code available @ https://github.com/stratospark/keras-multiprocess-image-data-generator**\n\n* [Introduction](#Introduction)\n* [Benchmark: CIFAR10 - In Memory Performance, Image Generation Only](#Benchmark:-CIFAR10---In-Memory-Performance,-Image-Generation-Only)\n* [Benchmark: CIFAR10 - In Memory Performance, Image Generation with GPU Training](#Benchmark:-CIFAR10---In-Memory-Performance,-Image-Generation-with-GPU-Training)\n* [Benchmark: Dogs vs. Cats - On Disk Performance, Image Generation witih GPU Training](#Benchmark:-Dogs-vs.-Cats---On-Disk-Performance,-Image-Generation-witih-GPU-Training)\n\n## Introduction\n\n**TLDR: By adding multiprocessing support to Keras ImageDataGenerator, benchmarking on a 6-core i7-6850K and 12GB TITAN X Pascal: 3.5x speedup of training with image augmentation on in memory datasets, 3.9x speedup of training with image augmentation on datasets streamed from disk.**\n\nWhen exploring Deep Learning models, it isn't only beneficial to have good performance for the final training run. Accelerating training speed means more network models can be tried and more hyperparameter settings can be explored in the same amount of time. **The more that we can experiment, the better our results can become.**\n\nIn my experience with [training a moderately sized network](http://blog.stratospark.com/deep-learning-applied-food-classification-deep-learning-keras.html) on my home desktop, I found one bottleneck to be creating additional images to augment my dataset. Keras provides an [ImageDataGenerator](https://keras.io/preprocessing/image/) class that can take images, in memory or on disk, and create many different variations based on a set of parameters: rotations, flips, zooms, altering colors, etc. For reference, here is a [great tutorial](https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html) on improving network accuracy with image augmentation.\n\n![cat images augmented](https://blog.keras.io/img/imgclf/cat_data_augmentation.png)\n\nWhile training my initial models, I was waiting upwards of an entire day to see enough results to decide what to change. I saw that I was taking nowhere near full advantage of my CPU or GPU. As a result, I decided to add some Python multiprocessing support to a fork of ImageDataGenerator. I was able to drastically cut my training time and was finally able to steer my experiments in the right direction!\n\nFor reference, I am using:\n* Intel Core i7-6850K\n* NVIDIA TITAN X Pascal 12GB\n* 96GB RAM\n* 64-bit Ubuntu 16.04\n* Python 2.7.13 :: Continuum Analytics, Inc.\n* Keras 1.2.1\n* Tensorflow 0.12.1\n\nYou can use the multiprocessing-enabled ImageDataGenerator that is included with this repo as a drop-in replacement for the version that currently ships with Keras. If it makes sense, the code may get incorporated into the main branch at some point.\n\n\n```python\nimport numpy as np\nimport pandas as pd\nimport keras as K\nimport matplotlib.pyplot as plt\nimport multiprocessing\nimport time\nimport collections\nimport sys\nimport signal\n\n%matplotlib inline\n```\n\n    Using TensorFlow backend.\n\n\n\n```python\n# The original class can be imported like this:\n# from keras.preprocessing.image import ImageDataGenerator\n\n# We access the modified version through T.ImageDataGenerator\nimport tools.image as T\n\n# Useful for checking the output of the generators after code change\ntry:\n    from importlib import reload\n    reload(T)\nexcept:\n    reload(T)\n```\n\nThese are helper methods used throughout the notebook.\n\n\n```python\ndef preprocess_img(img):\n    img = img.astype(np.float32) / 255.0\n    img -= 0.5\n    return img * 2\n```\n\n\n```python\ndef plot_images(img_gen, title):\n    fig, ax = plt.subplots(6, 6, figsize=(10, 10))\n    plt.suptitle(title, size=32)\n    plt.setp(ax, xticks=[], yticks=[])\n    plt.tight_layout(rect=[0, 0.03, 1, 0.95])\n    for (imgs, labels) in img_gen:\n        for i in range(6):\n            for j in range(6):\n                if i*6 + j \u003c 32:\n                    ax[i][j].imshow(imgs[i*6 + j])\n        break    \n```\n\n## Benchmark: CIFAR10 - In Memory Performance, Image Generation Only\n\n[CIFAR10](https://www.cs.toronto.edu/~kriz/cifar.html) is a toy dataset that includes 50,000 training images and 10,000 test images of shape 32x32x3.\n\nIt includes the following 10 classes: **airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck**\n\n\n```python\nfrom keras.datasets.cifar10 import load_data\nfrom keras.utils.np_utils import to_categorical\n\n(X_train, y_train), (X_test, y_test) = load_data()\n\ny_train_cat = to_categorical(y_train)\ny_test_cat = to_categorical(y_test)\n```\n\nHere is an example of how to set up a `multiprocessing.Pool` and add it as an argument to the ImageDataGenerator constructor. This is the only change to the class' public interface. If you leave out the `pool` parameter or set it to `None`, the generator will operate in its original single process mode.\n\n\n```python\ntry:\n    pool.terminate()\nexcept:\n    pass\nn_process = 4\n    \npool = multiprocessing.Pool(processes=n_process)\nstart = time.time()\ngen = T.ImageDataGenerator(\n     featurewise_center=False,\n     samplewise_center=False,\n     featurewise_std_normalization=False,\n     samplewise_std_normalization=False,\n     zca_whitening=False,\n     rotation_range=45,\n     width_shift_range=.1,\n     height_shift_range=.1,\n     shear_range=0.,\n     zoom_range=0,\n     channel_shift_range=0,\n     fill_mode='nearest',\n     cval=0.,\n     horizontal_flip=True,\n     vertical_flip=False,\n     rescale=1/255.,\n     #preprocessing_function=preprocess_img, # disable for nicer visualization\n     dim_ordering='default',\n     pool=pool # \u003c-------------- Only change needed!\n)\n\ngen.fit(X_train)\nX_train_aug = gen.flow(X_train, y_train_cat, seed=0)\n\nprint('{} process, duration: {}'.format(4, time.time() - start))\nplot_images(X_train_aug, 'Augmented Images generated with {} processes'.format(n_process))\n\npool.terminate()\n```\n\n    4 process, duration: 0.0404160022736\n\n\n\n![png](Accelerating%20Deep%20Learning%20with%20Multiprocess%20Image%20Augmentation%20in%20Keras_files/Accelerating%20Deep%20Learning%20with%20Multiprocess%20Image%20Augmentation%20in%20Keras_18_1.png)\n\n\nNow that we have verified that the images are being properly generated with multiple processes, we want to benchmark how the number of processes affects performance. Idealy, we would like to see speedups scale linearly with the number of processes added. However, as explained by [Amdahl's Law](https://en.wikipedia.org/wiki/Amdahl%27s_law), there are diminishing returns due to additional overhead.\n\nThe following benchmark will first test image augmentation without multiprocessing, then do a test for an increasing number of processes, up to a max of the number of logical CPUs your system has. It does multiple rounds of these tests so that we may average the results.\n\n\n```python\ndurs = collections.defaultdict(list)\nnum_cores = 2\ntry:\n    num_cores = multiprocessing.cpu_count()\nexcept:\n    pass\n\nfor j in range(10):\n    print('Round', j)\n    \n    for num_p in range(0, num_cores + 1):\n        pool = None\n        if num_p \u003e 0:\n            pool = multiprocessing.Pool(processes=num_p)\n            \n        start = time.time()\n        gen = T.ImageDataGenerator(\n             featurewise_center=False,\n             samplewise_center=False,\n             featurewise_std_normalization=False,\n             samplewise_std_normalization=False,\n             zca_whitening=False,\n             rotation_range=45,\n             width_shift_range=.1,\n             height_shift_range=.1,\n             shear_range=0.,\n             zoom_range=0,\n             channel_shift_range=0,\n             fill_mode='nearest',\n             cval=0.,\n             horizontal_flip=True,\n             vertical_flip=False,\n             rescale=None,\n             preprocessing_function=preprocess_img,\n             dim_ordering='default',\n             pool=pool\n        )\n\n        gen.fit(X_train)\n        X_train_aug = gen.flow(X_train, y_train_cat, seed=0)\n\n        for i, (imgs, labels) in enumerate(X_train_aug):\n            if i == 1000:\n                break\n\n        dur = time.time() - start\n        #print(num_p, dur)\n        sys.stdout.write('{}: {} ... '.format(num_p, dur))\n        sys.stdout.flush()\n        \n        durs[num_p].append(dur)\n\n        if pool:\n            pool.terminate()\n```\n\n    ('Round', 0)\n    0: 6.84576511383 ... 1: 9.6486890316 ... 2: 6.03799390793 ... 3: 4.88081693649 ... 4: 4.66870999336 ... 5: 3.70913481712 ... 6: 3.27630805969 ... 7: 3.48509907722 ... 8: 3.64657878876 ... 9: 3.74150896072 ... 10: 3.57441878319 ... 11: 3.60130214691 ... 12: 3.47499299049 ... ('Round', 1)\n    0: 6.75701498985 ... 1: 9.94960093498 ... 2: 5.64250087738 ... 3: 5.06900811195 ... 4: 4.61409282684 ... 5: 4.57506585121 ... 6: 3.48270392418 ... 7: 3.51494693756 ... 8: 3.88235402107 ... 9: 3.62926697731 ... 10: 3.91224503517 ... 11: 3.59025716782 ... 12: 3.5045068264 ... ('Round', 2)\n    0: 6.90472793579 ... 1: 9.55179905891 ... 2: 6.57418012619 ... 3: 5.2566280365 ... 4: 4.55560803413 ... 5: 4.45380306244 ... 6: 3.54513192177 ... 7: 3.21149206161 ... 8: 3.78789710999 ... 9: 3.67751908302 ... 10: 3.74882698059 ... 11: 3.98700881004 ... 12: 3.64187002182 ... ('Round', 3)\n    0: 6.82807612419 ... 1: 9.48674917221 ... 2: 5.57596802711 ... 3: 4.74470591545 ... 4: 4.18711090088 ... 5: 3.89195489883 ... 6: 3.22924613953 ... 7: 3.17622900009 ... 8: 4.07523298264 ... 9: 3.59954690933 ... 10: 3.7366130352 ... 11: 3.52489495277 ... 12: 3.82451415062 ... ('Round', 4)\n    0: 6.73704409599 ... 1: 9.2156291008 ... 2: 6.23566198349 ... 3: 5.13580393791 ... 4: 4.71229195595 ... 5: 3.35283398628 ... 6: 3.24846291542 ... 7: 3.79010605812 ... 8: 3.74294400215 ... 9: 3.76095604897 ... 10: 3.7142059803 ... 11: 3.54178500175 ... 12: 3.72024703026 ... ('Round', 5)\n    0: 6.75245904922 ... 1: 10.7912859917 ... 2: 6.79878306389 ... 3: 4.67795395851 ... 4: 4.7692129612 ... 5: 3.99766302109 ... 6: 3.45177388191 ... 7: 3.30268979073 ... 8: 3.92767882347 ... 9: 3.69342398643 ... 10: 3.52480602264 ... 11: 3.46998000145 ... 12: 3.60531187057 ... ('Round', 6)\n    0: 6.94973492622 ... 1: 9.72229290009 ... 2: 6.76698184013 ... 3: 5.28792905807 ... 4: 4.44634389877 ... 5: 4.34274101257 ... 6: 3.94904899597 ... 7: 3.34885692596 ... 8: 3.69488501549 ... 9: 3.87995219231 ... 10: 3.78279495239 ... 11: 3.49752092361 ... 12: 3.56351184845 ... ('Round', 7)\n    0: 6.71522402763 ... 1: 10.2026801109 ... 2: 6.04175400734 ... 3: 5.20836210251 ... 4: 4.35653805733 ... 5: 4.39560294151 ... 6: 3.74392104149 ... 7: 3.19262504578 ... 8: 3.89874505997 ... 9: 3.41301083565 ... 10: 3.79124188423 ... 11: 3.90449810028 ... 12: 3.74271798134 ... ('Round', 8)\n    0: 6.8355588913 ... 1: 9.49789810181 ... 2: 5.33640003204 ... 3: 5.41973185539 ... 4: 4.42942810059 ... 5: 4.30604100227 ... 6: 3.22810721397 ... 7: 3.24005103111 ... 8: 3.61394405365 ... 9: 3.50949716568 ... 10: 3.62207698822 ... 11: 3.84033894539 ... 12: 3.85311603546 ... ('Round', 9)\n    0: 6.74057507515 ... 1: 10.3358399868 ... 2: 6.02810311317 ... 3: 5.41968894005 ... 4: 4.69001197815 ... 5: 3.6060628891 ... 6: 3.84348988533 ... 7: 3.67217493057 ... 8: 4.02522802353 ... 9: 3.74887800217 ... 10: 4.08099198341 ... 11: 3.81078886986 ... 12: 3.46359109879 ... \n\n\n```python\ndf = pd.DataFrame(durs)\ndf\n```\n\n\n\n\n\u003cdiv\u003e\n\u003ctable border=\"1\" class=\"dataframe\"\u003e\n  \u003cthead\u003e\n    \u003ctr style=\"text-align: right;\"\u003e\n      \u003cth\u003e\u003c/th\u003e\n      \u003cth\u003e0\u003c/th\u003e\n      \u003cth\u003e1\u003c/th\u003e\n      \u003cth\u003e2\u003c/th\u003e\n      \u003cth\u003e3\u003c/th\u003e\n      \u003cth\u003e4\u003c/th\u003e\n      \u003cth\u003e5\u003c/th\u003e\n      \u003cth\u003e6\u003c/th\u003e\n      \u003cth\u003e7\u003c/th\u003e\n      \u003cth\u003e8\u003c/th\u003e\n      \u003cth\u003e9\u003c/th\u003e\n      \u003cth\u003e10\u003c/th\u003e\n      \u003cth\u003e11\u003c/th\u003e\n      \u003cth\u003e12\u003c/th\u003e\n    \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n    \u003ctr\u003e\n      \u003cth\u003e0\u003c/th\u003e\n      \u003ctd\u003e6.845765\u003c/td\u003e\n      \u003ctd\u003e9.648689\u003c/td\u003e\n      \u003ctd\u003e6.037994\u003c/td\u003e\n      \u003ctd\u003e4.880817\u003c/td\u003e\n      \u003ctd\u003e4.668710\u003c/td\u003e\n      \u003ctd\u003e3.709135\u003c/td\u003e\n      \u003ctd\u003e3.276308\u003c/td\u003e\n      \u003ctd\u003e3.485099\u003c/td\u003e\n      \u003ctd\u003e3.646579\u003c/td\u003e\n      \u003ctd\u003e3.741509\u003c/td\u003e\n      \u003ctd\u003e3.574419\u003c/td\u003e\n      \u003ctd\u003e3.601302\u003c/td\u003e\n      \u003ctd\u003e3.474993\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003e1\u003c/th\u003e\n      \u003ctd\u003e6.757015\u003c/td\u003e\n      \u003ctd\u003e9.949601\u003c/td\u003e\n      \u003ctd\u003e5.642501\u003c/td\u003e\n      \u003ctd\u003e5.069008\u003c/td\u003e\n      \u003ctd\u003e4.614093\u003c/td\u003e\n      \u003ctd\u003e4.575066\u003c/td\u003e\n      \u003ctd\u003e3.482704\u003c/td\u003e\n      \u003ctd\u003e3.514947\u003c/td\u003e\n      \u003ctd\u003e3.882354\u003c/td\u003e\n      \u003ctd\u003e3.629267\u003c/td\u003e\n      \u003ctd\u003e3.912245\u003c/td\u003e\n      \u003ctd\u003e3.590257\u003c/td\u003e\n      \u003ctd\u003e3.504507\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003e2\u003c/th\u003e\n      \u003ctd\u003e6.904728\u003c/td\u003e\n      \u003ctd\u003e9.551799\u003c/td\u003e\n      \u003ctd\u003e6.574180\u003c/td\u003e\n      \u003ctd\u003e5.256628\u003c/td\u003e\n      \u003ctd\u003e4.555608\u003c/td\u003e\n      \u003ctd\u003e4.453803\u003c/td\u003e\n      \u003ctd\u003e3.545132\u003c/td\u003e\n      \u003ctd\u003e3.211492\u003c/td\u003e\n      \u003ctd\u003e3.787897\u003c/td\u003e\n      \u003ctd\u003e3.677519\u003c/td\u003e\n      \u003ctd\u003e3.748827\u003c/td\u003e\n      \u003ctd\u003e3.987009\u003c/td\u003e\n      \u003ctd\u003e3.641870\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003e3\u003c/th\u003e\n      \u003ctd\u003e6.828076\u003c/td\u003e\n      \u003ctd\u003e9.486749\u003c/td\u003e\n      \u003ctd\u003e5.575968\u003c/td\u003e\n      \u003ctd\u003e4.744706\u003c/td\u003e\n      \u003ctd\u003e4.187111\u003c/td\u003e\n      \u003ctd\u003e3.891955\u003c/td\u003e\n      \u003ctd\u003e3.229246\u003c/td\u003e\n      \u003ctd\u003e3.176229\u003c/td\u003e\n      \u003ctd\u003e4.075233\u003c/td\u003e\n      \u003ctd\u003e3.599547\u003c/td\u003e\n      \u003ctd\u003e3.736613\u003c/td\u003e\n      \u003ctd\u003e3.524895\u003c/td\u003e\n      \u003ctd\u003e3.824514\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003e4\u003c/th\u003e\n      \u003ctd\u003e6.737044\u003c/td\u003e\n      \u003ctd\u003e9.215629\u003c/td\u003e\n      \u003ctd\u003e6.235662\u003c/td\u003e\n      \u003ctd\u003e5.135804\u003c/td\u003e\n      \u003ctd\u003e4.712292\u003c/td\u003e\n      \u003ctd\u003e3.352834\u003c/td\u003e\n      \u003ctd\u003e3.248463\u003c/td\u003e\n      \u003ctd\u003e3.790106\u003c/td\u003e\n      \u003ctd\u003e3.742944\u003c/td\u003e\n      \u003ctd\u003e3.760956\u003c/td\u003e\n      \u003ctd\u003e3.714206\u003c/td\u003e\n      \u003ctd\u003e3.541785\u003c/td\u003e\n      \u003ctd\u003e3.720247\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003e5\u003c/th\u003e\n      \u003ctd\u003e6.752459\u003c/td\u003e\n      \u003ctd\u003e10.791286\u003c/td\u003e\n      \u003ctd\u003e6.798783\u003c/td\u003e\n      \u003ctd\u003e4.677954\u003c/td\u003e\n      \u003ctd\u003e4.769213\u003c/td\u003e\n      \u003ctd\u003e3.997663\u003c/td\u003e\n      \u003ctd\u003e3.451774\u003c/td\u003e\n      \u003ctd\u003e3.302690\u003c/td\u003e\n      \u003ctd\u003e3.927679\u003c/td\u003e\n      \u003ctd\u003e3.693424\u003c/td\u003e\n      \u003ctd\u003e3.524806\u003c/td\u003e\n      \u003ctd\u003e3.469980\u003c/td\u003e\n      \u003ctd\u003e3.605312\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003e6\u003c/th\u003e\n      \u003ctd\u003e6.949735\u003c/td\u003e\n      \u003ctd\u003e9.722293\u003c/td\u003e\n      \u003ctd\u003e6.766982\u003c/td\u003e\n      \u003ctd\u003e5.287929\u003c/td\u003e\n      \u003ctd\u003e4.446344\u003c/td\u003e\n      \u003ctd\u003e4.342741\u003c/td\u003e\n      \u003ctd\u003e3.949049\u003c/td\u003e\n      \u003ctd\u003e3.348857\u003c/td\u003e\n      \u003ctd\u003e3.694885\u003c/td\u003e\n      \u003ctd\u003e3.879952\u003c/td\u003e\n      \u003ctd\u003e3.782795\u003c/td\u003e\n      \u003ctd\u003e3.497521\u003c/td\u003e\n      \u003ctd\u003e3.563512\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003e7\u003c/th\u003e\n      \u003ctd\u003e6.715224\u003c/td\u003e\n      \u003ctd\u003e10.202680\u003c/td\u003e\n      \u003ctd\u003e6.041754\u003c/td\u003e\n      \u003ctd\u003e5.208362\u003c/td\u003e\n      \u003ctd\u003e4.356538\u003c/td\u003e\n      \u003ctd\u003e4.395603\u003c/td\u003e\n      \u003ctd\u003e3.743921\u003c/td\u003e\n      \u003ctd\u003e3.192625\u003c/td\u003e\n      \u003ctd\u003e3.898745\u003c/td\u003e\n      \u003ctd\u003e3.413011\u003c/td\u003e\n      \u003ctd\u003e3.791242\u003c/td\u003e\n      \u003ctd\u003e3.904498\u003c/td\u003e\n      \u003ctd\u003e3.742718\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003e8\u003c/th\u003e\n      \u003ctd\u003e6.835559\u003c/td\u003e\n      \u003ctd\u003e9.497898\u003c/td\u003e\n      \u003ctd\u003e5.336400\u003c/td\u003e\n      \u003ctd\u003e5.419732\u003c/td\u003e\n      \u003ctd\u003e4.429428\u003c/td\u003e\n      \u003ctd\u003e4.306041\u003c/td\u003e\n      \u003ctd\u003e3.228107\u003c/td\u003e\n      \u003ctd\u003e3.240051\u003c/td\u003e\n      \u003ctd\u003e3.613944\u003c/td\u003e\n      \u003ctd\u003e3.509497\u003c/td\u003e\n      \u003ctd\u003e3.622077\u003c/td\u003e\n      \u003ctd\u003e3.840339\u003c/td\u003e\n      \u003ctd\u003e3.853116\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003e9\u003c/th\u003e\n      \u003ctd\u003e6.740575\u003c/td\u003e\n      \u003ctd\u003e10.335840\u003c/td\u003e\n      \u003ctd\u003e6.028103\u003c/td\u003e\n      \u003ctd\u003e5.419689\u003c/td\u003e\n      \u003ctd\u003e4.690012\u003c/td\u003e\n      \u003ctd\u003e3.606063\u003c/td\u003e\n      \u003ctd\u003e3.843490\u003c/td\u003e\n      \u003ctd\u003e3.672175\u003c/td\u003e\n      \u003ctd\u003e4.025228\u003c/td\u003e\n      \u003ctd\u003e3.748878\u003c/td\u003e\n      \u003ctd\u003e4.080992\u003c/td\u003e\n      \u003ctd\u003e3.810789\u003c/td\u003e\n      \u003ctd\u003e3.463591\u003c/td\u003e\n    \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003c/div\u003e\n\n\n\n\n```python\ndf_mean = pd.DataFrame(df.mean(axis=0))\nplt.figure(figsize=(10,5))\nplt.plot(df_mean, marker='o')\nplt.xlabel('# Processes')\nplt.ylabel('Seconds')\nplt.title('Image Augmentation time vs. # Processes')\n```\n\n\n\n\n    \u003cmatplotlib.text.Text at 0x7fcd54312a10\u003e\n\n\n\n\n![png](Accelerating%20Deep%20Learning%20with%20Multiprocess%20Image%20Augmentation%20in%20Keras_files/Accelerating%20Deep%20Learning%20with%20Multiprocess%20Image%20Augmentation%20in%20Keras_22_1.png)\n\n\n\n```python\nspeedups = 1 / (df_mean / df_mean[0][0])\nplt.figure(figsize=(10,5))\nplt.plot(speedups, marker='o')\nplt.xlabel('# Processes')\nplt.ylabel('Speedup')\nplt.hlines(1, -1, df_mean.shape[0], colors='red', linestyles='dashed')\nplt.title('Image Augmentation speedup vs. # Processes')\n```\n\n\n\n\n    \u003cmatplotlib.text.Text at 0x7fcd543f6950\u003e\n\n\n\n\n![png](Accelerating%20Deep%20Learning%20with%20Multiprocess%20Image%20Augmentation%20in%20Keras_files/Accelerating%20Deep%20Learning%20with%20Multiprocess%20Image%20Augmentation%20in%20Keras_23_1.png)\n\n\n\n```python\nbest_ix = np.argmax(speedups.values)\nprint('Best speedup: {0:.2f}x with {1} processes.'.format(speedups.values[best_ix][0], best_ix))\n```\n\n    Best speedup: 2.01x with 7 processes.\n\n\nAs we can see, we are able to cut image generation time in half. However, does the speedup remain when we are also sending the images to the GPU for network trianing?\n\n## Benchmark: CIFAR10 - In Memory Performance, Image Generation with GPU Training\n\n\n```python\nimport tools.sysmonitor as SM\nreload(SM)\n```\n\n\n\n\n    \u003cmodule 'tools.sysmonitor' from 'tools/sysmonitor.pyc'\u003e\n\n\n\nLet us take a model from one of the [Keras examples](https://github.com/fchollet/keras/blob/master/examples/cifar10_cnn.py):\n\n\n```python\nfrom keras.models import Sequential\nfrom keras.layers import Conv2D, Activation, MaxPooling2D, Dropout, Flatten, Dense\n\nmodel = Sequential()\nmodel.add(Conv2D(32, 3, 3, border_mode='same',\n                        input_shape=(32, 32, 3)))\nmodel.add(Activation('relu'))\nmodel.add(Conv2D(32, 3, 3))\nmodel.add(Activation('relu'))\nmodel.add(MaxPooling2D(pool_size=(2, 2)))\nmodel.add(Dropout(0.25))\n\nmodel.add(Conv2D(64, 3, 3, border_mode='same'))\nmodel.add(Activation('relu'))\nmodel.add(Conv2D(64, 3, 3))\nmodel.add(Activation('relu'))\nmodel.add(MaxPooling2D(pool_size=(2, 2)))\nmodel.add(Dropout(0.25))\n\nmodel.add(Flatten())\nmodel.add(Dense(512))\nmodel.add(Activation('relu'))\nmodel.add(Dropout(0.5))\nmodel.add(Dense(10))\nmodel.add(Activation('softmax'))\n\nmodel.summary()\nmodel.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])\n```\n\n    ____________________________________________________________________________________________________\n    Layer (type)                     Output Shape          Param #     Connected to                     \n    ====================================================================================================\n    convolution2d_8 (Convolution2D)  (None, 32, 32, 32)    896         convolution2d_input_3[0][0]      \n    ____________________________________________________________________________________________________\n    activation_12 (Activation)       (None, 32, 32, 32)    0           convolution2d_8[0][0]            \n    ____________________________________________________________________________________________________\n    convolution2d_9 (Convolution2D)  (None, 30, 30, 32)    9248        activation_12[0][0]              \n    ____________________________________________________________________________________________________\n    activation_13 (Activation)       (None, 30, 30, 32)    0           convolution2d_9[0][0]            \n    ____________________________________________________________________________________________________\n    maxpooling2d_6 (MaxPooling2D)    (None, 15, 15, 32)    0           activation_13[0][0]              \n    ____________________________________________________________________________________________________\n    dropout_5 (Dropout)              (None, 15, 15, 32)    0           maxpooling2d_6[0][0]             \n    ____________________________________________________________________________________________________\n    convolution2d_10 (Convolution2D) (None, 15, 15, 64)    18496       dropout_5[0][0]                  \n    ____________________________________________________________________________________________________\n    activation_14 (Activation)       (None, 15, 15, 64)    0           convolution2d_10[0][0]           \n    ____________________________________________________________________________________________________\n    convolution2d_11 (Convolution2D) (None, 13, 13, 64)    36928       activation_14[0][0]              \n    ____________________________________________________________________________________________________\n    activation_15 (Activation)       (None, 13, 13, 64)    0           convolution2d_11[0][0]           \n    ____________________________________________________________________________________________________\n    maxpooling2d_7 (MaxPooling2D)    (None, 6, 6, 64)      0           activation_15[0][0]              \n    ____________________________________________________________________________________________________\n    dropout_6 (Dropout)              (None, 6, 6, 64)      0           maxpooling2d_7[0][0]             \n    ____________________________________________________________________________________________________\n    flatten_3 (Flatten)              (None, 2304)          0           dropout_6[0][0]                  \n    ____________________________________________________________________________________________________\n    dense_5 (Dense)                  (None, 512)           1180160     flatten_3[0][0]                  \n    ____________________________________________________________________________________________________\n    activation_16 (Activation)       (None, 512)           0           dense_5[0][0]                    \n    ____________________________________________________________________________________________________\n    dropout_7 (Dropout)              (None, 512)           0           activation_16[0][0]              \n    ____________________________________________________________________________________________________\n    dense_6 (Dense)                  (None, 10)            5130        dropout_7[0][0]                  \n    ____________________________________________________________________________________________________\n    activation_17 (Activation)       (None, 10)            0           dense_6[0][0]                    \n    ====================================================================================================\n    Total params: 1,250,858\n    Trainable params: 1,250,858\n    Non-trainable params: 0\n    ____________________________________________________________________________________________________\n\n\nWhen we are running lengthier training sessions, we may want to interrupt training to try a different approach: tweak hyperparameters, choose a different optimizer, adjust the network architecture, etc. In order to handle this gracefully with multiprocessing, we need to tell the child processes to ignore the interrupt signals. The parent process will catch the KeyboardInterrupt exception allow us to continue working interactively in the Notebook. Without this infrastructure, the processes will remain in limbo as detailed [here](http://noswap.com/blog/python-multiprocessing-keyboardinterrupt).\n\n\n```python\npool = None\n\ndef init_worker():\n    signal.signal(signal.SIGINT, signal.SIG_IGN)\n```\n\n\n```python\ndef setup_generator(processes=None, batch_size=32):\n    global pool\n    try:\n        pool.terminate()\n    except:\n        pass\n\n    if processes: \n        pool = multiprocessing.Pool(processes=processes, initializer=init_worker)\n    else:\n        pool = None\n\n    gen = T.ImageDataGenerator(\n         featurewise_center=False,\n         samplewise_center=False,\n         featurewise_std_normalization=False,\n         samplewise_std_normalization=False,\n         zca_whitening=False,\n         rotation_range=45,\n         width_shift_range=.1,\n         height_shift_range=.1,\n         shear_range=0.,\n         zoom_range=[.8, 1],\n         channel_shift_range=20,\n         fill_mode='nearest',\n         cval=0.,\n         horizontal_flip=True,\n         vertical_flip=False,\n         rescale=None,\n         preprocessing_function=preprocess_img,\n         dim_ordering='default',\n         pool=pool\n    )\n    test_gen = T.ImageDataGenerator(\n        preprocessing_function=preprocess_img,\n        pool=pool\n    )\n    \n    gen.fit(X_train)\n    test_gen.fit(X_train)\n    \n    X_train_aug = gen.flow(X_train, y_train_cat, seed=0, batch_size=batch_size)\n    X_test_aug = test_gen.flow(X_test, y_test_cat, seed=0, batch_size=batch_size)\n    \n    return X_train_aug, X_test_aug\n```\n\n\n```python\ndef run_benchmark(processes=None, batch_size=32, vert=True, plot=True):\n    X_train_aug, X_test_aug = setup_generator(processes=processes, batch_size=batch_size)\n    sys_mon = SM.SysMonitor()\n    sys_mon.start()\n    \n    try:\n        model.fit_generator(X_train_aug, samples_per_epoch=50000, nb_epoch=5, \n                        validation_data=X_test_aug, nb_val_samples=10000)\n    except KeyboardInterrupt:\n        print '\\n\\nTraining Interrupted\\n'\n        return None\n\n    sys_mon.stop()\n    \n    title = None\n    if not processes:\n        title = '{0:.2f} seconds of computation, no multiprocessing, batch size = {1}'.format(sys_mon.duration, batch_size)\n    else:\n        title = '{0:.2f} seconds of computation, using {1} processes, batch size = {2}'.format(sys_mon.duration, processes, batch_size)\n    \n    if plot:\n        sys_mon.plot(title, vert)\n    \n    if not processes:\n        processes = 0\n        \n    return {\n        'processes': processes,\n        'batch_size': batch_size,\n        'duration': sys_mon.duration,\n        'title': title\n    }\n```\n\n\n```python\nrun_benchmark(processes=None, batch_size=32)\n```\n\n    Epoch 1/5\n     3552/50000 [=\u003e............................] - ETA: 29s - loss: 2.1171 - acc: 0.2030\n    \n    Training Interrupted\n    \n\n\n\n```python\nrun_benchmark(processes=7, batch_size=32)\n```\n\n    Epoch 1/5\n    11136/50000 [=====\u003e........................] - ETA: 8s - loss: 1.8752 - acc: 0.3081\n    \n    Training Interrupted\n    \n\n\nNow let's try a variety of different test scenarios:\n\n\n```python\nruns = []\n```\n\n\n```python\nruns.append(run_benchmark(processes=None, batch_size=32))\n```\n\n    Epoch 1/5\n    50000/50000 [==============================] - 22s - loss: 1.1598 - acc: 0.5941 - val_loss: 0.8368 - val_acc: 0.7077\n    Epoch 2/5\n    50000/50000 [==============================] - 21s - loss: 1.1457 - acc: 0.6003 - val_loss: 0.8865 - val_acc: 0.6907\n    Epoch 3/5\n    50000/50000 [==============================] - 21s - loss: 1.1311 - acc: 0.6031 - val_loss: 0.8255 - val_acc: 0.7190\n    Epoch 4/5\n    50000/50000 [==============================] - 21s - loss: 1.1232 - acc: 0.6060 - val_loss: 0.8367 - val_acc: 0.7142\n    Epoch 5/5\n    50000/50000 [==============================] - 22s - loss: 1.1075 - acc: 0.6116 - val_loss: 0.8358 - val_acc: 0.7054\n\n\n\n![png](Accelerating%20Deep%20Learning%20with%20Multiprocess%20Image%20Augmentation%20in%20Keras_files/Accelerating%20Deep%20Learning%20with%20Multiprocess%20Image%20Augmentation%20in%20Keras_38_1.png)\n\n\n\n```python\nruns.append(run_benchmark(processes=7, batch_size=32))\n```\n\n    Epoch 1/5\n    50000/50000 [==============================] - 11s - loss: 1.0912 - acc: 0.6165 - val_loss: 0.8329 - val_acc: 0.7103\n    Epoch 2/5\n    50000/50000 [==============================] - 11s - loss: 1.0838 - acc: 0.6232 - val_loss: 0.8299 - val_acc: 0.7053\n    Epoch 3/5\n    50000/50000 [==============================] - 11s - loss: 1.0736 - acc: 0.6245 - val_loss: 0.8385 - val_acc: 0.7092\n    Epoch 4/5\n    50000/50000 [==============================] - 11s - loss: 1.0671 - acc: 0.6258 - val_loss: 0.7994 - val_acc: 0.7238\n    Epoch 5/5\n    50000/50000 [==============================] - 11s - loss: 1.0670 - acc: 0.6283 - val_loss: 0.8347 - val_acc: 0.7133\n\n\n\n![png](Accelerating%20Deep%20Learning%20with%20Multiprocess%20Image%20Augmentation%20in%20Keras_files/Accelerating%20Deep%20Learning%20with%20Multiprocess%20Image%20Augmentation%20in%20Keras_39_1.png)\n\n\n\n```python\nruns[0]['duration'] / runs[1]['duration']\n```\n\n\n\n\n    1.8832975152491378\n\n\n\nAs we can see, we can get a **1.8x speedup by using 7 processes**. The GPU and CPU utilization is markedly higher and more consistent.\n\nLet's see if batch size affects the outcome:\n\n\n```python\nruns.append(run_benchmark(processes=None, batch_size=256))\n```\n\n    Epoch 1/5\n    50000/50000 [==============================] - 19s - loss: 1.0319 - acc: 0.6400 - val_loss: 0.7463 - val_acc: 0.7389\n    Epoch 2/5\n    50000/50000 [==============================] - 17s - loss: 1.0013 - acc: 0.6495 - val_loss: 0.7436 - val_acc: 0.7416\n    Epoch 3/5\n    50000/50000 [==============================] - 17s - loss: 0.9910 - acc: 0.6537 - val_loss: 0.7253 - val_acc: 0.7484\n    Epoch 4/5\n    50000/50000 [==============================] - 17s - loss: 0.9824 - acc: 0.6582 - val_loss: 0.7271 - val_acc: 0.7499\n    Epoch 5/5\n    50000/50000 [==============================] - 17s - loss: 0.9752 - acc: 0.6600 - val_loss: 0.6967 - val_acc: 0.7607\n\n\n\n![png](Accelerating%20Deep%20Learning%20with%20Multiprocess%20Image%20Augmentation%20in%20Keras_files/Accelerating%20Deep%20Learning%20with%20Multiprocess%20Image%20Augmentation%20in%20Keras_42_1.png)\n\n\n\n```python\nruns.append(run_benchmark(processes=7, batch_size=256))\n```\n\n    Epoch 1/5\n    50000/50000 [==============================] - 5s - loss: 0.9585 - acc: 0.6660 - val_loss: 0.7220 - val_acc: 0.7495\n    Epoch 2/5\n    50000/50000 [==============================] - 5s - loss: 0.9553 - acc: 0.6671 - val_loss: 0.7071 - val_acc: 0.7546\n    Epoch 3/5\n    50000/50000 [==============================] - 5s - loss: 0.9502 - acc: 0.6690 - val_loss: 0.6920 - val_acc: 0.7640\n    Epoch 4/5\n    50000/50000 [==============================] - 5s - loss: 0.9525 - acc: 0.6687 - val_loss: 0.7103 - val_acc: 0.7558\n    Epoch 5/5\n    50000/50000 [==============================] - 5s - loss: 0.9452 - acc: 0.6713 - val_loss: 0.6999 - val_acc: 0.7565\n\n\n\n![png](Accelerating%20Deep%20Learning%20with%20Multiprocess%20Image%20Augmentation%20in%20Keras_files/Accelerating%20Deep%20Learning%20with%20Multiprocess%20Image%20Augmentation%20in%20Keras_43_1.png)\n\n\n\n```python\nruns[2]['duration'] / runs[3]['duration']\n```\n\n\n\n\n    3.3318531284663795\n\n\n\nWith a batch size of 256, we get an **even larger speedup of 3.3x**\n\n\n```python\nruns.append(run_benchmark(processes=None, batch_size=1024))\n```\n\n    Epoch 1/5\n    50000/50000 [==============================] - 18s - loss: 0.9383 - acc: 0.6709 - val_loss: 0.6876 - val_acc: 0.7634\n    Epoch 2/5\n    50000/50000 [==============================] - 15s - loss: 0.9310 - acc: 0.6733 - val_loss: 0.6851 - val_acc: 0.7626\n    Epoch 3/5\n    50000/50000 [==============================] - 16s - loss: 0.9226 - acc: 0.6794 - val_loss: 0.6783 - val_acc: 0.7701\n    Epoch 4/5\n    50000/50000 [==============================] - 15s - loss: 0.9230 - acc: 0.6785 - val_loss: 0.6884 - val_acc: 0.7651\n    Epoch 5/5\n    50000/50000 [==============================] - 15s - loss: 0.9152 - acc: 0.6809 - val_loss: 0.6682 - val_acc: 0.7695\n\n\n\n![png](Accelerating%20Deep%20Learning%20with%20Multiprocess%20Image%20Augmentation%20in%20Keras_files/Accelerating%20Deep%20Learning%20with%20Multiprocess%20Image%20Augmentation%20in%20Keras_46_1.png)\n\n\n\n```python\nruns.append(run_benchmark(processes=7, batch_size=1024))\n```\n\n    Epoch 1/5\n    50000/50000 [==============================] - 5s - loss: 0.9137 - acc: 0.6815 - val_loss: 0.6798 - val_acc: 0.7661\n    Epoch 2/5\n    50000/50000 [==============================] - 4s - loss: 0.9161 - acc: 0.6814 - val_loss: 0.6771 - val_acc: 0.7649\n    Epoch 3/5\n    50000/50000 [==============================] - 4s - loss: 0.9125 - acc: 0.6812 - val_loss: 0.6759 - val_acc: 0.7691\n    Epoch 4/5\n    50000/50000 [==============================] - 4s - loss: 0.9133 - acc: 0.6814 - val_loss: 0.6786 - val_acc: 0.7673\n    Epoch 5/5\n    50000/50000 [==============================] - 4s - loss: 0.9139 - acc: 0.6812 - val_loss: 0.6574 - val_acc: 0.7707\n\n\n\n![png](Accelerating%20Deep%20Learning%20with%20Multiprocess%20Image%20Augmentation%20in%20Keras_files/Accelerating%20Deep%20Learning%20with%20Multiprocess%20Image%20Augmentation%20in%20Keras_47_1.png)\n\n\n\n```python\nruns[4]['duration'] / runs[5]['duration']\n```\n\n\n\n\n    3.4816410549495163\n\n\n\nWith a batch size of 1024, we get **a speedup of 3.48%**. We also notice an interesting phenomenon. Without multiprocessing, the GPU is interittently going to 0 utilization. However, with 7 processes, we can see consistent \u003e60% GPU utilization with a long initial period of \u003e80%. Notice that with this batch size, we are able to get to lower losses a lot quicker than with lower batch sizes. This pattern will not necessarily continue with additional epochs, but it may be promising in some cases.\n\n\n```python\nruns.append(run_benchmark(processes=None, batch_size=4096))\n```\n\n    Epoch 1/5\n    50000/50000 [==============================] - 19s - loss: 0.9059 - acc: 0.6812 - val_loss: 0.6704 - val_acc: 0.7696\n    Epoch 2/5\n    50000/50000 [==============================] - 16s - loss: 0.9116 - acc: 0.6829 - val_loss: 0.6654 - val_acc: 0.7666\n    Epoch 3/5\n    50000/50000 [==============================] - 14s - loss: 0.9002 - acc: 0.6867 - val_loss: 0.6626 - val_acc: 0.7719\n    Epoch 4/5\n    50000/50000 [==============================] - 16s - loss: 0.8984 - acc: 0.6863 - val_loss: 0.6678 - val_acc: 0.7688\n    Epoch 5/5\n    50000/50000 [==============================] - 15s - loss: 0.9041 - acc: 0.6847 - val_loss: 0.6647 - val_acc: 0.7663\n\n\n\n![png](Accelerating%20Deep%20Learning%20with%20Multiprocess%20Image%20Augmentation%20in%20Keras_files/Accelerating%20Deep%20Learning%20with%20Multiprocess%20Image%20Augmentation%20in%20Keras_50_1.png)\n\n\n\n```python\nruns.append(run_benchmark(processes=7, batch_size=4096))\n```\n\n    Epoch 1/5\n    50000/50000 [==============================] - 6s - loss: 0.9057 - acc: 0.6823 - val_loss: 0.6678 - val_acc: 0.7680\n    Epoch 2/5\n    50000/50000 [==============================] - 4s - loss: 0.9003 - acc: 0.6881 - val_loss: 0.6596 - val_acc: 0.7687\n    Epoch 3/5\n    50000/50000 [==============================] - 4s - loss: 0.8993 - acc: 0.6866 - val_loss: 0.6560 - val_acc: 0.7734\n    Epoch 4/5\n    50000/50000 [==============================] - 4s - loss: 0.9034 - acc: 0.6857 - val_loss: 0.6641 - val_acc: 0.7713\n    Epoch 5/5\n    50000/50000 [==============================] - 4s - loss: 0.9023 - acc: 0.6862 - val_loss: 0.6670 - val_acc: 0.7653\n\n\n\n![png](Accelerating%20Deep%20Learning%20with%20Multiprocess%20Image%20Augmentation%20in%20Keras_files/Accelerating%20Deep%20Learning%20with%20Multiprocess%20Image%20Augmentation%20in%20Keras_51_1.png)\n\n\n\n```python\nruns[6]['duration'] / runs[7]['duration']\n```\n\n\n\n\n    3.4145451599892525\n\n\n\nA larger batch size of 4096 may or may not be a good choice in all cases, but when it comes to measuring system performance, we can see that the GPU usage is not consistent in the single process case. On the other hand, we are getting between 80-100% GPU utilization with 7 processes.\n\nLet's do a final experiment with this dataset to see how Image Augmentation + GPU Training time scales with process count:\n\n\n```python\nprocesses_counts = [None]\nprocesses_counts.extend(range(1, 13))\n\nresults = []\n\nfor pc in processes_counts:\n    print('process count', pc)\n    results.append(run_benchmark(processes=pc, batch_size=4096, plot=False))\n```\n\n    ('process count', None)\n    Epoch 1/5\n    50000/50000 [==============================] - 19s - loss: 0.8994 - acc: 0.6885 - val_loss: 0.6619 - val_acc: 0.7704\n    Epoch 2/5\n    50000/50000 [==============================] - 15s - loss: 0.9035 - acc: 0.6864 - val_loss: 0.6609 - val_acc: 0.7706\n    Epoch 3/5\n    50000/50000 [==============================] - 15s - loss: 0.8930 - acc: 0.6883 - val_loss: 0.6613 - val_acc: 0.7730\n    Epoch 4/5\n    50000/50000 [==============================] - 17s - loss: 0.8894 - acc: 0.6879 - val_loss: 0.6648 - val_acc: 0.7705\n    Epoch 5/5\n    50000/50000 [==============================] - 14s - loss: 0.8942 - acc: 0.6870 - val_loss: 0.6639 - val_acc: 0.7706\n    ('process count', 1)\n    Epoch 1/5\n    50000/50000 [==============================] - 26s - loss: 0.8941 - acc: 0.6873 - val_loss: 0.6590 - val_acc: 0.7727\n    Epoch 2/5\n    50000/50000 [==============================] - 19s - loss: 0.8953 - acc: 0.6900 - val_loss: 0.6597 - val_acc: 0.7690\n    Epoch 3/5\n    50000/50000 [==============================] - 18s - loss: 0.8889 - acc: 0.6888 - val_loss: 0.6528 - val_acc: 0.7775\n    Epoch 4/5\n    50000/50000 [==============================] - 20s - loss: 0.8924 - acc: 0.6879 - val_loss: 0.6627 - val_acc: 0.7714\n    Epoch 5/5\n    50000/50000 [==============================] - 21s - loss: 0.8962 - acc: 0.6873 - val_loss: 0.6599 - val_acc: 0.7704\n    ('process count', 2)\n    Epoch 1/5\n    50000/50000 [==============================] - 15s - loss: 0.8916 - acc: 0.6884 - val_loss: 0.6598 - val_acc: 0.7725\n    Epoch 2/5\n    50000/50000 [==============================] - 11s - loss: 0.8925 - acc: 0.6888 - val_loss: 0.6544 - val_acc: 0.7716\n    Epoch 3/5\n    50000/50000 [==============================] - 11s - loss: 0.8869 - acc: 0.6898 - val_loss: 0.6505 - val_acc: 0.7768\n    Epoch 4/5\n    50000/50000 [==============================] - 11s - loss: 0.8917 - acc: 0.6895 - val_loss: 0.6578 - val_acc: 0.7735\n    Epoch 5/5\n    50000/50000 [==============================] - 11s - loss: 0.8890 - acc: 0.6888 - val_loss: 0.6614 - val_acc: 0.7701\n    ('process count', 3)\n    Epoch 1/5\n    50000/50000 [==============================] - 10s - loss: 0.8788 - acc: 0.6938 - val_loss: 0.6580 - val_acc: 0.7730\n    Epoch 2/5\n    50000/50000 [==============================] - 7s - loss: 0.8887 - acc: 0.6901 - val_loss: 0.6564 - val_acc: 0.7694\n    Epoch 3/5\n    50000/50000 [==============================] - 7s - loss: 0.8817 - acc: 0.6919 - val_loss: 0.6488 - val_acc: 0.7756\n    Epoch 4/5\n    50000/50000 [==============================] - 7s - loss: 0.8852 - acc: 0.6923 - val_loss: 0.6549 - val_acc: 0.7731\n    Epoch 5/5\n    50000/50000 [==============================] - 7s - loss: 0.8833 - acc: 0.6904 - val_loss: 0.6574 - val_acc: 0.7729\n    ('process count', 4)\n    Epoch 1/5\n    50000/50000 [==============================] - 8s - loss: 0.8780 - acc: 0.6919 - val_loss: 0.6539 - val_acc: 0.7742\n    Epoch 2/5\n    50000/50000 [==============================] - 6s - loss: 0.8839 - acc: 0.6914 - val_loss: 0.6511 - val_acc: 0.7696\n    Epoch 3/5\n    50000/50000 [==============================] - 5s - loss: 0.8782 - acc: 0.6936 - val_loss: 0.6481 - val_acc: 0.7741\n    Epoch 4/5\n    50000/50000 [==============================] - 5s - loss: 0.8792 - acc: 0.6940 - val_loss: 0.6529 - val_acc: 0.7736\n    Epoch 5/5\n    50000/50000 [==============================] - 6s - loss: 0.8844 - acc: 0.6907 - val_loss: 0.6586 - val_acc: 0.7713\n    ('process count', 5)\n    Epoch 1/5\n    50000/50000 [==============================] - 7s - loss: 0.8776 - acc: 0.6930 - val_loss: 0.6514 - val_acc: 0.7749\n    Epoch 2/5\n    50000/50000 [==============================] - 5s - loss: 0.8779 - acc: 0.6919 - val_loss: 0.6521 - val_acc: 0.7697\n    Epoch 3/5\n    50000/50000 [==============================] - 5s - loss: 0.8692 - acc: 0.6957 - val_loss: 0.6453 - val_acc: 0.7769\n    Epoch 4/5\n    50000/50000 [==============================] - 5s - loss: 0.8792 - acc: 0.6944 - val_loss: 0.6520 - val_acc: 0.7753\n    Epoch 5/5\n    50000/50000 [==============================] - 5s - loss: 0.8804 - acc: 0.6926 - val_loss: 0.6561 - val_acc: 0.7738\n    ('process count', 6)\n    Epoch 1/5\n    50000/50000 [==============================] - 6s - loss: 0.8702 - acc: 0.6945 - val_loss: 0.6512 - val_acc: 0.7739\n    Epoch 2/5\n    50000/50000 [==============================] - 4s - loss: 0.8708 - acc: 0.6949 - val_loss: 0.6470 - val_acc: 0.7715\n    Epoch 3/5\n    50000/50000 [==============================] - 4s - loss: 0.8686 - acc: 0.6964 - val_loss: 0.6417 - val_acc: 0.7766\n    Epoch 4/5\n    50000/50000 [==============================] - 4s - loss: 0.8683 - acc: 0.6966 - val_loss: 0.6495 - val_acc: 0.7763\n    Epoch 5/5\n    50000/50000 [==============================] - 4s - loss: 0.8692 - acc: 0.6997 - val_loss: 0.6525 - val_acc: 0.7752\n    ('process count', 7)\n    Epoch 1/5\n    50000/50000 [==============================] - 6s - loss: 0.8676 - acc: 0.6960 - val_loss: 0.6477 - val_acc: 0.7746\n    Epoch 2/5\n    50000/50000 [==============================] - 4s - loss: 0.8634 - acc: 0.6985 - val_loss: 0.6442 - val_acc: 0.7714\n    Epoch 3/5\n    50000/50000 [==============================] - 4s - loss: 0.8656 - acc: 0.6988 - val_loss: 0.6398 - val_acc: 0.7769\n    Epoch 4/5\n    50000/50000 [==============================] - 4s - loss: 0.8694 - acc: 0.6967 - val_loss: 0.6495 - val_acc: 0.7749\n    Epoch 5/5\n    50000/50000 [==============================] - 4s - loss: 0.8617 - acc: 0.6994 - val_loss: 0.6511 - val_acc: 0.7761\n    ('process count', 8)\n    Epoch 1/5\n    50000/50000 [==============================] - 5s - loss: 0.8600 - acc: 0.6989 - val_loss: 0.6462 - val_acc: 0.7754\n    Epoch 2/5\n    50000/50000 [==============================] - 4s - loss: 0.8620 - acc: 0.6997 - val_loss: 0.6404 - val_acc: 0.7743\n    Epoch 3/5\n    50000/50000 [==============================] - 4s - loss: 0.8563 - acc: 0.6982 - val_loss: 0.6389 - val_acc: 0.7768\n    Epoch 4/5\n    50000/50000 [==============================] - 4s - loss: 0.8639 - acc: 0.6981 - val_loss: 0.6457 - val_acc: 0.7772\n    Epoch 5/5\n    50000/50000 [==============================] - 4s - loss: 0.8653 - acc: 0.6995 - val_loss: 0.6504 - val_acc: 0.7762\n    ('process count', 9)\n    Epoch 1/5\n    50000/50000 [==============================] - 5s - loss: 0.8581 - acc: 0.6996 - val_loss: 0.6442 - val_acc: 0.7769\n    Epoch 2/5\n    50000/50000 [==============================] - 4s - loss: 0.8603 - acc: 0.6989 - val_loss: 0.6437 - val_acc: 0.7727\n    Epoch 3/5\n    50000/50000 [==============================] - 4s - loss: 0.8557 - acc: 0.7032 - val_loss: 0.6374 - val_acc: 0.7794\n    Epoch 4/5\n    50000/50000 [==============================] - 4s - loss: 0.8620 - acc: 0.6998 - val_loss: 0.6439 - val_acc: 0.7776\n    Epoch 5/5\n    50000/50000 [==============================] - 4s - loss: 0.8594 - acc: 0.6969 - val_loss: 0.6474 - val_acc: 0.7774\n    ('process count', 10)\n    Epoch 1/5\n    50000/50000 [==============================] - 5s - loss: 0.8528 - acc: 0.7036 - val_loss: 0.6420 - val_acc: 0.7759\n    Epoch 2/5\n    50000/50000 [==============================] - 4s - loss: 0.8518 - acc: 0.7028 - val_loss: 0.6378 - val_acc: 0.7756\n    Epoch 3/5\n    50000/50000 [==============================] - 4s - loss: 0.8491 - acc: 0.7034 - val_loss: 0.6332 - val_acc: 0.7793\n    Epoch 4/5\n    50000/50000 [==============================] - 4s - loss: 0.8555 - acc: 0.7023 - val_loss: 0.6421 - val_acc: 0.7777\n    Epoch 5/5\n    50000/50000 [==============================] - 4s - loss: 0.8520 - acc: 0.7011 - val_loss: 0.6459 - val_acc: 0.7750\n    ('process count', 11)\n    Epoch 1/5\n    50000/50000 [==============================] - 5s - loss: 0.8518 - acc: 0.7010 - val_loss: 0.6389 - val_acc: 0.7795\n    Epoch 2/5\n    50000/50000 [==============================] - 4s - loss: 0.8506 - acc: 0.7038 - val_loss: 0.6398 - val_acc: 0.7746\n    Epoch 3/5\n    50000/50000 [==============================] - 4s - loss: 0.8438 - acc: 0.7056 - val_loss: 0.6339 - val_acc: 0.7812\n    Epoch 4/5\n    50000/50000 [==============================] - 4s - loss: 0.8553 - acc: 0.7020 - val_loss: 0.6393 - val_acc: 0.7784\n    Epoch 5/5\n    50000/50000 [==============================] - 4s - loss: 0.8475 - acc: 0.7057 - val_loss: 0.6449 - val_acc: 0.7785\n    ('process count', 12)\n    Epoch 1/5\n    50000/50000 [==============================] - 5s - loss: 0.8450 - acc: 0.7028 - val_loss: 0.6371 - val_acc: 0.7784\n    Epoch 2/5\n    50000/50000 [==============================] - 4s - loss: 0.8444 - acc: 0.7047 - val_loss: 0.6353 - val_acc: 0.7773\n    Epoch 3/5\n    50000/50000 [==============================] - 4s - loss: 0.8418 - acc: 0.7074 - val_loss: 0.6290 - val_acc: 0.7809\n    Epoch 4/5\n    50000/50000 [==============================] - 4s - loss: 0.8447 - acc: 0.7049 - val_loss: 0.6392 - val_acc: 0.7783\n    Epoch 5/5\n    50000/50000 [==============================] - 4s - loss: 0.8457 - acc: 0.7011 - val_loss: 0.6417 - val_acc: 0.7781\n\n\n\n```python\ndurs_4096 = pd.DataFrame([x['duration'] for x in results])\nplt.figure(figsize=(10,5))\nplt.plot(durs_4096, marker='o')\nplt.xlabel('# Processes')\nplt.ylabel('Seconds')\nplt.title('Image Augmentation + GPU Training time vs. # Processes')\n```\n\n\n\n\n    \u003cmatplotlib.text.Text at 0x7fcd485fec90\u003e\n\n\n\n\n![png](Accelerating%20Deep%20Learning%20with%20Multiprocess%20Image%20Augmentation%20in%20Keras_files/Accelerating%20Deep%20Learning%20with%20Multiprocess%20Image%20Augmentation%20in%20Keras_55_1.png)\n\n\n\n```python\nspeedups_4096 = 1 / (durs_4096 / durs_4096.ix[0])\n\nplt.figure(figsize=(10,5))\nplt.plot(speedups_4096, marker='o')\nplt.xlabel('# Processes')\nplt.ylabel('Speedup')\nplt.hlines(1, -1, speedups_4096.shape[0], colors='red', linestyles='dashed')\nplt.title('Image Augmentation + GPU Training speedup vs. # Processes')\n```\n\n\n\n\n    \u003cmatplotlib.text.Text at 0x7fcd48790c90\u003e\n\n\n\n\n![png](Accelerating%20Deep%20Learning%20with%20Multiprocess%20Image%20Augmentation%20in%20Keras_files/Accelerating%20Deep%20Learning%20with%20Multiprocess%20Image%20Augmentation%20in%20Keras_56_1.png)\n\n\n\n```python\nbest_ix = np.argmax(speedups_4096.values)\nprint('Best speedup: {0:.2f}x with {1} processes.'.format(speedups_4096.values[best_ix][0], best_ix))\n```\n\n    Best speedup: 3.51x with 9 processes.\n\n\n## Benchmark: Dogs vs. Cats - On Disk Performance, Image Generation witih GPU Training\n\nUsing the images in the [dogs vs. cats dataset](https://www.kaggle.com/c/dogs-vs-cats/data) provided by Kaggle, we can test the performance of image augmentation on images loaded from disk on the fly.\n\nTo follow along, unzip the downloaded training zip file, then create a `data/train/cat`, `/data/train/dog`, `data/validation/cat`, and `data/validation/dog` folders. \n\nThen move the images that have indicies starting with 8 into the appropriate validation folders.\n\n`\nmv cat.8* data/validation/cat/\nmv dog.8* data/validation/dog/\n`\n\n\n```python\nimport os\n\npaths = sorted(os.listdir('./data/train/cat'))\n\nfig, ax = plt.subplots(5, 5, figsize=(15, 15))\nfor i in range(5):\n    for j in range(5):\n        ix = i*5 + j\n        img = plt.imread('./data/train/cat/' + paths[ix])\n        ax[i][j].imshow(img)\n\n```\n\n\n![png](Accelerating%20Deep%20Learning%20with%20Multiprocess%20Image%20Augmentation%20in%20Keras_files/Accelerating%20Deep%20Learning%20with%20Multiprocess%20Image%20Augmentation%20in%20Keras_60_0.png)\n\n\n\n```python\npaths = sorted(os.listdir('./data/train/dog'))\n\nfig, ax = plt.subplots(5, 5, figsize=(15, 15))\nfor i in range(5):\n    for j in range(5):\n        ix = i*5 + j\n        img = plt.imread('./data/train/dog/' + paths[ix])\n        ax[i][j].imshow(img)\n\n```\n\n\n![png](Accelerating%20Deep%20Learning%20with%20Multiprocess%20Image%20Augmentation%20in%20Keras_files/Accelerating%20Deep%20Learning%20with%20Multiprocess%20Image%20Augmentation%20in%20Keras_61_0.png)\n\n\n\n```python\ngen = T.ImageDataGenerator(\n     featurewise_center=False,\n     samplewise_center=False,\n     featurewise_std_normalization=False,\n     samplewise_std_normalization=False,\n     zca_whitening=False,\n     rotation_range=45,\n     width_shift_range=.1,\n     height_shift_range=.1,\n     shear_range=0.,\n     zoom_range=[.8, 1],\n     channel_shift_range=0,\n     fill_mode='nearest',\n     cval=0.,\n     horizontal_flip=True,\n     vertical_flip=False,\n     rescale=1/255.,\n#      preprocessing_function=preprocess_img,\n     #dim_ordering='default',\n#      pool=None\n)\ntest_gen = T.ImageDataGenerator(\n    preprocessing_function=preprocess_img,\n#     pool=None\n)\n\n\ntrain_generator = gen.flow_from_directory(\n    'data/train',\n    target_size=(150, 150),\n    batch_size=32,\n    class_mode='binary')\n\ntest_generator = gen.flow_from_directory(\n    'data/validation',\n    target_size=(150, 150),\n    batch_size=32,\n    class_mode='binary')\n\nfig, ax = plt.subplots(6, 6, figsize=(15, 15))\nfor (imgs, labels) in train_generator:\n    for i in range(6):\n        for j in range(6):\n            if i*6 + j \u003c 32:\n                ax[i][j].imshow(imgs[i*6 + j])\n    break\n\n```\n\n    Found 22778 images belonging to 2 classes.\n    Found 2222 images belonging to 2 classes.\n\n\n\n![png](Accelerating%20Deep%20Learning%20with%20Multiprocess%20Image%20Augmentation%20in%20Keras_files/Accelerating%20Deep%20Learning%20with%20Multiprocess%20Image%20Augmentation%20in%20Keras_62_1.png)\n\n\n\n```python\nfrom keras.models import Sequential\nfrom keras.layers import Convolution2D, MaxPooling2D\nfrom keras.layers import Activation, Dropout, Flatten, Dense\n\nmodel = Sequential()\nmodel.add(Convolution2D(32, 3, 3, input_shape=(299, 299, 3)))\nmodel.add(Activation('relu'))\nmodel.add(MaxPooling2D(pool_size=(2, 2)))\n\nmodel.add(Convolution2D(32, 3, 3))\nmodel.add(Activation('relu'))\nmodel.add(MaxPooling2D(pool_size=(2, 2)))\n\nmodel.add(Convolution2D(64, 3, 3))\nmodel.add(Activation('relu'))\nmodel.add(MaxPooling2D(pool_size=(2, 2)))\n\nmodel.add(Flatten())  # this converts our 3D feature maps to 1D feature vectors\nmodel.add(Dense(64))\nmodel.add(Activation('relu'))\nmodel.add(Dropout(0.5))\nmodel.add(Dense(1))\nmodel.add(Activation('sigmoid'))\n\nmodel.compile(loss='binary_crossentropy',\n              optimizer='rmsprop',\n              metrics=['accuracy'])\n\n```\n\n\n```python\ndef setup_cat_dog_generator(processes=None, batch_size=32):\n    global pool\n    try:\n        pool.terminate()\n    except:\n        pass\n\n    if processes: \n        pool = multiprocessing.Pool(processes=processes, initializer=init_worker)\n    else:\n        pool = None\n\n    gen = T.ImageDataGenerator(\n         featurewise_center=False,\n         samplewise_center=False,\n         featurewise_std_normalization=False,\n         samplewise_std_normalization=False,\n         zca_whitening=False,\n         rotation_range=45,\n         width_shift_range=.1,\n         height_shift_range=.1,\n         shear_range=0.,\n         zoom_range=[.8, 1],\n         channel_shift_range=20,\n         fill_mode='nearest',\n         cval=0.,\n         horizontal_flip=True,\n         vertical_flip=False,\n         rescale=None,\n         preprocessing_function=preprocess_img,\n         dim_ordering='default',\n         pool=pool\n    )\n    test_gen = T.ImageDataGenerator(\n        preprocessing_function=preprocess_img,\n        pool=pool\n    )\n    \n    gen.fit(X_train)\n    test_gen.fit(X_train)\n    \n    X_train_aug = gen.flow_from_directory(\n        'data/train',\n        target_size=(299, 299),\n        batch_size=batch_size,\n        class_mode='binary')\n\n    X_test_aug = gen.flow_from_directory(\n        'data/validation',\n        target_size=(299, 299),\n        batch_size=batch_size,\n        class_mode='binary')\n    \n    return X_train_aug, X_test_aug\n```\n\n\n```python\ndef run_cat_dog_benchmark(processes=None, batch_size=32, vert=True, plot=True):\n    \n    X_train_aug, X_test_aug = setup_cat_dog_generator(processes=processes, batch_size=batch_size)\n    sys_mon = SM.SysMonitor()\n    sys_mon.start()\n    \n    try:        \n        model.fit_generator(\n                X_train_aug,\n                samples_per_epoch=22778,\n                nb_epoch=2,\n                validation_data=X_test_aug,\n                nb_val_samples=2222)\n    except KeyboardInterrupt:\n        print '\\n\\nTraining Interrupted\\n'\n        return None\n\n    sys_mon.stop()\n    \n    title = None\n    if not processes:\n        title = '{0:.2f} seconds of computation, no multiprocessing, batch size = {1}'.format(sys_mon.duration, batch_size)\n    else:\n        title = '{0:.2f} seconds of computation, using {1} processes, batch size = {2}'.format(sys_mon.duration, processes, batch_size)\n    \n    if plot:\n        sys_mon.plot(title, vert)\n    \n    if not processes:\n        processes = 0\n        \n    return {\n        'processes': processes,\n        'batch_size': batch_size,\n        'duration': sys_mon.duration,\n        'title': title\n    }\n```\n\nIn the following benchmark runs, you can see how inconsistent the GPU is being used without multiprocessing. Even with multiprocessing, the CPU is struggling to get enough data to the GPU to keep the GPU utilization stable. However, it's averaging out to be much higher than before.\n\nBefore running each benchmark, I run: \n\n`sync; echo 3 \u003e /proc/sys/vm/drop_caches` \n\nin the shell. This clears any diles that may be cached in memory that could be skewing the benchmarking results.\n\n\n```python\nruns = []\n```\n\n\n```python\nruns.append(run_cat_dog_benchmark(processes=None, batch_size=64))\n```\n\n    Found 22778 images belonging to 2 classes.\n    Found 2222 images belonging to 2 classes.\n    Epoch 1/2\n    22778/22778 [==============================] - 326s - loss: 0.6311 - acc: 0.6492 - val_loss: 0.5449 - val_acc: 0.7151\n    Epoch 2/2\n    22778/22778 [==============================] - 313s - loss: 0.5782 - acc: 0.7043 - val_loss: 0.5174 - val_acc: 0.7480\n\n\n\n![png](Accelerating%20Deep%20Learning%20with%20Multiprocess%20Image%20Augmentation%20in%20Keras_files/Accelerating%20Deep%20Learning%20with%20Multiprocess%20Image%20Augmentation%20in%20Keras_68_1.png)\n\n\n\n```python\nruns.append(run_cat_dog_benchmark(processes=7, batch_size=64))\n```\n\n    Found 22778 images belonging to 2 classes.\n    Found 2222 images belonging to 2 classes.\n    Epoch 1/2\n    22778/22778 [==============================] - 90s - loss: 0.5475 - acc: 0.7274 - val_loss: 0.4989 - val_acc: 0.7610\n    Epoch 2/2\n    22778/22778 [==============================] - 87s - loss: 0.5318 - acc: 0.7417 - val_loss: 0.4973 - val_acc: 0.7610\n\n\n\n![png](Accelerating%20Deep%20Learning%20with%20Multiprocess%20Image%20Augmentation%20in%20Keras_files/Accelerating%20Deep%20Learning%20with%20Multiprocess%20Image%20Augmentation%20in%20Keras_69_1.png)\n\n\n\n```python\nruns.append(run_cat_dog_benchmark(processes=11, batch_size=64))\n```\n\n    Found 22778 images belonging to 2 classes.\n    Found 2222 images belonging to 2 classes.\n    Epoch 1/2\n    22778/22778 [==============================] - 81s - loss: 0.5181 - acc: 0.7514 - val_loss: 0.5052 - val_acc: 0.7520\n    Epoch 2/2\n    22778/22778 [==============================] - 80s - loss: 0.5086 - acc: 0.7591 - val_loss: 0.4665 - val_acc: 0.7691\n\n\n\n![png](Accelerating%20Deep%20Learning%20with%20Multiprocess%20Image%20Augmentation%20in%20Keras_files/Accelerating%20Deep%20Learning%20with%20Multiprocess%20Image%20Augmentation%20in%20Keras_70_1.png)\n\n\n\n```python\nruns[0]['duration'] / runs[2]['duration']\n```\n\n\n\n\n    3.9467718654410233\n\n\n\nAs we can see, we can get an even bigger performance gain when flowing from disk. Using 11 processes, we are getting 3.94x performance over single threaded. This will really help a lot when working with larger than memory datasets.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstratospark%2Fkeras-multiprocess-image-data-generator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstratospark%2Fkeras-multiprocess-image-data-generator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstratospark%2Fkeras-multiprocess-image-data-generator/lists"}