{"id":15638149,"url":"https://github.com/tlatkowski/inpainting-gmcnn-keras","last_synced_at":"2025-06-14T04:07:37.616Z","repository":{"id":39737907,"uuid":"175416574","full_name":"tlatkowski/inpainting-gmcnn-keras","owner":"tlatkowski","description":"Keras implementation of \"Image Inpainting via Generative Multi-column Convolutional Neural Networks\" paper published at NIPS 2018 ","archived":false,"fork":false,"pushed_at":"2023-02-14T20:53:13.000Z","size":7957,"stargazers_count":94,"open_issues_count":12,"forks_count":39,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-04-15T06:15:31.772Z","etag":null,"topics":["cnn","computer-vision","convolutional-neural-networks","deep-learning","deep-neural-networks","gan","generative-adversarial-network","image-inpainting","improved-wasserstein","improved-wgan","inpainting","keras","neurips-2018","nips-2018","nvidia","places365","python3","tensorflow","wasserstein-gan","wgan-gp"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tlatkowski.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2019-03-13T12:29:05.000Z","updated_at":"2025-01-12T17:53:11.000Z","dependencies_parsed_at":"2024-01-27T01:05:22.068Z","dependency_job_id":"77460322-8572-4996-be67-7ee93234be9f","html_url":"https://github.com/tlatkowski/inpainting-gmcnn-keras","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/tlatkowski/inpainting-gmcnn-keras","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tlatkowski%2Finpainting-gmcnn-keras","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tlatkowski%2Finpainting-gmcnn-keras/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tlatkowski%2Finpainting-gmcnn-keras/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tlatkowski%2Finpainting-gmcnn-keras/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tlatkowski","download_url":"https://codeload.github.com/tlatkowski/inpainting-gmcnn-keras/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tlatkowski%2Finpainting-gmcnn-keras/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259756888,"owners_count":22906680,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cnn","computer-vision","convolutional-neural-networks","deep-learning","deep-neural-networks","gan","generative-adversarial-network","image-inpainting","improved-wasserstein","improved-wgan","inpainting","keras","neurips-2018","nips-2018","nvidia","places365","python3","tensorflow","wasserstein-gan","wgan-gp"],"created_at":"2024-10-03T11:19:30.411Z","updated_at":"2025-06-14T04:07:37.579Z","avatar_url":"https://github.com/tlatkowski.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"![](https://img.shields.io/badge/Python-3.6-blue.svg) ![](https://img.shields.io/badge/Keras-2.2.4-blue.svg) ![](https://img.shields.io/badge/TensorFlow-1.12.0-blue.svg) ![](https://img.shields.io/badge/License-MIT-blue.svg)\n\n# Generative Multi-column Convolutional Neural Networks inpainting model in Keras\n**Keras** implementation of **GMCNN** (Generative Multi-column Convolutional Neural Networks) inpainting model originally proposed at NIPS 2018:\n[Image Inpainting via Generative Multi-column Convolutional Neural Networks](https://arxiv.org/pdf/1810.08771.pdf)\n\n\n## Model architecture\n![GMCNN model](./pics/models/gmcnn_model.png)\n\n![](https://img.shields.io/badge/-convolution-yellow.svg)\n![](https://img.shields.io/badge/-dilated_convolution-red.svg)\n![](https://img.shields.io/badge/-up_scaling-blue.svg)\n![](https://img.shields.io/badge/-concatention-lightgrey.svg)\n## Installation\n* Code from this repository was tested on **Python 3.6** and **Ubuntu 14.04**\n* All required dependencies are stored in **requirements.txt**, **requirements-cpu.txt** and **requirements-gpu.txt** files.\n\nCode download:\n```bash\ngit clone https://github.com/tlatkowski/inpainting-gmcnn-keras.git\ncd inpainting-gmcnn-keras\n```\n\nTo install requirements, create Python virtual environment and install dependencies from files:\n```bash\nvirtualenv -p /usr/bin/python3.6 .venv\nsource .venv/bin/activate\npip install -r requirements/requirements.txt\n```\nIn case of using GPU support:\n```bash\npip install -r requirements/requirements-gpu.txt\n```\nOtherwise (CPU usage):\n```bash\npip install -r requirements/requirements-cpu.txt\n```\n\n\n## Datasets\n\n### Image dataset\nModel was trained with usage of high-resolution images from Places365-Standard dataset.\nIt can be found [here](http://places2.csail.mit.edu/download.html)\n\n\n### Mask dataset\nThe mask dataset used for model training comes from NVIDIA's paper: [Image Inpainting for Irregular Holes Using Partial Convolutions](https://eccv2018.org/openaccess/content_ECCV_2018/papers/Guilin_Liu_Image_Inpainting_for_ECCV_2018_paper.pdf)\n\nNVIDIA's mask dataset is available [here](http://masc.cs.gmu.edu/wiki/partialconv)\n\n**Please note that the model training was performed on testing irregular mask dataset containing  12,000 masks.**\n \n \n**./samples** folder contains exemplary structure of dataset catalogs:\n```bash\nsamples\n |-masks\n    |-nvidia_masks\n |-images\n    |-places365\n```\n**nvidia_masks** catalog contains 5 sample masks from NVIDIA's test set.\n\n**places365** catalog contains 5 sample images form Places365 validation set.\n## Model training\nThe main configuration file is placed in **./config/main_config.ini**. It contains training and model parameters. You can tweak those parameters before model running.\n\nThe default configuration looks as follows:\n```ini\n[TRAINING]\nWGAN_TRAINING_RATIO = 5\nNUM_EPOCHS = 5\nBATCH_SIZE = 4\nIMG_HEIGHT = 256\nIMG_WIDTH = 256\nNUM_CHANNELS = 3\nLEARNING_RATE = 0.0001\nSAVE_MODEL_STEPS_PERIOD = 1000\n\n[MODEL]\nADD_MASK_AS_GENERATOR_INPUT = False\nGRADIENT_PENALTY_LOSS_WEIGHT = 10\nID_MRF_LOSS_WEIGHT = 0.05\nADVERSARIAL_LOSS_WEIGHT = 0.001\nNN_STRETCH_SIGMA = 0.5\nVGG_16_LAYERS = 3,6,10\nID_MRF_STYLE_WEIGHT = 1.0\nID_MRF_CONTENT_WEIGHT = 1.0\nNUM_GAUSSIAN_STEPS = 3\nGAUSSIAN_KERNEL_SIZE = 32\nGAUSSIAN_KERNEL_STD = 40.0\n```\n\nAfter the dependencies installation you can perform training dry-run using image and mask samples provided in **samples** directory. To do so, execute the following command:\n\n**NOTE: Set BATCH_SIZE to 1 before executing the below command.**\n```bash\npython runner.py --train_path ./samples/images --mask_path ./samples/masks --experiment_name \"dry-run-test\"\n```\nIf everything goes correct you should be able to see the progress bar logging the basic training metrics.\n\nIn order to run GMCNN model training on your training data you have to provide paths to your datasets:\n```bash\npython runner.py --train_path /path/to/training/images --mask_path /path/to/mask/images --experiment_name \"experiment_name\"\n```\n\n### Warm-up generator training\nAccording to the best practices of the usage of GAN frameworks, first we should train the generator model for a while. In order to train the generator only in the first line run the following command (additional flag **warm_up_generator** is set):\n```bash\npython runner.py --train_path /path/to/training/images --mask_path /path/to/mask/images -warm_up_generator\n```\nIn this mode the generator will be trained with only confidence-driven reconstruction loss.\n\nBelow picture presents GMCNN outcome after 5 epochs training in warm-up generator mode\n![](https://github.com/tlatkowski/inpainting-gmcnn/blob/master/pics/outputs/warm_up_generator_5_epochs.jpg)\n\n### WGAN-GP training\nIn order to continue training with full WGAN-GP framework (GMCNN generator, local and global discriminators), execute:\n```bash\npython runner.py --train_path /path/to/training/images --mask_path /path/to/mask/images --experiment_name \"experiment_name\" -from_weights\n```\n\nRunning training with additional **from_weights** flag will force pipeline to load the latest models checkpoints from **./outputs/weights/** directory. \n\n### GMCNN model training in Google Colab notebook\nIf you don't have an access to workstation with GPU, you can use the below exemplary Google Colab notebook for training your GMCNN model on Places365 validation data and NVIDIA's testing mask with usage of K80 GPU available within Google Colab backend: [GMCNN in Google Colab](https://github.com/tlatkowski/inpainting-gmcnn-keras/blob/master/colab/Image_Inpainting_with_GMCNN_model.ipynb)\n\n\n\n### Pipeline outcomes\n\nDuring the training procedure the pipeline logs additional results to the **outputs** directory:\n* **outputs/experiment_name/logs** contains TensorBoard logs\n* **outputs/experiment_name/predicted_pics/warm_up_generator** contains the model predictions for the specific steps in the warm up generator training mode\n* **outputs/experiment_name/predicted_pics/wgan** contains the model predictions for the specific steps in the WGAN-GP training mode\n* **outputs/experiment_name/weights** contains the generator and critics models weights\n* **outputs/experiment_name/summaries** contains the generator and critics models summaries\n\nYou can track the metrics during the training with usage of TensorBoard:\n```bash\ntensorboard --logdir=./outputs/experiment_name/logs\n```\n\n## Implementation differences from original paper\n\n1. This model is trained using NVIDIA's irregular mask test set whereas the original model is trained using randomly generated rectangle masks. \n2. The current version of pipeline uses the higher-order features extracted from VGG16 model whereas the original model utilizes VGG19.\n\n## Visualization of Gaussian blurring masks \n\nBelow you can find the visualization of applying Gaussian blur to the training masks for the different number of convolution steps (number of iteration steps over the input raw mask). \n\n#### Large mask\nOriginal | 1 step | 2 steps | 3 steps | 4 steps | 5 steps | 10 steps\n------- |  ------- | ------- | ------- | ------- | ------- | ------- \n![](https://github.com/tlatkowski/inpainting-gmcnn/blob/master/pics/masks/large_mask_original.png) | ![](https://github.com/tlatkowski/inpainting-gmcnn/blob/master/pics/masks/large_blurred_mask_1_step.png) | ![](https://github.com/tlatkowski/inpainting-gmcnn/blob/master/pics/masks/large_blurred_mask_2_step.png) | ![](https://github.com/tlatkowski/inpainting-gmcnn/blob/master/pics/masks/large_blurred_mask_3_step.png) | ![](https://github.com/tlatkowski/inpainting-gmcnn/blob/master/pics/masks/large_blurred_mask_4_step.png) | ![](https://github.com/tlatkowski/inpainting-gmcnn/blob/master/pics/masks/large_blurred_mask_5_step.png) | ![](https://github.com/tlatkowski/inpainting-gmcnn/blob/master/pics/masks/large_blurred_mask_10_step.png)\n\n\n#### Small mask\nOriginal | 1 step | 2 steps | 3 steps | 4 steps | 5 steps | 10 steps\n------- | ------- | ------- | ------- | ------- | ------- | -------  \n![](https://github.com/tlatkowski/inpainting-gmcnn/blob/master/pics/masks/small_mask_original.png) | ![](https://github.com/tlatkowski/inpainting-gmcnn/blob/master/pics/masks/small_blurred_mask_1_step.png) | ![](https://github.com/tlatkowski/inpainting-gmcnn/blob/master/pics/masks/small_blurred_mask_2_step.png) | ![](https://github.com/tlatkowski/inpainting-gmcnn/blob/master/pics/masks/small_blurred_mask_3_step.png) | ![](https://github.com/tlatkowski/inpainting-gmcnn/blob/master/pics/masks/small_blurred_mask_4_step.png) | ![](https://github.com/tlatkowski/inpainting-gmcnn/blob/master/pics/masks/small_blurred_mask_5_step.png) | ![](https://github.com/tlatkowski/inpainting-gmcnn/blob/master/pics/masks/small_blurred_mask_10_step.png)\n\n#### Rectangle mask\nOriginal | 1 step | 2 steps | 3 steps | 4 steps | 5 steps | 10 steps\n------- | ------- | ------- | ------- | ------- | ------- | ------- \n![](https://github.com/tlatkowski/inpainting-gmcnn/blob/master/pics/masks/rectangle_mask_original.png) | ![](https://github.com/tlatkowski/inpainting-gmcnn/blob/master/pics/masks/rectangle_blurred_mask_1_step.png) | ![](https://github.com/tlatkowski/inpainting-gmcnn/blob/master/pics/masks/rectangle_blurred_mask_2_step.png) | ![](https://github.com/tlatkowski/inpainting-gmcnn/blob/master/pics/masks/rectangle_blurred_mask_3_step.png) | ![](https://github.com/tlatkowski/inpainting-gmcnn/blob/master/pics/masks/rectangle_blurred_mask_4_step.png) | ![](https://github.com/tlatkowski/inpainting-gmcnn/blob/master/pics/masks/rectangle_blurred_mask_5_step.png) | ![](https://github.com/tlatkowski/inpainting-gmcnn/blob/master/pics/masks/rectangle_blurred_mask_10_step.png)\n\n\n## Visualization of training losses\n\nAfter activating TensorBoard you can monitor the following training metrics:\n1. For the generator: confidence reconstruction loss, global wasserstein loss, local wasserstein loss, id mrf loss and total loss\n2. For the local and global discriminators: fake loss, real loss, gradient penalty loss and total loss\n\n![](https://github.com/tlatkowski/inpainting-gmcnn/blob/master/pics/tb_log.png)\n\n## Code References\n\n1. ID-MRF loss function was implemented with usage of original Tensorflow implementation: [GMCNN in Tensorflow](https://github.com/shepnerd/inpainting_gmcnn)\n2. Improved Wasserstain GAN was implemented based on: [Wasserstein GAN with gradient penalty in Keras](https://github.com/keras-team/keras-contrib/blob/master/examples/improved_wgan.py)\n3. Model architecture diagram was done with usage of PlotNeuralNet: [PlotNeuralNet on GitHub](https://github.com/HarisIqbal88/PlotNeuralNet)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftlatkowski%2Finpainting-gmcnn-keras","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftlatkowski%2Finpainting-gmcnn-keras","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftlatkowski%2Finpainting-gmcnn-keras/lists"}