{"id":13719233,"url":"https://github.com/shekkizh/Colorization.tensorflow","last_synced_at":"2025-05-07T11:31:22.933Z","repository":{"id":45751845,"uuid":"71087232","full_name":"shekkizh/Colorization.tensorflow","owner":"shekkizh","description":"Image colorization using CNNs in tensorflow","archived":false,"fork":false,"pushed_at":"2018-05-17T16:58:38.000Z","size":1850,"stargazers_count":73,"open_issues_count":1,"forks_count":20,"subscribers_count":6,"default_branch":"master","last_synced_at":"2024-11-13T09:09:07.344Z","etag":null,"topics":["convolutional-networks","image-colorization","tensorflow"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/shekkizh.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-10-17T00:52:20.000Z","updated_at":"2024-10-16T20:42:33.000Z","dependencies_parsed_at":"2022-09-22T22:00:46.867Z","dependency_job_id":null,"html_url":"https://github.com/shekkizh/Colorization.tensorflow","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shekkizh%2FColorization.tensorflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shekkizh%2FColorization.tensorflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shekkizh%2FColorization.tensorflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shekkizh%2FColorization.tensorflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/shekkizh","download_url":"https://codeload.github.com/shekkizh/Colorization.tensorflow/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224592519,"owners_count":17337106,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["convolutional-networks","image-colorization","tensorflow"],"created_at":"2024-08-03T01:00:44.778Z","updated_at":"2024-11-14T08:31:31.190Z","avatar_url":"https://github.com/shekkizh.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Image Colorization using Convolutional Networks\nThis is a long overdue project I have been trying to get working ever since I read the article by Ryan Dahl on the topic. The idea is to sematically understand concepts in an image and color them with statistical prior knowledge gained from training with a dataset.\n\nThe model was initially tested for architecture with the flowers dataset from tensorflow and later trained on the [LaMem dataset](http://memorability.csail.mit.edu) from MIT.\n\n1. [Results](#results)\n2. [Observations] (#observations)\n3. [Useful Links] (#useful-links)\n \n \n## Results \nThe idea for the model architecture is very similar to that of \"Residual Autoencoder\" model by Ryan with modifications. The intiution is to have the imagenet model output layer from convolution (encoder) and then do transpose convolution and fuse with pool layers. Unlike ryan I tried to fuse only the higher pooling layers and didn't want the lower level layers for color prediction - the thought being higher level layers are compositional and capture concepts and the lower level layers are closely related to the modalities of the image. The below is crude representation of the architecture.\n\nArchitecture:\n\u003e **Image (Only L value)** -\u003e 3x3 n64s1 -\u003e ReLu -\u003e VGG layer conv1_2 -\u003e .. -\u003e pool1 -\u003e .. -\u003e pool2 -\u003e .. -\u003e pool3 -\u003e .. -\u003e pool4 -\u003e .. -\u003e VGG relu5_4 -\u003e conv_T 3x3 -\u003e fuse with pool4 -\u003e conv_T 3x3 -\u003e fuse with pool3 -\u003e conv_T 3x3 -\u003e **a, b value prediction for Image**\n\nNote that there is no non-linearites added after the conv transpose operation. The loss was formulated as a simple MSE between the predicted a,b and the original a,b values for the image. Training was done for about 25 epochs with the learning rate halved every 3 epochs are so. Adam Optimizer was used and the initial learning rate was set to 1e-4. As seen below the training is very slow but produces reasonable results.\n\n![](results/loss.png)\n\n***Lab*** color space split for original colored image (top) and predicted colored image (bottom)\n\n![](results/lab_split1.png)\n![](results/lab_split2.png)\n\nFrom the above plot we can infer that the network predicted color values comes from some conceptual understanding of the object. A few colorization results are as below. Notice how in the case of the child face the model predicts color for an image that has no color. \n\n**Left: Input - Middle: Predicted Color - Right: Original Color**\n\n![](results/result1.png)\n\n![](results/result2.png)\n\n![](results/result3.png)\n\n![](results/result4.png)\n\n![](results/result5.png)\n\n![](results/result6.png)\n\n![](results/result7.png)\n\n## Observations\n- Creating a model from scratch and training was painfully slow. Tried the residual block architecture from Perceptual Losses for Real-Time Style Transfer by Johnson et. al which was incidentally used by folks from Twitter in their paper on Photo-Realistic Single Image Super-Resolution and Super-Resolution. \n  - The trained model from this architecture was able to produce green and blue colors at the right places after a few epochs but the training plateaued after that. \n- There were failures in the model based on VGG as can be seen from the image of truck and the toy although the colors were always consitent in the final model and didn't produce any patched color ouputs.\n- Prominent colors such as those for skies, trees and white background were the first colors to be learnt. Also skin tone for humans was noticeably learnt with good predictions amongst other things. \n\n----\nSome interesting outputs I got while working with flowers dataset. The outputs below are the results of applying the algorithm. The original colors for these flowers were nothing like that which was predicted :v:\n\n![](results/flowers/pred29.png) ![](results/flowers/pred30.png) ![](results/flowers/pred35.png) ![](results/flowers/pred39.png)\n\n## Useful Links\n- Colorful Image Colorization, Richard Zang et. al. - [link](http://richzhang.github.io/colorization/)\n- Ryan's blog post - [Automatic Colorization](http://tinyclouds.org/colorize/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshekkizh%2FColorization.tensorflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshekkizh%2FColorization.tensorflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshekkizh%2FColorization.tensorflow/lists"}