{"id":13422668,"url":"https://github.com/pathak22/context-encoder","last_synced_at":"2025-04-04T11:11:13.962Z","repository":{"id":41192361,"uuid":"56881233","full_name":"pathak22/context-encoder","owner":"pathak22","description":"[CVPR 2016] Unsupervised Feature Learning by Image Inpainting using GANs","archived":false,"fork":false,"pushed_at":"2020-07-19T21:37:48.000Z","size":6442,"stargazers_count":894,"open_issues_count":102,"forks_count":202,"subscribers_count":36,"default_branch":"master","last_synced_at":"2025-03-28T10:04:52.337Z","etag":null,"topics":["computer-graphics","computer-vision","context-encoders","dcgan","deep-learning","gan","generative-adversarial-network","image-inpainting","machine-learning","unsupervised-learning"],"latest_commit_sha":null,"homepage":"https://www.cs.cmu.edu/~dpathak/context_encoder/","language":"Lua","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pathak22.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-04-22T19:48:36.000Z","updated_at":"2025-03-21T14:53:29.000Z","dependencies_parsed_at":"2022-07-14T23:30:40.073Z","dependency_job_id":null,"html_url":"https://github.com/pathak22/context-encoder","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pathak22%2Fcontext-encoder","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pathak22%2Fcontext-encoder/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pathak22%2Fcontext-encoder/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pathak22%2Fcontext-encoder/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pathak22","download_url":"https://codeload.github.com/pathak22/context-encoder/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247166168,"owners_count":20894654,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-graphics","computer-vision","context-encoders","dcgan","deep-learning","gan","generative-adversarial-network","image-inpainting","machine-learning","unsupervised-learning"],"created_at":"2024-07-30T23:00:50.117Z","updated_at":"2025-04-04T11:11:13.936Z","avatar_url":"https://github.com/pathak22.png","language":"Lua","readme":"## Context Encoders: Feature Learning by Inpainting\n#### CVPR 2016\n#### [[Project Website]](https://www.cs.cmu.edu/~dpathak/context_encoder/) [[Imagenet Results]](https://www.cs.cmu.edu/~dpathak/context_encoder/#extraResults)\nSample results on held-out images: \n\n![teaser](images/teaser.jpg \"Sample inpainting results on held-out images\")\n\nThis is the training code for our [CVPR 2016 paper on Context Encoders](https://www.cs.cmu.edu/~dpathak/context_encoder/) for learning deep feature representation in an unsupervised manner by image inpainting. Context Encoders are trained jointly with reconstruction and adversarial loss. This repo contains quick demo, training/testing code for center region inpainting and training/testing code for arbitray random region inpainting. This code is adapted from an initial fork of [Soumith's DCGAN](https://github.com/soumith/dcgan.torch) implementation. Scroll down to try out a quick demo or train your own inpainting models!\n\nIf you find [Context Encoders](https://www.cs.cmu.edu/~dpathak/context_encoder/) useful in your research, please cite:\n\n    @inproceedings{pathakCVPR16context,\n        Author = {Pathak, Deepak and Kr\\\"ahenb\\\"uhl, Philipp and Donahue, Jeff and Darrell, Trevor and Efros, Alexei},\n        Title = {Context Encoders: Feature Learning by Inpainting},\n        Booktitle = {Computer Vision and Pattern Recognition ({CVPR})},\n        Year = {2016}\n    }\n\n### Contents\n1. [Semantic Inpainting Demo](#1-semantic-inpainting-demo)\n2. [Train Context Encoders](#2-train-context-encoders)\n3. [Download Features Caffemodel](#3-download-features-caffemodel)\n4. [TensorFlow Implementation](#4-tensorflow-implementation)\n5. [Project Website](#5-project-website)\n6. [Download Dataset](#6-paris-street-view-dataset)\n\n### 1) Semantic Inpainting Demo\n\n1. Install Torch:  http://torch.ch/docs/getting-started.html#_\n\n2. Clone the repository\n  ```Shell\n  git clone https://github.com/pathak22/context-encoder.git\n  ```\n  \n3. Demo\n  ```Shell\n  cd context-encoder\n  bash ./models/scripts/download_inpaintCenter_models.sh\n  # This will populate the `./models/` folder with trained models.\n\n  net=models/inpaintCenter/paris_inpaintCenter.t7 name=paris_result imDir=images/paris overlapPred=4 manualSeed=222 batchSize=21 gpu=1 th demo.lua\n  net=models/inpaintCenter/imagenet_inpaintCenter.t7 name=imagenet_result imDir=images/imagenet overlapPred=4 manualSeed=222 batchSize=21 gpu=1 th demo.lua\n  net=models/inpaintCenter/paris_inpaintCenter.t7 name=ucberkeley_result imDir=images/ucberkeley overlapPred=4 manualSeed=222 batchSize=4 gpu=1 th demo.lua\n  # Note: If you are running on cpu, use gpu=0\n  # Note: samples given in ./images/* are held-out images\n  ```\n\n### 2) Train Context Encoders\n\nIf you could successfully run the above demo, run following steps to train your own context encoder model for image inpainting.\n\n0. [Optional] Install Display Package as follows. If you don't want to install it, then set `display=0` in `train.lua`.\n  ```Shell\n  luarocks install https://raw.githubusercontent.com/szym/display/master/display-scm-0.rockspec\n  cd ~\n  th -ldisplay.start 8000\n  # if working on server machine create tunnel: ssh -f -L 8000:localhost:8000 -N server_address.com\n  # on client side, open in browser: http://localhost:8000/\n  ```\n\n1. Make the dataset folder.\n  ```Shell\n  mkdir -p /path_to_wherever_you_want/mydataset/train/images/\n  # put all training images inside mydataset/train/images/\n  mkdir -p /path_to_wherever_you_want/mydataset/val/images/\n  # put all val images inside mydataset/val/images/\n  cd context-encoder/\n  ln -sf /path_to_wherever_you_want/mydataset dataset\n  ```\n\n2. Train the model\n  ```Shell\n  # For training center region inpainting model, run:\n  DATA_ROOT=dataset/train display_id=11 name=inpaintCenter overlapPred=4 wtl2=0.999 nBottleneck=4000 niter=500 loadSize=350 fineSize=128 gpu=1 th train.lua\n\n  # For training random region inpainting model, run:\n  DATA_ROOT=dataset/train display_id=11 name=inpaintRandomNoOverlap useOverlapPred=0 wtl2=0.999 nBottleneck=4000 niter=500 loadSize=350 fineSize=128 gpu=1 th train_random.lua\n  # or use fineSize=64 to train to generate 64x64 sized image (results are better):\n  DATA_ROOT=dataset/train display_id=11 name=inpaintRandomNoOverlap useOverlapPred=0 wtl2=0.999 nBottleneck=4000 niter=500 loadSize=350 fineSize=64 gpu=1 th train_random.lua\n  ```\n\n3. Test the model\n  ```Shell\n  # For training center region inpainting model, run:\n  DATA_ROOT=dataset/val net=checkpoints/inpaintCenter_500_net_G.t7 name=test_patch overlapPred=4 manualSeed=222 batchSize=30 loadSize=350 gpu=1 th test.lua\n  DATA_ROOT=dataset/val net=checkpoints/inpaintCenter_500_net_G.t7 name=test_full overlapPred=4 manualSeed=222 batchSize=30 loadSize=129 gpu=1 th test.lua\n\n  # For testing random region inpainting model, run (with fineSize=64 or 124, same as training):\n  DATA_ROOT=dataset/val net=checkpoints/inpaintRandomNoOverlap_500_net_G.t7 name=test_patch_random useOverlapPred=0 manualSeed=222 batchSize=30 loadSize=350 gpu=1 th test_random.lua\n  DATA_ROOT=dataset/val net=checkpoints/inpaintRandomNoOverlap_500_net_G.t7 name=test_full_random useOverlapPred=0 manualSeed=222 batchSize=30 loadSize=129 gpu=1 th test_random.lua\n  ```\n\n### 3) Download Features Caffemodel\n\nFeatures for context encoder trained with reconstruction loss.\n\n- [Prototxt](https://www.cs.cmu.edu/~dpathak/context_encoder/resources/ce_features.prototxt)\n- [Caffemodel](https://www.cs.cmu.edu/~dpathak/context_encoder/resources/ce_features.caffemodel)\n\n### 4) TensorFlow Implementation\n\nCheckout the TensorFlow implementation of our paper by Taeksoo [here](https://github.com/jazzsaxmafia/Inpainting). However, it does not implement full functionalities of our paper.\n\n### 5) Project Website\n\nClick [here](https://www.cs.cmu.edu/~dpathak/context_encoder/).\n\n### 6) Paris Street-View Dataset\n\nPlease email me if you need the dataset and I will share a private link with you. I can't post the public link to this dataset due to the policy restrictions from Google Street View.\n","funding_links":[],"categories":["Cool Projects","Fundamental MIM Methods","Model Zoo"],"sub_categories":["MIM for Transformers and CNNs","ETC"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpathak22%2Fcontext-encoder","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpathak22%2Fcontext-encoder","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpathak22%2Fcontext-encoder/lists"}