{"id":13877861,"url":"https://github.com/paarthneekhara/text-to-image","last_synced_at":"2025-05-15T20:07:44.921Z","repository":{"id":49384659,"uuid":"65850354","full_name":"paarthneekhara/text-to-image","owner":"paarthneekhara","description":"Text to image synthesis using thought vectors","archived":false,"fork":false,"pushed_at":"2018-01-30T16:29:57.000Z","size":6325,"stargazers_count":2168,"open_issues_count":46,"forks_count":399,"subscribers_count":84,"default_branch":"master","last_synced_at":"2025-04-08T04:12:42.902Z","etag":null,"topics":["deep-learning","generative-adversarial-network","skip-thought-vectors","tensorflow"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/paarthneekhara.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-08-16T20:12:54.000Z","updated_at":"2025-04-02T02:04:28.000Z","dependencies_parsed_at":"2022-09-24T08:11:11.483Z","dependency_job_id":null,"html_url":"https://github.com/paarthneekhara/text-to-image","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paarthneekhara%2Ftext-to-image","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paarthneekhara%2Ftext-to-image/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paarthneekhara%2Ftext-to-image/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paarthneekhara%2Ftext-to-image/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/paarthneekhara","download_url":"https://codeload.github.com/paarthneekhara/text-to-image/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254414501,"owners_count":22067272,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","generative-adversarial-network","skip-thought-vectors","tensorflow"],"created_at":"2024-08-06T08:01:32.200Z","updated_at":"2025-05-15T20:07:39.889Z","avatar_url":"https://github.com/paarthneekhara.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# Text To Image Synthesis Using Thought Vectors\n\n[![Join the chat at https://gitter.im/text-to-image/Lobby](https://badges.gitter.im/text-to-image/Lobby.svg)](https://gitter.im/text-to-image/Lobby?utm_source=badge\u0026utm_medium=badge\u0026utm_campaign=pr-badge\u0026utm_content=badge)\n\nThis is an experimental tensorflow implementation of synthesizing images from captions using [Skip Thought Vectors][1]. The images are synthesized using the GAN-CLS Algorithm from the paper [Generative Adversarial Text-to-Image Synthesis][2]. This implementation is built on top of the excellent [DCGAN in Tensorflow][3]. The following is the model architecture. The blue bars represent the Skip Thought Vectors for the captions.\n\n![Model architecture](http://i.imgur.com/dNl2HkZ.jpg)\n\nImage Source : [Generative Adversarial Text-to-Image Synthesis][2] Paper\n\n## Requirements\n- Python 2.7.6\n- [Tensorflow][4]\n- [h5py][5]\n- [Theano][6] : for skip thought vectors\n- [scikit-learn][7] : for skip thought vectors\n- [NLTK][8] : for skip thought vectors\n\n## Datasets\n- All the steps below for downloading the datasets and models can be performed automatically by running `python download_datasets.py`. Several gigabytes of files will be downloaded and extracted.\n- The model is currently trained on the [flowers dataset][9]. Download the images from [this link][9] and save them in ```Data/flowers/jpg```. Also download the captions from [this link][10]. Extract the archive, copy the ```text_c10``` folder and paste it in ```Data/flowers```.\n- Download the pretrained models and vocabulary for skip thought vectors as per the instructions given [here][13]. Save the downloaded files in ```Data/skipthoughts```.\n- Make empty directories in Data, ```Data/samples```,  ```Data/val_samples``` and ```Data/Models```. They will be used for sampling the generated images and saving the trained models.\n\n## Usage\n- \u003cb\u003eData Processing\u003c/b\u003e : Extract the skip thought vectors for the flowers data set using :\n```\npython data_loader.py --data_set=\"flowers\"\n```\n- \u003cb\u003eTraining\u003c/b\u003e\n  * Basic usage `python train.py --data_set=\"flowers\"`\n  * Options\n      - `z_dim`: Noise Dimension. Default is 100.\n      - `t_dim`: Text feature dimension. Default is 256.\n      - `batch_size`: Batch Size. Default is 64.\n      - `image_size`: Image dimension. Default is 64.\n      - `gf_dim`: Number of conv in the first layer generator. Default is 64.\n      - `df_dim`: Number of conv in the first layer discriminator. Default is 64.\n      - `gfc_dim`: Dimension of gen untis for for fully connected layer. Default is 1024.\n      - `caption_vector_length`: Length of the caption vector. Default is 1024.\n      - `data_dir`: Data Directory. Default is `Data/`.\n      - `learning_rate`: Learning Rate. Default is 0.0002.\n      - `beta1`: Momentum for adam update. Default is 0.5.\n      - `epochs`: Max number of epochs. Default is 600.\n      - `resume_model`: Resume training from a pretrained model path.\n      - `data_set`: Data Set to train on. Default is flowers.\n      \n- \u003cb\u003eGenerating Images from Captions\u003c/b\u003e\n  * Write the captions in text file, and save it as ```Data/sample_captions.txt```. Generate the skip thought vectors for these captions using:\n  ```\n  python generate_thought_vectors.py --caption_file=\"Data/sample_captions.txt\"\n  ```\n  * Generate the Images for the thought vectors using:\n  ```\n  python generate_images.py --model_path=\u003cpath to the trained model\u003e --n_images=8\n  ```\n   ```n_images``` specifies the number of images to be generated per caption. The generated images will be saved in ```Data/val_samples/```. ```python generate_images.py --help``` for more options.\n\n## Sample Images Generated\nFollowing are the images generated by the generative model from the captions.\n\n| Caption        | Generated Images  |\n| ------------- | -----:|\n| the flower shown has yellow anther red pistil and bright red petals        | ![](http://i.imgur.com/SknZ3Sg.jpg)   |\n| this flower has petals that are yellow, white and purple and has dark lines        | ![](http://i.imgur.com/8zsv9Nc.jpg)   |\n| the petals on this flower are white with a yellow center        | ![](http://i.imgur.com/vvzv1cE.jpg)   |\n| this flower has a lot of small round pink petals.        | ![](http://i.imgur.com/w0zK1DC.jpg)   |\n| this flower is orange in color, and has petals that are ruffled and rounded.        | ![](http://i.imgur.com/VfBbRP1.jpg)   |\n| the flower has yellow petals and the center of it is brown        | ![](http://i.imgur.com/IAuOGZY.jpg)   |\n\n\n## Implementation Details\n- Only the uni-skip vectors from the skip thought vectors are used. I have not tried training the model with combine-skip vectors.\n- The model was trained for around 200 epochs on a GPU. This took roughly 2-3 days.\n- The images generated are 64 x 64 in dimension.\n- While processing the batches before training, the images are flipped horizontally with a probability of 0.5.\n- The train-val split is 0.75.\n\n## Pre-trained Models\n- Download the pretrained model from [here][14] and save it in ```Data/Models```. Use this path for generating the images.\n\n## TODO\n- Train the model on the MS-COCO data set, and generate more generic images.\n- Try different embedding options for captions(other than skip thought vectors). Also try to train the caption embedding RNN along with the GAN-CLS model. \n\n## References\n- [Generative Adversarial Text-to-Image Synthesis][2] Paper\n- [Generative Adversarial Text-to-Image Synthesis][11] Code\n- [Skip Thought Vectors][1] Paper\n- [Skip Thought Vectors][12] Code\n- [DCGAN in Tensorflow][3]\n- [DCGAN in Tensorlayer][15]\n\n## Alternate Implementations\n- [Text to Image in Torch by Scot Reed][11]\n- [Text to Image in Tensorlayer by Dong Hao][16]\n\n## License\nMIT\n\n\n[1]:http://arxiv.org/abs/1506.06726\n[2]:http://arxiv.org/abs/1605.05396\n[3]:https://github.com/carpedm20/DCGAN-tensorflow\n[4]:https://github.com/tensorflow/tensorflow\n[5]:http://www.h5py.org/\n[6]:https://github.com/Theano/Theano\n[7]:http://scikit-learn.org/stable/index.html\n[8]:http://www.nltk.org/\n[9]:http://www.robots.ox.ac.uk/~vgg/data/flowers/102/\n[10]:https://drive.google.com/file/d/0B0ywwgffWnLLcms2WWJQRFNSWXM/view\n[11]:https://github.com/reedscot/icml2016\n[12]:https://github.com/ryankiros/skip-thoughts\n[13]:https://github.com/ryankiros/skip-thoughts#getting-started\n[14]:https://bitbucket.org/paarth_neekhara/texttomimagemodel/raw/74a4bbaeee26fe31e148a54c4f495694680e2c31/latest_model_flowers_temp.ckpt\n[15]:https://github.com/zsdonghao/dcgan\n[16]:https://github.com/zsdonghao/text-to-image\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpaarthneekhara%2Ftext-to-image","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpaarthneekhara%2Ftext-to-image","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpaarthneekhara%2Ftext-to-image/lists"}