Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/google/sg2im
Code for "Image Generation from Scene Graphs", Johnson et al, CVPR 2018
https://github.com/google/sg2im
Last synced: 28 days ago
JSON representation
Code for "Image Generation from Scene Graphs", Johnson et al, CVPR 2018
- Host: GitHub
- URL: https://github.com/google/sg2im
- Owner: google
- License: apache-2.0
- Created: 2018-06-29T16:32:17.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2024-07-25T10:13:10.000Z (4 months ago)
- Last Synced: 2024-10-01T12:06:12.017Z (about 1 month ago)
- Language: Python
- Homepage:
- Size: 1.19 MB
- Stars: 1,294
- Watchers: 43
- Forks: 231
- Open Issues: 14
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project
- awesome-gcn - google/sg2im
README
# sg2im
This is the code for the paper
**Image Generation from Scene Graphs**
Justin Johnson,
Agrim Gupta,
Li Fei-Fei
Presented at [CVPR 2018](http://cvpr2018.thecvf.com/)
Please note that this is not an officially supported Google product.
A **scene graph** is a structured representation of a visual scene where nodes represent *objects* in the scene and edges represent *relationships* between objects. In this paper we present and end-to-end neural network model that inputs a scene graph and outputs an image.
Below we show some example scene graphs along with images generated from those scene graphs using our model. By modifying the input scene graph we can exercise fine-grained control over the objects in the generated image.
If you find this code useful in your research then please cite
```
@inproceedings{johnson2018image,
title={Image Generation from Scene Graphs},
author={Johnson, Justin and Gupta, Agrim and Fei-Fei, Li},
booktitle={CVPR},
year={2018}
}
```
## Model
The input scene graph is processed with a *graph convolution network* which passes information along edges to compute embedding vectors for all objects. These vectors are used to predict bounding boxes and segmentation masks for all objects, which are combined to form a coarse *scene layout*. The layout is passed to a *cascaded refinement network* (Chen an Koltun, ICCV 2017) which generates an output image at increasing spatial scales. The model is trained adversarially against a pair of *discriminator networks* which ensure that output images look realistic.
## Setup
All code was developed and tested on Ubuntu 16.04 with Python 3.5 and PyTorch 0.4.
You can setup a virtual environment to run the code like this:
```bash
python3 -m venv env # Create a virtual environment
source env/bin/activate # Activate virtual environment
pip install -r requirements.txt # Install dependencies
echo $PWD > env/lib/python3.5/site-packages/sg2im.pth # Add current directory to python path
# Work for a while ...
deactivate # Exit virtual environment
```
## Pretrained Models
You can download pretrained models by running the script `bash scripts/download_models.sh`. This will download the following models, and will require about 355 MB of disk space:
- `sg2im-models/coco64.pt`: Trained to generate 64 x 64 images on the COCO-Stuff dataset. This model was used to generate the COCO images in Figure 5 from the paper.
- `sg2im-models/vg64.pt`: Trained to generate 64 x 64 images on the Visual Genome dataset. This model was used to generate the Visual Genome images in Figure 5 from the paper.
- `sg2im-models/vg128.pt`: Trained to generate 128 x 128 images on the Visual Genome dataset. This model was used to generate the images in Figure 6 from the paper.
Table 1 in the paper presents an ablation study where we disable various components of the full model. You can download the additional models used in this ablation study by running the script `bash scripts/download_ablated_models.sh`. This will download 12 additional models, requiring and additional 1.25 GB of disk space.
## Running Models
You can use the script `scripts/run_model.py` to easily run any of the pretrained models on new scene graphs using a simple human-readable JSON format. For example you can replicate the sheep images above like this:
```bash
python scripts/run_model.py \
--checkpoint sg2im-models/vg128.pt \
--scene_graphs scene_graphs/figure_6_sheep.json \
--output_dir outputs
```
The generated images will be saved to the directory specified by the `--output_dir` flag. You can control whether the model runs on CPU or GPU using py passing the flag `--device cpu` or `--device gpu`.
We provide JSON files and pretrained models allowing you to recreate all images from Figures 5 and 6 from the paper.
#### (Optional): GraphViz
This script can also draw images for the scene graphs themselves using [GraphViz](http://www.graphviz.org/); to enable this option just add the flag `--draw_scene_graphs 1` and the scene graph images will also be saved in the output directory. For this option to work you must install GraphViz; on Ubuntu 16.04 you can simply run `sudo apt-get install graphviz`.
## Training new models
Instructions for training new models can be [found here](TRAINING.md).