{"id":18600831,"url":"https://github.com/autonomousvision/giraffe","last_synced_at":"2025-05-16T13:08:43.537Z","repository":{"id":37637729,"uuid":"351774576","full_name":"autonomousvision/giraffe","owner":"autonomousvision","description":"This repository contains the code for the CVPR 2021 paper \"GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields\"","archived":false,"fork":false,"pushed_at":"2022-02-09T08:06:14.000Z","size":30978,"stargazers_count":1243,"open_issues_count":8,"forks_count":156,"subscribers_count":21,"default_branch":"main","last_synced_at":"2025-04-12T10:57:50.902Z","etag":null,"topics":["cvpr2021","generative-adversarial-network","generative-model","generative-modelling","implicit-surfaces","nerf","neural-scene-representations"],"latest_commit_sha":null,"homepage":"https://m-niemeyer.github.io/project-pages/giraffe/index.html","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/autonomousvision.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-03-26T12:26:16.000Z","updated_at":"2025-04-04T06:32:31.000Z","dependencies_parsed_at":"2022-07-10T18:18:26.299Z","dependency_job_id":null,"html_url":"https://github.com/autonomousvision/giraffe","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/autonomousvision%2Fgiraffe","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/autonomousvision%2Fgiraffe/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/autonomousvision%2Fgiraffe/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/autonomousvision%2Fgiraffe/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/autonomousvision","download_url":"https://codeload.github.com/autonomousvision/giraffe/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254535828,"owners_count":22087399,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cvpr2021","generative-adversarial-network","generative-model","generative-modelling","implicit-surfaces","nerf","neural-scene-representations"],"created_at":"2024-11-07T02:05:47.299Z","updated_at":"2025-05-16T13:08:38.522Z","avatar_url":"https://github.com/autonomousvision.png","language":"Python","funding_links":[],"categories":["Papers"],"sub_categories":["NeRF Related Tasks"],"readme":"# GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields\n## [Project Page](https://m-niemeyer.github.io/project-pages/giraffe/index.html) | [Paper](http://www.cvlibs.net/publications/Niemeyer2021CVPR.pdf) | [Supplementary](http://www.cvlibs.net/publications/Niemeyer2021CVPR_supplementary.pdf) | [Video](http://www.youtube.com/watch?v=fIaDXC-qRSg\u0026vq=hd1080\u0026autoplay=1) | [Slides](https://m-niemeyer.github.io/slides/talks/giraffe/index.html) | [Blog](https://autonomousvision.github.io/giraffe/) | [Talk](https://www.youtube.com/watch?v=scnXyCSMJF4)\n![Add Clevr](gfx/add_clevr6.gif)\n![Tranlation Horizontal Cars](gfx/tr_d_cars.gif)\n![Interpolate Shape Faces](gfx/rotation_celebahq.gif)\n\nIf you find our code or paper useful, please cite as\n\n    @inproceedings{GIRAFFE,\n        title = {GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields},\n        author = {Niemeyer, Michael and Geiger, Andreas},\n        booktitle = {Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)},\n        year = {2021}\n    }\n\n## TL; DR - Quick Start\n\n![Rotating Cars](gfx/rotation_cars.gif)\n![Tranlation Horizontal Cars](gfx/tr_h_cars.gif)\n![Tranlation Horizontal Cars](gfx/tr_d_cars.gif)\n\nFirst you have to make sure that you have all dependencies in place. The simplest way to do so, is to use [anaconda](https://www.anaconda.com/).\n\nYou can create an anaconda environment called `giraffe` using\n```\nconda env create -f environment.yml\nconda activate giraffe\n```\n\nYou can now test our code on the provided pre-trained models.\nFor example, simply run\n```\npython render.py configs/256res/cars_256_pretrained.yaml\n```\nThis script should create a model output folder `out/cars256_pretrained`.\nThe animations are then saved to the respective subfolders in `out/cars256_pretrained/rendering`.\n\n## Usage\n\n### Datasets\n\nTo train a model from scratch or to use our ground truth activations for evaluation, you have to download the respective dataset.\n\nFor this, please run\n```\nbash scripts/download_dataset.sh\n```\nand following the instructions. This script should download and unpack the data automatically into the `data/` folder.\n\n\n### Controllable Image Synthesis\n\nTo render images of a trained model, run\n```\npython render.py CONFIG.yaml\n```\nwhere you replace `CONFIG.yaml` with the correct config file.\nThe easiest way is to use a pre-trained model.\nYou can do this by using one of the config files which are indicated with `*_pretrained.yaml`. \n\nFor example, for our model trained on Cars at 256x256 pixels, run\n```\npython render.py configs/256res/cars_256_pretrained.yaml\n```\nor for celebA-HQ at 256x256 pixels, run\n```\npython render.py configs/256res/celebahq_256_pretrained.yaml\n```\nOur script will automatically download the model checkpoints and render images.\nYou can find the outputs in the `out/*_pretrained` folders.\n\nPlease note that the config files  `*_pretrained.yaml` are only for evaluation or rendering, not for training new models: when these configs are used for training, the model will be trained from scratch, but during inference our code will still use the pre-trained model.\n\n### FID Evaluation\nFor evaluation of the models, we provide the script `eval.py`. You can run it using\n```\npython eval.py CONFIG.yaml\n```\nThe script generates 20000 images and calculates the FID score.\n\nNote: For some experiments, the numbers in the paper might slightly differ because we used the evaluation protocol from [GRAF](https://github.com/autonomousvision/graf) to fairly compare against the methods reported in GRAF.\n\n### Training\nFinally, to train a new network from scratch, run\n```\npython train.py CONFIG.yaml\n```\nwhere you replace `CONFIG.yaml` with the name of the configuration file you want to use.\n\nYou can monitor on \u003chttp://localhost:6006\u003e the training process using [tensorboard](https://www.tensorflow.org/guide/summaries_and_tensorboard):\n```\ncd OUTPUT_DIR\ntensorboard --logdir ./logs\n```\nwhere you replace `OUTPUT_DIR` with the respective output directory. For available training options, please take a look at `configs/default.yaml`.\n\n## 2D-GAN Baseline\n\nFor convinience, we have implemented a 2D-GAN baseline which closely follows this [GAN_stability repo](https://github.com/LMescheder/GAN_stability). For example, you can train a 2D-GAN on CompCars at 64x64 pixels similar to our GIRAFFE method by running\n```\npython train.py configs/64res/cars_64_2dgan.yaml\n```\n\n## Using Your Own Dataset\n\nIf you want to train a model on a new dataset, you first need to generate ground truth activations for the intermediate or final FID calculations.\nFor this, you can use the script in `scripts/calc_fid/precalc_fid.py`.\nFor example, if you want to generate an FID file for the comprehensive cars dataset at 64x64 pixels, you need to run\n```\npython scripts/precalc_fid.py  \"data/comprehensive_cars/images/*.jpg\" --regex True --gpu 0 --out-file \"data/comprehensive_cars/fid_files/comprehensiveCars_64.npz\" --img-size 64\n```\nor for LSUN churches, you need to run\n```\npython scripts/precalc_fid.py path/to/LSUN --class-name scene_categories/church_outdoor_train_lmdb --lsun True --gpu 0 --out-file data/church/fid_files/church_64.npz --img-size 64\n```\n\nNote: We apply the same transformations to the ground truth images for this FID calculation as we do during training. If you want to use your own dataset, you need to adjust the image transformations in the script accordingly. Further, you might need to adjust the object-level and camera transformations to your dataset. \n\n## Evaluating Generated Images\n\nWe provide the script `eval_files.py` for evaluating the FID score of your own generated images.\nFor example, if you would like to evaluate your images on CompCars at 64x64 pixels, save them to an npy file and run\n```\npython eval_files.py --input-file \"path/to/your/images.npy\" --gt-file \"data/comprehensive_cars/fid_files/comprehensiveCars_64.npz\"\n```\n\n# Futher Information\n\n## More Work on Implicit Representations\nIf you like the GIRAFFE project, please check out related works on neural representions from our group:\n- [Schwarz et. al. - GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis (NeurIPS'20)](https://github.com/autonomousvision/graf)\n- [Niemeyer et. al. - DVR: Learning Implicit 3D Representations without 3D Supervision (CVPR'20)](https://github.com/autonomousvision/differentiable_volumetric_rendering)\n- [Oechsle et. al. - Learning Implicit Surface Light Fields (3DV'20)](https://arxiv.org/abs/2003.12406)\n- [Peng et. al. - Convolutional Occupancy Networks (ECCV'20)](https://arxiv.org/abs/2003.04618)\n- [Niemeyer et. al. - Occupancy Flow: 4D Reconstruction by Learning Particle Dynamics (ICCV'19)](https://avg.is.tuebingen.mpg.de/publications/niemeyer2019iccv)\n- [Oechsle et. al. - Texture Fields: Learning Texture Representations in Function Space (ICCV'19)](https://avg.is.tuebingen.mpg.de/publications/oechsle2019iccv)\n- [Mescheder et. al. - Occupancy Networks: Learning 3D Reconstruction in Function Space (CVPR'19)](https://avg.is.tuebingen.mpg.de/publications/occupancy-networks)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fautonomousvision%2Fgiraffe","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fautonomousvision%2Fgiraffe","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fautonomousvision%2Fgiraffe/lists"}