{"id":18682525,"url":"https://github.com/postech-cvlab/nvsadapter","last_synced_at":"2025-04-12T04:21:43.062Z","repository":{"id":212009701,"uuid":"730489736","full_name":"POSTECH-CVLab/nvsadapter","owner":"POSTECH-CVLab","description":"[ECCV2024] Official Implementation of \"NVS-Adapter: Plug-and-Play Novel View Synthesis from a Single Image\"","archived":false,"fork":false,"pushed_at":"2024-12-04T07:30:37.000Z","size":70262,"stargazers_count":25,"open_issues_count":2,"forks_count":2,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-03-25T23:51:20.298Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://postech-cvlab.github.io/nvsadapter/","language":"CSS","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/POSTECH-CVLab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE-CODE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-12T03:19:04.000Z","updated_at":"2025-03-13T06:03:07.000Z","dependencies_parsed_at":null,"dependency_job_id":"38ddb91a-194b-4ec8-bcd7-756709347f37","html_url":"https://github.com/POSTECH-CVLab/nvsadapter","commit_stats":null,"previous_names":["postech-cvlab/nvsadapter.github.io"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/POSTECH-CVLab%2Fnvsadapter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/POSTECH-CVLab%2Fnvsadapter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/POSTECH-CVLab%2Fnvsadapter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/POSTECH-CVLab%2Fnvsadapter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/POSTECH-CVLab","download_url":"https://codeload.github.com/POSTECH-CVLab/nvsadapter/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248514436,"owners_count":21116963,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T10:12:14.418Z","updated_at":"2025-04-12T04:21:43.055Z","avatar_url":"https://github.com/POSTECH-CVLab.png","language":"CSS","funding_links":[],"categories":[],"sub_categories":[],"readme":"# NVS-Adapter: Plug-and-Play Novel View Synthesis from a Single Image\n\n\u003cp align=\"center\"\u003e\n  \u003cimg width=\"330\" height=\"330\" src=\"./assets/teaser.png\"\u003e\n\u003c/p\u003e\n\n## Abstract\n\nTransfer learning of large-scale Text-to-Image (T2I) models has recently shown impressive potential for Novel View Synthesis (NVS) of diverse objects from a single image. While previous methods typically train large models on multi-view datasets for NVS, fine-tuning the whole parameters of T2I models not only demands a high cost but also reduces the generalization capacity of T2I models in generating diverse images in a new domain. In this study, we propose an effective method, dubbed NVS-Adapter, which is a plug-and-play module for a T2I model, to synthesize novel multi-views of visual objects while fully exploiting the generalization capacity of T2I models. NVS-Adapter consists of two main components; view-consistency cross-attention learns the visual correspondences to align the local details of view features, and global semantic conditioning aligns the semantic structure of generated views with the reference view. Experimental results demonstrate that the NVS-Adapter can effectively synthesize geometrically consistent multi-views and also achieve high performance on benchmarks without full fine-tuning of T2I models.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg width=\"640\" height=\"460\" src=\"./assets/main_framework.png\"\u003e\n\u003c/p\u003e\n\n\n## Preparation\n\n### Installation\n\n```shell\nconda create -n nvs_adapter python=3.10 -c anaconda\nconda activate nvs_adapter\npip3 install -r requirements.txt\n```\n\n### Pretrained Checkpoints\n\nOur code requires several pretrained checkpoints depending on the model options. Our implementation assumes all the pretrained models are located inside the `checkpoints/` directory. \n\nAs a default, our model uses **pretrained SD2.1 checkpoint** and you can manually download the file [here](https://huggingface.co/stabilityai/stable-diffusion-2-1-base/blob/main/v2-1_512-ema-pruned.safetensors) and put the file inside `./checkpoints/`. Otherwise, you can use the script below:\n```\nmkdir -p ./checkpoints\nwget https://huggingface.co/stabilityai/stable-diffusion-2-1-base/resolve/main/v2-1_512-ema-pruned.safetensors?download=true -O ./checkpoints/v2-1_512-ema-pruned.safetensors\n```\n\nWhen you are training the model with **pretrained SD1.5 checkpoint**, you can instead use the file [here](https://huggingface.co/runwayml/stable-diffusion-v1-5/blob/main/v1-5-pruned-emaonly.ckpt). Otherwise, you can use the script below:\n```\nmkdir -p ./checkpoints\nwget https://huggingface.co/runwayml/stable-diffusion-v1-5/resolve/main/v1-5-pruned-emaonly.ckpt?download=true -O ./checkpoints/v1-5-pruned-emaonly.ckpt\n```\n\nWhen combining our model with ControlNet, two more pretrained checkpoints are required.\n- DPT Hybrid model: [here](https://github.com/intel-isl/DPT/releases/download/1_0/dpt_hybrid-midas-501f0c75.pt)\n- ControlNet (Depth) SD1.5 model: [here](https://huggingface.co/lllyasviel/ControlNet/resolve/main/models/control_sd15_depth.pth?download=true)\n\n```\nmkdir -p ./checkpoints\nwget https://github.com/intel-isl/DPT/releases/download/1_0/dpt_hybrid-midas-501f0c75.pt -O ./checkpoints/dpt_hybrid-midas-501f0c75.pt\nwget https://huggingface.co/lllyasviel/ControlNet/resolve/main/models/control_sd15_depth.pth?download=true -O ./checkpoints/control_sd15_depth.pth\n```\n\nDouble check that our model is compatible with ControlNet when the SD version is the same.\n\n### Dataset\n\nDownload objaverse renderings from [Zero-1-to-3 repository](https://github.com/cvlab-columbia/zero123?tab=readme-ov-file#dataset-objaverse-renderings).\n\nConvert objaverse renderings to WebDataset for training. As an example:\n```\npython3 scripts/objaverse_renderings_to_webdataset.py --paths_json_path=[path to json of file paths] --basepath=[basepath]\n\n# Example\npython3 scripts/objaverse_renderings_to_webdataset.py --paths_json_path=/data/objaverse/valid_paths.json --basepath=/data/objaverse/\n```\n\nWe will soon release our GSO rendered dataset.\n\n## Training\n\nFollowing the implementation from [SD-XL](https://github.com/Stability-AI/generative-models), we can put multiple configuration files when running a script. As an example:\n```\npython3 main.py -b [config_path1] [config_path2] [config_path3] -- [config.key=new_val] [config.key=new_val]\n\n# Example \npython3 main.py -b configs/base.yaml configs/ablation/camera_extrinsic.yaml configs/num_queries/2_queries.yaml -- trainer.devices=0, data.batch_size=4\n```\n\nOur 4-query base model:\n```\npython3 main.py -b configs/base.yaml\n```\n\nFor more scripts for other experiments, please refer to the scripts below.\n\u003cdetails open\u003e\n\n  \u003csummary\u003eMore scripts for training \u003c/summary\u003e\n\n  ```\n  # Ours with learnable query scale 0.5 (Table 3)\n  python3 main.py -b configs/base.yaml configs/ablation/query_emb_scale_half.yaml configs/num_queries/1_queries.yaml\n\n  # Ours with learnable query scale 2.0 (Table 3)\n  python3 main.py -b configs/base.yaml configs/ablation/query_emb_scale_2.yaml configs/num_queries/1_queries.yaml\n\n  # Ours 1 query model (Table 4)\n  python3 main.py -b configs/base.yaml configs/num_queries/1_queries.yaml\n\n  # Ours 2 query model (Table 4)\n  python3 main.py -b configs/base.yaml configs/num_queries/2_queries.yaml\n\n  # Ours 6 query model (Table 4)\n  python3 main.py -b configs/base.yaml configs/num_queries/6_queries.yaml\n\n  # Ours w/o GSC (Table 5)\n  python3 main.py -b configs/base.yaml configs/ablation/no_image_attn.yaml configs/num_queries/1_queries.yaml\n\n  # Ours with ray offset and ray direction (Table 6)\n  python3 main.py -b configs/base.yaml configs/ablation/rayo_rayd.yaml configs/num_queries/1_queries.yaml\n\n  # Ours with extrinsic camera params (Table 6)\n  python3 main.py -b configs/base.yaml configs/ablation/camera_extrinsic.yaml configs/num_queries/1_queries.yaml\n\n  # Ours with full fine-tuning (Table 7)\n  python3 main.py -b configs/base.yaml configs/options/full_finetune.yaml\n  ```\n\u003c/details\u003e\n\n## Evaluation\n\nOurs pretrained 4 query model and config files can be downloaded with the scrips below.\n\n```\nmkdir -p ./checkpoints\nwget https://twg.kakaocdn.net/brainrepo/models/nvsadapter/base_query_4_step_200000.ckpt -O ./checkpoints/base_query_4_step_200000.ckpt\nwget https://twg.kakaocdn.net/brainrepo/models/nvsadapter/base_query_4.yaml -O ./checkpoints/base_query_4.yaml\n```\n\nYou can synthesize novel view with the following script:\n```\npython3 scripts/novel_view_sampling.py --name nvs --input_image_path [path to image] --config_path [path to config] --ckpt_path [path to checkpoint] --elevations [elevation values in degree] --azimuths [azimuth values in degree] --cfg_scale [classifier free guidance scale in float] --use_ema\n\n# Example\npython3 scripts/novel_view_sampling.py --name nvs --input_image_path sample/deer.png --config_path checkpoints/base_query_4.yaml --ckpt_path checkpoints/base_query_4_step_200000.ckpt --elevations 0 0 0 0 --azimuths 30 60 90 120 --cfg_scale 3.0 --use_ema\n```\n\n## Demo\n\nWe also offer a demo that lets users easily test our model using any input image or viewpoint they choose.\nTo run the demo, you should first download the checkpoint and pass the downloaded path as an argument. \nYou can simply run the gradio demo with the command below:\n```\ngit submodule init\ngradio demo.py --ckpt_path [path to ckpt]\n\n# Example\ngradio demo.py --config checkpoints/base_query_4.yaml --ckpt_path checkpoints/base_query_4_step_200000.ckpt \n```\n\n## 3D Reconstruction using Score Distillation Sampling\n\nWe implemented score distillation sampling based on [threestudio](https://github.com/threestudio-project/threestudio).\n\nExample script for score distillation sampling:\n```\n# in 3drec/\npython3 launch.py --config configs/nvsadapter.yaml --train --gpu 0 data.image_path=[path to image]\n\n# Example\npython3 launch.py --config configs/nvsadapter.yaml --train --gpu 0 data.image_path=./load/images/a_beautiful_rainbow_fish_512_rgba.png\n```\n\nYou can find the results in `3drec/outputs`.\n\n## LoRA Injection\n\nWe can also combine our model with LoRA models to inject specific styles on generated images. \nFollow the step below to download LoRA models and inject it into our model.\n\n1. Search for your LoRA models in that are publicly available such as [civitai](https://civitai.com/tag/lora).\n2. Convert the LoRA model to a Diffuser-like format using `scripts/convert_lora_safetensor_to_diffusers.py` .\n3. Convert the Diffuer-like format to our format using `scripts/conver_diffusers_to_original_stalbe_diffusion.py`\n4. Run the script below\n```\npython3 scripts/novel_view_sampling.py --config_path configs/base_sd15.yaml --ckpt_path [path to our model] --use_ema --cfg_scale 11.0 --input_image_path [path to image] -n [store name] --elevation 0 0 0 0 --azimuths 72 144 216 288 --prompt \"\" -c [LoRA config path] --seed $SEED\n```\n5. Be sure your LoRA model is trained with the same version with our backbone SD model. \n\n\n## Licence\n\nThis project including the codes is distributed under the [MIT License](LICENSE-CODE), equivalent version of [Generative Models by Stability AI](https://github.com/Stability-AI/generative-models/blob/main/LICENSE-CODE), \nand the weights are distributed under the [Open RAIL++-M License](LICENSE-MODEL), equivalent version of [Stable Diffusion by Stability AI and Runway](https://github.com/CompVis/stable-diffusion/blob/main/LICENSE).\n\n## Citation\nIf you find this research useful, please cite:\n```\n@article{jeong2023nvsadapter,\n  title    = {NVS-Adapter: Plug-and-Play Novel View Synthesis from a Single Image},\n  author   = {Yoonwoo Jeong, Jinwoo Lee, Chiheon Kim, Minsu Cho, and Doyup Lee},\n  journal  = {arXiv preprint arXiv:2309.03453},\n  year     = {2023}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpostech-cvlab%2Fnvsadapter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpostech-cvlab%2Fnvsadapter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpostech-cvlab%2Fnvsadapter/lists"}