{"id":13723919,"url":"https://github.com/ubc-vision/vivid123","last_synced_at":"2025-05-07T17:32:06.075Z","repository":{"id":211331129,"uuid":"724352437","full_name":"ubc-vision/vivid123","owner":"ubc-vision","description":"[CVPR 2024 Highlight] ViVid-1-to-3: Novel View Synthesis with Video Diffusion Models","archived":false,"fork":false,"pushed_at":"2024-07-24T01:41:47.000Z","size":27931,"stargazers_count":132,"open_issues_count":1,"forks_count":6,"subscribers_count":11,"default_branch":"main","last_synced_at":"2024-08-04T01:23:16.879Z","etag":null,"topics":["diffusion-models","novel-view-synthesis","single-view-3d","stable-diffusion","video-diffusion"],"latest_commit_sha":null,"homepage":"https://ubc-vision.github.io/vivid123/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ubc-vision.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-11-27T22:48:17.000Z","updated_at":"2024-08-03T03:29:44.000Z","dependencies_parsed_at":"2024-01-10T07:27:05.108Z","dependency_job_id":"8e513aec-5cc0-47dc-947a-69505520dbcd","html_url":"https://github.com/ubc-vision/vivid123","commit_stats":null,"previous_names":["ubc-vision/vivid123"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ubc-vision%2Fvivid123","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ubc-vision%2Fvivid123/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ubc-vision%2Fvivid123/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ubc-vision%2Fvivid123/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ubc-vision","download_url":"https://codeload.github.com/ubc-vision/vivid123/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224628215,"owners_count":17343293,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["diffusion-models","novel-view-synthesis","single-view-3d","stable-diffusion","video-diffusion"],"created_at":"2024-08-03T01:01:47.168Z","updated_at":"2024-11-14T13:31:17.953Z","avatar_url":"https://github.com/ubc-vision.png","language":"Python","funding_links":[],"categories":["\u003cspan id=\"model\"\u003e3D Model\u003c/span\u003e"],"sub_categories":["\u003cspan id=\"tool\"\u003eLLM (LLM \u0026 Tool)\u003c/span\u003e"],"readme":"# ViVid-1-to-3: Novel View Synthesis with Video Diffusion Models\n\nThis repository is a reference implementation for ViVid-1-to-3. It combines video diffusion with novel-view synthesis diffusion models for increased pose and appearace consistency.\n\n[[arXiv]](https://arxiv.org/abs/2312.01305), [[project page]](https://ubc-vision.github.io/vivid123/)\n\n## Requirements\n```bash\npip install torch \"diffusers==0.24\" transformers accelerate einops kornia imageio[ffmpeg] opencv-python pydantic scikit-image lpips\n```\n\n## Run single generation task\nPut the reference image to $IMAGE_PATH, and set the `input_image_path` in `scripts/task_example.yaml` to it. Then run\n```bash\npython run_generation.py --task_yaml_path=scripts/task_example.yaml\n```\n\n## Run batch generation tasks\nWe have supported running batch generation tasks on both PC and SLURM clusters.\n### Prepare batch generation config yaml file\nWe tested our method on 100 [GSO](https://app.gazebosim.org/GoogleResearch/fuel/collections/Scanned%20Objects%20by%20Google%20Research) objects. The list of the objects is in `scripts/gso_metadata_object_prompt_100.csv`, along with our labeled text prompts if you would like to test prompt-based generation yourself. We have rendered the 100 objects beforehand. It can be downloaded [here](https://drive.google.com/file/d/1A9PJDRD27igX5p88slWVF_QSDKxaZDCZ/view?usp=sharing). You can decompress the content into `gso-100`. Then simply run the following line to prepare a batch generation job on a PC:\n```bash\npython -m scripts.job_config_yaml_generation \n```\nOr run the following line to prepare a batch generation job on a SLURM cluster, which will move temporary stuff to `$SLURM_TMPDIR` of your cluster:\n```\npython -m scripts.job_config_yaml_generation --run_on_slurm\n```\nAll the yaml files will be generated in a new folder called `tasks_gso`.\n\nIf you want to run customized batch generation, simply add an entry in the `job_specs` list in the beginning of `scripts/job_config_yaml_generation.py` and run it with the same bash command. An example has been commented out in it.\n\n\n### Batch generation\nFor batch generation, run\n```bash\npython run_batch_generation.py --task_yamls_dir=tasks_gso --dataset_dir=gso-100 --output_dir=outputs --obj_csv_file=scripts/gso_metadata_object_prompt_100.csv\n```\n\n### Tips for scheduling batch generation on SLURM clusters\nIt takes about 1min30s to run one generation on a v100 gpu. If the number of generations is too large for each job you can schedule on a SLURM cluster, \nyou can split the dataset for each job using the `--run_from_obj_index` and `--run_to_obj_index` options. For example\n```bash\npython run_batch_generation.py --task_yamls_dir=tasks_gso --dataset_dir=gso-100 --output_dir=outputs --obj_csv_file=scripts/gso_metadata_object_prompt_100.csv --run_from_obj_index=0 --run_to_obj_index=50\n```\n\n### Run evaluation\n#### Get metrics for each object\nTo run evaluation for a batch generation, put the experiments you want to evaluate in the `eval_specs` list in `run_evaluation.py`. Make sure the `exp_name` key has the same value as that of your batch generation. Also, you should modify the `expdir` and `savedir` in `run_evaluation.py`. Suppose you want to run the $EXP_ID-th experiment in the list, then do the following:\n```bash\npython run_evaluation.py --exp_id $EXP_ID\n```\nAfter the evaluation is run, intermediate results on PSNR, SSIM, LPIPS, FOR_8, FOR_16 for each object will be put to `savedir`.\n#### Get stats for this experiment\nFinally, you can use `run_calculate_stats.py` to get the PSNR, SSIM, LPIPS, FOR_8, FOR_16 stats for this experiment on your whole dataset. Make sure to modify the `psnr_save_dir`, `lpips_save_dir`, `ssim_save_dir`, `for_8_save_dir`, `for_16_save_dir` in `run_calculate_stats.py` to match the folder storing the intermediate results from the last step.\n```bash\npython run_calculate_stats.py\n```\n\n\n\n## Acknowledgement\nThis repo is based on the Huggingface community [implementation](https://github.com/huggingface/diffusers/blob/main/examples/community/pipeline_zero1to3.py) and [converted weights](https://huggingface.co/bennyguo/zero123-xl-diffusers) of [Zero-1-to-3](https://github.com/cvlab-columbia/zero123), as well as the Huggingface community text-to-video model [Zeroscope v2](https://huggingface.co/cerspense/zeroscope_v2_576w). Thanks for their awesome works.\n\n## Citation\n\nIf you use this code in your research, please cite our paper:\n```\n@inproceedings{kwak2024vivid,\n  title={Vivid-1-to-3: Novel view synthesis with video diffusion models},\n  author={Kwak, Jeong-gi and Dong, Erqun and Jin, Yuhe and Ko, Hanseok and Mahajan, Shweta and Yi, Kwang Moo},\n  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},\n  pages={6775--6785},\n  year={2024}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fubc-vision%2Fvivid123","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fubc-vision%2Fvivid123","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fubc-vision%2Fvivid123/lists"}