{"id":20213575,"url":"https://github.com/adobe-research/supergaussian","last_synced_at":"2025-09-20T03:35:12.506Z","repository":{"id":257421741,"uuid":"851844644","full_name":"adobe-research/SuperGaussian","owner":"adobe-research","description":"ECCV 2024 SuperGaussian for generic 3D upsampling","archived":false,"fork":false,"pushed_at":"2024-11-02T14:48:42.000Z","size":37096,"stargazers_count":59,"open_issues_count":4,"forks_count":2,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-03-24T12:39:08.507Z","etag":null,"topics":["3d","eccv2024","generative-modeling","generic-3d-upsampling","super-resolution","video-prior"],"latest_commit_sha":null,"homepage":"https://supergaussian.github.io/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/adobe-research.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-03T19:44:17.000Z","updated_at":"2025-03-21T02:06:05.000Z","dependencies_parsed_at":"2024-09-16T17:09:35.731Z","dependency_job_id":null,"html_url":"https://github.com/adobe-research/SuperGaussian","commit_stats":null,"previous_names":["adobe-research/supergaussian"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adobe-research%2FSuperGaussian","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adobe-research%2FSuperGaussian/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adobe-research%2FSuperGaussian/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adobe-research%2FSuperGaussian/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/adobe-research","download_url":"https://codeload.github.com/adobe-research/SuperGaussian/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248228530,"owners_count":21068706,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d","eccv2024","generative-modeling","generic-3d-upsampling","super-resolution","video-prior"],"created_at":"2024-11-14T06:10:00.971Z","updated_at":"2025-09-20T03:35:07.434Z","avatar_url":"https://github.com/adobe-research.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# SuperGaussian: Repurposing Video Models for 3D Super Resolution\n\nYuan Shen, Duygu Ceylan, Paul Guerrero, Zexiang Xu, Niloy J. Mitra, Shenlong Wang, and Anna Frühstück\n\nECCV 2024\n\n[Paper](https://arxiv.org/abs/2406.00609) │ [Project Page](https://supergaussian.github.io/)\n\nTLDR: Instead of image prior, we use video upsampling prior to achieve 3D upsampling on generic low-res 3D representations. \n\n![Teaser](assets/teaser.gif)\n\nIn this codebase, we provide a SuperGaussian implementation on a third-parties video upsampling prior, [RealBasicVSR](https://github.com/ckkelvinchan/RealBasicVSR). \nAdditionally, our codebase provides 3D upsampling evaluations on the MVImgNet testset given the upsampled image sequences from the image prior GigaGAN (image) and video prior Videogigagan (video).\nHopefully, this could help the community to reproduce our results and compare with other methods.\n\n## Dependencies\n\n### 0. Hardware Requirements\n- NVIDIA GPU with CUDA support 11.8. \n- The code has been tested with NVIDIA A6000 on Ubuntu 20.04.\n### 1. Install with Conda\n```bash\nconda create -n super_gaussian_eccv24 python=3.8 -y\n\nconda activate super_gaussian_eccv24\npip install torch==2.0.1 torchvision==0.15.2 --index-url https://download.pytorch.org/whl/cu118\n\nconda install cuda -c nvidia/label/cuda-11.8.0 -y\npip install tqdm plyfile==0.8.1 hydra-core==1.3.1 h5py==3.8.0 autolab_core==1.1.1 pyiqa timm==0.9.10 rich wandb lpips boto3 tyro\nexport CUDA_HOME=$(dirname $(dirname $(which python)))\ncd third_parties/gaussian-splatting/submodules/diff-gaussian-rasterization\nrm -rf build \u0026\u0026 pip install -e .\ncd ../simple-knn\nrm -rf build \u0026\u0026 pip install -e .\n\n# install RealBasicvsr in another conda env\nconda create -n realbasicvsr python=3.8 -y\nconda activate realbasicvsr\npip install torch==1.7.1+cu110 torchvision==0.8.2+cu110 torchaudio==0.7.2 -f https://download.pytorch.org/whl/torch_stable.html # it needs to be torch 1.7.1. For CuDA, it depends on your GPU compatibility.\npip install mmcv-full==1.5.0\npip install mmedit==0.15.0\n\n# install Evaluation dependencies in a third conda env\nconda create -n supergaussian_evaluation python=3.10 -y\nconda activate supergaussian_evaluation\npip install torch==2.0.1 torchvision==0.15.2 --index-url https://download.pytorch.org/whl/cu118\npip install pyiqa torchmetrics\npip install torch-fidelity\n\nconda activate super_gaussian_eccv24\n\n# change the python environment path in ``sg_utils/sg_helpers.py:20`` and ``sg_utils/sg_helpers.py:47`` to your conda environment path\n```\n\n### 2. Install with Docker (Recommended)\n```bash\nexport DATA_PATH=[your data path (see testdata downloading section)]\ndocker pull yshen47/adobe_supergaussian:latest\ndocker run -it -v $DATA_PATH:/mnt/data --shm-size=64g --gpus all yshen47/adobe_supergaussian:latest bash\ncd /root/SuperGaussian # By default, you should be at this directory\n```\n\n## Test data Download\n\n1. Download the [MVImgNet test set](https://uofi.box.com/s/d13gifxwz573cr6li37r1m39cczrvzhm) with 523 scenes. Unzip the dataset in the project root (using jar xf data.zip, otherwise unzip command has fork bomb complaints), i.e., ``SuperGaussian/data``. This should be the DATA_PATH to mount in your docker container. \nEach scene are grouped first by its category id and then scene id, defined by MVImgNet.\nThe test scenes are selected for its rich details, diversity and challenging scenarios. Within each scene folder, the directory are organized as follows. Please use our provided dataloader to load data (``dataset/mvimg_test_dataset.py``)\n```     \n--- [category_id]/[scene_id]\n    --- cam_extrinsics.pkl          # camera extrinsics\n    --- cam_intrinsics.pkl          # camera intrinsics\n    --- rgb.pkl                     # low-res 3DGS rgb\n    --- xyz.pkl                     # low-res 3DGS xyz(position)\n    --- novel_trajectory_poses      # we prepare several novel trajectory poses for evaluation to avoid overfit to gt trajectory. Trajectory specified in 000.json is used for test evaluation. \n        --- 0000.json\n        --- 0001.json\n        ...\n    --- gt_rgb                      # The original ground_truth RGB images in MVImgNet\n        --- 0000.png\n        --- 0001.png\n        ...\n    --- LR_131072_gaussian          # The low-res 3DGS renderings we prepaed in MVImgNet \n        --- 0000.png                # images without prefix are original RGB sequences in MVImgNet\n        --- 0001.png\n        ...\n        --- traj_0_000.png          # images with prefix 'traj_0' are the novel trajectory images in MVImgNet. traj_0 is used for test evaluation.\n        --- traj_0_001.png\n        ...\n        --- traj_1_000.png          # images with prefix 'traj_1' are the novel trajectory images in MVImgNet. traj_1 is added into the upsampling and 3D lifting.\n        --- traj_1_001.png\n        ...\n    --- HR_131072_gaussian          # The high-res 3DGS renderings we prepared in MVImgNet ( so it can be used as 'gt' on the test trajectory)\n        --- 0000.png                # images without prefix are original RGB sequences in MVImgNet\n        --- 0001.png\n        ...\n        --- traj_0_000.png          # images with prefix 'traj_0' are the novel trajectory images in MVImgNet. traj_0 is used for test evaluation.\n        --- traj_0_001.png\n        ...\n        --- traj_1_000.png          # images with prefix 'traj_1' are the novel trajectory images in MVImgNet. traj_1 is added into the upsampling and 3D lifting.\n        --- traj_1_001.png\n        ...\n        \n    --- gigagan_image               # The upsampled RGB images using image upsampling prior, [GigaGAN](https://mingukkang.github.io/GigaGAN/)\n        --- 0000.png                # images without prefix are original RGB sequences in MVImgNet\n        --- 0001.png\n        ...\n        --- traj_0_000.png          # images with prefix 'traj_0' are the novel trajectory images in MVImgNet. traj_0 is used for test evaluation.\n        --- traj_0_001.png\n        ...\n        --- traj_1_000.png          # images with prefix 'traj_1' are the novel trajectory images in MVImgNet. traj_1 is added into the upsampling and 3D lifting.\n        --- traj_1_001.png\n        ...\n    ... upsampled_rgb_videogigagan  # The upsampled RGB images using video upsampling prior, [VideoGigaGAN](https://videogigagan.github.io/)\n        --- 0000.png                # images without prefix are original RGB sequences in MVImgNet\n        --- 0001.png\n        ...\n        --- traj_0_000.png          # images with prefix 'traj_0' are the novel trajectory images in MVImgNet. traj_0 is used for test evaluation.\n        --- traj_0_001.png\n        ...\n        --- traj_1_000.png          # images with prefix 'traj_1' are the novel trajectory images in MVImgNet. traj_1 is added into the upsampling and 3D lifting.\n        --- traj_1_001.png\n        ...\n```\n## RealBasicVSR Checkpoints Download\nDownload the pre-trained weights to `third_parties/RealBasicVSR/checkpoints/`. ([Dropbox](https://www.dropbox.com/s/eufigxmmkv5woop/RealBasicVSR.pth?dl=0) / [Google Drive](https://drive.google.com/file/d/1OYR1J2GXE90Zu2gVU5xc0t0P_UmKH7ID/view) / [OneDrive](https://entuedu-my.sharepoint.com/:u:/g/personal/chan0899_e_ntu_edu_sg/EfMvf8H6Y45JiY0xsK4Wy-EB0kiGmuUbqKf0qsdoFU3Y-A?e=9p8ITR))\n\n## Evaluation\n1. To run inference on all our test scenes, you can use the following command. If you do not feel like running the inference, go to 2.\n```bash\nconda activate super_gaussian_eccv24\npython main_super_gaussian.py    # change upsampling_prior variable in Line 23 to switch between different priors.\nconda activate supergaussian_evaluation\npython evaluation.py             # change target variable in Line 94 to switch between different priors.\n```\nNote ``main_super_gaussian.py`` will create a folder in the root directory to store the upsampled 3D represented in 3DGS and final renderings for each scene. It also outputs a performance json averaged across each scene. And ``evaluation.py`` calculates performance by averaging image pairs across entire test set (instead of at scene-levels) \n\n2. We provide the above evaluation results for all priors, if you hope to make an Apple-to-Apple comparison between our SuperGaussian using VideoGigaGAN with your method. \nYou can download from links in the table below to get all our inference results on GigaGAN, VideoGigaGAN and RealBasicVSR. You are able to access 3D upsampled gaussians \nand final 4x renderings with all poses in the test scenes.\n\nWe recommend also trying to use Upscale-a-video video prior in our SuperGaussian framework, one alternative to VideoGigaGAN which should give much better performance than RealBasicVSR. \n\n| Priors       | Results after running ``main_super_gaussian.py``                                | Results after running ``evaluation.py`` |\n|--------------|-----------------------------------------------------------------------------| ---------|\n| GigaGAN      | [downloading link](https://uofi.box.com/s/cjqqlr0zfjw0ew02p1m5cm8ge6v73oow) | gigagan_prior folder in [link](https://uofi.box.com/s/cwjeo5sp6t2d81wcof0okqv6fn7aje8p) |\n| VideoGigaGAN | [downloading link](https://uofi.box.com/s/xma4iqhirebmtzzj01s40jnjdg12unnk) | videogigagan_prior folder in [link](https://uofi.box.com/s/cwjeo5sp6t2d81wcof0okqv6fn7aje8p) (the same link as the above, but different folder after unzipping) |\n| RealBasicVSR | [downloading link](https://uofi.box.com/s/lzakjip07upgullx7xunu6sausdydd95) | realbasicvsr_prior folder in [link](https://uofi.box.com/s/cwjeo5sp6t2d81wcof0okqv6fn7aje8p) (the same link as the above, but different folder after unzipping) |\n\n2.1 Here is detailed information on the output file structure after running ``main_super_gaussian.py``\n```\n--- [category_id]/[scene_id]\n    --- 64x64               # low-res images\n        --- 0000.png\n        --- 0001.png\n        ...\n    --- gt/high_res_images   # high-res images\n        --- 0000.png\n        --- 0001.png\n        ...\n    --- step_1_upsampling    # intermediate results after running the first step of upsampling, either image upsampling or video upsampling\n        --- 256x256           \n            --- 0000.png\n            --- 0001.png\n            ...\n    --- step_2_fitting_with_3dgs  # final results after running the second step of 3DDGS\n        --- point_cloud\n            --- iteration_0\n            --- iteration_2000\n                --- predicted              # renderings from upsampled 3DGS\n                    --- 000.png\n                    --- 001.png\n                    ...\n                point_cloud.ply            # upsampled 3DGS\n                performance_novel.json     # reference based performance averaged within this scene\n        --- cameras.json                 # camera extrinsics and intrinsics \n        --- input.ply                    # low-res 3DGS, you can also used color point cloud to initalize. \n    --- configs.json                     # configurations used for this scene   \n    --- transforms.json                  # camera info used for this scene\n    --- surface_pcd_131072_seed_0.ply    # low-res 3DGS, same as the one in input.ply (which is copied from this file)\n```\n2.2 Here is detailed information on the output file structure after running ``evaluation.py``\n```\n--- evaluations\n    --- gigagan_prior/realbasicvsr_prior/videogigagan_prior\n        --- gt                  # pseudo gt images rendered from high-res 3DGS on novel trajectory traj_0 sampled from all scenes for evaluation\n            --- XXX.png\n            --- XXX.png\n            ...\n        --- pred                # rendering from 3d upsampled gaussians on novel trajectory traj_0 sampled from all scenes for evaluation\n            --- XXX.png\n            --- XXX.png\n            ...\n            \n        --- gt_fid              # gt images in the original MVImgNet which are sampled from all scenes for evaluation on FID\n            --- XXX.png\n            --- XXX.png\n            ...\n```\n3. We provide the quantitative results using the above cached upsampling images below. Note RealBasicVSR is a third-party method, which we newly benchmarked. Meanwhile, we recommend also trying to use Upscale-a-video video prior in our SuperGaussian framework, one alternative to VideoGigaGAN which should give much better performance than RealBasicVSR.\n```\n| Priors       | LPIPS ↓ | NIQE  ↓ | FID  ↓ | \n|--------------|---------|---------|--------|\n| GigaGAN      | 0.1522  | 7.65    | 27.04  |\n| VideoGigaGAN | 0.1290  | 6.80    | 24.24  | \n| RealBasicVSR | 0.1924  | 7.58    | 41.40  |\n```\nNote some of the above numbers for GigaGAN and VideoGigaGAN are slightly better than the reported results in the paper, as we re-generated the above cached upsampled results in different dev environment used during our submission (some seeds might be different).\n\n## Citation\nIf you use this code for your research, please cite our paper.\n```\n@inproceedings{Shen2024SuperGaussian,\n  title = {SuperGaussian: Repurposing Video Models for 3D Super Resolution},\n  author = {Shen, Yuan and Ceylan, Duygu and Guerrero, Paul and Xu, Zexiang and Mitra, {Niloy J.} and Wang, Shenlong and Fr{\\“u}hst{\\“u}ck, Anna},\n  booktitle = {European Conference on Computer Vision (ECCV)},\n  year = {2024}\n}\n```\n## Acknowledgements\nThis project was the product of an internship by Yuan Shen with Adobe Research.\nWe would like to thank Yiran Xu, Difan Liu, and Taesung Park for their discussions related to VideoGigaGAN. Their support has been essential to\nthe development and success of SuperGaussian. We also thank Nathan Carr for providing Gaussian Splat examples. \nFinally, we thank the authors of RealBasicVSR and 3D Gaussian Splats for open-sourcing their codebases, which we modified on top of for this codebase. \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fadobe-research%2Fsupergaussian","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fadobe-research%2Fsupergaussian","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fadobe-research%2Fsupergaussian/lists"}