{"id":18936175,"url":"https://github.com/aim-uofa/frozenrecon","last_synced_at":"2025-06-16T14:08:35.706Z","repository":{"id":187518840,"uuid":"674045145","full_name":"aim-uofa/FrozenRecon","owner":"aim-uofa","description":"[ICCV2023] 🧊FrozenRecon: Pose-free 3D Scene Reconstruction with Frozen Depth Models","archived":false,"fork":false,"pushed_at":"2024-08-23T00:43:38.000Z","size":8079,"stargazers_count":126,"open_issues_count":6,"forks_count":5,"subscribers_count":15,"default_branch":"main","last_synced_at":"2025-04-28T15:55:02.114Z","etag":null,"topics":["3d-reconstruction","3d-scene-reconstruction","monocular-depth-estimation","pose-estimation"],"latest_commit_sha":null,"homepage":"https://aim-uofa.github.io/FrozenRecon/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aim-uofa.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-08-03T02:48:56.000Z","updated_at":"2025-04-28T07:12:20.000Z","dependencies_parsed_at":"2023-12-26T15:12:03.692Z","dependency_job_id":"3e273ba5-25b8-4968-81c8-376b84b68f57","html_url":"https://github.com/aim-uofa/FrozenRecon","commit_stats":null,"previous_names":["aim-uofa/frozenrecon"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/aim-uofa/FrozenRecon","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aim-uofa%2FFrozenRecon","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aim-uofa%2FFrozenRecon/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aim-uofa%2FFrozenRecon/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aim-uofa%2FFrozenRecon/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aim-uofa","download_url":"https://codeload.github.com/aim-uofa/FrozenRecon/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aim-uofa%2FFrozenRecon/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260173939,"owners_count":22969870,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-reconstruction","3d-scene-reconstruction","monocular-depth-estimation","pose-estimation"],"created_at":"2024-11-08T12:06:15.350Z","updated_at":"2025-06-16T14:08:35.682Z","avatar_url":"https://github.com/aim-uofa.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n\u003ch1\u003e[ICCV2023] 🧊 FrozenRecon: Pose-free 3D Scene Reconstruction with Frozen Depth Models \u003c/h1\u003e\n\n[Guangkai Xu](https://github.com/guangkaixu/)\u003csup\u003e1*\u003c/sup\u003e, \u0026nbsp; \n[Wei Yin](https://yvanyin.net/)\u003csup\u003e2*\u003c/sup\u003e, \u0026nbsp; \n[Hao Chen](https://stan-haochen.github.io/)\u003csup\u003e3\u003c/sup\u003e, \u0026nbsp;\n[Chunhua Shen](https://cshen.github.io/)\u003csup\u003e3\u003c/sup\u003e, \u0026nbsp;\n[Kai Cheng](https://cklibra.github.io/)\u003csup\u003e1\u003c/sup\u003e, \u0026nbsp;\n[Feng Zhao](https://scholar.google.co.uk/citations?user=r6CvuOUAAAAJ\u0026hl=en/)\u003csup\u003e1\u003c/sup\u003e\n\n\u003csup\u003e1\u003c/sup\u003eUniversity of Science and Technology of China \u0026nbsp;\u0026nbsp; \n\u003csup\u003e2\u003c/sup\u003eDJI Technology \u0026nbsp;\u0026nbsp; \n\u003csup\u003e3\u003c/sup\u003eZhejiang University, China\n\n\n### [Project Page](https://aim-uofa.github.io/FrozenRecon/),   [arXiv](https://arxiv.org/abs/2308.05733), [Paper](https://openaccess.thecvf.com/content/ICCV2023/papers/Xu_FrozenRecon_Pose-free_3D_Scene_Reconstruction_with_Frozen_Depth_Models_ICCV_2023_paper.pdf), [Supplementary](https://openaccess.thecvf.com/content/ICCV2023/supplemental/Xu_FrozenRecon_Pose-free_3D_ICCV_2023_supplemental.pdf)\n\n#### Reconstruct your pose-free video with 🧊 FrozenRecon in ~20 minutes\n\u003c/div\u003e\n\u003cdiv align=\"center\"\u003e\n\u003cimg width=\"800\" alt=\"image\" src=\"figs/frozenrecon-demo.png\"\u003e\n\u003c/div\u003e\nWe propose a novel test-time optimization approach that can transfer the robustness of affine-invariant depth models such as LeReS to challenging diverse scenes while ensuring inter-frame consistency, with only dozens of parameters to optimize per video frame. Specifically, our approach involves freezing the pre-trained affine-invariant depth model's depth predictions, rectifying them by optimizing the unknown scale-shift values with a geometric consistency alignment module, and employing the resulting scale-consistent depth maps to robustly obtain camera poses and camera intrinsic simultaneously. Dense scene reconstruction demo is shown as below.\n\n\n## Prerequisite\n\n### Pre-trained Checkpoints\nIn this project, we use LeReS to predict affine-invariant depth maps. Please download the pre-trained [checkpoint](https://pan.baidu.com/s/1o2oVMiLRu770Fdpa65Pdbw?pwd=g3yi) of [LeReS](https://github.com/aim-uofa/AdelaiDepth/tree/main), and place it in `FrozenRecon/LeReS/res101.pth`. If optimize outdoor scenes, the [checkpoint](https://connecthkuhk-my.sharepoint.com/:f:/g/personal/xieenze_connect_hku_hk/Ept_oetyUGFCsZTKiL_90kUBy5jmPV65O5rJInsnRCDWJQ?e=CvGohw) of [Segformer](https://github.com/NVlabs/SegFormer) should also be downloaded and placed in `FrozenRecon/SegFormer/segformer.b3.512x512.ade.160k.pth`\n\n### Demo Data\nWe provide one demo data for each scene, and another in-the-wild video captured from iPhone14 Pro without any lidar sensor information. Download from [BaiduNetDisk](https://pan.baidu.com/s/1bSH8G0-ZQ8LpytZdZ6zIQw?pwd=e6hc), and place it in `FrozenRecon/demo_data`.\n\n### Installation\n```bash\ngit clone --recursive https://github.com/aim-uofa/FrozenRecon.git\ncd FrozenRecon\nconda create -y -n frozenrecon python=3.8\nconda activate frozenrecon\npip install torch==1.7.1+cu110 torchvision==0.8.2+cu110 -f https://download.pytorch.org/whl/torch_stable.html # pytorch 1.7.1 for SegFormer\npip install -r requirements.txt\n\n# (Optional) For outdoor scenes, we recommand to mask the sky regions and cars (potential dynamic objects)\npip install timm==0.3.2\npip install --upgrade mmcv-full==1.2.7 -f https://download.openmmlab.com/mmcv/dist/cu110/torch171/index.html\n# pip install \"mmsegmentation==0.11.0\"\npip install ipython attr \ngit clone https://github.com/NVlabs/SegFormer.git\ncd SegFormer \u0026\u0026 pip install -e . \u0026 cd ..\n# After installing SegFormer, please downlaod segformer.b3.512x512.ade.160k.pth checkpoint following https://github.com/NVlabs/SegFormer, and place it in SegFormer/\n\n# (Optional) Install lietorch. It can make optimization faster.\ngit clone --recursive https://github.com/princeton-vl/lietorch.git\ncd lietorch \u0026\u0026 python setup.py install \u0026 cd ..\n```\n\n## Optimization\n\n### 1. In-the-wild Video Input\n```bash\n# Take demo data as an example\npython src/optimize.py --video_path demo_data/IMG_8765.MOV\n\n# # For self-captured videos\n# python src/optimize.py --video_path PATH_TO_VIDEO --scene_name SCENE_NAME\n```\n\n### 2. In-the-wild Extracted Images Input\n```bash\npython src/optimize.py --img_root PATH_TO_IMG_FOLDER --scene_name SCENE_NAME\n```\n\n### 3. Datasets (Optional with GT Priors)\n```bash\n# Export ground-truth data root here.\nexport GT_ROOT='./demo_data' # PATH_TO_GT_DATA_ROOT, you can download demo_data following \"Data\" subsection.\n\n# FrozenRecon with datasets, take NYUDepthVideo classroom_0004 as example.\npython src/optimize.py --dataset_name NYUDepthVideo --gt_root $GT_ROOT --scene_name classroom_0004 \n\n# FrozenRecon with GT priors.\npython src/optimize.py --dataset_name NYUDepthVideo --gt_root $GT_ROOT --gt_intrinsic_flag --save_suffix gt_intrinsic --scene_name classroom_0004\npython src/optimize.py --dataset_name NYUDepthVideo --gt_root $GT_ROOT --gt_pose_flag --save_suffix gt_pose --scene_name classroom_0004\npython src/optimize.py --dataset_name NYUDepthVideo --gt_root $GT_ROOT --gt_depth_flag --save_suffix gt_depth --scene_name classroom_0004\npython src/optimize.py --dataset_name NYUDepthVideo --gt_root $GT_ROOT --gt_intrinsic_flag --gt_pose_flag --save_suffix gt_intrinsic_gt_pose --scene_name classroom_0004\npython src/optimize.py --dataset_name NYUDepthVideo --gt_root $GT_ROOT --gt_pose_flag --gt_depth_flag --save_suffix gt_depth_gt_pose --scene_name classroom_0004\npython src/optimize.py --dataset_name NYUDepthVideo --gt_root $GT_ROOT --gt_intrinsic_flag --gt_depth_flag --save_suffix gt_intrinsic_gt_depth --scene_name classroom_0004\npython src/optimize.py --dataset_name NYUDepthVideo --gt_root $GT_ROOT --gt_intrinsic_flag --gt_pose_flag --gt_depth_flag --save_suffix gt_intrinsic_gt_depth_gt_pose --scene_name classroom_0004\n```\n\n### 4. Outdoor Scenes\n```bash\nexport GT_ROOT='./demo_data' # PATH_TO_GT_DATA_ROOT, you can download demo_data following \"Data\" subsection.\n# We suggest to use GT intrinsic for stable optimization.\npython src/optimize.py --dataset_name NYUDepthVideo --scene_name SCENE_NAME --gt_root $GT_ROOT --gt_intrinsic_flag --scene_name 2011_09_26_drive_0001_sync --outdoor_scenes \n```\n\n\n\n## 🎫 License\nFor non-commercial academic use, this project is licensed under [the 2-clause BSD License](https://opensource.org/license/bsd-2-clause). \nFor commercial use, please contact [Chunhua Shen](mailto:chhshen@gmail.com).\n\n\n\n## 🖊️ Citation\n\nIf you find this project useful in your research, please cite:\n\n\n```BibTeX\n@inproceedings{xu2023frozenrecon,\n  title={FrozenRecon: Pose-free 3D Scene Reconstruction with Frozen Depth Models},\n  author={Xu, Guangkai and Yin, Wei and Chen, Hao and Shen, Chunhua and Cheng, Kai and Zhao, Feng},\n  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},\n  pages={9310--9320},\n  year={2023}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faim-uofa%2Ffrozenrecon","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faim-uofa%2Ffrozenrecon","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faim-uofa%2Ffrozenrecon/lists"}