{"id":13442966,"url":"https://github.com/vincentfung13/MINE","last_synced_at":"2025-03-20T15:31:42.227Z","repository":{"id":37787993,"uuid":"389264363","full_name":"vincentfung13/MINE","owner":"vincentfung13","description":"Code and models for our ICCV 2021 paper \"MINE: Towards Continuous Depth MPI with NeRF for Novel View Synthesis\"","archived":false,"fork":false,"pushed_at":"2021-09-19T11:13:02.000Z","size":7584,"stargazers_count":408,"open_issues_count":15,"forks_count":43,"subscribers_count":16,"default_branch":"main","last_synced_at":"2024-10-28T06:00:17.933Z","etag":null,"topics":["3d-reconstruction","3d-vision","computer-vision","deep-learning","depth-estimation","nerf","novel-view-synthesis"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vincentfung13.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-07-25T04:50:31.000Z","updated_at":"2024-10-19T10:05:45.000Z","dependencies_parsed_at":"2022-08-09T12:40:49.151Z","dependency_job_id":null,"html_url":"https://github.com/vincentfung13/MINE","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vincentfung13%2FMINE","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vincentfung13%2FMINE/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vincentfung13%2FMINE/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vincentfung13%2FMINE/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vincentfung13","download_url":"https://codeload.github.com/vincentfung13/MINE/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244640073,"owners_count":20485978,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-reconstruction","3d-vision","computer-vision","deep-learning","depth-estimation","nerf","novel-view-synthesis"],"created_at":"2024-07-31T03:01:53.985Z","updated_at":"2025-03-20T15:31:39.383Z","avatar_url":"https://github.com/vincentfung13.png","language":"Python","funding_links":[],"categories":["Python","Papers"],"sub_categories":["NeRF Related Tasks"],"readme":"# MINE: Continuous-Depth MPI with Neural Radiance Fields\n### [Project Page](https://vincentfung13.github.io/projects/nemi/) | [YouTube](https://youtu.be/I_92BXju350) | [bilibili](https://www.bilibili.com/video/BV1qR4y1H7F1/)\nPyTorch implementation for our ICCV 2021 paper.\u003cbr\u003e\u003cbr\u003e\n[MINE: Towards Continuous Depth MPI with NeRF for Novel View Synthesis](https://vincentfung13.github.io/projects/nemi/)  \n [Jiaxin Li](https://www.jiaxinli.me/)\\*\u003csup\u003e1\u003c/sup\u003e,\n [Zijian Feng](https://vincentfung13.github.io/)\\*\u003csup\u003e1\u003c/sup\u003e,\n [Qi She](http://scholar.google.com/citations?user=iHoGTt4AAAAJ\u0026hl=en)\u003csup\u003e1\u003c/sup\u003e,\n [Henghui Ding](https://henghuiding.github.io/)\u003csup\u003e1\u003c/sup\u003e,\n [Changhu Wang](http://scholar.google.com.sg/citations?user=DsVZkjAAAAAJ\u0026hl=en)\u003csup\u003e1\u003c/sup\u003e,\n [Gim Hee Lee](https://www.comp.nus.edu.sg/~leegh)\u003csup\u003e2\u003c/sup\u003e \u003cbr\u003e\n \u003csup\u003e1\u003c/sup\u003eByteDance, \u003csup\u003e2\u003c/sup\u003eNational University of Singapore  \n  \\*denotes equal contribution  \n\nOur MINE takes a single image as input and densely reconstructs the frustum of the camera, through which we can easily render novel views of the given scene:\n\n![ferngif](resources/teasers.gif)\n\nThe overall architecture of our method:\n\n\u003cimg src='resources/pipeline.png'/\u003e\n\n## Run training on the LLFF dataset:\n\nFirstly, set up your conda environment:\n```\nconda env create -f environment.yml \nconda activate MINE\n```\n\nDownload the pre-downsampled version of the LLFF dataset from [Google Drive](https://drive.google.com/file/d/1sV7ioO_bintNg4U33YfUpFDD782OY8NI/view?usp=sharing), unzip it and put it in the root of the project, then start training by running the following command:\n```\nsh start_training.sh MASTER_ADDR=\"localhost\" MASTER_PORT=1234 N_NODES=1 GPUS_PER_NODE=2 NODE_RANK=0 WORKSPACE=/run/user/3861/vs_tmp DATASET=llff VERSION=debug EXTRA_CONFIG='{\"training.gpus\": \"0,1\"}'\n```\n\nYou may find the tensorboard logs and checkpoints in the sub-working directory (WORKSPACE + VERSION). \n\nApart from the LLFF dataset, we experimented on the RealEstate10K, KITTI Raw and the Flowers Light Fields datasets - the data pre-processing codes and training flow for these datasets will be released later.\n\n## Running our pretrained models:\n\nWe release the pretrained models trained on the RealEstate10K, KITTI and the Flowers datasets:\n\n|    Dataset    |  N | Input Resolution | Download Link |\n|:-------------:|:--:|:----------------:|:-------------:|\n| RealEstate10K | 32 |      384x256     |  [Google Drive](https://drive.google.com/drive/folders/1otJH4O_p6v96r-PHw_8c7dS-ketKHi2o?usp=sharing) |\n| RealEstate10K | 64 |      384x256     |  [Google Drive](https://drive.google.com/drive/folders/1bD-DRjoX7UcKTI2WjoDaU3lCXZBzoI7n?usp=sharing) |\n|     KITTI     | 32 |      768x256     |  [Google Drive](https://drive.google.com/drive/folders/1z91uK68D0NJOoWODm3_t1i7PGV6VitbN?usp=sharing) |\n|     KITTI     | 64 |      768x256     |  [Google Drive](https://drive.google.com/drive/folders/11VFBhycjLfycZI8IfL44pk9TwuqN8n0q?usp=sharing) |\n|    Flowers    | 32 |      512x384     |  [Google Drive](https://drive.google.com/drive/folders/10BHWynkL1XYMjGMtCwtUJ0zsIhhpMOnv?usp=sharing) |\n|    Flowers    | 64 |      512x384     |  [Google Drive](https://drive.google.com/drive/folders/1kjhGrLznurjaBk5zcibyMSG2UC7Hb-jr?usp=sharing) |\n\nTo run the models, download the checkpoint and the hyper-parameter yaml file and place them in the same directory, then run the following script:\n```\npython3 visualizations/image_to_video.py --checkpoint_path MINE_realestate10k_384x256_monodepth2_N64/checkpoint.pth --gpus 0 --data_path visualizations/home.jpg --output_dir .\n```\n\n\n## Citation\n\nIf you find our work helpful to your research, please cite our paper:\n```\n@inproceedings{mine2021,\n  title={MINE: Towards Continuous Depth MPI with NeRF for Novel View Synthesis},\n  author={Jiaxin Li and Zijian Feng and Qi She and Henghui Ding and Changhu Wang and Gim Hee Lee},\n  year={2021},\n  booktitle={ICCV},\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvincentfung13%2FMINE","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvincentfung13%2FMINE","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvincentfung13%2FMINE/lists"}