{"id":20516386,"url":"https://github.com/nianticlabs/implicit-depth","last_synced_at":"2025-10-07T19:32:35.555Z","repository":{"id":164597242,"uuid":"630405389","full_name":"nianticlabs/implicit-depth","owner":"nianticlabs","description":"[CVPR 2023] Virtual Occlusions Through Implicit Depth","archived":false,"fork":false,"pushed_at":"2024-05-30T12:04:47.000Z","size":15351,"stargazers_count":81,"open_issues_count":1,"forks_count":4,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-03-27T14:51:50.856Z","etag":null,"topics":["augmented-reality","computer-vision","cvpr2023","depth-estimation","machine-learning","occlusion","pytorch"],"latest_commit_sha":null,"homepage":"https://nianticlabs.github.io/implicit-depth/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nianticlabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-04-20T10:03:06.000Z","updated_at":"2025-01-17T14:46:38.000Z","dependencies_parsed_at":null,"dependency_job_id":"2fb57ad4-c2c4-453e-a662-7da2226c5fe4","html_url":"https://github.com/nianticlabs/implicit-depth","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nianticlabs%2Fimplicit-depth","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nianticlabs%2Fimplicit-depth/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nianticlabs%2Fimplicit-depth/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nianticlabs%2Fimplicit-depth/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nianticlabs","download_url":"https://codeload.github.com/nianticlabs/implicit-depth/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248802504,"owners_count":21163858,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["augmented-reality","computer-vision","cvpr2023","depth-estimation","machine-learning","occlusion","pytorch"],"created_at":"2024-11-15T21:28:35.776Z","updated_at":"2025-10-07T19:32:30.523Z","avatar_url":"https://github.com/nianticlabs.png","language":"Python","readme":"# Virtual Occlusions Through Implicit Depth – CVPR 2023\n\nThis is the reference PyTorch implementation for training and testing our implcit depth estimation system:\n\n\u003e **Virtual Occlusions Through Implicit Depth**\n\u003e\n\u003e [Jamie Watson](https://www.linkedin.com/in/jamie-watson-544825127/), [Mohamed Sayed](https://masayed.com), [Zawar Qureshi](https://qureshizawar.github.io/), [Gabriel J Brostow](http://www0.cs.ucl.ac.uk/staff/g.brostow/),\n[Sara Vicente](https://scholar.google.co.uk/citations?user=7wWsNNcAAAAJ\u0026hl=en), [Oisin Mac Aodha](https://homepages.inf.ed.ac.uk/omacaod/) and [Michael Firman](http://www.michaelfirman.co.uk)\n\u003e\n\u003e [Paper, CVPR 2023 (arXiv pdf w/ supp)](https://arxiv.org/pdf/2305.07014.pdf), [Project Page](https://nianticlabs.github.io/implicit-depth/), [Video](https://www.youtube.com/watch?v=-hcvZrXO8wM)\n\nhttps://user-images.githubusercontent.com/6114258/233341766-a6b465a1-f796-4ca4-a9b4-a497bf75eb1a.mp4\n\nThis code is for non-commercial use; please see the [license file](LICENSE) for terms. If you do find any part of this codebase helpful, please cite our paper using the BibTex below and link to this repo. Thanks!\n\n## ⚙️ Setup\n\nAssuming a fresh [Anaconda](https://www.anaconda.com/download/) distribution, you can install dependencies with:\n\n```shell\nconda env create -f binarydepth_env.yml\nconda activate binarydepth\n```\n\n## Data\nTo download and prepare the ScanNetv2 dataset, please follow the instructions on the [SimpleRecon](https://github.com/nianticlabs/simplerecon) repo.\nYou will need to update the data configs to point to the location of your \nScanNetv2 data. You can do this by setting `dataset_path: \u003cYOUR_DATA_LOCATION\u003e` in the six \n`configs/data/scannet_*.yaml` files.\n\n# Hypersim\n\nTo Download the Hypersim dataset, please follow the the intructions in the [Hypersim Datatset](https://github.com/apple/ml-hypersim) repo.\n\nOnce the dataset has been downloaded and extracted, please update the `dataset_path` argument for Hypersim data configs in `configs/data/` to point to the extracted dataset.\n\nNote that the depth maps provided as part of the dataset are not planar depths and need to be planarised. We have provided helper functions to planarise the depth maps (see `_get_prependicular_depths` method in `datasets/hypersim_dataset.py`). The planarised depth maps can be generated with the help of the `data_scripts/generate_hypersim_planar_depths.py` script:\n\n```bash\n# train\npython ./data_scripts/generate_hypersim_planar_depths.py \\\n    --data_config configs/data/hypersim_default_train.yaml \\\n    --num_workers 8\n\n# val\npython ./data_scripts/generate_hypersim_planar_depths.py \\\n    --data_config configs/data/hypersim_default_val.yaml \\\n    --num_workers 8\n```\n\nNext, we need to generate the frame tuples similarly to ScanNetv2 dataset:\n\n```bash\n# train\npython ./data_scripts/generate_train_tuples.py\n    --data_config configs/data/hypersim_default_train.yaml\n    --num_workers 8\n\n# val\npython ./data_scripts/generate_val_tuples.py\n    --data_config configs/data/hypersim_default_val.yaml\n    --num_workers 8\n```\n\nAfter the tuple generation, you should be ready to train on hypersim using the provided configs!\n\nWe provide the train and val splits we used for our experiments (see `data_splits/hypersim/bd_split/train_files_bd.json` and `data_splits/hypersim/bd_split/val_files_bd.json`)\n## 📦 Models\n\nWe provide the following pretrained models for you to try out - we suggest using the HyperSim trained model to obtain the best qualitative results.\n\n| Model Type                                                                                                                                                                 | Training Data | Temporal Smoothing |\n|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|--------------------|\n| [Implicit Depth (Ours - best qualitative results)](https://storage.googleapis.com/niantic-lon-static/research/implicit-depth/models/implicit_depth_temporal_hypersim.ckpt) | HyperSim      | Yes                |\n| [Implicit Depth (Ours)](https://storage.googleapis.com/niantic-lon-static/research/implicit-depth/models/implicit_depth_temporal.ckpt)                                     | ScanNet       | Yes                |\n| [Implicit Depth (Ours)](https://storage.googleapis.com/niantic-lon-static/research/implicit-depth/models/implicit_depth.ckpt)                                              | ScanNet       | No                 |\n| [Regression](https://storage.googleapis.com/niantic-lon-static/research/implicit-depth/models/regression.ckpt)                                                                                                                                                             | ScanNet       | No                 |\n\nDownload these to the `weights/` folder.\n\n## 📊 Evaluation\nAfter downloading the models, you can run our occlusion evaluation as described in the paper with:\n\n```bash\nCUDA_VISIBLE_DEVICES=0 python test_bd.py --name implicit_depth \\\n            --output_base_path outputs \\\n            --config_file configs/models/implicit_depth.yaml \\\n            --load_weights_from_checkpoint weights/implicit_depth.ckpt \\\n            --data_config configs/data/scannet_default_test.yaml \\\n            --num_workers 8 \\\n            --batch_size 4;\n\n```\nTo run depth evaluation add the flag `--binary_eval_depth`.\n\nTo instead run our temporal evaluation, use:\n```bash\nCUDA_VISIBLE_DEVICES=0 python test_bd.py --name implicit_depth \\\n            --output_base_path outputs \\\n            --config_file configs/models/implicit_depth.yaml \\\n            --load_weights_from_checkpoint weights/implicit_depth.ckpt \\\n            --data_config configs/data/scannet_default_test.yaml \\\n            --num_workers 8 \\\n            --batch_size 4 \\\n            --temporal_evaluation \\\n            --mv_tuple_file_suffix _eight_view_deepvmvs_dense.txt;\n\n```\n\nWe can also evaluate regression models using:\n```bash\nCUDA_VISIBLE_DEVICES=0 python test_reg.py --name regression \\\n            --output_base_path outputs \\\n            --config_file configs/models/regression.yaml \\\n            --load_weights_from_checkpoint weights/regression.ckpt \\\n            --data_config configs/data/scannet_default_test.yaml \\\n            --num_workers 8 \\\n            --batch_size 4 \\\n            --regression_plane_eval;\n\n```\n\n## ⚙️ Inference\n\nFirst, download our example data from [here](https://storage.googleapis.com/niantic-lon-static/research/implicit-depth/example_data.zip), and extract in the current folder. This should create an `example_data` folder here.\n\nThen run the following steps to prepare the dataset.\n\n```bash\n# Create a config and txt file specifically for this single sequence\npython -m inference.make_config_and_txt_file \\\n    --input-sequence-dir example_data/scans/garden_chair \\\n    --save-dir example_data/scans/config\n\n# Find keyframes (test tuples) for this sequence, and save to disk\npython ./data_scripts/generate_test_tuples.py \\\n    --data_config example_data/scans/config/config.yaml \\\n    --num_workers 16\n```\n\nNext we can run inference on each frame in the sequence with the following.\n\nOur implicit depth model requires a depth map as input for each frame, for example as rendered from an AR asset in the scene.\nThese are loaded from the folder at `--rendered_depth_map_load_dir`.\nThe `--rendered_depth_map_load_dir` flag is optional; if it isn't specified, then a fixed plane at 2m from the camera is used as the depth input for each inference frame.\nYou should also download our hypersim model from [here](https://storage.googleapis.com/niantic-lon-static/research/implicit-depth/models/implicit_depth_temporal_hypersim.ckpt).\n\n```bash\nCUDA_VISIBLE_DEVICES=0 python -m inference.inference \\\n    --config configs/models/sr_bd_high_res_sup_pretrained_7525.yaml \\\n    --load_weights_from_checkpoint weights/implicit_depth_temporal_hypersim.ckpt \\\n    --data_config example_data/scans/config/config.yaml \\\n    --rendered_depth_map_load_dir example_data/renders \\\n    --use_prior \\\n    --output_base_path example_data/predictions/ \\\n    --dataset_path example_data/scans\n```\n\n## 🖼️ Compositing\n\n![Example render](inference/example_render.gif)\n\nTo make AR visualisations such as the above video, you need three things:\n\n1) A folder containing:\n    1) RGB images of a real scene, and\n    2) A json file with camera intrinsics and extrinsics (see example file below for expected format)\n2) A folder of RGB and depth renderings of a virtual object for the scene.\n    1) These are assumed to be `.png` and `.npy` files respectively, named according to the same convention as the real RGB images.\n3) Either:\n    1) A folder of compositing masks, e.g. as predicted by our inference code, or\n    2) A folder of depth maps, e.g. as predicted by a regression baseline.\n\nYou can download our example data [here](https://storage.googleapis.com/niantic-lon-static/research/implicit-depth/example_data.zip).\n\nHere are some example commands to run compositing. These should save `mp4` files into the `--out-dir` folders.\n\n```bash\nSEQUENCE_DIR=example_data/scans/garden_chair\nRENDERS_DIR=example_data/renders\n\n# Composite with predicted *masks* (e.g. from our implicit depth method)\npython -m inference.composite \\\n    --predicted-masks-dir example_data/reference_predictions/implicit_depth \\\n    --renders-dir $RENDERS_DIR \\\n    --vdr-dir $SEQUENCE_DIR \\\n    --out-dir example_data/composited/implicit_depth\n\n# Composite with predicted *depths* (e.g. from a depth regression baseline)\npython -m inference.composite \\\n    --predicted-depths-dir example_data/reference_predictions/simplerecon_regression \\\n    --renders-dir $RENDERS_DIR \\\n    --vdr-dir $SEQUENCE_DIR \\\n    --out-dir example_data/composited/regression_baseline\n\n# Composite with *lidar* (from apple iphone)\npython -m inference.composite \\\n    --renders-dir $RENDERS_DIR \\\n    --vdr-dir $SEQUENCE_DIR \\\n    --out-dir example_data/composited/lidar_baseline\n```\n\nThe commands above use reference depth and mask predictions given in our example data download.\nYou can instead use the predictions you made in the 'Inference' step above, by using:\n\n```bash\n    --predicted-masks-dir example_data/predictions/implicit_depth/render/garden_chair/ \\\n```\n\n## ⏳ Training\n\nBy default models and tensorboard event files are saved to `~/tmp/tensorboard/\u003cmodel_name\u003e`.\nThis can be changed with the `--log_dir` flag.\n\nWe train with a batch_size of 12 with 16-bit precision on two A100s on the default ScanNetv2 split.\n\nExample command to train with two GPUs:\n\n```shell\nCUDA_VISIBLE_DEVICES=0,1 python train_bd.py --name implicit_depth \\\n            --log_dir logs \\\n            --config_file configs/models/implicit_depth.yaml \\\n            --data_config configs/data/scannet_default_train.yaml \\\n            --lazy_load_weights_from_checkpoint weights/regression.ckpt \\\n            --gpus 2 \\\n            --batch_size 12;\n```\nNote that we initialise our implicit depth models using a trained regression network, so you will need to download those weights first (see above).\n\nAlternatively, you could train your own regression network from scratch using:\n```shell\nCUDA_VISIBLE_DEVICES=0,1 python train.py --name regression \\\n            --log_dir logs \\\n            --config_file configs/models/regression.yaml \\\n            --data_config configs/data/scannet_default_train.yaml \\\n            --gpus 2 \\\n            --batch_size 16;\n```\n\nThe code supports any number of GPUs for training.\nYou can specify which GPUs to use with the `CUDA_VISIBLE_DEVICES` environment.\n\n## 🔧 Other training and testing options\n\nSee `options.py` for the range of other training options, such as learning rates and ablation settings, and testing options.\n\n## 🙏 Acknowledgements\n\nMany thanks to Daniyar Turmukhambetov, Jamie Wynn, Clement Godard, and Filippo Aleotti for their valuable help and suggestions.\nWe'd also like to thank Niantic's infrastructure team for quick actions when we needed them. Thanks folks!\n\n## 📜 BibTeX\n\nIf you find our work useful in your research please consider citing our paper:\n\n```bibtex\n@inproceedings{watson2023implict,\n  title={Virtual Occlusions Through Implicit Depth},\n  author={Watson, Jamie and Sayed, Mohamed and Qureshi, Zawar and Brostow, Gabriel J and Vicente, Sara and Mac Aodha, Oisin and Firman, Michael},\n  booktitle={Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR)},\n  year={2023},\n}\n```\n\n## 👩‍⚖️ License\n\nCopyright © Niantic, Inc. 2023. Patent Pending.\nAll rights reserved.\nPlease see the [license file](LICENSE) for terms.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnianticlabs%2Fimplicit-depth","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnianticlabs%2Fimplicit-depth","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnianticlabs%2Fimplicit-depth/lists"}