{"id":20516404,"url":"https://github.com/nianticlabs/depth-hints","last_synced_at":"2026-02-28T16:31:04.340Z","repository":{"id":78486553,"uuid":"209591465","full_name":"nianticlabs/depth-hints","owner":"nianticlabs","description":"[ICCV 2019] Depth Hints are complementary depth suggestions which improve monocular depth estimation algorithms trained from stereo pairs","archived":false,"fork":false,"pushed_at":"2021-05-17T13:11:06.000Z","size":65567,"stargazers_count":187,"open_issues_count":0,"forks_count":20,"subscribers_count":27,"default_branch":"master","last_synced_at":"2025-08-22T05:44:40.526Z","etag":null,"topics":["computer-vision","deep-learning","depth-estimation","monodepth","neural-network","pytorch","self-supervision","stereo-matching"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nianticlabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2019-09-19T15:44:06.000Z","updated_at":"2025-02-27T03:13:59.000Z","dependencies_parsed_at":"2023-05-31T20:31:22.508Z","dependency_job_id":null,"html_url":"https://github.com/nianticlabs/depth-hints","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/nianticlabs/depth-hints","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nianticlabs%2Fdepth-hints","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nianticlabs%2Fdepth-hints/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nianticlabs%2Fdepth-hints/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nianticlabs%2Fdepth-hints/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nianticlabs","download_url":"https://codeload.github.com/nianticlabs/depth-hints/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nianticlabs%2Fdepth-hints/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29942787,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-28T13:49:17.081Z","status":"ssl_error","status_checked_at":"2026-02-28T13:48:50.396Z","response_time":90,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","deep-learning","depth-estimation","monodepth","neural-network","pytorch","self-supervision","stereo-matching"],"created_at":"2024-11-15T21:28:40.902Z","updated_at":"2026-02-28T16:31:04.307Z","avatar_url":"https://github.com/nianticlabs.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Self-Supervised Monocular Depth Hints\n\n\n**[Jamie Watson](https://scholar.google.com/citations?view_op=list_works\u0026hl=en\u0026user=5pC7fw8AAAAJ), [Michael Firman](http://www.michaelfirman.co.uk), [Gabriel J. Brostow](http://www0.cs.ucl.ac.uk/staff/g.brostow/) and [Daniyar Turmukhambetov](http://dantkz.github.io/) – ICCV 2019**\n\n[[Link to paper]](https://arxiv.org/abs/1909.09051)\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/kitti.gif\" alt=\"example input output gif\" width=\"700\" /\u003e\n\u003c/p\u003e\n\n**We introduce *Depth Hints*, which improve monocular depth estimation algorithms trained from stereo pairs.**\n\nWe find that photometric reprojection losses used with self-supervised learning typically have multiple local minima.   \nThis can restrict what a regression network learns, for example causing artifacts around thin structures.\n\n**Depth Hints are complementary depth suggestions obtained from simple off-the-shelf stereo algorithms**, e.g. Semi-Global Matching.\nThese hints are used during training to guide the network to learn better weights.\nThey require no additional data, and are assumed to be right only sometimes.\n\nCombined with other good practices, Depth Hints gives **state-of-the-art depth predictions on the KITTI benchmark** (see images above and results table below).\nWe show additional monocular depth estimation results on the [sceneflow dataset](https://lmb.informatik.uni-freiburg.de/resources/datasets/SceneFlowDatasets.en.html):\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/sceneflow.gif\" alt=\"example input output gif\" width=\"700\" /\u003e\n\u003c/p\u003e\n\n## ✏️ 📄 Citation\n\n\nIf you find our work useful or interesting, please consider citing [our paper](https://arxiv.org/abs/1909.09051):\n\n```\n@inproceedings{watson-2019-depth-hints,\n  title     = {Self-Supervised Monocular Depth Hints},\n  author    = {Jamie Watson and\n               Michael Firman and\n               Gabriel J. Brostow and\n               Daniyar Turmukhambetov},\n  booktitle = {The International Conference on Computer Vision (ICCV)},\n  month = {October},\n  year = {2019}\n}\n```\n\n\n## 📈 KITTI Results\n\n| Model name | Training modality | ImageNet pretrained | Resolution | Abs rel | Sq rel | 𝛿 \u003c 1.25 |\n| ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ----- |\n| `Ours Resnet50` | Stereo | Yes | 640 x 192 | 0.102 | 0.762 | 0.880 | \n| `Ours Resnet50 no pt` | Stereo | No | 640 x 192 | 0.118 | 0.941 | 0.850 |\n| `Ours HR Resnet50` | Stereo | Yes | 1024 x 320 | 0.096 | 0.710 | 0.890 |\n| `Ours HR Resnet50 no pt` | Stereo | No | 1024 x 320 | 0.112 | 0.857 | 0.861 |\n| `Ours HR` | Mono + Stereo | Yes | 1024 x 320 | 0.098 | 0.702 | 0.887 |\n\nPlease see the paper for full results. To download the weights and predictions for each model please follow the links below:\n\n| Model name | Training modality | ImageNet pretrained | Resolution | Weights | Eigen Predictions | \n| ---------- | ---------- | ---------- | ---------- | ---------- | ---------- |\n| `Ours Resnet50` | Stereo | Yes | 640 x 192 | [Download](https://storage.googleapis.com/niantic-lon-static/research/depth-hints/DH-Resnet50/models.zip) | [Download](https://storage.googleapis.com/niantic-lon-static/research/depth-hints/DH-Resnet50/disps_eigen_split.npy) |\n| `Ours Resnet50 no pt` | Stereo | No | 640 x 192 | [Download](https://storage.googleapis.com/niantic-lon-static/research/depth-hints/DH-Resnet50-nopt/models.zip)| [Download](https://storage.googleapis.com/niantic-lon-static/research/depth-hints/DH-Resnet50-nopt/disps_eigen_split.npy) | \n| `Ours HR Resnet50` | Stereo | Yes | 1024 x 320 | [Download](https://storage.googleapis.com/niantic-lon-static/research/depth-hints/DH-HR-Resnet50/models.zip) | [Download](https://storage.googleapis.com/niantic-lon-static/research/depth-hints/DH-HR-Resnet50/disps_eigen_split.npy) | \n| `Ours HR Resnet50 no pt` | Stereo | No | 1024 x 320 | [Download](https://storage.googleapis.com/niantic-lon-static/research/depth-hints/DH-HR-Resnet50-nopt/models.zip) | [Download](https://storage.googleapis.com/niantic-lon-static/research/depth-hints/DH-HR-Resnet50-nopt/disps_eigen_split.npy) |\n| `Ours HR` | Mono + Stereo | Yes | 1024 x 320 | [Download](https://storage.googleapis.com/niantic-lon-static/research/depth-hints/DH-HR-Mono%2BStereo/models.zip) | [Download](https://storage.googleapis.com/niantic-lon-static/research/depth-hints/DH-HR-Mono%2BStereo/disps_eigen_split.npy) |\n\n\n\n## ⚙️ Code\n\nThe code for ***Depth Hints*** builds upon [monodepth2](https://github.com/nianticlabs/monodepth2). If you have questions about running the code, please see the issues in that repository first.\n\nTo train using depth hints:\n  - Clone this repository\n  - Run `python precompute_depth_hints.py  --data_path \u003cyour_KITTI_path\u003e`, optionally setting `--save_path` (will default to \u003cdata_path\u003e/depth_hints) and `--filenames` (will default to training and validation images for the eigen split). This will create the \"fused\" depth hints referenced in the paper. This process takes approximately 4 hours on a GPU.\n  - Add the flag `--use_depth_hints` to your usual monodepth2 training command, optionally also setting `--depth_hint_path` (will default to \u003cdata_path\u003e/depth_hints). See below for a full command.\n  \n🎉 And that's it! 🎉\n\n## 👀 Reproducing Paper Results\n\nTo recreate the results from our paper, run:\n\n```\npython train.py\n  --data_path \u003cyour_KITTI_path\u003e\n  --log_dir \u003cyour_save_path\u003e\n  --model_name stereo_depth_hints\n  --use_depth_hints\n  --depth_hint_path \u003cyour_depth_hint_path\u003e\n  --frame_ids 0  --use_stereo\n  --scheduler_step_size 5\n  --split eigen_full\n  --disparity_smoothness 0\n```\n\nAdditionally:\n  - For Resnet50 models, add `--num_layers 50`\n  - Add `--height 320  --width 1024` for High Resolution models (you may also have to set `--batch_size 6` depending on the size of your GPU) \n  - For Mono + Stereo add `--frame_ids 0 -1 1 ` and remove `--split eigen_full`\n\nThe results above and in the main paper arise from evaluating on the KITTI sparse LiDAR point cloud, using the Eigen Test split.\n\nTo test on KITTI, run:\n\n```\npython evaluate_depth.py\n  --data_path \u003cyour_KITTI_path\u003e\n  --load_weights_folder \u003cyour_model_path\u003e\n  --use_stereo\n```\n\nMake sure you have run `export_gt_depth.py` to extract ground truth files. \n\nAdditionally, if you see `ValueError: Object arrays cannot be loaded when allow_pickle=False`, then either downgrade numpy, or change line 166 in `evaluate_depth.py` to\n\n```\ngt_depths = np.load(gt_path, fix_imports=True, encoding='latin1', allow_pickle=True)[\"data\"]\n```\n\n## 🖼 Running on your own images\n\nTo run on your own images, run:\n\n```\npython test_simple.py\n  --image_path \u003cyour_image_path\u003e\n  --model_path \u003cyour_model_path\u003e\n  --num_layers \u003c18 or 50\u003e\n```\n\nThis will save a numpy array of depths, and a colormapped depth image.\n\n## 👩‍⚖️ License\nCopyright © Niantic, Inc. 2020. Patent Pending.\nAll rights reserved.\nPlease see the [license file](LICENSE) for terms.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnianticlabs%2Fdepth-hints","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnianticlabs%2Fdepth-hints","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnianticlabs%2Fdepth-hints/lists"}