{"id":20516391,"url":"https://github.com/nianticlabs/stereo-from-mono","last_synced_at":"2025-10-11T04:22:03.578Z","repository":{"id":39668697,"uuid":"284949095","full_name":"nianticlabs/stereo-from-mono","owner":"nianticlabs","description":"[ECCV 2020] Learning stereo from single images using monocular depth estimation networks","archived":false,"fork":false,"pushed_at":"2021-07-02T10:26:52.000Z","size":15137,"stargazers_count":401,"open_issues_count":3,"forks_count":55,"subscribers_count":34,"default_branch":"master","last_synced_at":"2025-03-30T08:11:19.491Z","etag":null,"topics":["deep-learning","deeplearning","depth-estimation","megadepth","monocular-depth-estimation","monodepth","single-image-depth-prediction","stereo","stereo-algorithms","stereo-matching"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nianticlabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-08-04T10:36:24.000Z","updated_at":"2025-02-07T05:52:35.000Z","dependencies_parsed_at":"2022-08-10T09:03:50.924Z","dependency_job_id":null,"html_url":"https://github.com/nianticlabs/stereo-from-mono","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nianticlabs%2Fstereo-from-mono","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nianticlabs%2Fstereo-from-mono/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nianticlabs%2Fstereo-from-mono/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nianticlabs%2Fstereo-from-mono/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nianticlabs","download_url":"https://codeload.github.com/nianticlabs/stereo-from-mono/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247457803,"owners_count":20941906,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","deeplearning","depth-estimation","megadepth","monocular-depth-estimation","monodepth","single-image-depth-prediction","stereo","stereo-algorithms","stereo-matching"],"created_at":"2024-11-15T21:28:36.761Z","updated_at":"2025-10-11T04:21:58.548Z","avatar_url":"https://github.com/nianticlabs.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# [Learning Stereo from Single Images](https://arxiv.org/abs/2008.01484)\n\n**[Jamie Watson](https://scholar.google.com/citations?view_op=list_works\u0026hl=en\u0026user=5pC7fw8AAAAJ), [Oisin Mac Aodha](https://homepages.inf.ed.ac.uk/omacaod/), [Daniyar Turmukhambetov](http://dantkz.github.io/about), [Gabriel J. Brostow](http://www0.cs.ucl.ac.uk/staff/g.brostow/) and [Michael Firman](http://www.michaelfirman.co.uk) – ECCV 2020 (Oral presentation)**\n\n\n[Link to paper](https://arxiv.org/abs/2008.01484)  \n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://storage.googleapis.com/niantic-lon-static/research/stereo-from-mono/short-video.mp4\"\u003e\n  \u003cimg src=\"assets/2min.png\" alt=\"2 minute ECCV presentation video link\" width=\"400\"\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://storage.googleapis.com/niantic-lon-static/research/stereo-from-mono/long-video.mp4\"\u003e\n  \u003cimg src=\"assets/10min.png\" alt=\"10 minute ECCV presentation video link\" width=\"400\"\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/teaser.png\" alt=\"Training data and results qualitative comparison\" width=\"600\" /\u003e\n\u003c/p\u003e\n\nSupervised deep networks are among the best methods for finding correspondences in stereo image pairs. Like all supervised approaches, these networks require ground truth data during training. However, collecting large quantities of accurate dense correspondence data is very challenging. We propose that it is unnecessary to have such a high reliance on ground truth depths or even corresponding stereo pairs.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/method.png\" alt=\"Overview of our stereo data generation approach\" width=\"600\" /\u003e\n\u003c/p\u003e\n\nInspired by recent progress in monocular depth estimation, we generate plausible disparity maps from single images. In turn, we use those flawed disparity maps in a carefully designed pipeline to generate stereo training pairs. Training in this manner makes it possible to convert any collection of single RGB images into stereo training data. This results in a significant reduction in human effort, with no need to collect real depths or to hand-design synthetic data. We can consequently train a stereo matching network from scratch on datasets like COCO, which were previously hard to exploit for stereo. \n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/results.png\" alt=\"Depth maps produced by stereo networks trained with Sceneflow and our method\" width=\"600\" /\u003e\n\u003c/p\u003e\n\nThrough extensive experiments we show that our approach outperforms stereo networks trained with standard synthetic datasets, when evaluated on  KITTI, ETH3D, and Middlebury. \n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/table.png\" alt=\"Quantitative comparison of stereo networks trained with Sceneflow and our method\" width=\"500\" /\u003e\n\u003c/p\u003e\n\n## ✏️ 📄 Citation\n\nIf you find our work useful or interesting, please consider citing [our paper](https://arxiv.org/abs/2008.01484):\n\n```\n@inproceedings{watson-2020-stereo-from-mono,\n title   = {Learning Stereo from Single Images},\n author  = {Jamie Watson and\n            Oisin Mac Aodha and\n            Daniyar Turmukhambetov and\n            Gabriel J. Brostow and\n            Michael Firman\n           },\n booktitle = {European Conference on Computer Vision ({ECCV})},\n year = {2020}\n}\n```\n\n\n## 📊 Evaluation\n\nWe evaluate our performance on several datasets: \nKITTI ([2015](http://www.cvlibs.net/datasets/kitti/eval_scene_flow.php?benchmark=stereo) and [2012](http://www.cvlibs.net/datasets/kitti/eval_stereo_flow.php?benchmark=stereo)), \n[Middlebury](https://vision.middlebury.edu/stereo/submit3/) (full resolution) and [ETH3D](https://www.eth3d.net/datasets#high-res-multi-view) (Low res two view). \nTo run inference on these datasets first download them, and update  `paths_config.yaml` to point to these locations.\n\nNote that we report scores on the *training sets* of each dataset since we never see these images during training.\n\nRun evaluation using:\n\n```\nCUDA_VISIBLE_DEVICES=X  python main.py \\\n  --mode inference \\\n  --load_path \u003cdownloaded_model_path\u003e \n\n```\noptionally setting `--test_data_types` and `--save_disparities`.\n\nA trained model can be found [HERE](https://storage.googleapis.com/niantic-lon-static/research/stereo-from-mono/models/hourglass_midas_release.zip). \n\n\n## 🎯 Training\n\nTo train a new model, you will need to download several datasets: \n[ADE20K](https://groups.csail.mit.edu/vision/datasets/ADE20K/), [DIODE](https://diode-dataset.org/), \n[Depth in the Wild](http://www-personal.umich.edu/~wfchen/depth-in-the-wild/), \n[Mapillary](https://www.mapillary.com/dataset/vistas?pKey=1GyeWFxH_NPIQwgl0onILw)\n and  [COCO](https://github.com/nightrome/cocostuff). After doing so, update `paths_config.yaml` to point to these directories.\n \n Additionally you will need some precomputed monocular depth estimates for these images. \n We provide these for MiDaS: [ADE20K](https://storage.googleapis.com/niantic-lon-static/research/stereo-from-mono/data/ADE20K.zip), [DIODE](https://storage.googleapis.com/niantic-lon-static/research/stereo-from-mono/data/diode.zip), \n[Depth in the Wild](https://storage.googleapis.com/niantic-lon-static/research/stereo-from-mono/data/diw.zip), \n[Mapillary](https://storage.googleapis.com/niantic-lon-static/research/stereo-from-mono/data/mapillary.zip)\n and  [COCO](https://storage.googleapis.com/niantic-lon-static/research/stereo-from-mono/data/mscoco.zip).\n Download these and put them in the corresponding data paths (i.e. your paths specified in `paths_config.yaml`).\n \n Now you can train a new model using:\n ```\nCUDA_VISIBLE_DEVICES=X  python  main.py --mode train \\\n  --log_path \u003cwhere_to_save_your_model\u003e \\\n  --model_name \u003cname_of_your_model\u003e\n\n```\nPlease see `options.py` for full list of training options.\n\n# 👩‍⚖️ License\nCopyright © Niantic, Inc. 2020. Patent Pending. All rights reserved. Please see the license file for terms.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnianticlabs%2Fstereo-from-mono","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnianticlabs%2Fstereo-from-mono","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnianticlabs%2Fstereo-from-mono/lists"}