{"id":24518177,"url":"https://github.com/div99/w-stereo-disp","last_synced_at":"2025-08-01T00:04:02.213Z","repository":{"id":42086868,"uuid":"305293595","full_name":"Div99/W-Stereo-Disp","owner":"Div99","description":"(NeurIPS 2020 Spotlight) Wasserstein Distances for Stereo Disparity Estimation","archived":false,"fork":false,"pushed_at":"2022-04-13T00:40:05.000Z","size":37472,"stargazers_count":103,"open_issues_count":3,"forks_count":15,"subscribers_count":8,"default_branch":"main","last_synced_at":"2025-03-27T23:05:34.470Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://div99.github.io/W-Stereo-Disp/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Div99.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-10-19T07:08:49.000Z","updated_at":"2025-03-25T08:47:17.000Z","dependencies_parsed_at":"2022-08-12T04:51:06.969Z","dependency_job_id":null,"html_url":"https://github.com/Div99/W-Stereo-Disp","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Div99%2FW-Stereo-Disp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Div99%2FW-Stereo-Disp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Div99%2FW-Stereo-Disp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Div99%2FW-Stereo-Disp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Div99","download_url":"https://codeload.github.com/Div99/W-Stereo-Disp/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248856283,"owners_count":21172737,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-01-22T01:39:25.822Z","updated_at":"2025-04-14T09:43:20.093Z","avatar_url":"https://github.com/Div99.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Wasserstein Distances for Stereo Disparity Estimation\n\n[**[Project Page](https://div99.github.io/W-Stereo-Disp)**] Accepted in NeurIPS 2020 as Spotlight\n\n[Wasserstein Distances for Stereo Disparity Estimation](https://arxiv.org/abs/2007.03085) \n\nby [Divyansh Garg](https://divyanshgarg.com), [Yan Wang](https://www.cs.cornell.edu/~yanwang/), [Bharath Hariharan](http://home.bharathh.info/), [Mark Campbell](https://campbell.mae.cornell.edu/), [Kilian Q. Weinberger](http://kilian.cs.cornell.edu/) and [Wei-Lun Chao](http://www-scf.usc.edu/~weilunc/)\n\n![Figure](figures/neurips2020-pipeline.png)\n\n### Citation\n```\n@inproceedings{div2020wstereo,\n  title={Wasserstein Distances for Stereo Disparity Estimation},\n  author={Garg, Divyansh and Wang, Yan and Hariharan, Bharath and Campbell, Mark and Weinberger, Kilian and Chao, Wei-Lun},\n  booktitle={NeurIPS},\n  year={2020}\n}\n```\n\n## Introduction\nExisting approaches to depth or disparity estimation output a distribution over a\nset of pre-defined discrete values. This leads to inaccurate results when the true\ndepth or disparity does not match any of these values. The fact that this distribution\nis usually learned indirectly through a regression loss causes further problems in\nambiguous regions around object boundaries. We address these issues using a new\nneural network architecture that is capable of outputting arbitrary depth values,\nand a new loss function that is derived from the Wasserstein distance between\nthe true and the predicted distributions. We validate our approach on a variety\nof tasks, including stereo disparity and depth estimation, and the downstream 3D\nobject detection. Our approach drastically reduces the error in ambiguous regions,\nespecially around object boundaries that greatly affect the localization of objects in\n3D, achieving the state-of-the-art in 3D object detection for autonomous driving.\n\n## Update\n  - 1 June 2021: Released all pretrained models. Added script to download KITTI Object Detection dataset.\n\n## Contents\n\nOur Wasserstein loss [W_loss](https://github.com/Div99/W-Stereo-Disp/blob/85cbf2bdc199a632ffe6b960f91bdc8e02046b77/src/main_disp.py#L123) can be easily plugged in existing stereo depth models to improve the training and obtain better results.\n\nWe release the code for CDN-PSMNet and CDN-SDN models. \n\n- [Requirements](#requirements)\n- [Pretrained Models](#pretrained-models)\n- [Training and Inference](#training-and-inference)\n\n## Requirements\n1. Python 3.7\n2. Pytorch 1.2.0+\n3. CUDA\n4. `pip install -r ./requirements.txt`\n5. [SceneFlow](https://lmb.informatik.uni-freiburg.de/resources/datasets/SceneFlowDatasets.en.html)\n5. [KITTI](http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=3d)\n\n## Pretrained Models\n\nPlace the checkpoint folders in `./results`.\n\nDepth Models\n- [Trained on Sceneflow](https://drive.google.com/drive/folders/1uav_MD0cJ3jaeqkfJVwAv37r_cS_sKDn?usp=sharing)\n- [Trained on KITTI Object Detection train set](https://drive.google.com/drive/folders/1c-YUzbuaS5wbBAmSznrP34Kn-lsWW56-?usp=sharing)\n- [Trained on KITTI Object Detection train+val set](https://drive.google.com/drive/folders/1gePafBBvHJDm1b4EpTa34C3XqoPOz757?usp=sharing)\n\n\nDisparity Models\n- [Trained on Sceneflow](https://drive.google.com/drive/folders/1E0QxohXyuGad96P64QwxQc7PVL99GGJU?usp=sharing)\n- [Trained on KITTI 2015 train set](https://drive.google.com/drive/folders/1B-Jpr6w8-IaFpw54vji2oxVh8PLtYhJo?usp=sharing)\n## Datasets\nYou have to download the [SceneFlow](https://lmb.informatik.uni-freiburg.de/resources/datasets/SceneFlowDatasets.en.html) and [KITTI](http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=3d) datasets. The structures of the datasets are shown in below. \n\nKITTI can be automatically downloaded using `./scripts/download_kitti.sh`\n#### SceneFlow Dataset Structure\n```\nSceneFlow\n    | monkaa\n        | frames_cleanpass\n        | disparity\n    | driving\n        | frames_cleanpass\n        | disparity\n    | flyingthings3d\n        | frames_cleanpass \n        | disparity\n```\n#### KITTI Object Detection Dataset Structure\n```\nKITTI\n    | training\n        | calib\n        | image_2\n        | image_3\n        | velodyne\n    | testing\n        | calib\n        | image_2\n        | image_3\n```\nGenerate soft-links of SceneFlow Datasets. The results will be saved in `./sceneflow` folder. Please change to fakepath `path-to-SceneFlow` to the SceneFlow dataset location before running the script.\n```bash\npython sceneflow.py --path path-to-SceneFlow --force\n```\n\nConvert the KITTI velodyne ground truths to depth maps. Please change to fakepath `path-to-KITTI` to the SceneFlow dataset location before running the script.\n```bash\npython ./src/preprocess/generate_depth_map.py --data_path path-to-KITTI/ --split_file ./split/trainval.txt\n```\n\nOptionally download [KITTI2015](http://www.cvlibs.net/datasets/kitti/eval_scene_flow.php?benchmark=stereo) datasets for evaluating stereo disparity models.\n\n## Training and Inference\n- [1 Train CDN from Scratch on SceneFlow Dataset](#1-train-sDNet-from-scratch-on-sceneflow-dataset)\n- [2 Train CDN on KITTI Dataset](#2-train-sdnet-on-kitti-dataset)\n- [3 Generate Predictions](#3-generate-predictions)\n- [4 Train 3D Detection with Pseudo-LiDAR](#7-train-3d-detection-with-pseudo-lidar)\n\nWe have provided all pretrained models [Pretrained Models](#pretrained-models). If you only want to generate the predictions, you can directly go to step [3](#3-generate-predictions). \n\nWe use config files to simplify argument parsing. The default setting requires four gpus to train. You can use smaller batch sizes which are `btrain` and `bval`, if you don't have enough gpus. \n\nWe provide code for both stereo disparity and stereo depth models.\n\nWe optionally use [Losswise](https://losswise.com) to visualize training metrics. An API key can be obtained and added to a config key to enable it.\n#### 1 Train CDN-SDN from Scratch on SceneFlow Dataset\n```bash\npython ./src/main_depth.py -c src/configs/sceneflow_w1.config\n```\nThe checkpoints are saved in `./results/stack_sceneflow_w1/`.\n\nFollow same procedure to train stereo disparity model, but use `src/main_disp.py` and change to a disparity config.\n\n#### 2 Train CDN-SDN on KITTI Dataset\n```bash\npython ./src/main_depth.py -c src/configs/kitti_w1.config \\\n    --pretrain ./results/sceneflow_w1/checkpoint.pth.tar --datapath  path-to-KITTI/training/\n```\nBefore running, please change the fakepath `path-to-KITTI/` to the correct one. `--pretrain` is the path to the  pretrained model on SceneFlow. The training results are saved in `./results/kitti_w1_train`.\n\nIf you are working on evaluating CDN on KITTI testing set, you might want to train CDN on *training+validation* sets. The training results will be saved in `./results/sdn_kitti_trainval`.\n```bash\npython ./src/main_depth.py -c src/configs/kitti_w1.config \\\n    --pretrain ./results/sceneflow_w1/checkpoint.pth.tar \\\n    --datapath  path-to-KITTI/training/ --split_train ./split/trainval.txt \\\n    --save_path ./results/sdn_kitti_trainval\n ```\nThe disparity models can also be trained on KITTI2015 datasets using `src/kitti2015_w1_disp.config`.\n\n#### 3 Generate Predictions\nPlease change the fakepath `path-to-KITTI`. Moreover, if you use the our provided checkpoint, please modify the value of `--resume` to the checkpoint location. \n\n* a. Using the model trained on KITTI training set, and generating predictions on training + validation sets.\n```bash\npython ./src/main_depth.py -c src/configs/kitti_w1.config \\\n    --resume ./results/sdn_kitti_train/checkpoint.pth.tar --datapath  path-to-KITTI/training/ \\\n    --data_list ./split/trainval.txt --generate_depth_map --data_tag trainval\n``` \nThe results will be saved in `./results/sdn_kitti_train/depth_maps_trainval/`.\n\n* b. Using the model trained on KITTI training + validation set, and generating predictions on testing sets. You will use them when you want to submit your results to the leaderboard.\n\nThe results will be saved in `./results/sdn_kitti_trainval_set/depth_maps_trainval/`.\n```bash\n# testing sets\npython ./src/main_depth.py -c src/configs/kitti_w1.config \\\n    --resume ./results/sdn_kitti_trainval/checkpoint.pth.tar --datapath  path-to-KITTI/testing/ \\\n    --data_list=./split/test.txt --generate_depth_map --data_tag test\n``` \nThe results will be saved in `./results/sdn_kitti_trainval/depth_maps_test/`.\n\n#### 4 Train 3D Detection with Pseudo-LiDAR\nFor training 3D object detection models, follow step 4 and after in the Pseudo-LiDAR_V2 repo https://github.com/mileyan/Pseudo_Lidar_V2.\n\n## Results\n\n### Results on the Stereo Disparity\n![Figure](figures/results_disp.png)\n\n### 3D Object Detection Results on KITTI leader board\n![Figure](figures/results_object3d.png)\n\n## Questions\nPlease feel free to email us if you have any questions. \n\nDivyansh Garg ([dg595@cornell.edu](mailto:dg595@cornell.edu?subject=[GitHub]%20W-Stereo_Disp)),\nYan Wang ([yw763@cornell.edu](mailto:yw763@cornell.edu?subject=[GitHub]%20W-Stereo_Disp)),\nWei-Lun Chao ([weilunchao760414@gmail.com](mailto:weilunchao760414@gmail.com?subject=[GitHub]%20W-Stereo_Disp))\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdiv99%2Fw-stereo-disp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdiv99%2Fw-stereo-disp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdiv99%2Fw-stereo-disp/lists"}