{"id":13528747,"url":"https://github.com/Huangying-Zhan/Depth-VO-Feat","last_synced_at":"2025-04-01T14:32:59.319Z","repository":{"id":49406552,"uuid":"124465517","full_name":"Huangying-Zhan/Depth-VO-Feat","owner":"Huangying-Zhan","description":"Unsupervised Learning of Monocular Depth Estimation and Visual Odometry with Deep Feature Reconstruction","archived":false,"fork":false,"pushed_at":"2020-11-10T00:59:41.000Z","size":1351,"stargazers_count":349,"open_issues_count":5,"forks_count":66,"subscribers_count":13,"default_branch":"master","last_synced_at":"2024-11-02T15:36:17.625Z","etag":null,"topics":["3d-vision","caffe","computer-vision","cvpr","deep-learning","depth-estimation","visual-odometry"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Huangying-Zhan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-03-09T00:38:12.000Z","updated_at":"2024-10-23T01:42:48.000Z","dependencies_parsed_at":"2022-08-28T00:12:14.354Z","dependency_job_id":null,"html_url":"https://github.com/Huangying-Zhan/Depth-VO-Feat","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Huangying-Zhan%2FDepth-VO-Feat","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Huangying-Zhan%2FDepth-VO-Feat/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Huangying-Zhan%2FDepth-VO-Feat/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Huangying-Zhan%2FDepth-VO-Feat/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Huangying-Zhan","download_url":"https://codeload.github.com/Huangying-Zhan/Depth-VO-Feat/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246655243,"owners_count":20812605,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-vision","caffe","computer-vision","cvpr","deep-learning","depth-estimation","visual-odometry"],"created_at":"2024-08-01T07:00:23.797Z","updated_at":"2025-04-01T14:32:56.665Z","avatar_url":"https://github.com/Huangying-Zhan.png","language":"Python","funding_links":[],"categories":["5. Learning based SLAM","2. 单目深度估计(半监督、无监督)","2. Monocular Depth (Semi- / Un-Supervised)"],"sub_categories":["5.2 Others","2.2 Multi View"],"readme":"# Introduction\n\nThis repo implements the system described in the CVPR-2018 paper:\n\n[**Unsupervised Learning of Monocular Depth Estimation and Visual Odometry with Deep Feature Reconstruction** \n](https://arxiv.org/abs/1803.03893) \n\nHuangying Zhan, Ravi Garg, Chamara Saroj Weerasekera, Kejie Li, Harsh Agarwal, Ian Reid\n\n```\n@InProceedings{Zhan_2018_CVPR,\nauthor = {Zhan, Huangying and Garg, Ravi and Saroj Weerasekera, Chamara and Li, Kejie and Agarwal, Harsh and Reid, Ian},\ntitle = {Unsupervised Learning of Monocular Depth Estimation and Visual Odometry With Deep Feature Reconstruction},\nbooktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},\nmonth = {June},\nyear = {2018}\n}\n```\nThis repo includes (1) the training procedure of our models;  (2) evaluation scripts for the results; (3) trained models and results.\n\n\n### Contents\n1. [Requirements](#part-1-requirements)\n2. [Prepare dataset](#part-2-prepare-dataset)\n3. [Depth](#part-3-depth)\n4. [Depth and odometry](#part-4-depth-and-odometry)\n5. [Feature Reconstruction Loss for Depth](#part-5-feature-reconstruction-loss-for-depth)\n6. [Depth, odometry and feature](#part-6-depth-odometry-and-feature)\n7. [Result evaluation](#part-7-result-evaluation)\n\n\n### Part 1. Requirements\n\nThis code was tested with Python 2.7, CUDA 8.0 and Ubuntu 14.04 using [Caffe](http://caffe.berkeleyvision.org/).\n\nCaffe: Add the required layers in `./caffe` into your own Caffe. Remember to enable Python Layers in the Caffe configuration.\n\nMost of our required models, trained models and results can be downloaded from [here](https://www.dropbox.com/sh/qxfqflrrzzwupua/AAAPA1mF0QaKwwR2Ds0jtDhYa?dl=0). The following instruction also includes specific links to the items.\n\n### Part 2. Download dataset and models\n\nThe main dataset used in this project is [KITTI Driving Dataset](http://www.cvlibs.net/datasets/kitti/raw_data.php). Please follow the instruction in `./data/README.md` to prepare the required dataset.\n\nFor our trained models and pre-requested models, please visit [here](https://www.dropbox.com/sh/60onn52jm9g2ygu/AADUkDRkwycS1STazstG5XOpa?dl=0) to download the models and put the models into the directory `./models`.\n\n### Part 3. Depth\n\nIn this part, the training of single view depth estimation network from stereo pairs is introduced. Photometric loss is used as the main supervision signal. Only stereo pairs are used in this experiment.\n\n1. Update `$YOUR_CAFFE_DIR` in `./experiments/depth/train.sh`. \n2. Run `bash ./expriments/depth/train.sh`. \n\nThe trained models are saved in `./snapshots/depth`\n\n### Part 4. Depth and odometry\n\nIn this part, the joint training of the depth estimation network and the visual odometry network is introduced. \nPhotometric losses for spatial pairs and temporal pairs are used as the main supervision signal. \nBoth spatial (stereo) pairs and temporal pairs (i.e. stereo sequences) are used in this experiment.\n\nTo facilitate the training, the model trained in the Depth experiment is used as an initialization.\n1. Update `$YOUR_CAFFE_DIR` in `./experiments/depth_odometry/train.sh`. \n2. Run `bash ./expriments/depth_odometry/train.sh`. \n\nThe trained models are saved in `./snapshots/depth_odometry`\n\n### Part 5. Feature Reconstruction Loss for Depth \n\nIn this part, the training of single view depth estimation network from stereo pairs is introduced. Both photometric loss and feature reconstruction loss are used as the main supervision signal. Only stereo pairs are used in this experiment. There are several features we have tried for this experiment. Currently, only the example of using **KITTI Feat.** is shown here. More details of using other features will be updated later.\n\nTo facilitate the training, the model trained in the Depth experiment is used as an initialization.\n1. Update `$YOUR_CAFFE_DIR` in `./experiments/depth_feature/train.sh`. \n2. Run `bash ./expriments/depth_feature/train.sh`. \n\nThe trained models are saved in `./snapshots/depth_feature`\n\n### Part 6. Depth, odometry and feature\n\nIn this part, we show the training including feature reconstruction loss.\nStereo sequences are used in this experiment.\n\nWith the feature extractor proposed in [Weerasekera et.al](https://arxiv.org/abs/1711.05919), we can finetune the trained depth model and/or odometry model with our proposed deep feature reconstruction loss.\n\n1. Update `$YOUR_CAFFE_DIR` in `./experiments/depth_odometry_feature/train.sh`. \n2. Run `bash ./expriments/depth_odometry_feature/train.sh`. \n \n**NOTE:** The link to download the feature extractor proposed in [Weerasekera et.al](https://arxiv.org/abs/1711.05919) will be released soon.\n\n### Part 7. Result evalution\n\nNote that the evaluation script provided here uses a different image interpolation for resizing input images (i.e. python's interpolation v.s. Caffe's interpolation), therefore the quantative result could be a little different from the published result. \n\n#### Depth estimation\n\nUsing the test set (697 image-depth pairs from 28 scenes) in Eigen Split is a common protocol to evaluate depth estimation result.\n\nWe basically use the evaluation script provided by [monodepth](https://github.com/mrharicot/monodepth) to evalute depth estimation results.\n\nIn order to run the evaluation, a `npy` file is required to store the predicted depths. Then run the script to evaluate the performance.\n\n1. Update `caffe_root` in `./tools/evaluation_tools.py`\n2. To generate the depth prediction and save it in a `npy` file. \n```\n python ./tools/evaluation_tools.py --func generate_depth_npy --dataset kitti_eigen --depth_net_def ./experiments/networks/depth_deploy.prototxt --model models/trained_models/eigen_split/Baseline.caffemodel --npy_dir ./result/depth/inv_depths_baseline.npy\n```\n\n3. To evalute the predictions.\n```\npython ./tools/eval_depth.py --split eigen --predicted_inv_depth_path ./result/depth/inv_depths_baseline.npy --gt_path data/kitti_raw_data/ --min_depth 1  --max_depth 50 --garg_crop\n```\n\nSome of our results (inverse depths) are released and can be downloaded from [here](https://www.dropbox.com/sh/1f6nkd4ezx0qfw4/AADmGuFLIxImtikz2UJrHeTOa?dl=0).\n\n#### Visual Odometry\n\n[KITTI Odometry benchmark](http://www.cvlibs.net/datasets/kitti/eval_odometry.php) contains 22 stereo sequences, in which 11 sequences are provided with ground truth. The 11 sequences are used for evaluation or training of visual odometry. \n\n1. Update `caffe_root` in `./tools/evaluation_tools.py`\n2. To generate the odometry predictions (relative camera motions), run the following script.\n\n```\npython ./tools/evaluation_tools.py --func generate_odom_result --model models/trained_models/odometry_split/Temporal.caffemodel --odom_net_def ./experiments/networks/odometry_deploy.prototxt --odom_result_dir ./result/odom_result\n```\n\n3. After getting the odometry predictions, we can evalute the performance by comparing with the ground truth poses.\n\n```\npython ./tools/evaluation_tools.py --func eval_odom --odom_result_dir ./result/odom_result\n```\n\n\nOur odometry results are released and can be downloaded from [here](https://www.dropbox.com/sh/qsb54kdpsp4i3wd/AAAht6__ssw3LlN168DsEqxca?dl=0).\n\n### License\nFor academic usage, the code is released under the permissive BSD license. For any commercial purpose, please contact the authors.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FHuangying-Zhan%2FDepth-VO-Feat","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FHuangying-Zhan%2FDepth-VO-Feat","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FHuangying-Zhan%2FDepth-VO-Feat/lists"}