https://github.com/isl-org/vi-depth
Code for Monocular Visual-Inertial Depth Estimation (ICRA 2023)
https://github.com/isl-org/vi-depth
Last synced: 3 months ago
JSON representation
Code for Monocular Visual-Inertial Depth Estimation (ICRA 2023)
- Host: GitHub
- URL: https://github.com/isl-org/vi-depth
- Owner: isl-org
- License: mit
- Created: 2023-03-06T16:35:57.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2024-08-26T12:33:14.000Z (almost 2 years ago)
- Last Synced: 2025-04-13T08:26:22.857Z (about 1 year ago)
- Language: Python
- Homepage:
- Size: 2.33 MB
- Stars: 176
- Watchers: 12
- Forks: 15
- Open Issues: 6
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# PROJECT NOT UNDER ACTIVE MANAGEMENT #
This project will no longer be maintained by Intel.
Intel will not provide or guarantee development of or support for this project, including but not limited to, maintenance, bug fixes, new releases or updates.
Patches to this project are no longer accepted by Intel.
If you have an ongoing need to use this project, are interested in independently developing it, or would like to maintain patches for the community, please create your own fork of the project.
# Monocular Visual-Inertial Depth Estimation
This repository contains code and models for our paper:
> [Monocular Visual-Inertial Depth Estimation](https://arxiv.org/abs/2303.12134)
> Diana Wofk, René Ranftl, Matthias Müller, Vladlen Koltun
For a quick overview of the work you can watch the [short talk](https://youtu.be/Ja4Nic3YYCg) and [teaser](https://youtu.be/IMwiKwSpshQ) on YouTube.
## Introduction
We present a visual-inertial depth estimation pipeline that integrates monocular depth estimation and visual-inertial odometry to produce dense depth estimates with metric scale. Our approach consists of three stages: (1) input processing, where RGB and IMU data feed into monocular depth estimation alongside visual-inertial odometry, (2) global scale and shift alignment, where monocular depth estimates are fitted to sparse depth from VIO in a least-squares manner, and (3) learning-based dense scale alignment, where globally-aligned depth is locally realigned using a dense scale map regressed by the ScaleMapLearner (SML). The images at the bottom in the diagram above illustrate a VOID sample being processed through our pipeline; from left to right: the input RGB, ground truth depth, sparse depth from VIO, globally-aligned depth, scale map scaffolding, dense scale map regressed by SML, final depth output.
## Setup
1) Setup dependencies:
```shell
conda env create -f environment.yaml
conda activate vi-depth
```
2) Pick one or more ScaleMapLearner (SML) models and download the corresponding weights to the `weights` folder.
| Depth Predictor | SML on VOID 150 | SML on VOID 500 | SML on VOID 1500 |
| :--- | :----: | :----: | :----: |
| DPT-BEiT-Large | [model](https://github.com/isl-org/VI-Depth/releases/download/v1/sml_model.dpredictor.dpt_beit_large_512.nsamples.150.ckpt) | [model](https://github.com/isl-org/VI-Depth/releases/download/v1/sml_model.dpredictor.dpt_beit_large_512.nsamples.500.ckpt) | [model](https://github.com/isl-org/VI-Depth/releases/download/v1/sml_model.dpredictor.dpt_beit_large_512.nsamples.1500.ckpt) |
| DPT-SwinV2-Large | [model](https://github.com/isl-org/VI-Depth/releases/download/v1/sml_model.dpredictor.dpt_swin2_large_384.nsamples.150.ckpt) | [model](https://github.com/isl-org/VI-Depth/releases/download/v1/sml_model.dpredictor.dpt_swin2_large_384.nsamples.500.ckpt) | [model](https://github.com/isl-org/VI-Depth/releases/download/v1/sml_model.dpredictor.dpt_swin2_large_384.nsamples.1500.ckpt) |
| DPT-Large | [model](https://github.com/isl-org/VI-Depth/releases/download/v1/sml_model.dpredictor.dpt_large.nsamples.150.ckpt) | [model](https://github.com/isl-org/VI-Depth/releases/download/v1/sml_model.dpredictor.dpt_large.nsamples.500.ckpt) | [model](https://github.com/isl-org/VI-Depth/releases/download/v1/sml_model.dpredictor.dpt_large.nsamples.1500.ckpt) |
| DPT-Hybrid | [model](https://github.com/isl-org/VI-Depth/releases/download/v1/sml_model.dpredictor.dpt_hybrid.nsamples.150.ckpt)* | [model](https://github.com/isl-org/VI-Depth/releases/download/v1/sml_model.dpredictor.dpt_hybrid.nsamples.500.ckpt) | [model](https://github.com/isl-org/VI-Depth/releases/download/v1/sml_model.dpredictor.dpt_hybrid.nsamples.1500.ckpt) |
| DPT-SwinV2-Tiny | [model](https://github.com/isl-org/VI-Depth/releases/download/v1/sml_model.dpredictor.dpt_swin2_tiny_256.nsamples.150.ckpt) | [model](https://github.com/isl-org/VI-Depth/releases/download/v1/sml_model.dpredictor.dpt_swin2_tiny_256.nsamples.500.ckpt) | [model](https://github.com/isl-org/VI-Depth/releases/download/v1/sml_model.dpredictor.dpt_swin2_tiny_256.nsamples.1500.ckpt) |
| DPT-LeViT | [model](https://github.com/isl-org/VI-Depth/releases/download/v1/sml_model.dpredictor.dpt_levit_224.nsamples.150.ckpt) | [model](https://github.com/isl-org/VI-Depth/releases/download/v1/sml_model.dpredictor.dpt_levit_224.nsamples.500.ckpt) | [model](https://github.com/isl-org/VI-Depth/releases/download/v1/sml_model.dpredictor.dpt_levit_224.nsamples.1500.ckpt) |
| MiDaS-small | [model](https://github.com/isl-org/VI-Depth/releases/download/v1/sml_model.dpredictor.midas_small.nsamples.150.ckpt) | [model](https://github.com/isl-org/VI-Depth/releases/download/v1/sml_model.dpredictor.midas_small.nsamples.500.ckpt) | [model](https://github.com/isl-org/VI-Depth/releases/download/v1/sml_model.dpredictor.midas_small.nsamples.1500.ckpt) |
*Also available with pretraining on TartanAir: [model](https://github.com/isl-org/VI-Depth/releases/download/v1/sml_model.dpredictor.dpt_hybrid.nsamples.150.pretrained.ckpt)
## Inference
1) Place inputs into the `input` folder. An input image and corresponding sparse metric depth map are expected:
```bash
input
├── image # RGB image
│ ├── .png
│ └── ...
└── sparse_depth # sparse metric depth map
├── .png # as 16b PNG
└── ...
```
The `load_sparse_depth` function in `run.py` may need to be modified depending on the format in which sparse depth is stored. By default, the depth storage method [used in the VOID dataset](https://github.com/alexklwong/void-dataset/blob/master/src/data_utils.py) is assumed.
2) Run the `run.py` script as follows:
```bash
DEPTH_PREDICTOR="dpt_beit_large_512"
NSAMPLES=150
SML_MODEL_PATH="weights/sml_model.dpredictor.${DEPTH_PREDICTOR}.nsamples.${NSAMPLES}.ckpt"
python run.py -dp $DEPTH_PREDICTOR -ns $NSAMPLES -sm $SML_MODEL_PATH --save-output
```
3) The `--save-output` flag enables saving outputs to the `output` folder. By default, the following outputs will be saved per sample:
```bash
output
├── ga_depth # metric depth map after global alignment
│ ├── .pfm # as PFM
│ ├── .png # as 16b PNG
│ └── ...
└── sml_depth # metric depth map output by SML
├── .pfm # as PFM
├── .png # as 16b PNG
└── ...
```
## Evaluation
Models provided in this repo were trained on the VOID dataset.
1) Download the VOID dataset following [the instructions in the VOID dataset repo](https://github.com/alexklwong/void-dataset#downloading-void).
2) To evaluate on VOID test sets, run the `evaluate.py` script as follows:
```bash
DATASET_PATH="/path/to/void_release/"
DEPTH_PREDICTOR="dpt_beit_large_512"
NSAMPLES=150
SML_MODEL_PATH="weights/sml_model.dpredictor.${DEPTH_PREDICTOR}.nsamples.${NSAMPLES}.ckpt"
python evaluate.py -ds $DATASET_PATH -dp $DEPTH_PREDICTOR -ns $NSAMPLES -sm $SML_MODEL_PATH
```
Results for the example shown above:
```
Averaging metrics for globally-aligned depth over 800 samples
Averaging metrics for SML-aligned depth over 800 samples
+---------+----------+----------+
| metric | GA Only | GA+SML |
+---------+----------+----------+
| RMSE | 191.36 | 142.85 |
| MAE | 115.84 | 76.95 |
| AbsRel | 0.069 | 0.046 |
| iRMSE | 72.70 | 57.13 |
| iMAE | 49.32 | 34.25 |
| iAbsRel | 0.071 | 0.048 |
+---------+----------+----------+
```
To evaluate on VOID test sets at different densities (void_150, void_500, void_1500), change the `NSAMPLES` argument above accordingly.
## Citation
If you reference our work, please consider citing the following:
```bib
@inproceedings{wofk2023videpth,
author = {{Wofk, Diana and Ranftl, Ren\'{e} and M{\"u}ller, Matthias and Koltun, Vladlen}},
title = {{Monocular Visual-Inertial Depth Estimation}},
booktitle = {{IEEE International Conference on Robotics and Automation (ICRA)}},
year = {{2023}}
}
```
## Acknowledgements
Our work builds on and uses code from [MiDaS](https://github.com/isl-org/MiDaS), [timm](https://github.com/rwightman/pytorch-image-models), and [PyTorch Lightning](https://lightning.ai/docs/pytorch/stable/). We'd like to thank the authors for making these libraries and frameworks available.
_Last revisited: August 2024_