https://github.com/prs-eth/deflow
[CVPRW 2023, Best Paper Award] DeFlow: Self-supervised 3D Motion Estimation of Debris Flow
https://github.com/prs-eth/deflow
opticalflow sceneflow selfsupervision sensorfusion
Last synced: 2 months ago
JSON representation
[CVPRW 2023, Best Paper Award] DeFlow: Self-supervised 3D Motion Estimation of Debris Flow
- Host: GitHub
- URL: https://github.com/prs-eth/deflow
- Owner: prs-eth
- License: mit
- Created: 2023-04-05T13:05:36.000Z (about 2 years ago)
- Default Branch: master
- Last Pushed: 2023-07-25T07:48:57.000Z (almost 2 years ago)
- Last Synced: 2025-03-24T18:11:25.261Z (3 months ago)
- Topics: opticalflow, sceneflow, selfsupervision, sensorfusion
- Language: Python
- Homepage:
- Size: 15.8 MB
- Stars: 31
- Watchers: 3
- Forks: 2
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
DeFlow: Self-supervised 3D Motion Estimation of Debris Flow :mountain:
Liyuan Zhu,
Yuru Jia,
Shengyu Huang,
Nicholas Meyer,
Andreas Wieser,
Konrad Schindler,
Jordan Aaron
ETH Zurich
Paper
| Website | Dataset
This repository is the official implementation of paper:
DeFlow: Self-supervised 3D Motion Estimation of Debris Flow, CVPRW 2023.
Existing work on scene flow estimation focuses on autonomous driving and mobile robotics, while automated solutions are lacking for motion in nature, such as that exhibited by debris flows. We propose \deflow, a model for 3D motion estimation of debris flows, together with a newly captured dataset. We adopt a novel multi-level sensor fusion architecture and self-supervision to incorporate the inductive biases of the scene. We further adopt a multi-frame temporal processing module to enable flow speed estimation over time. Our model achieves state-of-the-art optical flow and depth estimation on our dataset, and fully automates the motion estimation for debris flows.
## Installation :national_park:
First clone our repository:
```bash
git clone https://github.com/Zhu-Liyuan/DeFlow
cd DeFlow
```
You will need to install conda to build the environment.
```bash
conda create -n DeFlow python=3.9
conda activate DeFlow
pip install -r requirements.txt
```
## Dataset and pretrained model
We provide preprocessed debris flow dataset. The preprocessed dataset and checkpoint can be downloaded by running:
```shell
wget --no-check-certificate --show-progress https://share.phys.ethz.ch/~gsg/DeFlow/DeFlow_Dataset.zip
unzip DeFlow_Dataset.zip -d data
wget --no-check-certificate --show-progress https://share.phys.ethz.ch/~gsg/DeFlow/checkpoint.zip
unzip checkpoint.zip
```
You can also build your own dataset following the structure below
```Shell
├── Data
├── Cam1
├── 000001.jpg
├── 000002.jpg
.
.
├── 00000X.jpg
├── Cam2
├── 000001.jpg
├── 000002.jpg
.
.
├── 00000X.jpg
├── LiDAR
├── 000001.ply
├── 000002.ply
.
.
├── 00000X.ply
├── Transformations
├── cam_intrinxics.txt
├── LiCam_tranformations.txt
```
To train a model, run:
```bash
python main.py --config_path configs/deflow_default.yaml
```
and you can change the mode to eval in the config file for evaluation.
## Contact
If you have any questions, please let me know:
- Liyuan Zhu {[email protected]}
## Citation
If you use DeFlow for any academic work, please cite our original paper.
```bibtex
@InProceedings{zhu2023DeFlow,
author = {Liyuan Zhu and Yuru Jia and Shengyu Huang and Nicholas Meyer and Andreas Wieser and Konrad Schindler, Jordan Aaron},
title = {DEFLOW: Self-supervised 3D Motion Estimation of Debris Flow},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
month = {June},
year = {2023}
}
```
Additionally, we thank the respective developers of the following open-source projects:
- [PCAccumulation](https://github.com/prs-eth/PCAccumulation)
- [CamLiFlow](https://github.com/MCG-NJU/CamLiFlow)
- [Self-mono-sf](https://github.com/visinf/self-mono-sf)