https://github.com/tum-vision/tandem

[CoRL 21'] TANDEM: Tracking and Dense Mapping in Real-time using Deep Multi-view Stereo
https://github.com/tum-vision/tandem

3d-reconstruction multiview-stereo slam visual-odometry

Last synced: about 1 year ago
JSON representation

[CoRL 21'] TANDEM: Tracking and Dense Mapping in Real-time using Deep Multi-view Stereo

• Host: GitHub
• URL: https://github.com/tum-vision/tandem
• Owner: tum-vision
• Created: 2021-10-29T13:31:00.000Z (over 4 years ago)
• Default Branch: master
• Last Pushed: 2022-12-29T02:10:16.000Z (over 3 years ago)
• Last Synced: 2025-04-04T01:11:07.183Z (about 1 year ago)
• Topics: 3d-reconstruction, multiview-stereo, slam, visual-odometry
• Language: C++
• Homepage:
• Size: 62.1 MB
• Stars: 935
• Watchers: 39
• Forks: 153
• Open Issues: 36
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

          
TANDEM: Tracking and Dense Mapping
in Real-time using Deep Multi-view Stereo



    Lukas Koestler1*   

    Nan Yang1,2*,†   

    Niclas Zeller2,3   

    Daniel Cremers1,2





    *equal contribution   

    †corresponding author





    1Technical University of Munich   

    2Artisense


    3Karlsruhe University of Applied Sciences





    Conference on Robot Learning (CoRL) 2021, London, UK





    3DV 2021 Best Demo Award





    arXiv |

    Video |

    OpenReview |

    Project Page



## Code and Data

- [x] 📣 C++ code released before Christmas! Please check [tandem/](tandem/).

- [x] 📣 CVA-MVSNet released! Please check [cva_mvsnet/](cva_mvsnet/).

- [x] 📣 Replica training data released! Please check [replica/](replica/).

- [x] Minor improvements throughout January. **Contributions are highly welcomed!**

- [x] Release of the ScanNet-trained model 

- [ ] Docker image for TANDEM. **Contributions are highly welcomed!**

### Abstract

In this paper, we present TANDEM a real-time monocular tracking and dense mapping framework. For pose estimation, TANDEM performs photometric bundle adjustment based on a sliding window of keyframes. To increase the robustness, we propose a novel tracking front-end that performs dense direct image alignment using depth maps rendered from a global model that is built incrementally from dense depth predictions. To predict the dense depth maps, we propose Cascade View-Aggregation MVSNet (CVA-MVSNet) that utilizes the entire active keyframe window by hierarchically constructing 3D cost volumes with adaptive view aggregation to balance the different stereo baselines between the keyframes. Finally, the predicted depth maps are fused into a consistent global map represented as a truncated signed distance function (TSDF) voxel grid. Our experimental results show that TANDEM outperforms other state-of-the-art traditional and learning-based monocular visual odometry (VO) methods in terms of camera tracking. Moreover, TANDEM shows state-of-the-art real-time 3D reconstruction performance.


### Poster