https://github.com/hustvl/mim4d

MIM4D: Masked Modeling with Multi-View Video for Autonomous Driving Representation Learning
https://github.com/hustvl/mim4d

Last synced: 3 days ago
JSON representation

MIM4D: Masked Modeling with Multi-View Video for Autonomous Driving Representation Learning

• Host: GitHub
• URL: https://github.com/hustvl/mim4d
• Owner: hustvl
• License: apache-2.0
• Created: 2024-03-13T07:37:18.000Z (8 months ago)
• Default Branch: master
• Last Pushed: 2024-03-14T01:32:33.000Z (8 months ago)
• Last Synced: 2024-08-03T09:07:43.479Z (3 months ago)
• Size: 2.58 MB
• Stars: 57
• Watchers: 10
• Forks: 1
• Open Issues: 1
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        


MIM4D 

Masked Modeling with Multi-View Video for Autonomous Driving Representation Learning


[Jialv Zou](https://github.com/Doctor-James)1 \*, [Bencheng Liao](https://github.com/LegendBC)2,1 \*, [Qian Zhang](https://scholar.google.com/citations?user=pCY-bikAAAAJ&hl=zh-CN)3, [Wenyu Liu](http://eic.hust.edu.cn/professor/liuwenyu/)1, [Xinggang Wang](https://xwcv.github.io/)1 :email:

 

1  School of EIC, HUST, 2  Institute of Artificial Intelligence, HUST,   3 Horizon Robotics

(\*) equal contribution, (:email:) corresponding author.

 

ArXiv Preprint ([arXiv 2403.08760](https://arxiv.org/abs/2403.08760))



#

### News

* **` Mar. 14th, 2024`:** We released our paper on Arxiv. Code/Models are coming soon. Please stay tuned! ☕️

## Abstract

Learning robust and scalable visual representations from massive multi-view video data remains a challenge in computer vision and autonomous driving. Existing pre-training methods either rely on expensive supervised learning with 3D annotations, limiting the scalability, or

focus on single-frame or monocular inputs, neglecting the temporal information. We propose MIM4D, a novel pre-training paradigm based on dual masked image modeling (MIM). MIM4D leverages both spatial and temporal relations by training on masked multi-view video inputs. It 

constructs pseudo-3D features using continuous scene flow and projects them onto 2D plane for supervision. To address the lack of dense 3D supervision, MIM4D reconstruct pixels by employing 3D volumetric differentiable rendering to learn geometric representations. 

We demonstrate that MIM4D achieves state-of-the-art performance on the nuScenes dataset for visual representation learning in autonomous driving. It significantly improves existing methods on multiple downstream tasks, including BEV segmentation (8.7% IoU), 

3D object detection (3.5% mAP), and HD map construction (1.4% mAP). Our work offers a new choice for learning representation at scale in autonomous driving.







## Citation

If you find MIM4D is useful in your research or applications, please consider giving us a star 🌟 and citing it by the following BibTeX entry.

```bibtex

 @article{mim4d,

  title={MIM4D: Masked Modeling with Multi-View Video for Autonomous Driving Representation Learning},

  author={Jialv Zou and Bencheng Liao and Qian Zhang and Wenyu Liu and Xinggang Wang},

  journal={arXiv preprint arXiv:2403.08760},

  year={2024}

}

```