https://github.com/haomo-ai/ModaLink

[IROS 2024] This repository contains the implementation of our paper: ModaLink: Unifying Modalities for Efficient Image-to-PointCloud Place Recognition
https://github.com/haomo-ai/ModaLink

camera-to-lidar cross-modal-localization cross-modal-retrieval place-recognition

Last synced: 1 day ago
JSON representation

[IROS 2024] This repository contains the implementation of our paper: ModaLink: Unifying Modalities for Efficient Image-to-PointCloud Place Recognition

• Host: GitHub
• URL: https://github.com/haomo-ai/ModaLink
• Owner: haomo-ai
• License: gpl-3.0
• Created: 2024-03-14T15:59:51.000Z (7 months ago)
• Default Branch: main
• Last Pushed: 2024-09-07T03:14:32.000Z (26 days ago)
• Last Synced: 2024-09-07T05:03:39.158Z (26 days ago)
• Topics: camera-to-lidar, cross-modal-localization, cross-modal-retrieval, place-recognition
• Language: Python
• Homepage:
• Size: 38.1 KB
• Stars: 11
• Watchers: 6
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# [IROS 2024] ModaLink

This repository contains the implementation of our IROS 2024 paper:

**ModaLink: Unifying Modalities for Efficient Image-to-PointCloud Place Recognition**

[Weidong Xie](https://sites.google.com/view/dong-hao/), [Lun Luo](https://zjuluolun.github.io/), Nanfei Ye, Yi Ren, Shaoyi Du, Minhang Wang, Jintao Xu, Rui Ai, Weihao Gu and [Xieyuanli Chen](https://github.com/Chen-Xieyuanli)

[Link](https://arxiv.org/abs/2403.18762) to the arXiv version of the paper is available.

The main contributions of this work are:

*  We propose a lightweight cross-modal place recognition method called ModaLink based on FoV transformation. 

*  We introduce a Non-Negative Matrix Factorization-based module to extract extra potential semantic features to improve the distinctiveness of descriptors.

*  Extensive experimental results on the KITTI and a self-collected dataset show that our proposed method can achieve state-of-the-art performance while running in real-time of about 30Hz.

![main](https://github.com/haomo-ai/ModaLink/assets/47657625/28ab99f7-2eaa-4d96-9a7a-918719a69b8d)

## Citation

If you use our implementation in your academic work, please cite the corresponding [paper](https://arxiv.org/abs/2403.18762):

```

@inproceedings{xie2024modalink,

	author   = {Weidong Xie and Lun Luo and Nanfei Ye and Yi Ren and Shaoyi Du and Minhang Wang and Jintao Xu and Rui Ai and Weihao Gu and Xieyuanli Chen},

	title    = {{ModaLink: Unifying Modalities for Efficient Image-to-PointCloud Place Recognition}},

	booktitle  = {In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},

	year     = {2024},

}

```

## Dependencies

We use pytorch-gpu for neural networks. An Nvidia GPU is needed for faster retrieval.

To use a GPU, first, you need to install the Nvidia driver and CUDA.

- CUDA Installation guide: [link](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html)