https://github.com/runnanchen/Label-Free-Scene-Understanding

https://github.com/runnanchen/Label-Free-Scene-Understanding

Last synced: about 2 months ago
JSON representation

• Host: GitHub
• URL: https://github.com/runnanchen/Label-Free-Scene-Understanding
• Owner: runnanchen
• Created: 2023-06-06T16:18:15.000Z (over 1 year ago)
• Default Branch: main
• Last Pushed: 2023-11-19T09:37:36.000Z (about 1 year ago)
• Last Synced: 2023-11-19T10:33:10.753Z (about 1 year ago)
• Size: 27 MB
• Stars: 41
• Watchers: 10
• Forks: 1
• Open Issues: 2
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

• Awesome-Segment-Anything - [code

README

        
# Towards Label-free Scene Understanding by Vision Foundation Models (NeurIPS 2023)

![Overview of the method](./assets/teaser.jpeg)

We study how vision foundation models enable networks to comprehend 2D and 3D environments without relying on labelled data. To accomplish this, we introduce a novel framework called Cross-modality Noisy Supervision (CNS). By effectively harnessing the strengths of CLIP and

SAM, our approach simultaneously trains 2D and 3D networks, yielding remarkable performance. [[Preprint Paper]](https://arxiv.org/pdf/2306.03899.pdf) 

The codebase is adapted from [CLIP2Scene](https://github.com/runnanchen/CLIP2Scene). Codes will be released later this year.

# Qualitative Evaluation 

**scannet 2D.**

![Overview of the method](./assets/suplementary_scanNet2D.jpeg)

**scannet 3D.**

![Overview of the method](./assets/visual_scannet_3D.jpeg)

**nuImages.**

![Overview of the method](./assets/suplementary_nuImages2D.jpeg)

**nuScenes 2D.**

![Overview of the method](./assets/suplementary_nuScenes2D.jpeg)

**nuScenes 3D.**

![Overview of the method](./assets/suplementary_nuScenes3D.jpeg)

# Citation

```

@inproceedings{chen2023clip2scene,

  title={CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIP},

  author={Chen, Runnan and Liu, Youquan and Kong, Lingdong and Zhu, Xinge and Ma, Yuexin and Li, Yikang and Hou, Yuenan and Qiao, Yu and Wang, Wenping},

  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},

  pages={7020--7030},

  year={2023}

}

@inproceedings{chen2023towards,

  title={Towards label-free scene understanding by vision foundation models},

  author={Chen, Runnan and Liu, Youquan and Kong, Lingdong and Chen, Nenglun and Xinge, ZHU and Ma, Yuexin and Liu, Tongliang and Wang, Wenping},

  booktitle={Thirty-seventh Conference on Neural Information Processing Systems},

  year={2023}

}

```