https://github.com/jingkang50/psg4d
4D Panoptic Scene Graph Generation (NeurIPS'23 Spotlight)
https://github.com/jingkang50/psg4d
Last synced: about 1 month ago
JSON representation
4D Panoptic Scene Graph Generation (NeurIPS'23 Spotlight)
- Host: GitHub
- URL: https://github.com/jingkang50/psg4d
- Owner: Jingkang50
- Created: 2023-05-07T12:48:19.000Z (about 2 years ago)
- Default Branch: main
- Last Pushed: 2025-03-13T09:27:27.000Z (2 months ago)
- Last Synced: 2025-03-31T07:04:28.848Z (about 2 months ago)
- Language: Python
- Homepage:
- Size: 4.4 MB
- Stars: 105
- Watchers: 4
- Forks: 2
- Open Issues: 6
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# 4D Panoptic Scene Graph Generation
|  |
|:--:|
4D Panoptic Scene Graph Generation
Jingkang Yang,
Jun Cen,
Wenxuan Peng,
Shuai Liu,
Fangzhou Hong,
Xiangtai Li,
Kaiyang Zhou,
Qifeng Chen,
Ziwei Liu,
S-Lab, NTU & HKUST & BUPT & HKBU
---
## What is PSG4D Task?
The PSG4D (4D Panoptic Scene Graph Generation) Task is a novel task that aims to bridge the gap between raw visual inputs in a dynamic 4D world and high-level visual understanding. It involves generating a comprehensive 4D scene graph from RGB-D video sequences or point cloud video sequences.
## The PSG4D Dataset
We provide two dataset to facilitate PSG4D research. Each dataset is composed with RGB-D/3D videos. To access them, please checkout [`data/GTA`](data/GTA) and [`data/HOI`](data/HOI). If you find downloading PSG4D-GTA dataset challenging, please email `[email protected]` for some useful tips.
|  ||
|:--:|:--:|
| PSG4D-GTA Dataset Demo | PSG4D-HOI Dataset Demo |
## How to Run
|  |
|:--:|
|Illustration of the PSG4DFormer pipeline. The PSG4DFormer is a two stage pipeline. For Panoptic Segmentation part, please refer to `rgbd_seg` for RGB-D segmentation and `pc_seg` for point cloud segmentation. Then please refer to `*_track`. The relation modeling is identical to our previous work [OpenPVSG](https://github.com/LilyDaytoy/OpenPVSG?tab=readme-ov-file#training-and-testing). Each part can be considered as a standalone code, so please checkout the readme in each directory.|
## Citation
If you find our repository useful for your research, please consider citing our paper:
```bibtex
@inproceedings{yang2023psg4d,
author = {Yang, Jingkang and Cen, Jun and Peng, Wenxuan and Liu, Shuai amd Hong, Fangzhou and Li, Xiangtai and Zhou, Kaiyang and Chen, Qifeng and Liu, Ziwei}
title = {4D Panoptic Scene Graph Generation},
booktitle = {NeurIPS},
year = {2023},
}
```