https://github.com/vita-epfl/social-transmotion
[ICLR 2024] Official implementation of "Social-Transmotion: Promptable Human Trajectory Prediction" in PyTorch.
https://github.com/vita-epfl/social-transmotion
attention-mechanism deep-learning human-trajectory-prediction prompt
Last synced: 6 months ago
JSON representation
[ICLR 2024] Official implementation of "Social-Transmotion: Promptable Human Trajectory Prediction" in PyTorch.
- Host: GitHub
- URL: https://github.com/vita-epfl/social-transmotion
- Owner: vita-epfl
- License: agpl-3.0
- Created: 2023-12-25T15:12:40.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2024-04-13T16:24:46.000Z (about 2 years ago)
- Last Synced: 2024-04-14T06:53:15.434Z (about 2 years ago)
- Topics: attention-mechanism, deep-learning, human-trajectory-prediction, prompt
- Language: Python
- Homepage:
- Size: 221 KB
- Stars: 16
- Watchers: 2
- Forks: 1
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
Social-Transmotion:
Promptable Human Trajectory Prediction
Saeed Saadatnejad*, Yang Gao*, Kaouther Messaoud, Alexandre Alahi
International Conference on Learning Representations (ICLR), Austria, May 2024
[[Paper](https://arxiv.org/abs/2312.16168)] [[ICLR page](https://iclr.cc/virtual/2024/poster/18604)] [[Poster](docs/Poster.pdf)] [[Slides](docs/iclr_slides.pdf)]
Abstract
Accurate human trajectory prediction is crucial for applications such as autonomous vehicles, robotics, and surveillance systems. Yet, existing models often fail to fully leverage the non-verbal social cues human subconsciously communicate when navigating the space. To address this, we introduce Social-Transmotion, a generic Transformer-based model that exploits diverse and numerous visual cues to predict human behavior. We translate the idea of a prompt from Natural Language Processing (NLP) to the task of human trajectory prediction, where a prompt can be a sequence of x-y coordinates on the ground, bounding boxes in the image plane, or body pose keypoints in either 2D or 3D. This, in turn, augments trajectory data, leading to enhanced human trajectory prediction. Using masking technique, our model exhibits flexibility and adaptability by capturing spatiotemporal interactions between agents based on the available visual cues. We delve into the merits of using 2D versus 3D poses, and a limited set of poses. Additionally, we investigate the spatial and temporal attention map to identify which keypoints and time-steps in the sequence are vital for optimizing human trajectory prediction. Our approach is validated on multiple datasets, including JTA, JRDB, Pedestrians and Cyclists in Road Traffic, and ETH-UCY.
# Getting Started
Install the requirements using `pip`:
```
pip install -r requirements.txt
```
We have conveniently added the preprocessed data to the release section of the repository (for license details, please refer to the original papers).
Place the data subdirectory of JTA under `data/jta_all_visual_cues` and the data subdirectory of JRDB under `data/jrdb_2dbox` of the repository.
# Training and Testing
## JTA dataset
You can train the Social-Transmotion model on this dataset using the following command:
```
python train_jta.py --cfg configs/jta_all_visual_cues.yaml --exp_name jta
```
To evaluate the trained model, use the following command:
```
python evaluate_jta.py --ckpt ./experiments/jta/checkpoints/checkpoint.pth.tar --metric ade_fde --modality traj+all
```
Please note that the evaluation modality can be any of `[traj, traj+2dbox, traj+3dpose, traj+2dpose, traj+3dpose+3dbox, traj+all]`.
For the ease of use, we have also provided the trained model in the release section of this repo. In order to use that, you should pass the address of the saved checkpoint via `--ckpt`.
## JRDB dataset
You can train the Social-Transmotion model on this dataset using the following command:
```
python train_jrdb.py --cfg configs/jrdb_2dbox.yaml --exp_name jrdb
```
To evaluate the trained model, use the following command:
```
python evaluate_jrdb.py --ckpt ./experiments/jrdb/checkpoints/checkpoint.pth.tar --metric ade_fde --modality traj+2dbox
```
Please note that the evaluation modality can be one any of `[traj, traj+2dbox]`.
For the ease of use, we have also provided the trained model in the release section of this repo. In order to use that, you should pass the address of the saved checkpoint via `--ckpt`.
# Work in Progress
This repository is work-in-progress and will continue to get updated and improved over the coming months.
```
@InProceedings{saadatnejad2024socialtransmotion,
title={Social-Transmotion: Promptable Human Trajectory Prediction},
author={Saadatnejad, Saeed and Gao, Yang and Messaoud, Kaouther and Alahi, Alexandre},
booktitle={International Conference on Learning Representations (ICLR)},
year={2024},
}
```
Feel free to check the follow-up work, Multi-Transmotion, a pre-trained model for multimodal multitask human motion prediction.
```
@inproceedings{gao2024multi,
title={Multi-Transmotion: Pre-trained Model for Human Motion Prediction},
author={Gao, Yang and Luan, Po-Chien and Alahi, Alexandre},
booktitle={8th Annual Conference on Robot Learning},
year={2024}
}
```