Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/OpenDriveLab/ST-P3
[ECCV 2022] ST-P3, an end-to-end vision-based autonomous driving framework via spatial-temporal feature learning.
https://github.com/OpenDriveLab/ST-P3
end-to-end-autonomous-driving perception-prediction-planning
Last synced: about 2 months ago
JSON representation
[ECCV 2022] ST-P3, an end-to-end vision-based autonomous driving framework via spatial-temporal feature learning.
- Host: GitHub
- URL: https://github.com/OpenDriveLab/ST-P3
- Owner: OpenDriveLab
- License: apache-2.0
- Created: 2022-07-01T07:28:13.000Z (about 2 years ago)
- Default Branch: main
- Last Pushed: 2023-12-06T13:10:50.000Z (10 months ago)
- Last Synced: 2024-07-21T21:41:35.787Z (2 months ago)
- Topics: end-to-end-autonomous-driving, perception-prediction-planning
- Language: Python
- Homepage:
- Size: 1.83 MB
- Stars: 283
- Watchers: 9
- Forks: 34
- Open Issues: 12
-
Metadata Files:
- Readme: README.md
- Funding: .github/FUNDING.yml
- License: LICENSE
Awesome Lists containing this project
README
# ST-P3
> **ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learning**
> Shengchao Hu, [Li Chen](https://scholar.google.com/citations?hl=en&user=ulZxvY0AAAAJ), Penghao Wu, [Hongyang Li](https://lihongyang.info/), [Junchi Yan](https://thinklab.sjtu.edu.cn/), Dacheng Tao.
> - [arXiv Paper](https://arxiv.org/abs/2207.07601), ECCV 2022
> - Our [Blog](https://zhuanlan.zhihu.com/p/544387122) (in Chinese)
## Introduction
This reposity is the official PyTorch Lightning implementation for **ST-P3**.
**TL;DR**: we propose a spatial-temporal feature learning scheme towards a set of more representative features for perception, prediction and planning tasks simultaneously in autonomous driving, and thus devise an **explicit** pipeline to generate planning trajectories directly from raw sensor inputs.
## Get Started
### Setup
```
conda env create -f environment.yml
git clone https://github.com/OpenDriveLab/ST-P3.git
```
### Pre-trained models
- open-loop planning on nuScenes: [model](https://drive.google.com/file/d/1fPAzrgohTVeFfyXSUh5wUHB_US8v9HFa/view?usp=sharing).
- closed-loop planning on CARLA: [model](https://drive.google.com/file/d/17KAjamrzlN08XL-NdWkOd2BvpAJ4VZ9M/view?usp=sharing).
### Evaluation
To evaluate the model on nuScenes:
- Download the [nuScenes](https://www.nuscenes.org/download) dataset.
- Download the pretrained weights.
```
bash scripts/eval_plan.sh ${checkpoint} ${dataroot}
```
To evaluate the model on CARLA:
- Please refer to the [Transfuser](https://github.com/autonomousvision/transfuser) to set up the environment.
- Test with the `carla_agent.py` file and the pretrained weights.
### Training
```
# (recommended) perception module pretrain
bash scripts/train_perceive.sh ${configs} ${dataroot}
# (optional) prediction module training purpose, no need for e2e training
bash scripts/train_prediction.sh ${configs} ${dataroot} ${pretrained}
# entire model e2e training
bash scripts/train_plan.sh ${configs} ${dataroot} ${pretrained}
```
- To train the model from scratch on nuScenes, we recommend to train a perceptual weight first and use it to train subsequent tasks to prevent `nan` during training.
- If you would like to use the nuScenes depth data **(will be released very soon)**, put the depth folder in the dataroot directory and change `GT_DEPTH` in the config file to `True`.
## Benchmark
- Open-loop planning results on [nuScenes](https://github.com/nutonomy/nuscenes-devkit).
| Method | L2 (m) 1s | L2 (m) 2s | L2 (m) 3s | Collision (%) 1s | Collision (%) 2s | Collision (%) 3s |
|:---------:|:---------:|:---------:|:---------:|:----------------:|:----------------:|:----------------:|
| Vanilla | **0.50** | **1.25** | **2.80** | 0.68 | 0.98 | 2.76 |
| NMP | 0.61 | 1.44 | 3.18 | 0.66 | 0.90 | 2.34 |
| Freespace | 0.56 | 1.27 | 3.08 | 0.65 | 0.86 | 1.64 |
| **ST-P3** | 1.33 | 2.11 | 2.90 | **0.23** | **0.62** | **1.27** |
- Closed-loop simulation results on [CARLA](https://github.com/carla-simulator/carla).
| Method | Town05 Short DS | Town05 Short RC | Town05 Long DS | Tow05 Long RC |
|:----------:|:---------------:|:---------------:|:--------------:|:-------------:|
| CILRS | 7.47 | 13.40 | 3.68 | 7.19 |
| LBC | 30.97 | 55.01 | 7.05 | 32.09 |
| Transfuser | 54.52 | 78.41 | **33.15** | 56.36 |
| **ST-P3** | **55.14** | **86.74** | 11.45 | **83.15** |
## Visualization
- nuScenes visualization results
- CARLA visualization results
## Citation
If you find our repo or our paper useful, please use the following citation:
```
@inproceedings{hu2022stp3,
title={ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learning},
author={Shengchao Hu and Li Chen and Penghao Wu and Hongyang Li and Junchi Yan and Dacheng Tao},
booktitle={European Conference on Computer Vision (ECCV)},
year={2022}
}
```
## License
All code within this repository is under [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0).
## Acknowledgement
We thank Xiangwei Geng for his support on the depth map generation, and fruitful discussions from [Xiaosong Jia](https://jiaxiaosong1002.github.io/). We have many thanks to [FIERY](https://github.com/wayveai/fiery) team for their exellent open source project.