{"id":20731446,"url":"https://github.com/opendrivelab/st-p3","last_synced_at":"2025-04-05T10:08:38.454Z","repository":{"id":45812843,"uuid":"509361027","full_name":"OpenDriveLab/ST-P3","owner":"OpenDriveLab","description":"[ECCV 2022] ST-P3, an end-to-end vision-based autonomous driving framework via spatial-temporal feature learning.","archived":false,"fork":false,"pushed_at":"2024-08-14T02:09:32.000Z","size":1914,"stargazers_count":363,"open_issues_count":13,"forks_count":38,"subscribers_count":8,"default_branch":"main","last_synced_at":"2025-03-28T16:34:06.139Z","etag":null,"topics":["end-to-end-autonomous-driving","perception-prediction-planning"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/OpenDriveLab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["OpenDriveLab"],"patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"lfx_crowdfunding":null,"custom":null}},"created_at":"2022-07-01T07:28:13.000Z","updated_at":"2025-03-25T08:47:59.000Z","dependencies_parsed_at":"2024-01-18T14:42:13.123Z","dependency_job_id":"19b675a0-f42a-46b8-b6bb-65f9231fa246","html_url":"https://github.com/OpenDriveLab/ST-P3","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenDriveLab%2FST-P3","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenDriveLab%2FST-P3/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenDriveLab%2FST-P3/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenDriveLab%2FST-P3/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/OpenDriveLab","download_url":"https://codeload.github.com/OpenDriveLab/ST-P3/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247318744,"owners_count":20919484,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["end-to-end-autonomous-driving","perception-prediction-planning"],"created_at":"2024-11-17T05:14:52.848Z","updated_at":"2025-04-05T10:08:38.437Z","avatar_url":"https://github.com/OpenDriveLab.png","language":"Python","funding_links":["https://github.com/sponsors/OpenDriveLab"],"categories":[],"sub_categories":[],"readme":"# ST-P3\n\n![pipeline](imgs/pipeline.png)\n\n\u003e **ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learning**  \n\u003e Shengchao Hu, [Li Chen](https://scholar.google.com/citations?hl=en\u0026user=ulZxvY0AAAAJ), Penghao Wu, [Hongyang Li](https://lihongyang.info/), [Junchi Yan](https://thinklab.sjtu.edu.cn/), Dacheng Tao.       \n\u003e - [arXiv Paper](https://arxiv.org/abs/2207.07601), ECCV 2022\n\u003e - Our [Blog](https://zhuanlan.zhihu.com/p/544387122) (in Chinese) \n\n## Introduction\nThis reposity is the official PyTorch Lightning implementation for **ST-P3**.\n\n**TL;DR**: we propose a spatial-temporal feature learning scheme towards a set of more representative features for perception, prediction and planning tasks simultaneously in autonomous driving, and thus devise an **explicit** pipeline to generate planning trajectories directly from raw sensor inputs.\n\n## Get Started\n### Setup\n```\nconda env create -f environment.yml\ngit clone https://github.com/OpenDriveLab/ST-P3.git\n```\n\n### Pre-trained models\n- open-loop planning on nuScenes: [model](https://drive.google.com/file/d/1fPAzrgohTVeFfyXSUh5wUHB_US8v9HFa/view?usp=sharing).\n- closed-loop planning on CARLA: [model](https://drive.google.com/file/d/17KAjamrzlN08XL-NdWkOd2BvpAJ4VZ9M/view?usp=sharing).\n\n### Evaluation\nTo evaluate the model on nuScenes:\n- Download the [nuScenes](https://www.nuscenes.org/download) dataset.\n- Download the pretrained weights.\n\n```\nbash scripts/eval_plan.sh ${checkpoint} ${dataroot}\n```\n\nTo evaluate the model on CARLA:\n- Please refer to the [Transfuser](https://github.com/autonomousvision/transfuser) to set up the environment.\n- Test with the `carla_agent.py` file and the pretrained weights.\n\n\n### Training\n\n```\n# (recommended) perception module pretrain\nbash scripts/train_perceive.sh ${configs} ${dataroot}\n\n# (optional) prediction module training purpose, no need for e2e training\nbash scripts/train_prediction.sh ${configs} ${dataroot} ${pretrained}\n\n# entire model e2e training\nbash scripts/train_plan.sh ${configs} ${dataroot} ${pretrained}\n```\n\n- To train the model from scratch on nuScenes, we recommend to train a perceptual weight first and use it to train subsequent tasks to prevent `nan` during training. \n- If you would like to use the nuScenes depth data **(will be released very soon)**, put the depth folder in the dataroot directory and change `GT_DEPTH` in the config file to `True`.\n\n## Benchmark\n- Open-loop planning results on [nuScenes](https://github.com/nutonomy/nuscenes-devkit).\n\n| Method    | L2 (m) 1s | L2 (m) 2s | L2 (m) 3s | Collision (%) 1s | Collision (%) 2s | Collision (%) 3s |\n|:---------:|:---------:|:---------:|:---------:|:----------------:|:----------------:|:----------------:|\n| Vanilla   | **0.50**  | **1.25**  | **2.80**  | 0.68             | 0.98             | 2.76             |\n| NMP       | 0.61      | 1.44      | 3.18      | 0.66             | 0.90             | 2.34             |\n| Freespace | 0.56      | 1.27      | 3.08      | 0.65             | 0.86             | 1.64             |\n| **ST-P3** | 1.33      | 2.11      | 2.90      | **0.23**         | **0.62**         | **1.27**         |\n\n- Closed-loop simulation results on [CARLA](https://github.com/carla-simulator/carla).\n\n| Method     | Town05 Short DS | Town05 Short RC | Town05 Long DS | Tow05 Long RC |\n|:----------:|:---------------:|:---------------:|:--------------:|:-------------:|\n| CILRS      | 7.47            | 13.40           | 3.68           | 7.19          |\n| LBC        | 30.97           | 55.01           | 7.05           | 32.09         |\n| Transfuser | 54.52           | 78.41           | **33.15**      | 56.36         |\n| **ST-P3**  | **55.14**       | **86.74**       | 11.45          | **83.15**     |\n\n## Visualization\n- nuScenes visualization results\n\n\u003cimg src=imgs/nuScenes.png width=\"720\" height=\"360\" alt=\"nuscenes_vis\"/\u003e\u003cbr/\u003e\n\n- CARLA visualization results\n\n\u003cimg src=imgs/CARLA.png width=\"720\" height=\"240\" alt=\"CARLA_vis\"/\u003e\u003cbr/\u003e\n\n## Citation\n\nIf you find our repo or our paper useful, please use the following citation:\n\n```\n@inproceedings{hu2022stp3,\n title={ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learning}, \n author={Shengchao Hu and Li Chen and Penghao Wu and Hongyang Li and Junchi Yan and Dacheng Tao},\n booktitle={European Conference on Computer Vision (ECCV)},\n year={2022}\n}\n```\n\n## License\nAll code within this repository is under [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0).\n\n## Acknowledgement\nWe thank Xiangwei Geng for his support on the depth map generation, and fruitful discussions from [Xiaosong Jia](https://jiaxiaosong1002.github.io/). We have many thanks to [FIERY](https://github.com/wayveai/fiery) team for their exellent open source project.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopendrivelab%2Fst-p3","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopendrivelab%2Fst-p3","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopendrivelab%2Fst-p3/lists"}