{"id":13442692,"url":"https://github.com/OpenDriveLab/UniAD","last_synced_at":"2025-03-20T15:30:35.019Z","repository":{"id":89202217,"uuid":"575306900","full_name":"OpenDriveLab/UniAD","owner":"OpenDriveLab","description":"[CVPR 2023 Best Paper Award] Planning-oriented Autonomous Driving","archived":false,"fork":false,"pushed_at":"2024-08-28T09:01:41.000Z","size":10042,"stargazers_count":3449,"open_issues_count":77,"forks_count":382,"subscribers_count":38,"default_branch":"main","last_synced_at":"2024-10-28T02:37:53.661Z","etag":null,"topics":["autonomous-driving","autonomous-driving-framework","bev-segmentation","end-to-end-autonomous-driving","motion-planning","motion-prediction","multi-object-tracking","occupancy-prediction","perception-prediction-planning"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/OpenDriveLab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["OpenDriveLab"],"patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"lfx_crowdfunding":null,"custom":null}},"created_at":"2022-12-07T08:05:49.000Z","updated_at":"2024-10-28T02:20:48.000Z","dependencies_parsed_at":null,"dependency_job_id":"e931a4b6-218a-493d-ac1d-358703c936b6","html_url":"https://github.com/OpenDriveLab/UniAD","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenDriveLab%2FUniAD","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenDriveLab%2FUniAD/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenDriveLab%2FUniAD/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenDriveLab%2FUniAD/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/OpenDriveLab","download_url":"https://codeload.github.com/OpenDriveLab/UniAD/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221772536,"owners_count":16878124,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["autonomous-driving","autonomous-driving-framework","bev-segmentation","end-to-end-autonomous-driving","motion-planning","motion-prediction","multi-object-tracking","occupancy-prediction","perception-prediction-planning"],"created_at":"2024-07-31T03:01:49.246Z","updated_at":"2024-10-28T03:30:53.062Z","avatar_url":"https://github.com/OpenDriveLab.png","language":"Python","funding_links":["https://github.com/sponsors/OpenDriveLab"],"categories":["Python","Applications","Repos","💻 Open-Source Projects","1. 机器人项目 | Robots"],"sub_categories":["提示语（魔法）","Papers"],"readme":"\u003cdiv align=\"center\"\u003e   \n  \n# Planning-oriented Autonomous Driving\n\u003c/div\u003e\n\n\n\u003ch3 align=\"center\"\u003e\n  \u003ca href=\"https://arxiv.org/abs/2212.10156\"\u003earXiv\u003c/a\u003e |\n  \u003ca href=\"https://www.youtube.com/watch?v=cyrxJJ_nnaQ\"\u003eVideo\u003c/a\u003e |\n  \u003ca href=\"sources/cvpr23_uniad_poster.png\"\u003ePoster\u003c/a\u003e |\n  \u003ca href=\"https://opendrivelab.com/e2ead/UniAD_plenary_talk_slides.pdf\"\u003eSlides\u003c/a\u003e\n\u003c/h3\u003e\n\n\n\n\n\nhttps://github.com/OpenDriveLab/UniAD/assets/48089846/bcf685e4-2471-450e-8b77-e028a46bd0f7\n\n\n\n\n\n\n\u003cbr\u003e\u003cbr\u003e\n\n![teaser](sources/pipeline.png)\n\n## Table of Contents:\n1. [Highlights](#high)\n2. [News](#news)\n3. [Getting Started](#start)\n   - [Installation](docs/INSTALL.md)\n   - [Prepare Dataset](docs/DATA_PREP.md)\n   - [Evaluation Example](docs/TRAIN_EVAL.md#example)\n   - [GPU Requirements](docs/TRAIN_EVAL.md#gpu)\n   - [Train/Eval](docs/TRAIN_EVAL.md)\n4. [Results and Models](#models)\n5. [TODO List](#todos)\n6. [License](#license)\n7. [Citation](#citation)\n8. [🔥 See Also: GenAD \u0026 Vista](#see)\n\n## Highlights \u003ca name=\"high\"\u003e\u003c/a\u003e\n\n- :oncoming_automobile: **Planning-oriented philosophy**: UniAD is a Unified Autonomous Driving algorithm framework following a planning-oriented philosophy. Instead of standalone modular design and multi-task learning, we cast a series of tasks, including perception, prediction and planning tasks hierarchically.\n- :trophy: **SOTA performance**: All tasks within UniAD achieve SOTA performance, especially prediction and planning (motion: 0.71m minADE, occ: 63.4% IoU, planning: 0.31% avg.Col)\n\n## News \u003ca name=\"news\"\u003e\u003c/a\u003e\n\n- **`Paper Title Change`**: To avoid confusion with the \"goal-point\" navigation in Robotics, we change the title from \"Goal-oriented\" to \"Planning-oriented\" suggested by Reviewers. Thank you!\n\n- **`Planning Metric`**: Discussion [Ref: https://github.com/OpenDriveLab/UniAD/issues/29]: [Clarification](https://github.com/OpenDriveLab/UniAD/issues/29#issuecomment-1583070151) and [Notice](https://github.com/OpenDriveLab/UniAD/issues/29#issuecomment-1717594344) regarding open-loop planning results comparison.\n\n- **`2024/08/27`** New feature: Implementation for CARLA and closed-loop evaluation on CARLA Leaderboard 2.0 scenarios are available in [Bench2Drive](https://github.com/Thinklab-SJTU/Bench2Drive).\n\n- **`2023/08/03`** Bugfix [[Commit](https://github.com/OpenDriveLab/UniAD/commit/2e1380143d7af7c93bd67725a11d6960fa4347c6)]: Previously, the visualized planning results were in opposition on the x axis, compared to the ground truth. Now it's fixed.\n\n- **`2023/06/12`** Bugfix [Ref: https://github.com/OpenDriveLab/UniAD/issues/21]: Previously, the performance of the stage1 model (track_map) could not be replicated when trained from scratch, due to mistakenly adding `loss_past_traj` and freezing `img_neck` and `BN`. By removing `loss_past_traj` and unfreezing `img_neck` and `BN` in training, the reported results could be reproduced (AMOTA: 0.393, [stage1_train_log](https://github.com/OpenDriveLab/UniAD/releases/download/v1.0/uniad_reproduce_stage1_gpu16_train.log)).\n\n- **`2023/04/18`** New feature: You can replace BEVFormer with other BEV Encoding methods, e.g., LSS, as long as you provide the `bev_embed` and `bev_pos` in [track_train](https://github.com/OpenDriveLab/UniAD/blob/cb4e3dc336ac9f94897ef3c7d85edba85a507726/projects/mmdet3d_plugin/uniad/detectors/uniad_track.py#L394) and [track_inference](https://github.com/OpenDriveLab/UniAD/blob/cb4e3dc336ac9f94897ef3c7d85edba85a507726/projects/mmdet3d_plugin/uniad/detectors/uniad_track.py#L661). Make sure your bevs and ours are of the same shape.\n- **`2023/04/18`** Base-model checkpoints are released.\n\n\n- **`2023/03/29`** Code \u0026 model initial release `v1.0`.\n- **`2023/03/21`** 🌟🌟 UniAD is accepted by CVPR 2023, as an **Award Candidate** (12 out of 2360 accepted papers)!\n- **`2022/12/21`** UniAD [paper](https://arxiv.org/abs/2212.10156) is available on arXiv.\n\n\n\n## Getting Started \u003ca name=\"start\"\u003e\u003c/a\u003e\n- [Installation](docs/INSTALL.md)\n- [Prepare Dataset](docs/DATA_PREP.md)\n- [Evaluation Example](docs/TRAIN_EVAL.md#example)\n- [GPU Requirements](docs/TRAIN_EVAL.md#gpu)\n- [Train/Eval](docs/TRAIN_EVAL.md)\n\n## Results and Pre-trained Models \u003ca name=\"models\"\u003e\u003c/a\u003e\nUniAD is trained in two stages. Pretrained checkpoints of both stages will be released and the results of each model are listed in the following tables.\n\n### Stage1: Perception training\n\u003e We first train the perception modules (i.e., track and map) to obtain a stable weight initlization for the next stage. BEV features are aggregated with 5 frames (queue_length = 5).\n\n| Method | Encoder | Tracking\u003cbr\u003eAMOTA | Mapping\u003cbr\u003eIoU-lane | config | Download |\n| :---: | :---: | :---: | :---: | :---:|:---:| \n| UniAD-B | R101 | 0.390 | 0.297 |  [base-stage1](projects/configs/stage1_track_map/base_track_map.py) | [base-stage1](https://github.com/OpenDriveLab/UniAD/releases/download/v1.0/uniad_base_track_map.pth) |\n\n\n\n### Stage2: End-to-end training\n\u003e We optimize all task modules together, including track, map, motion, occupancy and planning. BEV features are aggregated with 3 frames (queue_length = 3).\n\n\u003c!-- \nPre-trained models and results under main metrics are provided below. We refer you to the [paper](https://arxiv.org/abs/2212.10156) for more details. --\u003e\n\n| Method | Encoder | Tracking\u003cbr\u003eAMOTA | Mapping\u003cbr\u003eIoU-lane | Motion\u003cbr\u003eminADE |Occupancy\u003cbr\u003eIoU-n. | Planning\u003cbr\u003eavg.Col. | config | Download |\n| :---: | :---: | :---: | :---: | :---:|:---:| :---: | :---: | :---: |\n| UniAD-B | R101 | 0.363 | 0.313 | 0.705 | 63.7 | 0.29 |  [base-stage2](projects/configs/stage2_e2e/base_e2e.py) | [base-stage2](https://github.com/OpenDriveLab/UniAD/releases/download/v1.0.1/uniad_base_e2e.pth) |\n\n### Checkpoint Usage\n* Download the checkpoints you need into `UniAD/ckpts/` directory.\n* You can evaluate these checkpoints to reproduce the results, following the `evaluation` section in [TRAIN_EVAL.md](docs/TRAIN_EVAL.md).\n* You can also initialize your own model with the provided weights. Change the `load_from` field to `path/of/ckpt` in the config and follow the `train` section in [TRAIN_EVAL.md](docs/TRAIN_EVAL.md) to start training.\n\n\n### Model Structure\nThe overall pipeline of UniAD is controlled by [uniad_e2e.py](projects/mmdet3d_plugin/uniad/detectors/uniad_e2e.py) which coordinates all the task modules in `UniAD/projects/mmdet3d_plugin/uniad/dense_heads`. If you are interested in the implementation of a specific task module, please refer to its corresponding file, e.g., [motion_head](projects/mmdet3d_plugin/uniad/dense_heads/motion_head.py).\n\n\n## License \u003ca name=\"license\"\u003e\u003c/a\u003e\n\nAll assets and code are under the [Apache 2.0 license](./LICENSE) unless specified otherwise.\n\n## Citation \u003ca name=\"citation\"\u003e\u003c/a\u003e\n\n\nIf you find our project useful for your research, please consider citing our paper and codebase with the following BibTeX:\n\n\n```bibtex\n@inproceedings{hu2023_uniad,\n title={Planning-oriented Autonomous Driving}, \n author={Yihan Hu and Jiazhi Yang and Li Chen and Keyu Li and Chonghao Sima and Xizhou Zhu and Siqi Chai and Senyao Du and Tianwei Lin and Wenhai Wang and Lewei Lu and Xiaosong Jia and Qiang Liu and Jifeng Dai and Yu Qiao and Hongyang Li},\n booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},\n year={2023},\n}\n```\n\n```bibtex\n@misc{contributors2023_uniadrepo,\n  title={Planning-oriented Autonomous Driving},\n  author={UniAD contributors},\n  howpublished={\\url{https://github.com/OpenDriveLab/UniAD}},\n  year={2023}\n}\n```\n## Related Resources\n\n[![Awesome](https://awesome.re/badge.svg)](https://awesome.re)\n- [BEVFormer](https://github.com/fundamentalvision/BEVFormer) (Ours!)\n- [ST-P3](https://github.com/OpenPerceptionX/ST-P3) (Ours!)\n- [FIERY](https://github.com/wayveai/fiery)\n- [MOTR](https://github.com/megvii-research/MOTR)\n\n\n## 🔥 See Also  \u003ca name=\"see\"\u003e\u003c/a\u003e\nWe are thrilled to launch our recent line of works: [GenAD](https://arxiv.org/abs/2403.09630) and [Vista](https://arxiv.org/abs/2405.17398), to advance  **driving world models** with the **largest driving video dataset** collected from the web - [OpenDV](https://github.com/OpenDriveLab/DriveAGI/tree/main/opendv).\n\n\n[GenAD](https://github.com/OpenDriveLab/DriveAGI): **Generalized Predictive Model for Autonomous Driving** (CVPR'24, Highlight ⭐)\n\n\u003cdiv id=\"top\" align=\"center\"\u003e\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"sources/opendv_dataset.png\" width=\"1000px\" \u003e\n\u003c/p\u003e\n\u003c/div\u003e\n\n\n[Vista](https://github.com/OpenDriveLab/Vista): **A Generalizable Driving World Model with High Fidelity and Versatile Controllability** 🌏\n\u003cdiv id=\"top\" align=\"center\"\u003e\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"sources/vista.gif\" width=\"1000px\" \u003e\n\u003c/p\u003e\n\u003c/div\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FOpenDriveLab%2FUniAD","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FOpenDriveLab%2FUniAD","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FOpenDriveLab%2FUniAD/lists"}