{"id":13430348,"url":"https://github.com/OpenDriveLab/DriveLM","last_synced_at":"2025-03-16T05:30:48.078Z","repository":{"id":190872154,"uuid":"676089005","full_name":"OpenDriveLab/DriveLM","owner":"OpenDriveLab","description":"[ECCV 2024 Oral] DriveLM: Driving with Graph Visual Question Answering","archived":false,"fork":false,"pushed_at":"2025-03-04T16:56:25.000Z","size":287123,"stargazers_count":998,"open_issues_count":21,"forks_count":65,"subscribers_count":22,"default_branch":"main","last_synced_at":"2025-03-10T14:11:26.934Z","etag":null,"topics":["autonomous-driving","chain-of-thought","graph-of-thoughts","large-language-models","llm","prompt-engineering","prompting","tree-of-thoughts","vision-language"],"latest_commit_sha":null,"homepage":"https://opendrivelab.com/DriveLM/","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/OpenDriveLab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["OpenDriveLab"],"patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"lfx_crowdfunding":null,"custom":null}},"created_at":"2023-08-08T12:07:33.000Z","updated_at":"2025-03-10T12:21:22.000Z","dependencies_parsed_at":"2024-07-26T12:56:00.698Z","dependency_job_id":null,"html_url":"https://github.com/OpenDriveLab/DriveLM","commit_stats":null,"previous_names":["opendrivelab/drivelm"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenDriveLab%2FDriveLM","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenDriveLab%2FDriveLM/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenDriveLab%2FDriveLM/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenDriveLab%2FDriveLM/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/OpenDriveLab","download_url":"https://codeload.github.com/OpenDriveLab/DriveLM/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243830912,"owners_count":20354848,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["autonomous-driving","chain-of-thought","graph-of-thoughts","large-language-models","llm","prompt-engineering","prompting","tree-of-thoughts","vision-language"],"created_at":"2024-07-31T02:00:52.509Z","updated_at":"2025-03-16T05:30:48.073Z","avatar_url":"https://github.com/OpenDriveLab.png","language":"HTML","funding_links":["https://github.com/sponsors/OpenDriveLab"],"categories":["3 Reasoning Tasks","Awesome Papers",":books: Papers","HTML","Autonomous Racing","Datasets","largemodel","Autonomous Driving"],"sub_categories":["3.8 Agent Reasoning","Datasets","Dataset \\\u0026 Benchmark","WRC Rally","Social \u0026 Human-Robot Interaction"],"readme":"\u003cdiv id=\"top\" align=\"center\"\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/images/repo/title_v2.jpg\"\u003e\n\u003c/p\u003e\n    \n**DriveLM:** *Driving with **G**raph **V**isual **Q**uestion **A**nswering*\n\n\u003c!-- Download dataset [**HERE**](docs/data_prep_nus.md) (serves as Official source for `Autonomous Driving Challenge 2024`) --\u003e\n`Autonomous Driving Challenge 2024` **Driving-with-Language** [Leaderboard](https://opendrivelab.com/challenge2024/#driving_with_language).\n\u003c/div\u003e\n\n\u003cdiv id=\"top\" align=\"center\"\u003e\n\n[![](https://img.shields.io/badge/Project%20Page-8A2BE2)](https://opendrivelab.com/DriveLM/)\n[![License: Apache2.0](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](#licenseandcitation)\n[![arXiv](https://img.shields.io/badge/arXiv-2312.14150-b31b1b.svg)](https://arxiv.org/abs/2312.14150)\n[![](https://img.shields.io/badge/Latest%20release-v1.1-yellow)](#gettingstarted)\n[![Hugging Face](https://img.shields.io/badge/Test%20Server-%F0%9F%A4%97-ffc107?color=ffc107\u0026logoColor=white)](https://huggingface.co/spaces/AGC2024/driving-with-language-official)\n\n\u003c!-- \u003ca href=\"https://opendrivelab.github.io/DriveLM\" target=\"_blank\"\u003e\n    \u003cimg alt=\"Github Page\" src=\"https://img.shields.io/badge/Project%20Page-white?logo=GitHub\u0026color=green\" /\u003e\n  \u003c/a\u003e --\u003e\n\n\u003c!-- [![Hugging Face](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-DriveLM-ffc107?color=ffc107\u0026logoColor=white)](https://huggingface.co/datasets/OpenDrive/DriveLM) --\u003e\n\n\u003c/div\u003e\n\n\n\u003c!-- \u003e https://github.com/OpenDriveLab/DriveLM/assets/103363891/67495435-4a32-4614-8d83-71b5c8b66443 --\u003e\n\n\u003c!-- \u003e above is old demo video. demo scene token: cc8c0bf57f984915a77078b10eb33198 --\u003e\n\nhttps://github.com/OpenDriveLab/DriveLM/assets/54334254/cddea8d6-9f6e-4e7e-b926-5afb59f8dce2\n\n\u003c!-- \u003e above is new demo video. demo scene token: cc8c0bf57f984915a77078b10eb33198 --\u003e\n\n## Highlights \u003ca name=\"highlight\"\u003e\u003c/a\u003e\n\n🔥 We instantiate datasets (**DriveLM-Data**) built upon nuScenes and CARLA, and propose a VLM-based baseline approach (**DriveLM-Agent**) for jointly performing **Graph VQA** and end-to-end driving. \n\n\u003c!-- 🔥 **The key insight** is that with our proposed suite, we obtain a suitable proxy task to mimic the human reasoning process during driving.  --\u003e\n\n🏁 **DriveLM** serves as a main track in the [**`CVPR 2024 Autonomous Driving Challenge`**](https://opendrivelab.com/challenge2024/#driving_with_language). Everything you need for the challenge is [HERE](https://github.com/OpenDriveLab/DriveLM/tree/main/challenge), including baseline, test data and submission format and evaluation pipeline!\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/images/repo/drivelm_teaser.jpg\"\u003e\n\u003c/p\u003e\n\n\n\u003c!-- ### Highlights of the DriveLM-Data --\u003e\n\n\u003c!-- #### In the view of full-stack autonomous driving\n- 🛣 Completeness in functionality (covering **Perception**, **Prediction**, and **Planning** QA pairs).\n\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/images/repo/point_1.png\"\u003e\n\u003c/p\u003e  --\u003e\n\n\n\u003c!-- - 🔜 Reasoning for future events that have not yet happened.\n  - Many **\"What If\"**-style questions: imagine the future by language.\n \n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/images/repo/point_2.png\" width=70%\u003e\n\u003c/p\u003e\n\n- ♻ Task-driven decomposition.\n  - **One** scene-level description into **many** frame-level trajectories \u0026 planning QA pairs.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/images/repo/point_3.png\"\u003e\n\u003c/p\u003e --\u003e\n\n\u003c!-- ### Highlights of the DriveLM-Agent --\u003e\n\n\u003c!-- #### In the view of the general Vision Language Models --\u003e\n\u003c!-- 🕸️ Multi-modal **Graph Visual Question Answering** (GVQA) benchmark for structured reasoning in the general Vision Language Models.\n\nhttps://github.com/OpenDriveLab/DriveLM-new/assets/75412366/78c32442-73c8-4f1d-ab69-34c15e7060af --\u003e\n\n\u003c!-- \u003e above is graph VQA demo video.  --\u003e\n\n## News \u003ca name=\"news\"\u003e\u003c/a\u003e\n\n- **`[2025/01/08]`** [Drive-Bench](https://drive-bench.github.io/) release! In-depth analysis in what are DriveLM really benchmarking. Take a look at [arxiv](https://arxiv.org/pdf/2501.04003).\n- **`[2024/07/16]`** DriveLM [official leaderboard](https://huggingface.co/spaces/AGC2024/driving-with-language-official) reopen!\n- **`[2024/07/01]`** DriveLM got accepted to ECCV 2024! Congrats to the team!\n- **`[2024/06/01]`** Challenge ended up! [See the final leaderboard](https://opendrivelab.com/challenge2024/#driving_with_language).\n- **`[2024/03/25]`** Challenge test server is online and the test questions are released. [Chekc it out!](https://github.com/OpenDriveLab/DriveLM/tree/main/challenge)\n- **`[2024/02/29]`** Challenge repo release. Baseline, data and submission format, evaluation pipeline. [Have a look!](https://github.com/OpenDriveLab/DriveLM/tree/main/challenge)\n- **`[2023/08/25]`** DriveLM-nuScenes demo released.\n- **`[2023/12/22]`** DriveLM-nuScenes full `v1.0` and [paper](https://arxiv.org/abs/2312.14150) released.\n\u003c!-- \u003e - **`[Early 2024]`** DriveLM-Agent inference code.  --\u003e\n\u003c!-- \u003e - **`Note:`** We plan to release a simple, flexible training code that supports multi-view inputs as a starter kit for the AD challenge (stay tuned for details).  --\u003e\n\n## Table of Contents\n1. [Highlights](#highlight)\n2. [Getting Started](#gettingstarted)\n   - [Prepare DriveLM-nuScenes](docs/data_prep_nus.md)  \n3. [Current Endeavors and Future Horizons](#timeline)\n4. [TODO List](#newsandtodolist)\n5. [DriveLM-Data](#drivelmdata)\n   - [Comparison and Stats](#comparison)\n   - [GVQA Details](docs/gvqa.md)\n   - [Annotation and Features](docs/data_details.md)\n6. [License and Citation](#licenseandcitation)\n7. [Other Resources](#otherresources)\n\n\u003c!--\n- [News](#news)\n- [DriveLM-Data](#drivelm-data)\n- [Getting Started](#getting-started)\n- [License and Citation](#license-and-citation)\n- [Other Resources](#other-resources) --\u003e\n\n\n## Getting Started \u003ca name=\"gettingstarted\"\u003e\u003c/a\u003e\nTo get started with DriveLM: \n- [Prepare DriveLM-nuScenes](/docs/data_prep_nus.md)\n- [Challenge devkit](/challenge/)\n- [More content coming soon](#todolist)\n\n\u003cp align=\"right\"\u003e(\u003ca href=\"#top\"\u003eback to top\u003c/a\u003e)\u003c/p\u003e\n\n\n## Current Endeavors and Future Directions  \u003ca name=\"timeline\"\u003e\u003c/a\u003e\n\u003e - The advent of GPT-style multimodal models in real-world applications motivates the study of the role of language in driving.\n\u003e - Date below reflects the arXiv submission date.\n\u003e - If there is any missing work, please reach out to us!\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/images/repo/drivelm_timeline_v3.jpg\"\u003e\n\u003c/p\u003e\n\nDriveLM attempts to address some of the challenges faced by the community.\n\n- **Lack of data**: DriveLM-Data serves as a comprehensive benchmark for driving with language.\n- **Embodiment**: GVQA provides a potential direction for embodied applications of LLMs / VLMs.\n- **Closed-loop**: DriveLM-CARLA attempts to explore closed-loop planning with language.\n\n\u003cp align=\"right\"\u003e(\u003ca href=\"#top\"\u003eback to top\u003c/a\u003e)\u003c/p\u003e\n\n## TODO List \u003ca name=\"newsandtodolist\"\u003e\u003c/a\u003e\n\n- [x] DriveLM-Data\n  - [x] DriveLM-nuScenes\n  - [x] DriveLM-CARLA\n- [x] DriveLM-Metrics\n  - [x] GPT-score\n- [ ] DriveLM-Agent\n  - [x] Inference code on DriveLM-nuScenes\n  - [ ] Inference code on DriveLM-CARLA\n\n\u003cp align=\"right\"\u003e(\u003ca href=\"#top\"\u003eback to top\u003c/a\u003e)\u003c/p\u003e\n\n## DriveLM-Data \u003ca name=\"drivelmdata\"\u003e\u003c/a\u003e\n\nWe facilitate the `Perception, Prediction, Planning, Behavior, Motion` tasks with human-written reasoning logic as a connection between them. We propose the task of [GVQA](docs/gvqa.md) on the DriveLM-Data. \n\n\u003c!-- DriveLM is an autonomous driving (**AD**) dataset incorporating linguistic information. Through DriveLM, we want to connect large language models and autonomous driving systems, and eventually introduce the reasoning ability of Large Language Models in autonomous driving (**AD**) to make decisions and ensure explainable planning. --\u003e\n\n\u003c!-- In DriveLM, we study how vision-language models (VLMs) trained on web-scale data can be integrated into end-to-end driving systems to boost generalization and enable interactivity with human users.\n\nSpecifically, we aim to facilitate `Perception, Prediction, Planning, Behavior, Motion` tasks with human-written reasoning logic as a connection. We propose the task of GVQA to connect the QA pairs in a graph-style structure. To support this novel task, we provide the DriveLM-Data.\n\n\n### What is GVQA?\nThe most exciting aspect of the dataset is that the questions and answers (`QA`) are connected in a graph-style structure, with QA pairs as every node and potential logical progression as the edges. The reason for doing this in the AD domain is that AD tasks are well-defined per stage, from raw sensor input to final control action through perception, prediction and planning.\n\nIts key difference to prior VQA tasks for AD is the availability of logical dependencies between QAs, which can be used to guide the answering process. --\u003e\n\n### 📊 Comparison and Stats \u003ca name=\"comparison\"\u003e\u003c/a\u003e\n**DriveLM-Data** is the *first* language-driving dataset facilitating the full stack of driving tasks with graph-structured logical dependencies.\n\u003c!-- \u003ccenter\u003e\n  \n| Language Dataset  | Base Dataset |      Language Form    |   Perspectives | Scale      |  Release?|\n|:---------:|:-------------:|:-------------:|:------:|:--------------------------------------------:|:----------:|\n| [BDD-X 2018](https://github.com/JinkyuKimUCB/explainable-deep-driving)  |  [BDD](https://bdd-data.berkeley.edu/)  | Description | Perception \u0026 Reasoning    | 8M frames, 20k text strings   |**:heavy_check_mark:**|\n| [HAD 2019](https://usa.honda-ri.com/had)  |  [HDD](https://usa.honda-ri.com/hdd)  | Advice | Goal-oriented \u0026 stimulus-driven advice | 5,675 video clips, 45k text strings   |**:heavy_check_mark:**|\n| [DRAMA 2022](https://usa.honda-ri.com/drama)   |    - | Description | Perception \u0026 Planning results | 18k frames, 100k text strings | **:heavy_check_mark:**|\n| [Rank2Tell 2023](https://arxiv.org/abs/2309.06597)   |   - | Perception \u0026 Planning results |  QA + Captions | 5k frames | :x: |\n| [nuScenes-QA 2023](https://arxiv.org/abs/2305.14836)   |   [nuScenes](https://www.nuscenes.org/)  | QA |  Perception Result     | 30k frames, 460k generated QA pairs|**:heavy_check_mark:**|\n| [nuPrompt 2023](https://arxiv.org/abs/2309.04379)   |   [nuScenes](https://www.nuscenes.org/)  | Object Description |  Perception Result     | 30k frames, 35k semi-generated QA pairs| :x:|\n| **DriveLM 2023** | [nuScenes](https://www.nuscenes.org/) | **:boom: QA + Scene Description** | **:boom:Perception, Prediction and Planning with Logic** | 30k frames, 360k annotated QA pairs |**:heavy_check_mark:** |\n\n\u003c/center\u003e --\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/images/repo/paper_data_comp.png\"\u003e\n\u003c/p\u003e\n\nLinks to details about [GVQA task](docs/gvqa.md), [Dataset Features](docs/data_details.md/#features), and [Annotation](docs/data_details.md/#annotation).\n\n\u003c!-- More details can be found [HERE](docs/data_details.md). --\u003e\n\u003c!--\n### What is included in the DriveLM-Data?\nDriveLM-Data comprises two distinct components: DriveLM-nuScenes and DriveLM-CARLA. In the case of DriveLM-nuScenes, we construct our dataset based on the prevailing nuScenes dataset. As for DriveLM-CARLA, we collect data from the CARLA simulator. The most central element of DriveLM is frame-based `multi-stage` `QA`. `Perception` questions require the model to recognize objects in the scene. `Prediction` questions ask the model to predict the future status of important objects in the scene. `Planning` questions prompt the model to give reasonable planning actions and avoid dangerous ones. We also include a `Behavior` question that provides behavior templates which aggregate the information from the other question types.\n\n\n### How about the annotation process?\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/images/repo/paper_data.jpg\"\u003e\n\u003c/p\u003e\n\n**For DriveLM-nuScenes:**\n\n1️⃣ Keyframe selection. Given all frames in one clip, the annotator selects the keyframes that need annotation. The criterion is that those frames should involve changes in ego-vehicle movement status (lane changes, sudden stops, start after a stop, etc.).\n\n2️⃣ Key objects selection. Given keyframes, the annotator needs to pick up key objects in the six surrounding images. The criterion is that those objects should be able to affect the action of the ego vehicle (traffic signals, pedestrians crossing the road, other vehicles that move in the direction of the ego vehicle, etc.).\n\n3️⃣ Question and answer annotation. Given those key objects, we automatically generate questions regarding single or multiple objects about perception, prediction, and planning. More details can be found in our data.\n\n**For DriveLM-CARLA:**\n\nWe collect data using CARLA 0.9.14 in the Leaderboard 2.0 framework with a privileged rule-based expert. We set up a series of routes in urban, residential, and rural areas and execute the expert on these routes. During this process, we collect the necessary sensor data, generate relevant QAs based on privileged information about objects and the scene, and organize the logical relationships to connect this series of QAs into a graph.\n\n--\u003e\n\n\u003cp align=\"right\"\u003e(\u003ca href=\"#top\"\u003eback to top\u003c/a\u003e)\u003c/p\u003e\n\n\n\n\n## License and Citation \u003ca name=\"licenseandcitation\"\u003e\u003c/a\u003e\nAll assets and code in this repository are under the [Apache 2.0 license](./LICENSE) unless specified otherwise. The language data is under [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/). Other datasets (including nuScenes) inherit their own distribution licenses. Please consider citing our paper and project if they help your research.\n\n```BibTeX\n@article{sima2023drivelm,\n  title={DriveLM: Driving with Graph Visual Question Answering},\n  author={Sima, Chonghao and Renz, Katrin and Chitta, Kashyap and Chen, Li and Zhang, Hanxue and Xie, Chengen and Luo, Ping and Geiger, Andreas and Li, Hongyang},\n  journal={arXiv preprint arXiv:2312.14150},\n  year={2023}\n}\n```\n\n```BibTeX\n@misc{contributors2023drivelmrepo,\n  title={DriveLM: Driving with Graph Visual Question Answering},\n  author={DriveLM contributors},\n  howpublished={\\url{https://github.com/OpenDriveLab/DriveLM}},\n  year={2023}\n}\n```\n\n\u003cp align=\"right\"\u003e(\u003ca href=\"#top\"\u003eback to top\u003c/a\u003e)\u003c/p\u003e\n\n## Other Resources \u003ca name=\"otherresources\"\u003e\u003c/a\u003e\n\u003ca href=\"https://twitter.com/OpenDriveLab\" target=\"_blank\"\u003e\n    \u003cimg alt=\"Twitter Follow\" src=\"https://img.shields.io/twitter/follow/OpenDriveLab?style=social\u0026color=brightgreen\u0026logo=twitter\" /\u003e\n  \u003c/a\u003e\n\n\u003c!-- \u003ca href=\"https://opendrivelab.com\" target=\"_blank\"\u003e\n  \u003cimg src=\"https://img.shields.io/badge/contact%40opendrivelab.com-white?style=social\u0026logo=gmail\"\u003e\n\u003c/a\u003e --\u003e\n\n\u003c!--\n [![Page Views Count](https://badges.toozhao.com/badges/01H9CR01K73G1S0AKDMF1ABC73/blue.svg)](https://badges.toozhao.com/stats/01H9CR01K73G1S0AKDMF1ABC73 \"Get your own page views count badge on badges.toozhao.com\")\n--\u003e\n\n**OpenDriveLab**\n- [DriveAGI](https://github.com/OpenDriveLab/DriveAGI) | [UniAD](https://github.com/OpenDriveLab/UniAD) | [OpenLane-V2](https://github.com/OpenDriveLab/OpenLane-V2) | [Survey on E2EAD](https://github.com/OpenDriveLab/End-to-end-Autonomous-Driving)\n- [Survey on BEV Perception](https://github.com/OpenDriveLab/BEVPerception-Survey-Recipe) | [BEVFormer](https://github.com/fundamentalvision/BEVFormer) | [OccNet](https://github.com/OpenDriveLab/OccNet)\n\n\u003ca href=\"https://twitter.com/AutoVisionGroup\" target=\"_blank\"\u003e\n    \u003cimg alt=\"Twitter Follow\" src=\"https://img.shields.io/twitter/follow/Awesome Vision Group?style=social\u0026color=brightgreen\u0026logo=twitter\" /\u003e\n  \u003c/a\u003e\n\n**Autonomous Vision Group**\n- [tuPlan garage](https://github.com/autonomousvision/tuplan_garage) | [CARLA garage](https://github.com/autonomousvision/carla_garage) | [Survey on E2EAD](https://github.com/OpenDriveLab/End-to-end-Autonomous-Driving)\n- [PlanT](https://github.com/autonomousvision/plant) | [KING](https://github.com/autonomousvision/king) | [TransFuser](https://github.com/autonomousvision/transfuser) | [NEAT](https://github.com/autonomousvision/neat)\n\n\u003cp align=\"right\"\u003e(\u003ca href=\"#top\"\u003eback to top\u003c/a\u003e)\u003c/p\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FOpenDriveLab%2FDriveLM","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FOpenDriveLab%2FDriveLM","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FOpenDriveLab%2FDriveLM/lists"}