Awesome-LLM4AD
A curated list of awesome LLM/VLM/VLA/World Model for Autonomous Driving(LLM4AD) resources (continually updated)
https://github.com/Thinklab-SJTU/Awesome-LLM4AD
Last synced: 14 days ago
JSON representation
Datasets
- Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving
- official
- DRAMA
- Language Prompt for Autonomous Driving
- Rank2Tell: A Multimodal Driving Dataset for Joint Importance Ranking and Reasoning
- WEDGE: A multi-weather autonomous driving dataset built from generative vision-language models
- NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenario
- DRAMA: Joint Risk Localization and Captioning in Driving
- Textual Explanations for Self-Driving Vehicles
- Grounding Human-To-Vehicle Advice for Self-Driving Vehicles
- DriveLM: Drive on Language
- BDD-X dataset
- WEDGE
- NuScenes-QA
- HAD dataset
- CoVLA: Comprehensive Vision-Language-Action Dataset for Autonomous Driving
- CoVLA
- DriveLM
- CARScenes: Semantic VLM Dataset for Safe Autonomous Driving
- CARScenes
- An interactive enhanced driving dataset for autonomous driving
ICLR 2024 Open Review
- Large Language Models as Decision Makers for Autonomous Driving
- DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model
- BEV-CLIP: Multi-modal BEV Retrieval Methodology for Complex Scene in Autonomous Driving
- GPT-Driver: Learning to Drive with GPT
- Radar Spectra-language Model for Automotive Scene Parsing
- GeoDiffusion: Text-Prompted Geometric Control for Object Detection Data Generation
- SPOT: Scalable 3D Pre-training via Occupancy Prediction for Autonomous Driving
- 3D DENSE CAPTIONING BEYOND NOUNS: A MIDDLEWARE FOR AUTONOMOUS DRIVING
- RADIal
Papers
- nuScenes
- LangProp: A code optimization framework using Language Models applied to driving
- CARLA
- DriveLM: Driving with Graph Visual Question Answering
- LingoQA: Video Question Answering for Autonomous Driving
- DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving
- Carla
- Large Language Models for Autonomous Driving: Real-World Experiments
- official
- LMDrive: Closed-Loop End-to-End Driving with Large Language Models
- Evaluation of Large Language Models for Decision Making in Autonomous Driving
- LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programs
- Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Driving
- ONCE
- Dolphins: Multimodal Language Model for Driving
- Dolphins
- GQA
- MSCOCO - VQA](https://okvqa.allenai.org/), TDIUC, Visual Genome dataset
- Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving
- Drive-WM
- Empowering Autonomous Driving with Large Language Models: A Safety Perspective
- GPT-4V Takes the Wheel: Evaluating Promise and Challenges for Pedestrian Behavior Prediction
- JAAD
- PIE
- ADriver-I: A General World Model for Autonomous Driving
- A Language Agent for Autonomous Driving
- UniAD
- Human-Centric Autonomous Systems With LLMs for User Command Reasoning
- On the Road with GPT-4V(ision): Early Explorations of Visual-Language Model on Autonomous Driving
- ChatGPT as Your Vehicle Co-Pilot: An Initial Attempt
- MagicDrive: Street View Generation with Diverse 3D Geometry Control
- MagicDrive
- Receive, Reason, and React: Drive as You Say with Large Language Models in Autonomous Vehicles
- video
- DrivingDiffusion: Layout-Guided multi-view driving scene video generation with latent diffusion model
- official
- official
- Talk2BEV: Language-enhanced Bird’s-eye View Maps for Autonomous Driving
- DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model
- official
- GPT-DRIVER: LEARNING TO DRIVE WITH GPT
- official
- GAIA-1: A Generative World Model for Autonomous Driving
- DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Models
- SurrealDriver: Designing Generative Driver Agent Simulation Framework in Urban Contexts based on Large Language Model
- Drive as You Speak: Enabling Human-Like Interaction with Large Language Models in Autonomous Vehicles
- DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving
- official
- Can you text what is happening? Integrating pre-trained language encoders into trajectory prediction models for autonomous driving
- TrafficGPT: Viewing, Processing and Interacting with Traffic Foundation Models
- HiLM-D: Towards High-Resolution Understanding in Multimodal Large Language Models for Autonomous Driving
- MTD-GPT: A Multi-Task Decision-Making GPT Model for Autonomous Driving at Unsignalized Intersections
- Domain Knowledge Distillation from Large Language Model: An Empirical Study in the Autonomous Driving Domain
- Drive Like a Human: Rethinking Autonomous Driving with Large Language Models
- Language-Guided Traffic Simulation via Scene-Level Diffusion
- ADAPT: Action-aware Driving Caption Transformer
- VLP: Vision Language Planning for Autonomous Driving
- DME-Driver: Integrating Human Decision Logic and 3D Scene Perception in Autonomous Driving
- Holistic Autonomous Driving Understanding by Bird’s-Eye-View Injected Multi-Modal Large Models
- LLM-ASSIST: Enhancing Closed-Loop Planning with Language-Based Reasoning
- LLM-ASSIST
- Driving Everywhere with Large Language Model Policy Adaptation
- RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Model
- DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models
- LimSim++
- AccidentGPT: Accident Analysis and Prevention from V2X Environmental Perception with Multi-modal Large Model
- AccidentGPT
- LLM-Assisted Light: Leveraging Large Language Model Capabilities for Human-Mimetic Traffic Signal Control in Complex Urban Environments
- DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generation
- DriveDreamer-2
- Editable Scene Simulation for Autonomous Driving via Collaborative LLM-Agents
- GenAD: Generative End-to-End Autonomous Driving
- AGENTSCODRIVER: Large Language Model Empowered Collaborative Driving with Lifelong Learning
- Multi-Frame, Lightweight & Efficient Vision-Language Models for Question Answering in Autonomous Driving
- official
- highD
- AIDE: An Automatic Data Engine for Object Detection in Autonomous Driving
- Mapillary - dataset.com/), nuImages, BDD100k, Waymo, KITTI
- Engineering Safety Requirements for Autonomous Driving with Large Language Models
- LeGo-Drive: Language-enhanced Goal-oriented Closed-Loop End-to-End Autonomous Driving
- LeGo-Drive
- LeGo-Drive
- Hybrid Reasoning Based on Large Language Models for Autonomous Car Driving
- Large Language Models Powered Context-aware Motion Prediction
- MTR
- Generalized Predictive Model for Autonomous Driving
- DriveAGI
- GPT-4 Enhanced Multimodal Grounding for Autonomous Driving: Leveraging Cross-Modal Attention with Large Language Models
- LanguageMPC: Large Language Models as Decision Makers for Autonomous Driving
- A Superalignment Framework in Autonomous Driving with Large Language Models
- PlanAgent: A Multi-modal Large Language Agent for Closed-loop Vehicle Motion Planning
- ChatScene: Knowledge-Enabled Safety-Critical Scenario Generation for Autonomous Vehicles
- Probing Multimodal LLMs as World Models for Driving
- OmniDrive: A Holistic LLM-Agent Framework for Autonomous Driving with 3D Perception Reasoning and Planning
- REvolve: Reward Evolution with Large Language Models for Autonomous Driving
- REvolve
- LC-LLM: Explainable Lane-Change Intention and Trajectory Predictions with Large Language Models
- ChatScene
- DriveSim
- OmniDrive
- AirSim
- HighwayEnv
- Large Language Models Powered Context-aware Motion Prediction
- WOMD
- LLM-Assisted-Light
- ChatSim
- DriveVLM
- GenAD
- RAG-Driver
- LLaDA
- LangProp
- official
- DriveLM: Driving with Graph Visual Question Answering
- OpenLane-V2
- official
- official
- official
- official
- official
- Talk2car
- Dolphins
- Drive-WM
- official
- WiDEVIEW
- LLaVA-7B-1.5 - diffusion](https://github.com/CompVis/latent-diffusion)
- Agent-Driver
- DriveCmd
- UCU Dataset
- official
- MagicDrive
- ComplexUrbanScenarios
- official
- official
- Talk2BEV
- GPT-Driver
- CitySim - Based vehicle trajectory dataset.
- official
- official
- official
- ADAPT
- AGENTSCODRIVER: Large Language Model Empowered Collaborative Driving with Lifelong Learning
- Testing Large Language Models on Driving Theory Knowledge and Skills for Connected Autonomous Vehicles
- UK Driving Theory Test Practice Questions and Answers
- KoMA: Knowledge-driven Multi-agent Framework for Autonomous Driving with Large Language Models
- KoMA
- Asynchronous Large Language Model Enhanced Planner for Autonomous Driving
- AsyncDriver
- nuPlan Closed-Loop Reactive Hard20
- Embodied Understanding of Driving Scenarios
- ELM
- LeGEND: A Top-Down Approach to Scenario Generation of Autonomous Driving Systems Assisted by Large Language Models
- LeGEND
- MiniDrive: More Efficient Vision-Language Models with Multi-Level 2D Features as Text Tokens for Autonomous Driving
- MiniDrive
- OccLLaMA: An Occupancy-Language-Action Generative World Model for Autonomous Driving
- ContextVLM: Zero-Shot and Few-Shot Context Understanding for Autonomous Driving using Vision Language Models
- ContextVLM
- DriveGenVLM: Real-world Video Generation for Vision Language Model based Autonomous Driving
- Edge-Cloud Collaborative Motion Planning for Autonomous Driving with Large Language Models
- EC-Drive
- V2X-VLM: End-to-End V2X Cooperative Autonomous Driving Through Large Vision-Language Models
- V2X-VLM
- V2X-VLM
- DAIR-V2X
- VLM-MPC: Vision Language Foundation Model (VLM)-Guided Model Predictive Controller (MPC) for Autonomous Driving
- SimpleLLM4AD: An End-to-End Vision-Language Model with Graph Visual Question Answering for Autonomous Driving
- WOMD-Reasoning: A Large-Scale Language Dataset for Interaction and Driving Intentions Reasoning
- WOMD-Reasoning
- Exploring the Potential of Multi-Modal AI for Driving Hazard Prediction
- DHPR