{"id":20731448,"url":"https://github.com/opendrivelab/elm","last_synced_at":"2025-04-04T12:06:26.505Z","repository":{"id":226606368,"uuid":"761992418","full_name":"OpenDriveLab/ELM","owner":"OpenDriveLab","description":"[ECCV 2024] Embodied Understanding of Driving Scenarios","archived":false,"fork":false,"pushed_at":"2025-01-02T11:57:24.000Z","size":5618,"stargazers_count":184,"open_issues_count":1,"forks_count":15,"subscribers_count":12,"default_branch":"main","last_synced_at":"2025-03-28T11:06:40.233Z","etag":null,"topics":["autonomous-driving","end-to-end-driving","vision-language-model"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/OpenDriveLab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["OpenDriveLab"],"patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"lfx_crowdfunding":null,"custom":null}},"created_at":"2024-02-22T21:50:05.000Z","updated_at":"2025-03-20T16:13:31.000Z","dependencies_parsed_at":"2024-03-31T08:19:54.497Z","dependency_job_id":"130fb1d2-161e-4f3b-8f59-65de7f2c4128","html_url":"https://github.com/OpenDriveLab/ELM","commit_stats":{"total_commits":52,"total_committers":9,"mean_commits":5.777777777777778,"dds":0.5,"last_synced_commit":"6dd5af79e8aacb7b498e0fcb678169f4d2fa808e"},"previous_names":["opendrivelab/elm"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenDriveLab%2FELM","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenDriveLab%2FELM/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenDriveLab%2FELM/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenDriveLab%2FELM/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/OpenDriveLab","download_url":"https://codeload.github.com/OpenDriveLab/ELM/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247174407,"owners_count":20896076,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["autonomous-driving","end-to-end-driving","vision-language-model"],"created_at":"2024-11-17T05:14:52.900Z","updated_at":"2025-04-04T12:06:26.485Z","avatar_url":"https://github.com/OpenDriveLab.png","language":"Python","funding_links":["https://github.com/sponsors/OpenDriveLab"],"categories":[],"sub_categories":[],"readme":"\u003cdiv id=\"top\" align=\"center\"\u003e\n\n# ELM: Embodied Understanding of Driving Scenarios\n\n**Revive driving scene understanding by delving into the embodiment philosophy**\n\n\u003ca href=\"https://arxiv.org/abs/2403.04593\"\u003e\u003cimg src=\"https://img.shields.io/badge/arXiv-Paper-\u003ccolor\u003e\"\u003e\u003c/a\u003e\n\u003ca href=\"https://opendrivelab.github.io/elm.github.io/\"\u003e\u003cimg src=\"https://img.shields.io/badge/Project-Page-orange\"\u003e\u003c/a\u003e\n\u003ca href=\"README.md\"\u003e\n  \u003cimg alt=\"ELM: v1.0\" src=\"https://img.shields.io/badge/ELM-v1.0-blueviolet\"/\u003e\n\u003c/a\u003e\n\u003ca href=\"#license-and-citation\"\u003e\n  \u003cimg alt=\"License: Apache2.0\" src=\"https://img.shields.io/badge/license-Apache%202.0-blue.svg\"/\u003e\n\u003c/a\u003e\n\n![](./assets/teaser.png \"Embodied Understanding of Driving Scenarios\")\n\n\u003c/div\u003e\n\n\u003e\n\u003e [Yunsong Zhou](https://zhouyunsong.github.io/), [Linyan Huang](https://github.com/DevLinyan), [Qingwen Bu](https://github.com/retsuh-bqw), Jia Zeng, Tianyu Li, Hang Qiu, Hongzi Zhu, Minyi Guo, Yu Qiao, and [Hongyang Li](https://lihongyang.info/)\n\u003e \n\u003e - Presented by [OpenDriveLab](https://opendrivelab.com/) and Shanghai AI Lab\n\u003e - :mailbox_with_mail: Primary contact: [Yunsong Zhou]((https://zhouyunsong-sjtu.github.io/)) ( zhouyunsong2017@gmail.com ) \n\u003e - [arXiv paper](https://arxiv.org/abs/2403.04593) | [Blog TODO]() | [Slides](https://drive.google.com/file/d/1hJ_cElQvGhqCq2GOlx_BnJaK5qumMmvh/view?usp=sharing)\n\u003e - [CVPR 2024 Autonomous Driving Challenge - Driving with Language](https://opendrivelab.com/challenge2024/)\n\n\n## Highlights \u003ca name=\"highlights\"\u003e\u003c/a\u003e\n\n:fire: The first **embodied language model** for understanding the long-horizon driving scenarios in `space` and `time`. \n\n:star2: **ELM** expands a wide spectrum of new tasks to fully leverage the capability of large language models in an embodiment setting and achieves significant improvements in various applications.\n\n![method](./assets/elm.png \"Architecture of ELM\")\n\n:trophy: Interpretable driving model, on the basis of language prompting, will be a main track in the `CVPR 2024 Autonomous Driving Challenge`. Please [stay tuned](https://opendrivelab.com/challenge2024/) for further details!\n\n## News \u003ca name=\"news\"\u003e\u003c/a\u003e\n\n- :fire: Interpretable driving model is launched. Please refer to the [link](https://opendrivelab.com/challenge2024/) for more details.\n- `[2024/03]` ELM [paper](https://arxiv.org/abs/2403.04593) released.\n- `[2024/03]` ELM code and data initially released.\n\n## Table of Contents\n\n1. [Highlights](#highlights)\n2. [News](#news)\n3. [TODO List](#todo)\n4. [Installation](#installation)\n5. [Dataset](#dataset)\n6. [Training and Inference](#training)\n7. [License and Citation](#license-and-citation)\n8. [Related Resources](#resources)\n\n## TODO List \u003ca name=\"todo\"\u003e\u003c/a\u003e\n\n- [x] Release fine-tuning code and data\n- [x] Release reference checkpoints\n- [x] Toolkit for label generation\n\n## Installation \u003ca name=\"installation\"\u003e\u003c/a\u003e\n\n1. (Optional) Creating conda environment\n\n```bash\nconda create -n elm python=3.8\nconda activate elm\n```\n\n2. install from [PyPI](https://pypi.org/project/salesforce-lavis/)\n```bash\npip install salesforce-lavis\n```\n    \n3. Or, for development, you may build from source\n\n```bash\ngit clone https://github.com/OpenDriveLab/ELM.git\ncd ELM\npip install -e .\n```\n\n## Dataset \u003ca name=\"dataset\"\u003e\u003c/a\u003e\n\n\n**Pre-training data.** We collect driving videos from YouTube, nuScenes, Waymo, and Ego4D. \nHere we provide a sample of 🔗 [YouTube video list](https://docs.google.com/spreadsheets/d/1HV-zOO6bh1sKjimhM1ZBcxWqPxgbalE3FDGyh2UHwPw/edit?usp=sharing) we used.\nFor privacy considerations, we are temporarily keeping the full-set data labels private. Part of pre-training data and reference checkpoints can be found in :floppy_disk: [google drive](https://drive.google.com/drive/folders/1n4S0A4k8_9yDFIPIPWH_JLTUQ6yFc8ME?usp=sharing).\n\n**Fine-tuning data.** \nThe full set of question and answer pairs for the benchmark can be obtained through this 🔗[data link](https://drive.google.com/drive/folders/1QFBIrKqxjn9lfv31XMC3wVIdaAbpMwDL?usp=sharing). You may need to download the corresponding image data from the official [nuScenes](https://www.nuscenes.org/download) and [Ego4D](https://ego4d-data.org/#download) channels. \nFor a `quick verification` of the pipeline, we recommend downloading the subset dataset of [DriveLM](https://github.com/OpenDriveLab/DriveLM/blob/main/docs/data_prep_nus.md) and organizing the data in line with the format.\n\nPlease make sure to soft link `nuScenes` and `ego4d` datasets under `data/xx` folder.\nYou may need to run `tools/video_clip_processor.py` to pre-process data first.\nBesides, we provide some script used during auto-labeling, you may use these as a reference if you want to customize data.\n\n\n## Training \u003ca name=\"training\"\u003e\u003c/a\u003e\n```bash\n# you can modify the lavis/projects/blip2/train/advqa_t5_elm.yaml\nbash scripts/train.sh\n```\n\n## Inference\nModify the  [advqa_t5_elm.yaml](lavis/projects/blip2/train/advqa_t5_elm.yaml#L71) to enable the evaluate as True.\n```bash\nbash scripts/train.sh\n```\nFor the evaluation of generated answers, please use the script in `scripts/qa_eval.py`.\n```bash\npython scripts/qa_eval.py \u003cdata_root\u003e \u003clog_name\u003e\n```\n\n\n## License and Citation\n\nAll assets and code in this repository are under the [Apache 2.0 license](./LICENSE) unless specified otherwise. The language data is under [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/). Other datasets (including nuScenes and Ego4D) inherit their own distribution licenses. Please consider citing our paper and project if they help your research.\n\n```BibTeX\n@article{zhou2024embodied,\n  title={Embodied Understanding of Driving Scenarios},\n  author={Zhou, Yunsong and Huang, Linyan and Bu, Qingwen and Zeng, Jia and Li, Tianyu and Qiu, Hang and Zhu, Hongzi and Guo, Minyi and Qiao, Yu and Li, Hongyang},\n  journal={arXiv preprint arXiv:2403.04593},\n  year={2024}\n}\n```\n\n## Related Resources \u003ca name=\"resources\"\u003e\u003c/a\u003e\n\nWe acknowledge all the open-source contributors for the following projects to make this work possible:\n\n- [Lavis](https://github.com/salesforce/LAVIS) | [DriveLM](https://github.com/OpenDriveLab/DriveLM)\n\n\n\u003ca href=\"https://twitter.com/OpenDriveLab\" target=\"_blank\"\u003e\n    \u003cimg alt=\"Twitter Follow\" src=\"https://img.shields.io/twitter/follow/OpenDriveLab?style=social\u0026color=brightgreen\u0026logo=twitter\" /\u003e\n  \u003c/a\u003e\n\n- [DriveAGI](https://github.com/OpenDriveLab/DriveAGI) | [Survey on BEV Perception](https://github.com/OpenDriveLab/BEVPerception-Survey-Recipe) | [Survey on E2EAD](https://github.com/OpenDriveLab/End-to-end-Autonomous-Driving)\n- [UniAD](https://github.com/OpenDriveLab/UniAD) | [OpenLane-V2](https://github.com/OpenDriveLab/OpenLane-V2) | [OccNet](https://github.com/OpenDriveLab/OccNet) | [OpenScene](https://github.com/OpenDriveLab/OpenScene)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopendrivelab%2Felm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopendrivelab%2Felm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopendrivelab%2Felm/lists"}