{"id":13445492,"url":"https://github.com/qiantianwen/NuScenes-QA","last_synced_at":"2025-03-20T21:30:39.908Z","repository":{"id":168911821,"uuid":"644732197","full_name":"qiantianwen/NuScenes-QA","owner":"qiantianwen","description":"[AAAI 2024] NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenario.","archived":false,"fork":false,"pushed_at":"2023-12-10T14:03:35.000Z","size":1629,"stargazers_count":141,"open_issues_count":3,"forks_count":1,"subscribers_count":15,"default_branch":"main","last_synced_at":"2024-08-01T05:14:57.601Z","etag":null,"topics":["autonomous-driving","vision-language","visual-question-answering"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/qiantianwen.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-05-24T06:28:17.000Z","updated_at":"2024-07-26T06:27:55.000Z","dependencies_parsed_at":"2023-12-10T14:30:52.920Z","dependency_job_id":"d308b17d-d58f-411c-b18d-c6d37bc8575f","html_url":"https://github.com/qiantianwen/NuScenes-QA","commit_stats":null,"previous_names":["qiantianwen/nuscenes-qa"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/qiantianwen%2FNuScenes-QA","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/qiantianwen%2FNuScenes-QA/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/qiantianwen%2FNuScenes-QA/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/qiantianwen%2FNuScenes-QA/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/qiantianwen","download_url":"https://codeload.github.com/qiantianwen/NuScenes-QA/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221807704,"owners_count":16883638,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["autonomous-driving","vision-language","visual-question-answering"],"created_at":"2024-07-31T05:00:34.731Z","updated_at":"2025-03-20T21:30:39.890Z","avatar_url":"https://github.com/qiantianwen.png","language":null,"funding_links":[],"categories":["Datasets","📋 Summary of Language-Enhanced Datasets",":books: Papers","Autonomous Driving"],"sub_categories":["Dataset \\\u0026 Benchmark","Social \u0026 Human-Robot Interaction"],"readme":"# [AAAI 2024] NuScenes-QA\n\nOfficial repository for the AAAI 2024 paper **[NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenario](https://arxiv.org/pdf/2305.14836.pdf)**.\n\n![DataConstruction](docs/data_construction.png)\n\n## :fire: News\n\n- `2024.11.01`  CenterPoint feature released. \n- `2024.10.11`  Training and Testing code released.\n- `2023.12.09`  Our paper is accepted by AAAI 2024! \n- `2023.09.04`  Our NuScenes-QA dataset v1.0 released.\n\n## :hourglass_flowing_sand: To Do\n\n- [x] Release question \u0026 anwswer data\n- [x] Release visual feature\n- [x] Release training and testing code\n\n## :running: Getting Started\n\n### Data Preparation\n\nWe have released our question-answer annotations, please download it from [HERE](https://drive.google.com/drive/folders/1jIkICT23wZWZYPrWCa0x-ubjpClSzOuU?usp=sharing).\n\nFor the visual data, you can download **CenterPoint** feature that we have extracted from [HERE](https://drive.google.com/file/d/1TNsK6cpQ4pd9fH1s7WFxTSXilrT2uNYb/view?usp=sharing). As an alternative, you can also download the origin nuScenes dataset from [HERE](https://www.nuscenes.org/download), and extract the object-level features refer to this [LINK](https://mmdetection3d.readthedocs.io/en/v0.16.0/datasets/nuscenes_det.html) with different backbones. For specific details on feature extraction, you can refer to the **Visual Feature Extraction** and **Object Embedding** sections of our paper.\n\nThe folder structure should be organized as follows before training.\n\n```\nNuScenes-QA\n+-- configs/\n|   +-- butd.yaml                    \n|   +-- mcan_small.yaml\n+-- data/\n|   +-- questions/\t\t\t\t# downloaded\n|   |   +-- NuScenes_train_questions.json\n|   |   +-- NuScenes_val_questions.json\n|   +-- features/ \t\t\t\t# downloaded or extracted\n|   |   +-- CenterPoint/\n|   |   |   +-- xxx.npz\n|   |   |   +-- ...\n|   |   +-- BEVDet/\n|   |   |   +-- xxx.npz\n|   |   |   +-- ...\n|   |   +-- MSMDFusion/\n|   |   |   +-- xxx.npz\n|   |   |   +-- ...\n+-- src/\n+-- run.py\n```\n\n### Installation\n\nThe following packages are required to build the project:\n\n```bash\npython \u003e= 3.5\nCUDA \u003e= 9.0\nPyTorch \u003e= 1.4.0\nSpaCy == 2.1.0\n```\n\nFor the SpaCy, you can install it by:\n\n```bash\nwget https://github.com/explosion/spacy-models/releases/download/en_core_web_lg-2.1.0/en_core_web_lg-2.1.0.tar.gz\npip install en_core_web_lg-2.1.0.tar.gz\n```\n\n### Training \n\nThe following script will start training a `man_small` model with `CenterPoint` feature on `2` GPUs:\n\n```bash\npython3 run.py --RUN='train' --MODEL='mcan_small' --VIS_FEAT='CenterPoint' --GPU='0, 1'\n```\n\nAll checkpoint files and the training logs will be saved to the following paths respectively:\n\n```bash\noutputs/ckpts/ckpt_\u003cVERSION\u003e/epoch\u003cEPOCH_INDEX\u003e.pkl\noutputs/log/log_run_\u003cVERSION\u003e.txt\n```\n\n### Testing\n\nFor testing, you can use the following script:\n\n```bash\npython3 run.py --RUN='val' --MODEL='mcan_small' --VIS_FEAT='CenterPoint' --CKPT_PATH'path/to/ckpt.pkl'\n```\n\nThe evaluation results and the answers for all questions will ba saved to the following paths respectively:\n\n```bash\noutputs/log/log_run_xxx.txt\noutputs/result/result_run_xxx.txt\n```\n\n## :star: Others\nIf you have any questions about the dataset and its generation or the object-level feature extraction, feel free to cantact me with `twqian19@fudan.edu.cn`.\n\n\n## :book: Citation\nIf you find our paper and project useful, please consider citing:\n```bibtex\n@article{qian2023nuscenes,\n  title={NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenario},\n  author={Qian, Tianwen and Chen, Jingjing and Zhuo, Linhai and Jiao, Yang and Jiang, Yu-Gang},\n  journal={arXiv preprint arXiv:2305.14836},\n  year={2023}\n}\n```\n\n## Acknowlegement\n\nWe sincerely thank the authors of [MMDetection3D](https://github.com/open-mmlab/mmdetection3d) and [OpenVQA](https://github.com/MILVLG/openvqa) for open sourcing their methods.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fqiantianwen%2FNuScenes-QA","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fqiantianwen%2FNuScenes-QA","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fqiantianwen%2FNuScenes-QA/lists"}