{"id":13435320,"url":"https://github.com/talk2car/Talk2Car","last_synced_at":"2025-03-18T02:31:49.877Z","repository":{"id":46524771,"uuid":"247939145","full_name":"talk2car/Talk2Car","owner":"talk2car","description":"The official Talk2Car dataset repo","archived":false,"fork":false,"pushed_at":"2024-07-25T11:16:15.000Z","size":18840,"stargazers_count":67,"open_issues_count":1,"forks_count":6,"subscribers_count":7,"default_branch":"master","last_synced_at":"2024-10-27T17:26:13.968Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/talk2car.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-03-17T10:19:21.000Z","updated_at":"2024-09-04T03:13:32.000Z","dependencies_parsed_at":"2024-10-27T17:23:35.706Z","dependency_job_id":null,"html_url":"https://github.com/talk2car/Talk2Car","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/talk2car%2FTalk2Car","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/talk2car%2FTalk2Car/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/talk2car%2FTalk2Car/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/talk2car%2FTalk2Car/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/talk2car","download_url":"https://codeload.github.com/talk2car/Talk2Car/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244144066,"owners_count":20405331,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T03:00:34.845Z","updated_at":"2025-03-18T02:31:46.049Z","avatar_url":"https://github.com/talk2car.png","language":"Python","funding_links":[],"categories":["Awesome Papers","Python","Papers","📋 Summary of Language-Enhanced Datasets"],"sub_categories":["Datasets"],"readme":"This repository contains the data and development kit for the Talk2Car dataset presented in our paper [Talk2Car: Taking Control of Your Self-Driving Car](https://arxiv.org/pdf/1909.10838).\nYou can visit the Talk2Car website [here](https://talk2car.github.io).\nProject financed by Internal Funds KU Leuven (C14/18/065). Talk2Car is part of the [MACCHINA](https://macchina-ai.cs.kuleuven.be/) project\nAn example image from Talk2Car given below for the following command: \"You can park up ahead behind \u003cb\u003ethe silver car, next to that lamppost with the orange sign on it\u003c/b\u003e\". The referred object is indicated in bold.\n\n\u003cp align=\"center\"\u003e\n\t\u003cimg src=\"static/example.png\" /\u003e\n\u003c/p\u003e\n\n\n# Overview\n\n\n- [Changelog](#changelog)\n- [Talk2Car](#talk2car_overview)\n  - [Requirements](#requirements)\n  - [Setup](#setup)\n  - [Evaluation](#evaluation)  \n  - [Talk2Car Leaderboard](#leaderboard)\n- [C4AV Challenge](#c4av_challenge)\n  - [C4AV Challenge Quick Start](#c4av_challenge_quick_start)\n- [Extensions](#extensions)\n- [Citation](#citation)\n- [License](#license)\n\n\n# \u003ca name=\"changelog\"\u003e\u003c/a\u003eChangelog\n\n- 8th January, 2023: Talk2Car-Trajectory support in Talk2Car class and added it to extensions section.\n- 13th September, 2022: Added Talk2Car-RegSeg dataset in extensions section.\n- 26th July, 2022: Added Talk2Car-Destination support in Talk2Car.\n- 3rd July, 2022: Added Talk2Car-Destination dataset in the extensions section.\n- 1st July, 2022: Added Talk2Car-Expr dataset in the extensions section.\n- 29th October, 2021: Added visualization script in utils to create videos up to and including the prediction of the referred object.\n- 6th October, 2021: Release of Talk2Car v1.1\n- 5th October, 2021: Making the Talk2Car class able to also load the nuScenes dataset or only loading the Talk2Car (Slim dataset version).\n- 2th June, 2021: Update code base to pytorch 1.8.1\n- 18th March, 2020: First release of Talk2Car.\n\n# \u003ca name=\"talk2car_overview\"\u003e\u003c/a\u003eTalk2Car\n\nThe Talk2Car dataset is built upon the [nuScenes](https://www.nuscenes.org/) dataset. \nHence, one can use all data provided by nuScenes when using Talk2Car (i.e. LIDAR, RADAR, Video, ...).\nHowever, if one wishes to do so, they need to download 300GB+ of data from nuScenes.\nTo make our dataset more accessible to researchers and self-driving car enthusiasts with limited hardware,\nwe also provide a dataset class that can only load the Talk2Car data. For this, one only needs \u003c2GB of storage.\nIn the following section we first describe the requirements and then we describe how to set up both versions of the dataset.\nWe also provide a google colab that can train our baseline model found [here](https://colab.research.google.com/drive/19wTtGskfdhZWvNIfIrUcpwe9IChRKBtQ?usp=sharing).\n\n## \u003ca name=\"requirements\"\u003e\u003c/a\u003e Requirements\n\nTo use Talk2Car, we recommend the following instructions:\n\nFirst, run\n```\nconda create --name talk2car python=3.6 -y\n```\n\nThen\n```\nsource activate talk2car\n```\n\nIn case you haven't copied the repo yet, run \n\n```\ngit clone https://github.com/talk2car/Talk2Car.git\n```\n\nThen run:\n\n```\ncd Talk2Car \u0026\u0026 pip install -r requirements.txt\n```\n\nFinally,\n\n```\npython -m spacy download en_core_web_sm\n```\n\n\n## \u003ca name=\"setup\"\u003e\u003c/a\u003eSetup\n\nWe first start with the dataset version that only uses the Talk2Car data. We will refer to this version as `Talk2CarSlim`.\nThe version that uses the nuScenes data will be referred to as `Talk2Car`.\n\n### Talk2CarSlim\n\n\nTo set up the data please first follow the requirements section.\nMake sure you are at the root directory of Talk2Car for the following instructions.\n\nActivate the `talk2car` environment if this is not done yet.\n```\nsource activate talk2car\n```\n\nthen,\n\n```\npip install gdown\n```\n\nNow download the images\n\n```\ngdown --id 1bhcdej7IFj5GqfvXGrHGPk2Knxe77pek\n```\n\nUnpack them,\n\n```\nunzip imgs.zip \u0026\u0026 mv imgs/ ./data/images\nrm imgs.zip\n```\n\nTo see if you have installed everything correctly, you should be able to run the following:\n\n```\ncd baseline\npython3 train.py --root ../data --lr 0.01 --nesterov --evaluate\n```\n\n### Talk2Car\n\nFirst, you need to download the complete nuScenes dataset. \nOn their download page, you will need to download all 10 parts.\nYou will need 300GB+ to download all data. \nThe `example.py` file provides an example for loading the data in PyTorch using the Talk2Car class from the `talktocar.py` file. \nWe advise to create a conda environment to run the code.\nThe nuscenes-devkit is required in addition to some popular python packages which can be easily installed through conda or pip. \nThe code can be run as follows:\n\nExport the path where you put the nuScenes data and install the nuscenes-devkit through pip.\n\n```\nexport NUSCENES='/path/to/downloaded/nuscenes'\n```\n\nCopy the Talk2Car json files to a directory named 'commands' in the nuScenes dataset folder.\n\n```\nexport COMMANDS=$NUSCENES'/commands/'\nmkdir -p $COMMANDS\ncp ./data/commands/* $COMMANDS\n```\n\nRun the example file.\n```\npython3 ./example.py --root $NUSCENES\n```\n\n## \u003ca name=\"evaluation\"\u003e\u003c/a\u003eEvaluation\nThe object referral task on the Talk2Car dataset requires to predict a bounding box for every command.\nThe models on Talk2Car are evaluated by checking if the Intersection over Union of the predicted object bounding box and the ground truth bounding box is above 0.5.\nThis metric can be referred to by many ways i.e.  IoU\u003csub\u003e0.5\u003c/sub\u003e, AP50, ...\n\nIf you want to try the evaluation locally on the validation set, you can do so by using `eval.py`.\nThe script can be used as follows:\n\n```\npython eval.py --root $NUSCENES --version val --predictions ./data/predictions.json\n```\n\nWhen replacing the `predictions.json` file by your own model predictions, you are required to follow the same format. \nSpecifically, the results need to be stored as a JSON file which contains a python dictionary of the following format {command_token: [x0, y0, w, h]}. Where x0 and y0 are the coordinates of the top left corner, and h, w the height and width of the predicted bounding box.  \n\nEvaluation of your models on the Talk2Car test set is possible [here](https://www.aicrowd.com/challenges/eccv-2020-commands-4-autonomous-vehicles).\nAn example of this can be seen in the `baseline/test.py`.\n\n## \u003ca name=\"leaderboard\"\u003e\u003c/a\u003eLeaderboard\n\nThe Talk2Car leaderboard can be found [here](leaderboard.md).\n\n# \u003ca name=\"c4av_challenge\"\u003e\u003c/a\u003eC4AV Challenge\n\nThe Talk2Car dataset is part of the [Commands for Autonomous Vehicles](https://www.aicrowd.com/challenges/eccv-2020-commands-4-autonomous-vehicles) challenge.\nThe challenge required to solve a visual grounding task.\nThe summary paper of the challenge can be found [here](https://link.springer.com/chapter/10.1007/978-3-030-66096-3_1).\n\n## \u003ca name=\"c4av_challenge_quick_start\"\u003e\u003c/a\u003eC4AV Challenge - Quick Start\n\nTo help participants get started in the C4AV challenge, we provide a [PyTorch code base](https://github.com/talk2car/Talk2Car/tree/master/baseline) that allows to train a baseline model on the Talk2Car dataset within minutes.\nAdditionally, we include the images and commands as separate files which avoids the need to download the entire nuScenes dataset first. \n\n# \u003ca name=\"extensions\"\u003e\u003c/a\u003eExtensions\n\n- Talk2Car-Expr - [Paper](https://www.sciencedirect.com/science/article/pii/S0952197621001044) - [Dataset](https://github.com/ThierryDeruyttere/Talk2Car-Expr)\n- Talk2Car-Destination - [Paper](https://www.aaai.org/AAAI22Papers/AAAI-8858.GrujicicD.pdf) - [Dataset](https://github.com/ThierryDeruyttere/Talk2Car-Destination)\n- Talk2Car-RegSeg - [Paper](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=\u0026arnumber=9636172) - [Dataset](https://rnr-t2c.github.io/)\n- Talk2Car-Trajectory - [Paper](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9961196) - [Dataset](https://github.com/ThierryDeruyttere/Talk2Car-Trajectory)\n\n# \u003ca name=\"citation\"\u003e\u003c/a\u003eCitation\n\n\nIf you use this work for your own research, please cite:\n```\n@inproceedings{deruyttere2019talk2car,\n  title={Talk2Car: Taking Control of Your Self-Driving Car},\n  author={Deruyttere, Thierry and Vandenhende, Simon and Grujicic, Dusan and Van Gool, Luc and Moens, Marie Francine},\n  booktitle={Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},\n  pages={2088--2098},\n  year={2019}\n}\n```\n\n# \u003ca name=\"license\"\u003e\u003c/a\u003eLicense \n\n\nThis software is released under an MIT license. For a commercial license please contact the authors.\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftalk2car%2FTalk2Car","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftalk2car%2FTalk2Car","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftalk2car%2FTalk2Car/lists"}