{"id":13443801,"url":"https://github.com/autonomousvision/transfuser","last_synced_at":"2025-05-15T23:05:06.667Z","repository":{"id":38539508,"uuid":"351720449","full_name":"autonomousvision/transfuser","owner":"autonomousvision","description":"[PAMI'23] TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving; [CVPR'21] Multi-Modal Fusion Transformer for End-to-End Autonomous Driving","archived":false,"fork":false,"pushed_at":"2024-06-28T11:10:59.000Z","size":44410,"stargazers_count":1286,"open_issues_count":1,"forks_count":201,"subscribers_count":24,"default_branch":"2022","last_synced_at":"2025-04-08T10:20:03.428Z","etag":null,"topics":["autonomous-driving","imitation-learning","sensor-fusion","transformers"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/autonomousvision.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-03-26T08:59:43.000Z","updated_at":"2025-04-08T03:45:32.000Z","dependencies_parsed_at":"2024-12-20T02:12:13.200Z","dependency_job_id":null,"html_url":"https://github.com/autonomousvision/transfuser","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/autonomousvision%2Ftransfuser","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/autonomousvision%2Ftransfuser/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/autonomousvision%2Ftransfuser/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/autonomousvision%2Ftransfuser/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/autonomousvision","download_url":"https://codeload.github.com/autonomousvision/transfuser/tar.gz/refs/heads/2022","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254436944,"owners_count":22070946,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["autonomous-driving","imitation-learning","sensor-fusion","transformers"],"created_at":"2024-07-31T03:02:10.552Z","updated_at":"2025-05-15T23:05:01.656Z","avatar_url":"https://github.com/autonomousvision.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving\n\n## [Paper](http://www.cvlibs.net/publications/Chitta2022PAMI.pdf) | [Supplementary](http://www.cvlibs.net/publications/Chitta2022PAMI_supplementary.pdf) | [Talk](https://www.youtube.com/watch?v=-GMhYcxOiEU) | [Poster](http://www.cvlibs.net/publications/Chitta2022PAMI_poster.pdf) | [Slides](https://kashyap7x.github.io/assets/pdf/talks/Chitta2022AIR.pdf)\n\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/transfuser-imitation-with-transformer-based/autonomous-driving-on-carla-leaderboard)](https://paperswithcode.com/sota/autonomous-driving-on-carla-leaderboard?p=transfuser-imitation-with-transformer-based)\n\n\u003cimg src=\"figures/demo.gif\"\u003e\n\nThis repository contains the code for the PAMI 2023 paper [TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving](https://arxiv.org/abs/2205.15997). This work is a journal extension of the CVPR 2021 paper [Multi-Modal Fusion Transformer for End-to-End Autonomous Driving](https://arxiv.org/abs/2104.09224). The code for the CVPR 2021 paper is available in the [cvpr2021](https://github.com/autonomousvision/transfuser/tree/cvpr2021) branch.\n\nIf you find our code or papers useful, please cite:\n\n```bibtex\n@article{Chitta2023PAMI,\n  author = {Chitta, Kashyap and\n            Prakash, Aditya and\n            Jaeger, Bernhard and\n            Yu, Zehao and\n            Renz, Katrin and\n            Geiger, Andreas},\n  title = {TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving},\n  journal = {Pattern Analysis and Machine Intelligence (PAMI)},\n  year = {2023},\n}\n```\n\n```bibtex\n@inproceedings{Prakash2021CVPR,\n  author = {Prakash, Aditya and\n            Chitta, Kashyap and\n            Geiger, Andreas},\n  title = {Multi-Modal Fusion Transformer for End-to-End Autonomous Driving},\n  booktitle = {Conference on Computer Vision and Pattern Recognition (CVPR)},\n  year = {2021}\n}\n```\n\nAlso, check out the code for other recent work on CARLA from our group:\n- [Jaeger et al., Hidden Biases of End-to-End Driving Models (ICCV 2023)](https://github.com/autonomousvision/carla_garage)\n- [Renz et al., PlanT: Explainable Planning Transformers via Object-Level Representations (CoRL 2022)](https://github.com/autonomousvision/plant)\n- [Hanselmann et al., KING: Generating Safety-Critical Driving Scenarios for Robust Imitation via Kinematics Gradients (ECCV 2022)](https://github.com/autonomousvision/king)\n- [Chitta et al., NEAT: Neural Attention Fields for End-to-End Autonomous Driving (ICCV 2021)](https://github.com/autonomousvision/neat)\n\n## Contents\n\n1. [Setup](#setup)\n2. [Dataset and Training](#dataset-and-training)\n3. [Evaluation](#evaluation)\n\n\n## Setup\n\nClone the repo, setup CARLA 0.9.10.1, and build the conda environment:\n\n```Shell\ngit clone https://github.com/autonomousvision/transfuser.git\ncd transfuser\ngit checkout 2022\nchmod +x setup_carla.sh\n./setup_carla.sh\nconda env create -f environment.yml\nconda activate tfuse\npip install torch-scatter -f https://data.pyg.org/whl/torch-1.11.0+cu102.html\npip install mmcv-full==1.5.3 -f https://download.openmmlab.com/mmcv/dist/cu102/torch1.11.0/index.html\n```\n\n## Dataset and Training\nOur dataset is generated via a privileged agent which we call the autopilot (`/team_code_autopilot/autopilot.py`) in 8 CARLA towns using the routes and scenario files provided in [this folder](./leaderboard/data/training/). See the [tools/dataset](./tools/dataset) folder for detailed documentation regarding the training routes and scenarios. You can download the dataset (210GB) by running:\n\n```Shell\nchmod +x download_data.sh\n./download_data.sh\n```\n\nThe dataset is structured as follows:\n```\n- Scenario\n    - Town\n        - Route\n            - rgb: camera images\n            - depth: corresponding depth images\n            - semantics: corresponding segmentation images\n            - lidar: 3d point cloud in .npy format\n            - topdown: topdown segmentation maps\n            - label_raw: 3d bounding boxes for vehicles\n            - measurements: contains ego-agent's position, velocity and other metadata\n```\n\n### Data generation\nIn addition to the dataset itself, we have provided the scripts for data generation with our autopilot agent. To generate data, the first step is to launch a CARLA server:\n\n```Shell\n./CarlaUE4.sh --world-port=2000 -opengl\n```\n\nFor more information on running CARLA servers (e.g. on a machine without a display), see the [official documentation.](https://carla.readthedocs.io/en/stable/carla_headless/) Once the server is running, use the script below for generating training data:\n```Shell\n./leaderboard/scripts/datagen.sh \u003ccarla root\u003e \u003cworking directory of this repo (*/transfuser/)\u003e\n```\n\nThe main variables to set for this script are `SCENARIOS` and `ROUTES`. \n\n### Training script\n\nThe code for training via imitation learning is provided in [train.py.](./team_code_transfuser/train.py) \\\nA minimal example of running the training script on a single machine:\n```Shell\ncd team_code_transfuser\npython train.py --batch_size 10 --logdir /path/to/logdir --root_dir /path/to/dataset_root/ --parallel_training 0\n```\nThe training script has many more useful features documented at the start of the main function. \nOne of them is parallel training. \nThe script has to be started differently when training on a multi-gpu node:\n```Shell\ncd team_code_transfuser\nCUDA_VISIBLE_DEVICES=0,1 OMP_NUM_THREADS=16 OPENBLAS_NUM_THREADS=1 torchrun --nnodes=1 --nproc_per_node=2 --max_restarts=0 --rdzv_id=1234576890 --rdzv_backend=c10d train.py --logdir /path/to/logdir --root_dir /path/to/dataset_root/ --parallel_training 1\n```\nEnumerate the GPUs you want to train on with CUDA_VISIBLE_DEVICES.\nSet the variable OMP_NUM_THREADS to the number of cpus available on your system.\nSet OPENBLAS_NUM_THREADS=1 if you want to avoid threads spawning other threads.\nSet --nproc_per_node to the number of available GPUs on your node.\n\nThe evaluation agent file is build to evaluate models trained with multiple GPUs. \nIf you want to evaluate a model trained with a single GPU you need to remove [this line](https://github.com/autonomousvision/transfuser/blob/a7d4db684c160095dec03851aff5ce92e36b2387/team_code_transfuser/submission_agent.py#LL95C18-L95C18).\n\n\n## Evaluation\n\n### Longest6 benchmark\nWe make some minor modifications to the CARLA leaderboard code for the Longest6 benchmark, which are documented [here](./leaderboard). See the [leaderboard/data/longest6](./leaderboard/data/longest6/) folder for a description of Longest6 and how to evaluate on it.\n\n### Pretrained agents\nPre-trained agent files for all 4 methods can be downloaded from [AWS](https://s3.eu-central-1.amazonaws.com/avg-projects/transfuser/models_2022.zip):\n\n```Shell\nmkdir model_ckpt\nwget https://s3.eu-central-1.amazonaws.com/avg-projects/transfuser/models_2022.zip -P model_ckpt\nunzip model_ckpt/models_2022.zip -d model_ckpt/\nrm model_ckpt/models_2022.zip\n```\n\n### Running an agent\nTo evaluate a model, we first launch a CARLA server:\n\n```Shell\n./CarlaUE4.sh --world-port=2000 -opengl\n```\n\nOnce the CARLA server is running, evaluate an agent with the script:\n```Shell\n./leaderboard/scripts/local_evaluation.sh \u003ccarla root\u003e \u003cworking directory of this repo (*/transfuser/)\u003e\n```\n\nBy editing the arguments in `local_evaluation.sh`, we can benchmark performance on the Longest6 routes. You can evaluate both privileged agents (such as [autopilot.py]) and sensor-based models. To evaluate the sensor-based models use [submission_agent.py](./team_code_transfuser/submission_agent.py) as the `TEAM_AGENT` and point to the folder you downloaded the model weights into for the `TEAM_CONFIG`. The code is automatically configured to use the correct method based on the args.txt file in the model folder.\n\nYou can look at qualitative examples of the expected driving behavior of TransFuser on the Longest6 routes [here](https://www.youtube.com/watch?v=DZS-U3-iV0s\u0026list=PL6LvknlY2HlQG3YQ2nMIx7WcnyzgK9meO).\n\n### Parsing longest6 results\nTo compute additional statistics from the results of evaluation runs we provide a parser script [tools/result_parser.py](./tools/result_parser.py).\n\n```Shell\n${WORK_DIR}/tools/result_parser.py --xml ${WORK_DIR}/leaderboard/data/longest6/longest6.xml --results /path/to/folder/with/json_results/ --save_dir /path/to/output --town_maps ${WORK_DIR}/leaderboard/data/town_maps_xodr\n```\n\nIt will generate a results.csv file containing the average results of the run as well as additional statistics. It also generates town maps and marks the locations where infractions occurred.\n\n### Submitting to the CARLA leaderboard\nTo submit to the CARLA leaderboard you need docker installed on your system.\nEdit the paths at the start of [make_docker.sh](./leaderboard/scripts/make_docker.sh).\nCreate the folder *team_code_transfuser/model_ckpt/transfuser*.\nCopy the *model.pth* files and *args.txt* that you want to evaluate to *team_code_transfuser/model_ckpt/transfuser*.\nIf you want to evaluate an ensemble simply copy multiple .pth files into the folder, the code will load all of them and ensemble the predictions.\n\n```Shell\ncd leaderboard\ncd scripts\n./make_docker.sh\n```\nThe script will create a docker image with the name transfuser-agent.\nFollow the instructions on the [leaderboard](https://leaderboard.carla.org/submit/) to make an account and install alpha.\n\n```Shell\nalpha login\nalpha benchmark:submit  --split 3 transfuser-agent:latest\n```\nThe command will upload the docker image to the cloud and evaluate it.\n\n\u003c!-- ### Building docker image\n\nAdd the following paths to your ```~/.bashrc```\n```\nexport CARLA_ROOT=\u003cpath_to_carla_root\u003e\nexport SCENARIO_RUNNER_ROOT=\u003cpath_to_scenario_runner_in_this_repo\u003e\nexport LEADERBOARD_ROOT=\u003cpath_to_leaderboard_in_this_repo\u003e\nexport PYTHONPATH=\"${CARLA_ROOT}/PythonAPI/carla/\":\"${SCENARIO_RUNNER_ROOT}\":\"${LEADERBOARD_ROOT}\":${PYTHONPATH}\n```\n\nEdit the contents of ```leaderboard/scripts/Dockerfile.master``` to specify the required dependencies, agent code and model checkpoints. Add all the required information in the area delimited by the tags ```BEGINNING OF USER COMMANDS``` and ```END OF USER COMMANDS```. The current Dockerfile works for all the models in this repository.\n\nSpecify a name for the docker image in ```leaderboard/scripts/make_docker.sh``` and run:\n```\nleaderboard/scripts/make_docker.sh\n```\n\nRefer to the Transfuser example for the directory structure and where to include the code and checkpoints.\n\n### Testing the docker image locally\n\nSpin up a CARLA server:\n```\nSDL_VIDEODRIVER=offscreen SDL_HINT_CUDA_DEVICE=0 ./CarlaUE4.sh -world-port=2000 -opengl\n```\n\nRun the docker container:  \nDocker 19:  \n```\ndocker run -it --rm --net=host --gpus '\"device=0\"' -e PORT=2000 \u003cdocker_image\u003e ./leaderboard/scripts/run_evaluation.sh\n```\nIf the docker container doesn't start properly, add another environment variable ```SDL_AUDIODRIVER=dsp```.\n\n### Submitting docker image to the leaderboard\n\nRegister on [AlphaDriver](https://app.alphadrive.ai/), create a team and apply to the CARLA Leaderboard.\n\nInstall AlphaDrive cli:\n```\ncurl http://dist.alphadrive.ai/install-ubuntu.sh | sh -\n```\n\nLogin to alphadrive and submit the docker image:\n```\nalpha login\nalpha benchmark:submit --split \u003c2/3\u003e \u003cdocker_image\u003e\n```\nUse ```split 2``` for MAP track and ```split 3``` for SENSORS track. --\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fautonomousvision%2Ftransfuser","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fautonomousvision%2Ftransfuser","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fautonomousvision%2Ftransfuser/lists"}