{"id":18600787,"url":"https://github.com/autonomousvision/carla_garage","last_synced_at":"2025-05-16T09:03:46.442Z","repository":{"id":175232808,"uuid":"650712727","full_name":"autonomousvision/carla_garage","owner":"autonomousvision","description":"[ICCV'23] Hidden Biases of End-to-End Driving Models \u0026 A starter kit for the CARLA leaderboard 2.0.","archived":false,"fork":false,"pushed_at":"2025-02-26T18:23:30.000Z","size":119788,"stargazers_count":351,"open_issues_count":0,"forks_count":28,"subscribers_count":10,"default_branch":"leaderboard_2","last_synced_at":"2025-04-12T04:51:27.682Z","etag":null,"topics":["autonomous-driving","carla-leaderboard","carla-simulator","end-to-end-driving","imitation-learning"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/autonomousvision.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-06-07T16:39:18.000Z","updated_at":"2025-04-11T06:37:14.000Z","dependencies_parsed_at":null,"dependency_job_id":"5e4a9a39-b21f-4ed1-94cf-0d7647b7f127","html_url":"https://github.com/autonomousvision/carla_garage","commit_stats":{"total_commits":53,"total_committers":2,"mean_commits":26.5,"dds":"0.13207547169811318","last_synced_commit":"6a28d332c1c27c8c0750a10da698bba595004b7f"},"previous_names":["autonomousvision/carla_garage"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/autonomousvision%2Fcarla_garage","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/autonomousvision%2Fcarla_garage/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/autonomousvision%2Fcarla_garage/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/autonomousvision%2Fcarla_garage/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/autonomousvision","download_url":"https://codeload.github.com/autonomousvision/carla_garage/tar.gz/refs/heads/leaderboard_2","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254501554,"owners_count":22081528,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["autonomous-driving","carla-leaderboard","carla-simulator","end-to-end-driving","imitation-learning"],"created_at":"2024-11-07T02:05:27.066Z","updated_at":"2025-05-16T09:03:41.432Z","avatar_url":"https://github.com/autonomousvision.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/carla_garage_white.png\" alt=\"CARLA garage\" width=\"500\"/\u003e\n  \u003ch3 align=\"center\"\u003e\n        \u003ca href=\"https://arxiv.org/abs/2306.07957.pdf\"\u003e Paper\u003c/a\u003e | \u003ca href=\"https://youtu.be/ChrPW8RdqQU\"\u003eVideo\u003c/a\u003e | \u003ca href=\"https://youtu.be/x_42Fji1Z2M?t=1073\"\u003eTalk\u003c/a\u003e | \u003ca href=\"https://www.cvlibs.net/shared/common_misconceptions.pdf\"\u003e Slides\u003c/a\u003e | \u003ca href=\"https://github.com/autonomousvision/carla_garage/tree/main/assets/Jaeger2023ICCV_Poster.pdf\"\u003ePoster\u003c/a\u003e\n  \u003c/h3\u003e\n\u003c/p\u003e\n    \n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/hidden-biases-of-end-to-end-driving-models/carla-longest6-on-carla)](https://paperswithcode.com/sota/carla-longest6-on-carla?p=hidden-biases-of-end-to-end-driving-models)\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/hidden-biases-of-end-to-end-driving-models/autonomous-driving-on-carla-leaderboard)](https://paperswithcode.com/sota/autonomous-driving-on-carla-leaderboard?p=hidden-biases-of-end-to-end-driving-models)\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/hidden-biases-of-end-to-end-driving-models/carla-leaderboard-2-0-on-carla)](https://paperswithcode.com/sota/carla-leaderboard-2-0-on-carla?p=hidden-biases-of-end-to-end-driving-models)\n\n\u003e [**Hidden Biases of End-to-End Driving Models**](https://arxiv.org/abs/2306.07957) \u003cbr\u003e\n\u003e [Bernhard Jaeger](https://kait0.github.io/), [Kashyap Chitta](https://kashyap7x.github.io/), [Andreas Geiger](https://www.cvlibs.net/) \u003cbr\u003e\n\u003e International Conference on Computer Vision (ICCV), 2023\n\u003e \n\u003e This repo contains the code for the paper [**Hidden Biases of End-to-End Driving Models**](https://arxiv.org/abs/2306.07957) . \\\n\u003e We provide clean, configurable code with documentation as well as pre-trained weights with strong performance. \\\n\u003e The repository can serve as a good starting point for end-to-end autonomous driving research on [CARLA](https://github.com/carla-simulator/carla).\n\n## Contents\n\n1. [Setup](#setup)\n2. [Pre-Trained Models](#pre-trained-models)\n3. [Evaluation](#evaluation)\n4. [Dataset](#dataset)\n4. [Data generation](#data-generation)\n5. [Training](#training)\n6. [Additional Documenation](#additional-documentation)\n7. [Citation](#citation)\n\n## Setup\n\nClone the repo, setup CARLA 0.9.10.1, and build the conda environment:\n\n```Shell\ngit clone https://github.com/autonomousvision/carla_garage.git\ncd carla_garage\nchmod +x setup_carla.sh\n./setup_carla.sh\nconda env create -f environment.yml\nconda activate garage\n```\nBefore running the code, you will need to add the following paths to PYTHONPATH on your system:\n```Shell\nexport CARLA_ROOT=/path/to/CARLA/root\nexport WORK_DIR=/path/to/carla_garage\nexport PYTHONPATH=$PYTHONPATH:${CARLA_ROOT}/PythonAPI\nexport PYTHONPATH=$PYTHONPATH:${CARLA_ROOT}/PythonAPI/carla\nexport PYTHONPATH=$PYTHONPATH:$CARLA_ROOT/PythonAPI/carla/dist/carla-0.9.10-py3.7-linux-x86_64.egg\nexport SCENARIO_RUNNER_ROOT=${WORK_DIR}/scenario_runner\nexport LEADERBOARD_ROOT=${WORK_DIR}/leaderboard\nexport PYTHONPATH=\"${CARLA_ROOT}/PythonAPI/carla/\":\"${SCENARIO_RUNNER_ROOT}\":\"${LEADERBOARD_ROOT}\":${PYTHONPATH}\n```\nYou can add this in your shell scripts or directly integrate it into your favorite IDE. \\\nE.g. in PyCharm: Settings -\u003e Project -\u003e Python Interpreter -\u003e Show all -\u003e garage (need to add from existing conda environment first) -\u003e Show Interpreter Paths -\u003e add all the absolute paths above (without pythonpath).\n\n## Pre-Trained Models\nWe provide a set of pretrained models [here](https://s3.eu-central-1.amazonaws.com/avg-projects-2/jaeger2023arxiv/models/pretrained_models.zip).\nThe models are licensed under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0).\nThese are the final model weights used in the paper, the folder indicates the benchmark.\nFor the training and validation towns, we provide 3 models which correspond to 3 different training seeds. The format is `approach_trainingsetting_seed`. Each folder has an `args.txt` containing the training settings in text, a `config.pickle` containing all hyperparameters for the code and a `model_0030.pth` containing the model weights. Additionally, there are training logs for most models.\n\n## Evaluation\n\nTo evaluate a model, you need to start a CARLA server:\n```Shell\ncd /path/to/CARLA/root\n./CarlaUE4.sh -opengl\n```\nAfterward, run [leaderboard_evaluator_local.py](leaderboard/leaderboard/leaderboard_evaluator_local.py) as the main python file.\nIt is a modified version of the original leaderboard_evaluator.py which has the configurations used in the benchmarks we consider and additionally provides extra logging functionality.\n\nSet the `--agent-config` option to a folder containing a `config.pickle` and `model_0030.pth`. \u003cbr\u003e\nSet the `--agent` to [sensor_agent.py](team_code/sensor_agent.py). \u003cbr\u003e\nThe `--routes` option should be set to [lav.xml](leaderboard/data/lav.xml) or [longest6.xml](leaderboard/data/longest6.xml). \u003cbr\u003e\nThe `--scenarios ` option should be set to [eval_scenarios.json](leaderboard/data/scenarios/eval_scenarios.json) for both benchmarks. \u003cbr\u003e\nSet `--checkpoint ` to `/path/to/results/result.json`\n\nTo evaluate on a benchmark, set the respective environment variable: `export BENCHMARK=lav` or `export BENCHMARK=longest6`. \u003cbr\u003e\nSet `export SAVE_PATH=/path/to/results` to save additional logs or visualizations\n\nModels have inference options that can be set via environment variables.\nFor the longest6 model you need to set `export UNCERTAINTY_THRESHOLD=0.33`, for the LAV model `export STOP_CONTROL=1` and for the leaderboard model `export DIRECT=0`.\nOther options are correctly set by default. \u003cbr\u003e\nFor an example, you can check out [local_evaluation.sh](leaderboard/scripts/local_evaluation.sh).\n\nAfter running the evaluation, you need to parse the results file with [result_parser.py](tools/result_parser.py).\nIt will recompute the metrics (the initial once are [incorrect](https://github.com/carla-simulator/leaderboard/issues/117)) compute additional statistics and optionally visualize infractions as short video clips.\n```Shell\npython ${WORK_DIR}/tools/result_parser.py --xml ${WORK_DIR}/leaderboard/data/lav.xml --results /path/to/results --log_dir /path/to/results\n```\n\nThe result parser can optionally create short video/gif clips showcasing re-renderings of infractions that happened during evaluation. The code was developed by Luis Winckelmann and Alexander Braun.\nTo use this feature, you need to prepare some map files once (that are too large to upload to GitHub). For that, start a CARLA sever on your computer and run [prepare_map_data.py](tools/proxy_simulator/prepare_map_data.py).\nAfterward, you can run the feature by using the `--visualize_infractions` flag in [result_parser.py](tools/result_parser.py). The feature requires logs to be available in your results folder, so you need to set `export SAVE_PATH=/path/to/results` during evaluation.\n\n### How to actually evaluate\nThe instructions above are what you will be using to debug the code. Actually evaluating challenging benchmarks such as longest6, that have over 108 long routes, is very slow in practice. Luckily, CARLA evaluations are [embarrassingly parallel](https://en.wikipedia.org/wiki/Embarrassingly_parallel). Each of the 108 routes can be evaluated independently in parallel. That means if you have 2 GPUs you can evaluate 2x faster, if you have 108 GPUs you can evaluate 108x faster. While using the same amount of overall compute. To do that, you need access to a scalable cluster system and some scripts to parallelize. We are using [SLURM](https://slurm.schedmd.com/overview.html) at our institute. To evaluate a model, we are using the script [evaluate_routes_slurm.py](evaluate_routes_slurm.py). It is intended to be run inside a tmux on an interactive node and will spawn evaluation jobs (up till the number set in [max_num_jobs.txt](max_num_jobs.txt)). It also monitors the jobs and resubmits jobs where it detected a crash. In the end, the script will run the result parser to aggregate the results. If you are using a different system, you can use this as guidance and write your own script. The CARLA leaderboard benchmarks are the most challenging in the driving scene right now, but if you don't have access to multiple GPUs you might want to use simulators that are less compute intensive for your research. NuPlan is a good option, and our group also provides [strong baselines for nuPlan](https://github.com/autonomousvision/nuplan_garage).\n\n## Dataset\nWe released the dataset we used to train our final models.\nThe dataset is licensed under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0).\nYou can download it using:\n```Shell\ncd /path/to/carla_garage/tools\nbash download_data.sh\n```\nThe script will download the data to `/path/to/carla_garage/data`. This is also the path you need to set `--root_dir` to for training. The script will download and unzip the data with 11 parallel processes. The download is roughly 350 GB large (will be a bit more after unzipping).\n\n## Data Generation\nDataset generation is similar to evaluation. You can generate a dataset by changing the `--agent` option to [data_agent.py](team_code/data_agent.py) and the `--track` option to `MAP`. In addition, you need to set the following environment flags:\n```Shell\nexport DATAGEN=1\nexport BENCHMARK=collection\nexport CHECKPOINT_ENDPOINT=/path/to/dataset/Routes_{route}_Repetition{repetition}/Dataset_generation_{route}_Repetition{repetition}.json\nexport SAVE_PATH=/path/to/dataset/Routes_{route}_Repetition{repetition}\n```\nAgain it is too slow to generate our dataset with a single computer, you should be using multiple GPUs. We provide a [python script](generate_dataset_slurm.py) for SLURM clusters, it works in the same fashion as the evaluation script.\n\n## Training\nAgents are trained via the file [train.py](team_code/train.py).\nExamples how to use it are provided for [shell](team_code/shell_train.sh) and [SLURM](team_code/slurm_train.sh).\nYou need to activate garage conda environment before running it.\nIt first sets the relevant environment variables and then launches the training with torchrun.\nTorchrun is a pytorch tool that handles multi-gpu training.\nIf you want to debug on a single gpu simply set `--nproc_per_node=1`.\nThe training script has many options to configure your training you can list them with `python train.py --help` or look through the code.\nThe most important once are: \n```Shell\n--id your_model_000 # Name of your experiment\n--batch_size 32 # Batch size per GPU\n--setting all # Which towns to withhold during training. Use 'all' for leaderboard, longest6 and '02_05_withheld' for LAV models.\n--root_dir /path/to/dataset # Path to the root_dir of your dataset\n--logdir /path/to/models # Root dir where the training files will be stored\n--use_controller_input_prediction 1 # Whether your model trains with a classification + path prediction head\n--use_wp_gru 0 # Whether you model trains with a waypoint head.\n--use_discrete_command 1 # Whether to use the navigational command as input to the model\n--use_tp 1  # Whether to use the target point as input to your model\n--cpu_cores 20 # Total number of cpu cores on your machine\n--num_repetitions 3 # How much data to train on (Options are 1,2,3). 1x corresponds to 185k in Table 5, 3x corresponds to 555k\n```\nAdditionally, to do the two stage training from Table 4 you need the `--continue_epoch` and `--load_file` option.\nYou need to train twice. First train a model with set `--use_wp_gru 0` and `--use_controller_input_prediction 0`, this will only train the perception backbone with auxiliary losses.\nThen, train a second model, set e.g. `--use_controller_input_prediction 1`, `--continue_epoch 0` and `--load_file /path/to/stage1/model_0030.pth`.\nThe `load_file` option is usually used to resume a crashed training, but with `--continue_epoch 0` the training will start from scratch with the pre-trained weights used for initialization.\n\n### Training in PyCharm\nYou can also run and debug torchrun in PyCharm. To do that you need to set your run/debug configuration as follows:\\\nSet the script path to: `/path/to/train.py` \\\nSet the interpreter options to:\n```Shell\n-m torch.distributed.run --nnodes=1 --nproc_per_node=1 --max_restarts=0 --rdzv_id=123456780 --rdzv_backend=c10d\n```\nTraining parameters should be set in the `Parameters:` field and environment variable in `Environment Variables:`.\nAdditionally, you need to set up conda environment (variables) as described above.\n\n## Submitting to the CARLA leaderboard\nTo submit to the CARLA leaderboard, you need docker installed on your system (as well as the nvidia-container-toolkit to test it). Create the folder `team_code/model_ckpt/transfuser`. Copy the model.pth files and config.pickle that you want to evaluate to team_code/model_ckpt/transfuser. If you want to evaluate an ensemble, simply copy multiple .pth files into the folder, the code will load all of them and ensemble the predictions.\nEdit the environment paths at the top of `tools/make_docker.sh` and then:\n\n```Shell\ncd tools\n./make_docker.sh\n```\n\nThe script will create a docker image with the name transfuser-agent.\nBefore submitting, you should locally test your image. To do that, start up a CARLA server on your computer (it will be able to communicate with the docker container via ports). Then start your docker container. An example is provided in [run_docker.sh](tools/run_docker.sh).\nInside the docker container start your agent using:\n```Shell\ncd leaderboard\ncd scripts\nbash run_evaluation.sh\n```\nYou can stop the evaluation, after confirming that there is no issue, using \"ctrl + c + \\\". \u003cbr\u003e\nTo submit, follow the instructions on the [leaderboard](https://leaderboard.carla.org/submit_v1/) to make an account and install alpha.\n\n```Shell\nalpha login\nalpha benchmark:submit  --split 3 transfuser-agent:latest\n```\nThe command will upload the docker image to the cloud and evaluate it.\n\n## Additional Documentation\n- **Coordinate systems** in CARLA repositories are usually a big mess. In this project, we addressed this by changing all data into a unified coordinate frame. Further information about the coordinate system can be found [here](docs/coordinate_systems.md).\n\n- The TransFuser model family has grown quite a lot with different variants, which can be confusing for new community members. The **[history](docs/history.md)** file explains the different versions and which paper you should cite to refer to them.\n\n- Building a full autonomous driving stack involves quite some [**engineering**](docs/engineering.md). The documentation explains some of the techniques and design philosophies we used in this project.\n\n- The codebase can run any experiment presented in the paper. It also supports some additional features that we did not end up using. They are documented [here](docs/additional_features.md).\n\n## Contact\nIf you have any questions or suggestions, please feel free to open an issue or contact us at bernhard.jaeger@uni-tuebingen.de.\n\n## Citation\nIf you find CARLA garage useful, please consider giving us a star \u0026#127775; and citing our paper with the following BibTeX entry.\n\n```BibTeX\n@InProceedings{Jaeger2023ICCV,\n  title={Hidden Biases of End-to-End Driving Models},\n  author={Bernhard Jaeger and Kashyap Chitta and Andreas Geiger},\n  booktitle={Proc. of the IEEE International Conf. on Computer Vision (ICCV)},\n  year={2023}\n}\n```\n\n\n## Acknowledgements\nOpen source code like this is build on the shoulders of many other open source repositories.\nIn particularly we would like to thank the following repositories for their contributions:\n* [simple_bev](https://github.com/aharley/simple_bev)\n* [transfuser](https://github.com/autonomousvision/transfuser)\n* [interfuser](https://github.com/opendilab/InterFuser)\n* [mmdet](https://github.com/open-mmlab/mmdetection)\n* [roach](https://github.com/zhejz/carla-roach/)\n* [plant](https://github.com/autonomousvision/plant)\n* [king](https://github.com/autonomousvision/king)\n* [wor](https://github.com/dotchen/WorldOnRails)\n* [tcp](https://github.com/OpenDriveLab/TCP)\n* [lbc](https://github.com/dotchen/LearningByCheating)\n\nWe also thank the creators of the numerous pip libraries we use. Complex projects like this would not be feasible without your contribution.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fautonomousvision%2Fcarla_garage","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fautonomousvision%2Fcarla_garage","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fautonomousvision%2Fcarla_garage/lists"}