{"id":18600775,"url":"https://github.com/autonomousvision/plant","last_synced_at":"2025-08-07T20:06:18.751Z","repository":{"id":62744220,"uuid":"551523316","full_name":"autonomousvision/plant","owner":"autonomousvision","description":"[CoRL'22] PlanT: Explainable Planning Transformers via Object-Level Representations","archived":false,"fork":false,"pushed_at":"2024-11-04T09:42:05.000Z","size":15841,"stargazers_count":260,"open_issues_count":6,"forks_count":35,"subscribers_count":10,"default_branch":"main","last_synced_at":"2025-05-16T22:01:37.754Z","etag":null,"topics":["autonomous-driving","carla-simulator","corl22","planning","self-driving","transformer"],"latest_commit_sha":null,"homepage":"https://katrinrenz.de/plant/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/autonomousvision.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-10-14T15:02:50.000Z","updated_at":"2025-05-15T02:12:23.000Z","dependencies_parsed_at":"2024-12-11T08:01:50.690Z","dependency_job_id":"0e444fa2-3f1e-464a-b1c3-58d1b43b2c4a","html_url":"https://github.com/autonomousvision/plant","commit_stats":{"total_commits":13,"total_committers":3,"mean_commits":4.333333333333333,"dds":"0.15384615384615385","last_synced_commit":"1b986a8b918b5c6f265d0d4931bce4f55f53a5f9"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/autonomousvision/plant","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/autonomousvision%2Fplant","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/autonomousvision%2Fplant/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/autonomousvision%2Fplant/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/autonomousvision%2Fplant/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/autonomousvision","download_url":"https://codeload.github.com/autonomousvision/plant/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/autonomousvision%2Fplant/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":269317236,"owners_count":24396846,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-07T02:00:09.698Z","response_time":73,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["autonomous-driving","carla-simulator","corl22","planning","self-driving","transformer"],"created_at":"2024-11-07T02:05:22.320Z","updated_at":"2025-08-07T20:06:18.726Z","avatar_url":"https://github.com/autonomousvision.png","language":"Python","funding_links":[],"categories":["💻 Open-Source Projects"],"sub_categories":["Papers"],"readme":"# PlanT: Explainable Planning Transformers via Object-Level Representations\n\n\n**News**: \\\n**19.01.2023:** We released the code to generate the attention visualization. \\\n**02.12.2022:** We released the perception checkpoint and the code for the SENSORS and MAP track agent. Conda environment needs to be updated. Checkpoints of the perception are in the checkpoint folder. Please download again. \\\n**11.11.2022:** We made some changes in the agent files to ensure compatibility with our perception PlanT. We therefore uploaded new checkpoint files. The old one does not work anymore with the current code.\n\n## [Project Page](http://www.katrinrenz.de/plant) | [Paper](https://arxiv.org/abs/2210.14222) | [Supplementary](https://www.katrinrenz.de/plant/resources/PlanT_supp.pdf) \n\n\nThis repository provides code for the following paper:\n\n- [Katrin Renz](https://www.katrinrenz.de), [Kashyap Chitta](https://kashyap7x.github.io/), [Otniel-Bogdan Mercea](https://merceaotniel.github.io/), [A. Sophia Koepke](https://www.eml-unitue.de/people/almut-sophia-koepke), [Zeynep Akata](https://www.eml-unitue.de/people/zeynep-akata) and [Andreas Geiger](http://www.cvlibs.net/),\n*PlanT: Explainable Planning Transformers via Object-Level Representations*, CoRL 2022.  \n\n\n![demo](gfx/plant_teaser.gif)\n\n# Content\n* [Setup](#setup)\n* [Data and models](#data-and-models)\n* [Data generation](#data-generation)\n* [Training](#training)\n* [Evaluation](#evaluation)\n* [Explainability](#explainability)\n* [Perception PlanT](#perception-plant)\n* [Citation](#citation)\n\n\n## Setup\nFirst, you have to install carla and the conda environment.\n\n``` bash\n# 1. Clone this repository\ngit clone https://github.com/autonomousvision/plant.git\ncd plant\n# 2. Setup Carla\n# if you have carla already installed, skip the next step AND\n# adapt the carla path in setup_env.sh before executing step 3.\nchmod +x setup_carla.sh\n./setup_carla.sh\n# 3. Setup conda environment\nchmod +x setup_env.sh\n./setup_env.sh\n\nconda activate plant\npip install -U openmim\nmim install mmcv-full==1.7.0\npip install mmdet\n```\n\n\n## Data and models\nYou can download our pretrained PlanT models by executing:\n``` bash\nchmod +x download.sh\n./download.sh\n```\n\nTo download our 3x dataset run:\n``` bash\nchmod +x download_data.sh\n./download_data.sh\n```\n\n\n## Data generation\nYou can download our dataset or generate your own dataset.\nIn order to generate your own one you first need to start a Carla server:\n```\n# with display\n./carla/CarlaUE4.sh --world-port=2000 -opengl\n```\n```\n# without display\nSDL_VIDEODRIVER=offscreen SDL_HINT_CUDA_DEVICE=0 ./carla/CarlaUE4.sh --world-port=2000 -opengl\n```\n\nTo generate the data for the route specified in `carla_agent_files/config/eval/train.yaml` you can run\n```\npython leaderboard/scripts/run_evaluation.py user=$USER experiments=datagen eval=train\n```\nIf you want to also save the sensor data that we used to train the perception module you can add the flag `experiments.SAVE_SENSORS=1`.\n\nTo generate the whole dataset you can use the `datagen.sh` file.\n\n\n## Training\nTo run the PlanT training on the 3x dataset, run:\n```\npython training/PlanT/lit_train.py user=$USER model=PlanT\n```\nTo change any hyperparameters have a look at `training/config/model/PlanT.yaml`. For general training settings (e.g., #GPUs) check `training/config/config.yaml`.\n\n\n## Evaluation\nThis evaluates the PlanT model on the specified benchmark (default: longest6). The config is specified in the folder `carla_agent_files/config`.\n\nStart a Carla server (see [Data generation](#data-generation)).\\\nWhen the server is running, start the evaluation with:\n```\npython leaderboard/scripts/run_evaluation.py user=$USER experiments=PlanTmedium3x eval=longest6\n```\nYou can find the results of the evaluation in a newly created evaluation folder inside the model folder. If you want to have a (very minimalistic) visualization you can set the `viz` flag (i.e., `python leaderboard/scripts/run_evaluation.py user=$USER experiments=PlanTmedium3x eval=longest6 viz=1`)\n\n\n## Explainability\nThe execution of the explainability agent contains two stages: (1) PlanT forwardpass (no execution of actions) to get attention weights. We filter the vehicles so that only the vehicles with the `topk` attention scores remain as input for the second step. (2) We execute either the expert or PlanT with the filtered input (the agent only sees `topk` vehicles instead of all).\n\nStart a Carla server (see [Data generation](#data-generation)). \\\nWhen the server is running, start the evaluation with:\n```\npython leaderboard/scripts/run_evaluation.py user=$USER experiments=PlanTExplainability experiments.exec_model=Expert experiments.topk=1\n```\n\nTo obtain the **attention visualization** set `experiments.topk=100000` and in addition add the flag `save_explainability_viz=True`. This saves a video per route in a `viz_vid` folder. The image resolution can be changed in `carla_agent_files/explainability_agent.py`. \\\n*Attention:* saving the videos slows the evaluation down.\n\n\n## Perception PlanT\nWe release two PlanT agents suitable for the two CARLA Leaderboard tracks. For the SENSORS track we predict the route with our perception module. In the MAP track model we get the route information from the map. The code is taken from the [TransFuser (PAMI 2022) repo](https://github.com/autonomousvision/transfuser) and adapted for our usecase. The config is specified in the folder `carla_agent_files/config`. The config for the perception model is in `training/Perception/config.py`.\n\n### SENSORS track\nStart a Carla server (see [Data generation](#data-generation)).\n\nWhen the server is running, start the evaluation with:\n```\npython leaderboard/scripts/run_evaluation.py user=$USER experiments=PlanTSubmission track=SENSORS eval=longest6 save_path=SENSORSagent\n```\nVisualization can be activated with the `viz` flag, and the unblocking from the TransFuser repo can be activated with the `experiments.unblock` flag.\n\n### MAP track\nStart a Carla server (see [Data generation](#data-generation)).\n\nWhen the server is running, start the evaluation with:\n```\npython leaderboard/scripts/run_evaluation.py user=$USER experiments=PlanTSubmissionMap track=MAP eval=longest6 save_path=MAPagent\n```\nVisualization can be activated with the `viz` flag, and the unblocking from the TransFuser repo can be activated with the `experiments.unblock` flag.\n\n## Citation\nIf you use this code and data, please cite the following:\n\n```bibtex\n@inproceedings{Renz2022CORL,\n    author       = {Katrin Renz and Kashyap Chitta and Otniel-Bogdan Mercea and A. Sophia Koepke and Zeynep Akata and Andreas Geiger},\n    title        = {PlanT: Explainable Planning Transformers via Object-Level Representations},\n    booktitle    = {Conference on Robotic Learning (CoRL)},\n    year         = {2022}\n}\n```\n\nAlso, check out the code for other recent work on CARLA from our group:\n- [Chitta et al., TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving (PAMI 2022)](https://github.com/autonomousvision/transfuser)\n- [Hanselmann et al., KING: Generating Safety-Critical Driving Scenarios for Robust Imitation via Kinematics Gradients (ECCV 2022)](https://github.com/autonomousvision/king)\n- [Chitta et al., NEAT: Neural Attention Fields for End-to-End Autonomous Driving (ICCV 2021)](https://github.com/autonomousvision/neat)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fautonomousvision%2Fplant","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fautonomousvision%2Fplant","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fautonomousvision%2Fplant/lists"}