{"id":23691934,"url":"https://github.com/justin900429/autonomous_driving_with_diffusion_model","last_synced_at":"2025-04-11T09:50:53.370Z","repository":{"id":244209488,"uuid":"807207344","full_name":"Justin900429/autonomous_driving_with_diffusion_model","owner":"Justin900429","description":"Using diffusion model to reach controllable end-to-end driving with Carla simulation environment.","archived":false,"fork":false,"pushed_at":"2025-03-18T13:25:41.000Z","size":143207,"stargazers_count":10,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-18T13:46:51.383Z","etag":null,"topics":["autonomous-driving","carla-simulator","controllable-generation","deep-learning","diffusion-model","end-to-end-driving","pytorch"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Justin900429.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-28T16:57:15.000Z","updated_at":"2025-03-18T13:25:44.000Z","dependencies_parsed_at":"2024-12-30T03:09:36.517Z","dependency_job_id":null,"html_url":"https://github.com/Justin900429/autonomous_driving_with_diffusion_model","commit_stats":null,"previous_names":["justin900429/carla_diffusion","justin900429/autonomous_driving_with_diffusion_model"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Justin900429%2Fautonomous_driving_with_diffusion_model","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Justin900429%2Fautonomous_driving_with_diffusion_model/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Justin900429%2Fautonomous_driving_with_diffusion_model/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Justin900429%2Fautonomous_driving_with_diffusion_model/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Justin900429","download_url":"https://codeload.github.com/Justin900429/autonomous_driving_with_diffusion_model/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248370661,"owners_count":21092854,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["autonomous-driving","carla-simulator","controllable-generation","deep-learning","diffusion-model","end-to-end-driving","pytorch"],"created_at":"2024-12-30T02:59:35.837Z","updated_at":"2025-04-11T09:50:53.364Z","avatar_url":"https://github.com/Justin900429.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Autonomous Driving with Diffusion Model\n\nThis repo is made to acheive controllable end-to-end autonomous driving through the Diffusion model. We provide the code for training the model and interacting with the model in the Carla simulator. We also provide the pre-trained model for users to evaluate the performance on the Carla leaderboard 1.0.\n\n|                 No Guidance                 |                    Classifier-free Guidance                     |                  Classifier Guidance                  |\n| :-----------------------------------------: | :-------------------------------------------------------------: | :---------------------------------------------------: |\n| ![no guidance gif](./assets/no_guide.gif) | ![classifier-free guidance gif](./assets/classifier_free.gif) | ![classifier guidance gif](./assets/classifier.gif) |\n\n## A. Setup 🚀\n\nPlease ensure you have installed the Carla simulator and the Python API.\n\n\u003e [!WARNING]\n\u003e You should checkout your own cuda version and install your own PyTorch version. We provide an example of PyTorch 2.2.2 with cuda 12.1.\n\n```shell\ngit clone https://github.com/Justin900429/carla_diffusion.git\nconda create -n carla-diffusion python=3.8 -y\nconda activate carla-diffusion\nconda install pytorch==2.2.2 torchvision==0.17.2 pytorch-cuda=12.1 -c pytorch -c nvidia -y\npip install -r requirements.txt\n```\n\nIf you would like to collect data by yourself, please install the Carla Python API and [Carla simulator](https://github.com/carla-simulator/carla):\n\n```shell\n# Carla version \u003c 0.9.12\neasy_install install {CARLA_ROOT}/PythonAPI/carla/dist/carla-{CARLA_VERSION}-py{CHECK_THIS_VERSION}-linux-x86_64.egg\n\n# Carla version \u003e= 0.9.12\npip install carla=={CARLA_VERSION}\n```\n\nAfterwards, please modify the `carla_sh_path` in `config/train_rl.yaml` to yours.\n\n## B. Data setup 📊\n\n\u003e[!NOTE]\n\u003eUsers can choose to download the provided data or collect the data by themselves.\n\n### (Option 1) Downloading the data 📦\n\n```shell\nwget https://huggingface.co/datasets/Justin900/carla_image_and_path/resolve/main/data.zip\nunzip -q data.zip \u0026\u0026 rm data.zip\n```\n\n### (Option 2) Collecting the data 📡\n\n```shell\npython misc/data_collect.py --save-path {PLACE_TO_SAVE_DATA} --save-num {NUM_OF_DATA}\n\n# Concrete example\npython misc/data_collect.py --save-path data/ --save-num 5000\n```\n\n\u003e If you would like to collect data under `off-screen` mode, please add the flag `--off-screen`.\n\nIf you find the data collection process fail during the simulation, please try the following way (this always run in `off-screen` mode):\n\n```shell\npython misc/collect_loop.py --save-path {PLACE_TO_SAVE_DATA} --save-num {NUM_OF_DATA}\n\n# Concrete example\npython misc/collect_loop.py --save-path data/ --save-num 5000\n```\n\nThis helps restart the simulation when the simulation is crashed but the number of data does not reach the target.\n\n## C. Usage 🛠\n\n### C-1. Model training 🧠\n\nUsers can choose the config file as shown below to train the model.\n\n| Config path                                 | Description                                                |\n| ------------------------------------------- | ---------------------------------------------------------- |\n| `configs/default.yaml`                      | Train the model without any guidance. (For ablation study) |\n| `configs/guidance/free_guidance.yaml`       | Train the model with classifier-free guidance.             |\n| `configs/guidance/classifier_guidance.yaml` | Train the model with classifier guidance.                  |\n\n```shell\n# with single-gpu\npython train.py --config {CONFIG_PATH}\n\n# with multi-gpus\naccelerate launch --multi_gpu --num_processes={NUM_OF_GPU} train.py --config {CONFIG_PATH}\n```\n\n### C-2. Interact with the model 🕹\n\n\u003e[!TIP]\n\u003e Check the description above to choose the config file.\n\n```shell\npython interact.py --config {CONFIG_PATH} --plot-on-world --save-bev-path {PATH_TO_SAVE_BEV_IMAGES} --opts EVAL.CHECKPOINT final.pth\n\n# Concrete example\n# 1. without any guidance\npython interact.py --config configs/default.yaml --plot-on-world --save-bev-path bev_images  --opts EVAL.CHECKPOINT final.pth\n\n# 2. with classifier-free guidance\npython interact.py --config configs/guidance/free_guidance.yaml --plot-on-world --save-bev-path bev_images  --opts EVAL.CHECKPOINT final.pth\n\n# 3. with classifier guidance\npython interact.py --config configs/guidance/classifier_guidance.yaml --plot-on-world --save-bev-path bev_images  --opts EVAL.CHECKPOINT final.pth\n```\n\n\u003e [!NOTE]\n\u003e Both `--plot-on-world` and `--save-bev-path` are optional.\n\n## D. Leaderboard 🏆\n\n### (Optional) D-0. Download the pre-trained model 📦\n\nWe provide our pre-trained model for users to evaluate the performance on the Carla leaderboard.\n\n| Model                                | Link                                                                                           |\n| :----------------------------------- | :--------------------------------------------------------------------------------------------- |\n| Classifier-free guidance             | [link](https://github.com/Justin900429/autonomous_driving_with_diffusion_model/releases/download/v0.0.0/free_guidance_plus.pth)    |\n| Classifier guidance                  | [link](https://github.com/Justin900429/autonomous_driving_with_diffusion_model/releases/download/v0.0.0/classifier_guidance.pth) |\n| Classifier guidance (with more data) | [link](https://github.com/Justin900429/autonomous_driving_with_diffusion_model/releases/download/v0.0.0/classifier_guidance_plus.pth) |\n\n```shell\nmkdir checkpoints\n\n# Classifier-free guidance\nwget https://github.com/Justin900429/autonomous_driving_with_diffusion_model/releases/download/v0.0.0/free_guidance_plus.pth -O checkpoints/free_guidance.pth\n\n# Classifier guidance\nwget https://github.com/Justin900429/autonomous_driving_with_diffusion_model/releases/download/v0.0.0/classifier_guidance.pth -O checkpoints/classifier_guidance.pth\n\n# Classifier gudiance (with more training data)\nwget https://github.com/Justin900429/autonomous_driving_with_diffusion_model/releases/download/v0.0.0/classifier_guidance_plus.pth -O checkpoints/classifier_guidance_plus.pth\n```\n\n### D-1. Environment setup\n\n\u003e[!TIP]\n\u003e This environment is different from the previous one and requires python 3.7 with Carla 0.9.10.\n\n```shell\nconda create -n carla-leaderboard python=3.7 -y\nconda activate carla-leaderboard\nconda install pytorch==1.13.0 torchvision==0.14.0 pytorch-cuda=11.7 -c pytorch -c nvidia -y\npip install -r requirements-leaderboard.txt\n```\n\n### D-2. Run the evaluator\n\nPlease check the below table and command to run different benchmarks.\n\n| Benchmark | Scenario file                                    | Route file                                             |\n| --------- | ------------------------------------------------ | ------------------------------------------------------ |\n| Longest 6 | `leaderboard/data/scenarios/eval_scenarios.json` | `leaderboard/data/evaluation_routes/longest6_tiny.xml` |\n\n```shell\n# Open this in another terminal\nbash {carla_server_root}/CarlaUE4.sh --world-port=2000 -opengl\n\nbash leaderboard/scripts/run_evaluation.sh \u003ccarla_server_root\u003e \u003cscenario_file\u003e \u003croute_file\u003e \u003cagent_config_file\u003e \u003csave_folder\u003e \u003csave_file\u003e\n\n# Concrete example (take longest 6 as an example)\nbash leaderboard/scripts/run_evaluation.sh \\\n     /path/to/carla/0.9.10\\\n     leaderboard/data/scenarios/eval_scenarios.json \\\n     leaderboard/data/evaluation_routes/longest6_tiny.xml \\\n     configs/guidance/free_guidance.yaml \\\n     free_guidance_longest_6 \\\n     free_guidance_longest_6/result_longest_6.json\n```\n\n\u003e[!IMPORTANT]\n\u003e Users should specify the checkpoint directly in the agent config file (see C-1) by setting `EVAL.CHECKPOINT`.\n\n```yaml\n...\nEVAL:\n    CHECKPOINT: /path/to/checkpoint.pth\n```\n\n### D-3. Generate the statistics\n\nAfter obtaining the `\u003csave_file\u003e` results, users can generate the statistics by running the following command:\n\n```shell\npython e2e_driving/statistics.py --json-file \u003csave_file\u003e\n\n# Concrete example\npython e2e_driving/statistics.py --json-file free_guidance_longest_6/result_longest_6.json\n```\n\n\u003ctable\u003e\n    \u003ctr\u003e\n      \u003cth rowspan=\"2\"\u003eApproach\u003c/th\u003e\n      \u003cth colspan=\"3\"\u003eScore\u003c/th\u003e\n      \u003cth colspan=\"3\"\u003eCollision\u003c/th\u003e\n      \u003cth rowspan=\"2\"\u003eRed light\u003c/th\u003e\n      \u003cth rowspan=\"2\"\u003eVehicle Blocked\u003c/th\u003e\n      \u003cth rowspan=\"2\"\u003eOutside Road\u003c/th\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd\u003eComposed\u003c/td\u003e\n      \u003ctd\u003ePenalty\u003c/td\u003e\n      \u003ctd\u003eRoute\u003c/td\u003e\n      \u003ctd\u003eLayout\u003c/td\u003e\n      \u003ctd\u003ePedestrian\u003c/td\u003e\n      \u003ctd\u003eVehicle\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd\u003e Classifier-free guidance \u003c/td\u003e\n      \u003ctd\u003e 0.00 \u003c/td\u003e\n      \u003ctd\u003e 0.00 \u003c/td\u003e\n      \u003ctd\u003e 100.00 \u003c/td\u003e\n      \u003ctd\u003e 0.53 \u003c/td\u003e\n      \u003ctd\u003e 0.76 \u003c/td\u003e\n      \u003ctd\u003e 8.77 \u003c/td\u003e\n      \u003ctd\u003e 2.60 \u003c/td\u003e\n      \u003ctd\u003e 0.00 \u003c/td\u003e\n      \u003ctd\u003e 0.17 \u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003eclassifier guidance\u003c/td\u003e\n        \u003ctd\u003e2.66\u003c/td\u003e\n        \u003ctd\u003e0.10\u003c/td\u003e\n        \u003ctd\u003e72.80\u003c/td\u003e\n        \u003ctd\u003e0.25\u003c/td\u003e\n        \u003ctd\u003e0.00\u003c/td\u003e\n        \u003ctd\u003e4.53\u003c/td\u003e\n        \u003ctd\u003e2.82\u003c/td\u003e\n        \u003ctd\u003e0.37\u003c/td\u003e\n        \u003ctd\u003e0.00\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003eclassifier guidance*\u003c/td\u003e\n        \u003ctd\u003e14.89\u003c/td\u003e\n        \u003ctd\u003e0.23\u003c/td\u003e\n        \u003ctd\u003e84.84\u003c/td\u003e\n        \u003ctd\u003e0.00\u003c/td\u003e\n        \u003ctd\u003e0.00\u003c/td\u003e\n        \u003ctd\u003e0.91\u003c/td\u003e\n        \u003ctd\u003e1.90\u003c/td\u003e\n        \u003ctd\u003e0.23\u003c/td\u003e\n        \u003ctd\u003e0.00\u003c/td\u003e\n    \u003c/tr\u003e\n\u003c/table\u003e\n\n## Acknowledgement 🙏\n\n* Our environment is adapted from [Carla Roach](https://github.com/zhejz/carla-roach).\n* The project template is obtained from [deep-learning-template](https://github.com/Justin900429/deep-learning-template).\n* The agent for carla leaderboard is adapted from [TCP](https://github.com/OpenDriveLab/TCP).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjustin900429%2Fautonomous_driving_with_diffusion_model","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjustin900429%2Fautonomous_driving_with_diffusion_model","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjustin900429%2Fautonomous_driving_with_diffusion_model/lists"}