{"id":20731451,"url":"https://github.com/opendrivelab/openpilot-deepdive","last_synced_at":"2025-04-06T12:11:15.537Z","repository":{"id":37347933,"uuid":"464766842","full_name":"OpenDriveLab/Openpilot-Deepdive","owner":"OpenDriveLab","description":"Our insights of Openpilot, a deepdive project on it","archived":false,"fork":false,"pushed_at":"2023-12-06T13:11:52.000Z","size":201563,"stargazers_count":254,"open_issues_count":4,"forks_count":73,"subscribers_count":9,"default_branch":"main","last_synced_at":"2025-04-06T03:38:45.356Z","etag":null,"topics":["end-to-end-autonomous-driving","perception-algorithm","planning-and-control"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/OpenDriveLab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null},"funding":{"github":["OpenDriveLab"],"patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"lfx_crowdfunding":null,"custom":null}},"created_at":"2022-03-01T06:11:30.000Z","updated_at":"2025-03-16T07:34:53.000Z","dependencies_parsed_at":"2023-12-06T14:29:57.555Z","dependency_job_id":"6f5b37fb-7a40-4cb4-b8f2-eebdc14536f7","html_url":"https://github.com/OpenDriveLab/Openpilot-Deepdive","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenDriveLab%2FOpenpilot-Deepdive","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenDriveLab%2FOpenpilot-Deepdive/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenDriveLab%2FOpenpilot-Deepdive/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenDriveLab%2FOpenpilot-Deepdive/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/OpenDriveLab","download_url":"https://codeload.github.com/OpenDriveLab/Openpilot-Deepdive/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247478324,"owners_count":20945266,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["end-to-end-autonomous-driving","perception-algorithm","planning-and-control"],"created_at":"2024-11-17T05:14:52.975Z","updated_at":"2025-04-06T12:11:15.520Z","avatar_url":"https://github.com/OpenDriveLab.png","language":"Python","funding_links":["https://github.com/sponsors/OpenDriveLab"],"categories":[],"sub_categories":[],"readme":"# \u003cp align=\"center\"\u003eOpenpilot-Deepdive\u003c/p\u003e\n**\u003cp align=\"center\"\u003eLevel 2 Autonomous Driving on a Single Device: Diving into the Devils of Openpilot\u003c/p\u003e**\n![image](./imgs/arXiv_fig1.png)\n[**Webpage**](https://sites.google.com/view/openpilot-deepdive/home) | [**Paper**](https://arxiv.org/abs/2206.08176) | [**Zhihu**](https://www.zhihu.com/people/PerceptionX)\n***\n# Introduction\n\nThis repository is the PyTorch implementation for our Openpilot-Deepdive.\nIn contrast to most traditional autonomous driving solutions where the perception, prediction, and planning module are apart, [Openpilot](https://github.com/commaai/openpilot) uses an end-to-end neural network to predict the trajectory directly from the camera images, which is called Supercombo. We try to reimplement the training details and test the pipeline on public benchmarks. Experimental results of OP-Deepdive on nuScenes, Comma2k19, CARLA, and in-house realistic scenarios (collected in Shanghai) verify that a low-cost device can indeed achieve most L2 functionalities and be on par with the original Supercombo model. We also test on CommaTwo device with a dual-model deploeyment framework, which is in this repo: [Openpilot-Deployment](https://github.com/OpenPerceptionX/Openpilot-Deepdive/tree/deploy/Openpilot-Deployment).\n\n* [Directory Structure](#directory-structure) \n* [Changelog](#changelog) \n* [Quick Start](#quick-start-examples) \n    * [Installation](#installation)\n    * [Dataset](#dataset)\n    * [Training and Testing](#training-and-testing)\n    * [Demo](#demo)\n* [Baselines](#baselines)\n* [Citation](#citation) \n* [License](#license)  \n\n***\n# Directory Structure\n\n```\nOpenpilot-Deepdive \n├── tools           - Tools to generate split on Comma2k19 and nuScenes datasets.  \n├── utils_comma2k19 - The utils provided by comma, copied from `commaai/comma2k19.git/utils`\n├── data\n      ├── nuscenes  -\u003e soft link to the nusSenes-all dataset\n      ├── comma2k19 -\u003e soft link to the Comma2k19 dataset\n```\n***\n# Changelog\n\n2022-6-17: We released the v1.0 code for Openpilot-Deepdive.\n\n2022-6-26: We fix some problems and update the readme for using the code on bare-metal machines. Thanks @EliomEssaim and @MicroHest!\n\n2022-7-13: We released the v1.0 code for [Openpilot-Deployment](https://github.com/OpenPerceptionX/Openpilot-Deepdive/tree/deploy/Openpilot-Deployment) for dual-model deployment in the Openpilot framework.\n\n***\n# Quick Start Examples\nBefore starting, we refer you to read the [arXiv](https://arxiv.org/abs/2206.08176) to understand the details of our work.\n## Installation\nClone repo and install requirements.txt in a [Python\u003e=3.7.0](https://www.python.org/) environment, including [PyTorch\u003e=1.7](https://pytorch.org/get-started/locally/).\n\n```\ngit clone https://github.com/OpenPerceptionX/Openpilot-Deepdive.git  # clone\ncd Openpilot-Deepdive\npip install -r requirements.txt  # install\n```\n## Dataset\nWe train and evaluate our model on two datasets, [nuScenes](https://www.nuscenes.org/nuscenes) and [Comma2k19](https://github.com/commaai/comma2k19).\nThe table shows some key features of them.\n\n| Dataset     | Raw\u003cbr\u003eFPS (Hz)  | Aligned\u0026\u003cbr\u003eFPS (Hz) | Length Per\u003cbr\u003eSequence\u003cbr\u003e(Frames/Second) | Altogether\u003cbr\u003eLength\u003cbr\u003e(Minutes) | Scenario | Locations |\n| :----:     |:----:|:----:|:----:|:----:|:----:|:----:|\n| nuScenes | 12 | 2 | 40 / 20 | 330 | Street | America\u003cbr\u003eSingapore |  \n| Comma2k19  | 20 | 20 | 1000 / 60 | 2000 | Highway | America | \n\nPlease create a `data` folder and create soft links to the datasets.\n\nFor dataset splits, you may create your own by running the scripts in the `tools` folder, or download it in https://github.com/OpenPerceptionX/Openpilot-Deepdive/issues/4.\n\n## Training and Testing\nBy default, the batch size is set to be 6 per GPU, which consumes 27 GB GPU memory. When using 8 V100 GPUs, it takes approximate 120 hours to train 100 epochs on Comma2k19 dataset. \n\n**Note**: Our lab use `slurm` to run and manage the tasks. Then, the PyTorch distributed training processes are initialized manually by `slurm`, since the automatic `mp.spawn` may cause unknown problems on slurm clusters. For most people who do not use a cluster, it's okay to launch the training process on bare-metal machines, but you will have to open multiple terminals and set some environmental variables manually if you want to use multiple GPUs. We will explain it below.\n\n**Warning**: Since we have to extract all the frames from the video before sending them into the network, the program is hungry for memory. The actual memory usage is related to `batch_size` and `n_workers`. By default, each process with `n_workers=4` and `batch_size=6` consumes around 40 to 50 GB memory. You'd better open an `htop` to monitor the memory usage, before the machine hangs.\n\n```[bash]\n# Training on a slurm cluster\nexport DIST_PORT = 23333  # You may use whatever you want\nexport NUM_GPUS = 8\nPORT=$DIST_PORT$ srun -p $PARTITION$ --job-name=openpilot -n $NUM_GPUS$ --gres=gpu:$NUM_GPUS$ --ntasks-per-node=$NUM_GPUS$ python main.py\n```\n\n```[bash]\n# Training on a bare-metal machine with a single GPU\nPORT=23333 SLURM_PROCID=0 SLURM_NTASKS=1 python main.py\n```\n\n```[bash]\n# Training on a bare-metal machine with multiple GPUs\n# You need to open multiple terminals\n\n# Let's use 4 GPUs for example\n# Terminal 1\nPORT=23333 SLURM_PROCID=0 SLURM_NTASKS=4 python main.py\n# Terminal 2\nPORT=23333 SLURM_PROCID=1 SLURM_NTASKS=4 python main.py\n# Terminal 3\nPORT=23333 SLURM_PROCID=2 SLURM_NTASKS=4 python main.py\n# Terminal 4\nPORT=23333 SLURM_PROCID=3 SLURM_NTASKS=4 python main.py\n# Then, the training process will start after all 4 processes are launched.\n```\n\nBy default, the program will not output anything once the training process starts, for the widely-used `tqdm` might be buggy on slurm clusters. So, you may see some debugging info like the one below and the program seems to be stuck.\n\n```\n[1656218909.68] starting job... 0 of 1\n[1656218911.53] DDP Initialized at localhost:23333 0 of 1\nComma2k19SequenceDataset: DEMO mode is on.\nLoaded pretrained weights for efficientnet-b2\n```\n\nDon't worry, you can open a tensorboard to see the loss and validation curves.\n```\ntensorboard --logdir runs --bind_all\n```\n\nOtherwise, you may want to parse `--tqdm=True` to show the progress bar in `Terminal 1`.\n\nBy default, the test process will be executed once every epoch. So we did not implement the independent test script.\n\n## Demo\nSee more demo and test cases on our [webpage](https://sites.google.com/view/openpilot-deepdive/home)！\n\nYou can generate your own demo video using `demo.py`. It will generate some frames in the `./vis` folder. (You may have to create it first.) Then, you can generate a video using `ffmpeg`.\n\nhttps://user-images.githubusercontent.com/20351686/174319920-35b3ad34-a15e-43d7-be23-b135c24712e2.mp4\n\n\n***\n# Baselines\nHere we list several baselines to perform trajectory prediction task on different datasets. You are welcome to pull request and add your work here!\n\n**nuScenes**\n| Method                | AP@0.5(0-10)  | AP@1(10-20) | AP@1(20-30) | AP@1(30-50) |\n| :----:                |:----:         |:----:       |:----:       |:----:       |\n| Supercombo            | 0.237         | 0.064       | 0.038       | 0.053       |\n| Supercombo-finetuned  | 0.305         | 0.162       | 0.088       | 0.050       |\n| OP-Deepdive (Ours)    | 0.28          | 0.14        | 0.067       |  0.038      |\n\n**Comma2k19**\n| Method                | AP@0.5(0-10)  | AP@1(10-20) | AP@1(20-30) | AP@1(30-50) | AP@2(50+) | Average Jerk* |\n| :----:                |:----:         |:----:       |:----:       |:----:       |:----:     |:----:         |\n| Supercombo            | 0.7966        | 0.6170      | 0.2661      | 0.0889      |  0.0062   |  2.2243       |\n| OP-Deepdive (Ours)    | 0.909         | 0.808       |  0.651      | 0.465       |  0.239    |  4.7959       | \n\n\\*: The lower, the better. To comparison, the average jerk of human driver's trajectories is  0.3232 m/s^2.\n\n***\n# Citation\nPlease use the following citation when referencing our repo or [arXiv](https://arxiv.org/abs/2206.08176).\n```\n@article{chen2022op,\n   title={Level 2 Autonomous Driving on a Single Device: Diving into the Devils of Openpilot},\n   author={Li Chen and Tutian Tang and Zhitian Cai and Yang Li and Penghao Wu and Hongyang Li and Jianping Shi and Junchi Yan and Yu Qiao},\n   journal={arXiv preprint arXiv:2206.08176},\n   year={2022}\n}\n```\n***\n# License\nAll code within this repository is under [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0).\n***\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopendrivelab%2Fopenpilot-deepdive","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopendrivelab%2Fopenpilot-deepdive","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopendrivelab%2Fopenpilot-deepdive/lists"}