{"id":13443531,"url":"https://github.com/fregu856/ebms_3dod","last_synced_at":"2025-03-20T16:32:07.622Z","repository":{"id":135362153,"uuid":"319594755","full_name":"fregu856/ebms_3dod","owner":"fregu856","description":"Official implementation of \"Accurate 3D Object Detection using Energy-Based Models\", CVPR Workshops 2021.","archived":false,"fork":false,"pushed_at":"2021-06-21T13:51:54.000Z","size":26694,"stargazers_count":59,"open_issues_count":2,"forks_count":10,"subscribers_count":7,"default_branch":"main","last_synced_at":"2024-08-01T03:43:54.322Z","etag":null,"topics":["3d-object-detection","computer-vision","deep-learning","energy-based-model","machine-learning","object-detection","pytorch"],"latest_commit_sha":null,"homepage":"http://www.fregu856.com","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fregu856.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2020-12-08T09:51:42.000Z","updated_at":"2024-01-04T16:52:56.000Z","dependencies_parsed_at":"2024-01-18T14:54:15.321Z","dependency_job_id":null,"html_url":"https://github.com/fregu856/ebms_3dod","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fregu856%2Febms_3dod","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fregu856%2Febms_3dod/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fregu856%2Febms_3dod/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fregu856%2Febms_3dod/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fregu856","download_url":"https://codeload.github.com/fregu856/ebms_3dod/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221780052,"owners_count":16879040,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-object-detection","computer-vision","deep-learning","energy-based-model","machine-learning","object-detection","pytorch"],"created_at":"2024-07-31T03:02:03.076Z","updated_at":"2025-03-20T16:32:07.610Z","avatar_url":"https://github.com/fregu856.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# ebms_3dod\n\n![overview image](ebms_3dod.jpg)\n\nOfficial implementation (PyTorch) of the paper: \\\n**Accurate 3D Object Detection using Energy-Based Models**, CVPR Workshops 2021 [[arXiv]](https://arxiv.org/abs/2012.04634) [[project]](http://www.fregu856.com/publication/ebms_3dod/). \\\n[Fredrik K. Gustafsson](http://www.fregu856.com/), [Martin Danelljan](https://martin-danelljan.github.io/), [Thomas B. Schön](http://user.it.uu.se/~thosc112/). \\\n_We apply energy-based models p(y|x; theta) to the task of 3D bounding box regression, extending the recent energy-based regression approach from 2D to 3D object detection. This is achieved by designing a differentiable pooling operator for 3D bounding boxes y, and adding an extra network branch to the state-of-the-art 3D object detector SA-SSD. We evaluate our proposed detector on the KITTI dataset and consistently outperform the SA-SSD baseline, demonstrating the potential of energy-based models for 3D object detection._\n\n[Youtube video](https://youtu.be/7JP6V818bh0) with qualitative results: \\\n[![demo video with qualitative results](https://img.youtube.com/vi/7JP6V818bh0/0.jpg)](https://youtu.be/7JP6V818bh0)\n\nIf you find this work useful, please consider citing:\n```\n@inproceedings{gustafsson2020accurate,\n  title={Accurate 3D Object Detection using Energy-Based Models},\n  author={Gustafsson, Fredrik K and Danelljan, Martin and Sch{\\\"o}n, Thomas B},\n  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},\n  year={2021}\n}\n```\n\n\n\n\n\n\n\n\n\n## Acknowledgements\n\n- The code is based on [SA-SSD](https://github.com/skyhehe123/SA-SSD) by [@skyhehe123](https://github.com/skyhehe123).\n\n\n\n\n\n\n\n\n\n## Index\n- [Usage](#usage)\n- [Documentation](#documentation)\n- [Pretrained model](#pretrained-model)\n***\n***\n***\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n***\n***\n***\n## Usage\n\nThe code has been tested on Ubuntu 16.04. Note that I had to use exactly the same versions for pytorch (1.1.0) and spconv (1.0) as in https://github.com/skyhehe123/SA-SSD for the code to work.\n\n- Installation:\n```\n$ pip install opencv-python\n$ pip install Shapely\n$ pip install mmcv==0.2.14 (NOTE! It did not work with the latest version)\n$ pip install terminaltables\n$ apt-get update\n$ apt-get install -y libsm6 libxext6 libxrender-dev\n$ pip install opencv-python\n$ pip install torch==1.1.0 torchvision==0.3.0 (NOTE! pytorch 1.1.0)\n$ pip install numba\n$ pip install Cython\n$ pip install pycocotools\n$ pip install scikit-image\n```\n- - Install spconv 1.0 (NOTE! spconv 1.0):\n```\n$ cd ebms_3dod/3dod\n$ git clone https://github.com/traveller59/spconv.git --recursive\n$ cd spconv\n$ git checkout 8da6f967fb9a054d8870c3515b1b44eca2103634 (this is the commit corresponding to spconv 1.0)\n$ apt-get update\n$ apt-get install libboost-all-dev\n$ python setup.py bdist_wheel\n$ cd dist\n$ pip install spconv-1.0-cp36-cp36m-linux_x86_64.whl (spconv-1.0-cp36-cp36m-linux_x86_64.whl was the name of the file at least for me)\n```\n```\n$ cd ebms_3dod/3dod\n$ pip install pybind11\n$ cd mmdet/ops/points_op\n$ python setup.py build_ext --inplace\n$ cd mmdet/ops/pointnet2\n$ python setup.py build_ext --inplace\n$ cd mmdet/ops/iou3d\n$ python setup.py build_ext --inplace\n```\n*\n- Download the pretrained SA-SSD model from https://drive.google.com/file/d/1WJnJDMOeNKszdZH3P077wKXcoty7XOUb/view, place the file epoch_50.pth in ebms_3dod/3dod.\n*\n- Create the folders ebms_3dod/3dod/data and ebms_3dod/3dod/data/KITTI.\n- Download the KITTI dataset, place the \"ImageSets\" and \"object\" folders in ebms_3dod/3dod/data/KITTI.\n*\n- Create cropped point clouds and sample for data augmentation:\n- - Create the folder ebms_3dod/3dod/data/KITTI/object/training/velodyne_reduced.\n- - Create the folder ebms_3dod/3dod/data/KITTI/object/testing/velodyne_reduced.\n- - $ cd ebms_3dod/3dod\n- - $ python create_data.py\n*\n*\n*\n*\n- Train model on KITTI train:\n- - $ cd ebms_3dod/3dod\n- - $ python train.py configs/car_cfg20.py\n*\n- Evaluate model on KITTI val:\n- - $ cd ebms_3dod/3dod\n- - $ python eval.py configs/car_cfg20_eval_ebm3.py saved_model_vehicle20/checkpoint_epoch_80.pth\n*\n- Run model on KITTI test:\n- - $ cd ebms_3dod/3dod\n- - $ python eval.py configs/car_cfg20_eval_ebm3_test.py saved_model_vehicle20/checkpoint_epoch_80.pth --out saved_model_vehicle20 _(this creates 000000.txt - 007517.txt in ebms_3dod/3dod/saved_model_vehicle20)_\n- - To evaluate on KITTI test:\n- - - Download all 7518 files, mark all files and compress to a zip file.\n- - - Upload the zip file to the KITTI evaluation server.\n***\n***\n***\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n***\n***\n***\n## Documentation\n\n- __3dod/mmdet/models/detectors/single_stage.py__: Code for defining the EBM network branch f_\\theta(x, y), the EBM loss and the gradient-based prediction procedure.\n\n- __3dod/viz_video.py__: Code for creating the [Youtube video](https://youtu.be/7JP6V818bh0) with qualitative results.\n\n- Also see [ebms_regression](https://github.com/fregu856/ebms_regression) for an illustrative 1D regression problem.\n\n\n***\n***\n***\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n***\n***\n***\n## Pretrained model\n\n- Model trained on KITTI train ($ python train.py configs/car_cfg20.py): https://drive.google.com/file/d/1hWKUZ4rx9h6Med3pI4A4wbHXEOey-8zI/view?usp=sharing\n*\n- Evaluate pretrained model on KITTI val:\n- - Download the file checkpoint_epoch_80.pth from above and place in ebms_3dod/3dod/pretrained.\n- - $ cd ebms_3dod/3dod\n- - $ python eval.py configs/car_cfg20_eval_ebm3.py pretrained/checkpoint_epoch_80.pth\n- - Expected output:\n```\nCar AP@0.90, 0.90, 0.90:\nbbox AP:39.30, 31.42, 29.55\nbev  AP:26.60, 22.03, 19.48\n3d   AP:3.45, 2.74, 2.26\naos  AP:39.30, 31.39, 29.51\nCar AP@0.85, 0.85, 0.85:\nbbox AP:82.14, 67.97, 64.99\nbev  AP:68.40, 58.62, 54.48\n3d   AP:31.02, 23.91, 21.95\naos  AP:82.08, 67.89, 64.87\nCar AP@0.80, 0.80, 0.80:\nbbox AP:95.75, 86.92, 82.20\nbev  AP:88.31, 80.06, 77.25\n3d   AP:66.70, 54.32, 51.36\naos  AP:95.69, 86.79, 81.99\nCar AP@0.75, 0.75, 0.75:\nbbox AP:99.05, 93.37, 90.79\nbev  AP:95.47, 87.54, 84.88\n3d   AP:87.85, 74.96, 71.95\naos  AP:98.99, 93.18, 90.45\nCar AP@0.70, 0.70, 0.70:\nbbox AP:99.38, 96.16, 93.69\nbev  AP:96.62, 92.93, 90.43\n3d   AP:95.50, 86.83, 82.23\naos  AP:99.32, 95.89, 93.25\nCar AP@0.50, 0.50, 0.50:\nbbox AP:99.38, 96.16, 93.69\nbev  AP:99.41, 96.35, 93.86\n3d   AP:99.39, 96.29, 93.81\naos  AP:99.32, 95.89, 93.25\n```\n*\n- Run pretrained model on KITTI test:\n- - Download the file checkpoint_epoch_80.pth from above and place in ebms_3dod/3dod/pretrained.\n- - $ cd ebms_3dod/3dod\n- - $ python eval.py configs/car_cfg20_eval_ebm3_test.py pretrained/checkpoint_epoch_80.pth --out pretrained _(this creates 000000.txt - 007517.txt in ebms_3dod/3dod/pretrained)_\n- - To evaluate on KITTI test:\n- - - Download all 7518 files, mark all files and compress to a zip file.\n- - - Upload the zip file to the KITTI evaluation server.\n- - - Expexted output:\n```\nBenchmark\t        Easy\tModerate\tHard\nCar (Detection)\t        96.81 %\t93.54 %\t88.33 %\nCar (Orientation)\t96.39 %\t92.88 %\t87.58 %\nCar (3D Detection)\t91.05 %\t80.12 %\t72.78 %\nCar (Bird's Eye View)\t95.64 %\t89.86 %\t84.56 %\n````\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffregu856%2Febms_3dod","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffregu856%2Febms_3dod","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffregu856%2Febms_3dod/lists"}