{"id":13438430,"url":"https://github.com/abhi1kumar/DEVIANT","last_synced_at":"2025-03-20T06:30:25.662Z","repository":{"id":49354436,"uuid":"510472617","full_name":"abhi1kumar/DEVIANT","owner":"abhi1kumar","description":"[ECCV 2022] Official PyTorch Code of DEVIANT: Depth Equivariant Network for Monocular 3D Object Detection","archived":false,"fork":false,"pushed_at":"2024-09-05T19:59:48.000Z","size":8557,"stargazers_count":203,"open_issues_count":0,"forks_count":29,"subscribers_count":6,"default_branch":"main","last_synced_at":"2024-10-28T00:22:59.905Z","etag":null,"topics":["3d-computer-vision","3d-object-detection","autonomous-driving","autonomous-vehicle","depth-equivariance","eccv","eccv-2022","eccv2022","equivariance","geometric-deep-learning","kitti-3d","monocular-3d-detection","monocular-3d-localization","nuscenes","object-detection","one-stage-detector","projective-geometry","projective-manifold","waymo"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2207.10758","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/abhi1kumar.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2022-07-04T18:57:39.000Z","updated_at":"2024-09-29T00:45:35.000Z","dependencies_parsed_at":"2023-12-06T07:55:27.267Z","dependency_job_id":"1a5bbb18-218e-4713-b40e-6b04648b77d2","html_url":"https://github.com/abhi1kumar/DEVIANT","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abhi1kumar%2FDEVIANT","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abhi1kumar%2FDEVIANT/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abhi1kumar%2FDEVIANT/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abhi1kumar%2FDEVIANT/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/abhi1kumar","download_url":"https://codeload.github.com/abhi1kumar/DEVIANT/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244564780,"owners_count":20473128,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-computer-vision","3d-object-detection","autonomous-driving","autonomous-vehicle","depth-equivariance","eccv","eccv-2022","eccv2022","equivariance","geometric-deep-learning","kitti-3d","monocular-3d-detection","monocular-3d-localization","nuscenes","object-detection","one-stage-detector","projective-geometry","projective-manifold","waymo"],"created_at":"2024-07-31T03:01:05.487Z","updated_at":"2025-03-20T06:30:20.628Z","avatar_url":"https://github.com/abhi1kumar.png","language":"C++","funding_links":[],"categories":["C++"],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n\u003cimg src=\"images/cvlab.png\" align=\"right\" width=8%\u003e\n\n# [DEVIANT: Depth EquiVarIAnt NeTwork for Monocular 3D Object Detection](https://arxiv.org/pdf/2207.10758.pdf)\n\n### [KITTI Demo](https://www.youtube.com/watch?v=2D73ZBrU-PA) | [KITTI Eqv Error Demo](https://www.youtube.com/watch?v=70DIjQkuZvw) | [Waymo Demo](https://www.youtube.com/watch?v=46S_OGxYFOM) | [Project](http://cvlab.cse.msu.edu/project-deviant.html) | [Talk](https://www.youtube.com/watch?v=yDrLYjmER4M) | [Slides](https://docs.google.com/presentation/d/1sSH1ArzqWvyswgmqEJCXlxAJBaLY1qxzZX0w4NojUAA/edit?usp=sharing) | [Poster](https://docs.google.com/presentation/d/1NB5YuPNDhctkA2EHYGJ1NZYe6ttPsumIll4ef9BDji4/edit?usp=sharing)\n\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/deviant-depth-equivariant-network-for/monocular-3d-object-detection-on-kitti-cars)](https://paperswithcode.com/sota/monocular-3d-object-detection-on-kitti-cars?p=deviant-depth-equivariant-network-for) \n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/deviant-depth-equivariant-network-for/3d-object-detection-from-monocular-images-on-6)](https://paperswithcode.com/sota/3d-object-detection-from-monocular-images-on-6?p=deviant-depth-equivariant-network-for)\n\n[![arXiv](http://img.shields.io/badge/arXiv-2207.10758-B31B1B.svg)](https://arxiv.org/abs/2207.10758)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n[![Visitors](https://api.visitorbadge.io/api/visitors?path=abhi1kumar%2FDEVIANT\u0026labelColor=%23FFFFFF\u0026countColor=%23721e82\u0026style=flat)](https://visitorbadge.io/status?path=abhi1kumar%2FDEVIANT)\n[![GitHub Stars](https://img.shields.io/github/stars/abhi1kumar/DEVIANT?style=social)](https://github.com/abhi1kumar/DEVIANT)\n\n\u003cp style=\"font-size:1.2em\"\u003e\n   \u003ca href=\"https://sites.google.com/view/abhinavkumar\"\u003e\u003cstrong\u003eAbhinav Kumar\u003c/strong\u003e\u003c/a\u003e\u003csup\u003e1\u003c/sup\u003e ·\n   \u003ca href=\"https://garrickbrazil.com\"\u003e\u003cstrong\u003eGarrick Brazil\u003c/strong\u003e\u003c/a\u003e\u003csup\u003e2\u003c/sup\u003e ·\n   \u003ca href=\"https://www.linkedin.com/in/enrique-corona-0752b84\"\u003e\u003cstrong\u003eEnrique Corona\u003c/strong\u003e\u003c/a\u003e\u003csup\u003e3\u003c/sup\u003e ·\n   \u003ca href=\"https://www.linkedin.com/in/parchami\"\u003e\u003cstrong\u003eArmin Parchami\u003c/strong\u003e\u003c/a\u003e\u003csup\u003e3\u003c/sup\u003e ·\n   \u003ca href=\"http://www.cse.msu.edu/~liuxm/index2.html\"\u003e\u003cstrong\u003eXiaoming Liu\u003c/strong\u003e\u003c/a\u003e\u003csup\u003e1\u003c/sup\u003e\u003cbr\u003e\n   \u003csup\u003e1\u003c/sup\u003eMichigan State University, \u003csup\u003e2\u003c/sup\u003eFAIR, Meta AI, \u003csup\u003e3\u003c/sup\u003eFord Motor Company\n\u003c/p\u003e\n\nin [ECCV 2022](https://eccv2022.ecva.net/)\n   \u003cp align=\"center\"\u003e\n      \u003cimg src=\"images/waymo_detection_demo.gif\" width=\"512\"\u003e\n   \u003c/p\u003e\n   \u003cp align=\"center\"\u003e\n      \u003cimg src=\"images/idea_overview.png\"\u003e\n   \u003c/p\u003e\n\u003c/div\u003e\n\n\u003e Modern neural networks use building blocks such as convolutions that are equivariant to arbitrary 2D translations $(t_u, t_v)$. However, these vanilla blocks are not equivariant to arbitrary 3D translations $(t_x, t_y, t_z)$ in the projective manifold. Even then, all monocular 3D detectors use vanilla blocks to obtain the 3D coordinates, a task for which the vanilla blocks are not designed for. This paper takes the first step towards convolutions equivariant to arbitrary 3D translations in the projective manifold. Since the depth is the hardest to estimate for monocular detection, this paper proposes Depth EquiVarIAnt NeTwork (DEVIANT) built with existing scale equivariant steerable blocks. As a result, DEVIANT is equivariant to the depth translations $(t_z)$ in the projective manifold whereas vanilla networks are not. The additional depth equivariance forces the DEVIANT to learn consistent depth estimates, and therefore, DEVIANT achieves state-of-the-art monocular 3D detection results on KITTI and Waymo datasets in the image-only category and performs competitively to methods using extra information. Moreover, DEVIANT works better than vanilla networks in cross-dataset evaluation.\n\n\nMuch of the codebase is based on [GUP Net](https://github.com/SuperMHP/GUPNet). Some implementations are from [GrooMeD-NMS](https://github.com/abhi1kumar/groomed_nms) and [PCT](https://github.com/amazon-research/progressive-coordinate-transforms). Scale Equivariant Steerable (SES) implementations are from [SiamSE](https://github.com/ISosnovik/SiamSE).\n\n## Citation\n\nIf you find our work useful in your research, please consider starring the repo and citing:\n\n```Bibtex\n@inproceedings{kumar2022deviant,\n   title={{DEVIANT: Depth EquiVarIAnt NeTwork for Monocular $3$D Object Detection}},\n   author={Kumar, Abhinav and Brazil, Garrick and Corona, Enrique and Parchami, Armin and Liu, Xiaoming},\n   booktitle={ECCV},\n   year={2022}\n}\n```\n\n## Setup\n\n- **Requirements**\n\n    1. Python 3.7\n    2. [PyTorch](http://pytorch.org) 1.10\n    3. Torchvision 0.11\n    4. Cuda 11.3\n    5. Ubuntu 18.04/Debian 8.9\n\nThis is tested with NVIDIA A100 GPU. Other platforms have not been tested. Clone the repo first. Unless otherwise stated, the below scripts and instructions assume the working directory is the directory `DEVIANT`:\n\n```bash\ngit clone https://github.com/abhi1kumar/DEVIANT.git\ncd DEVIANT\n```\n\n- **Cuda \u0026 Python**\n\nBuild the DEVIANT environment by installing the requirements:\n\n```bash\nconda create --name DEVIANT --file conda_GUP_environment_a100.txt\nconda activate DEVIANT\npip install opencv-python pandas\n```\n\n- **KITTI, nuScenes and Waymo Data**\n\nFollow instructions of [data_setup_README.md](data/data_setup_README.md) to setup KITTI, nuScenes and Waymo as follows:\n\n```bash\nDEVIANT\n├── data\n│      ├── KITTI\n│      │      ├── ImageSets\n│      │      ├── kitti_split1\n│      │      ├── training\n│      │      │     ├── calib\n│      │      │     ├── image_2\n│      │      │     └── label_2\n│      │      │\n│      │      └── testing\n│      │            ├── calib\n│      │            └── image_2\n│      │\n│      ├── nusc_kitti\n│      │      ├── ImageSets\n│      │      ├── training\n│      │      │     ├── calib\n│      │      │     ├── image\n│      │      │     └── label\n│      │      │\n│      │      └── validation\n│      │            ├── calib\n│      │            ├── image\n│      │            └── label\n│      │\n│      └── waymo\n│             ├── ImageSets\n│             ├── training\n│             │     ├── calib\n│             │     ├── image\n│             │     └── label\n│             │\n│             └── validation\n│                   ├── calib\n│                   ├── image\n│                   └── label\n│\n├── experiments\n├── images\n├── lib\n├── nuscenes-devkit        \n│ ...\n```\n\n\n- **AP Evaluation**\n\nRun the following to generate the KITTI binaries corresponding to `R40`:\n\n```bash\nsudo apt-get install libopenblas-dev libboost-dev libboost-all-dev gfortran\nsh data/KITTI/kitti_split1/devkit/cpp/build.sh\n```\n\nWe finally setup the Waymo evaluation. The Waymo evaluation is setup in a different environment `py36_waymo_tf` to avoid package conflicts with our DEVIANT environment:\n\n```bash\n# Set up environment\nconda create -n py36_waymo_tf python=3.7\nconda activate py36_waymo_tf\nconda install cudatoolkit=11.3 -c pytorch\n\n# Newer versions of tf are not in conda. tf\u003e=2.4.0 is compatible with conda.\npip install tensorflow-gpu==2.4\nconda install pandas\npip3 install waymo-open-dataset-tf-2-4-0 --user\n```\n\nTo verify that your Waymo evaluation is working correctly, pass the ground truth labels as predictions for a sanity check. Type the following:\n\n```bash\n/mnt/home/kumarab6/anaconda3/envs/py36_waymo_tf/bin/python -u data/waymo/waymo_eval.py --sanity\n```\n\nYou should see AP numbers as 100 in every entry after running this sanity check.\n\n\n## Training\n\nTrain the model:\n\n```bash\nchmod +x scripts_training.sh\n./scripts_training.sh\n```\n\nThe current Waymo config files use the full val set in training. For Waymo models, we had subsampled Waymo validation set by a factor of 10 (4k images) to save training time as in [DD3D](https://github.com/TRI-ML/dd3d#models). Change `val_split_name` from `'val'` to `'val_small'` in waymo configs to use subsampled Waymo val set.\n\n## Testing Pre-trained Models\n\n### Model Zoo\n\nWe provide logs/models/predictions for the main experiments on KITTI Val /KITTI Test/Waymo Val data splits available to download here.\n\n| Data_Splits | Method  | Config\u003cbr/\u003e(Run)                                          | Weight\u003cbr\u003e/Pred  | Metrics | All\u003cbr/\u003e(0.7) | Easy\u003cbr/\u003e(0.7) | Med\u003cbr/\u003e(0.7) | Hard\u003cbr/\u003e(0.7) |All\u003cbr/\u003e(0.5) | Easy\u003cbr/\u003e(0.5) | Med\u003cbr/\u003e(0.5) | Hard\u003cbr/\u003e(0.5) | \n|------------|---------|------------------------------------------------------------------|----------|--------|----------|-----------|----------|-----------|----------------|----|--------------|-----------|\n| KITTI Val  | GUP Net | [run_201](experiments/config_run_201_a100_v0_1.yaml) | [gdrive](https://drive.google.com/file/d/17qezmIjckRSAva1fNnYBmgR9LaY-dPnp/view?usp=sharing) | AP40  |   -   | 21.10 | 15.48 | 12.88 |   -   | 58.95 | 43.99 | 38.07 \n| KITTI Val  | DEVIANT | [run_221](experiments/run_221.yaml)                  | [gdrive](https://drive.google.com/file/d/1CBJf8keOutXVSAiu9Fj7XQPQftNYC1qv/view?usp=sharing)     | AP40  |   -   | 24.63 | 16.54 | 14.52 |   -   | 61.00 | 46.00 | 40.18\n| KITTI Test | DEVIANT | [run_250](experiments/run_250.yaml)                  | [gdrive](https://drive.google.com/file/d/1_79GfHcpAQR3wdvhj9GDHc7_c_ndf1Al/view?usp=sharing)     | AP40  |   -   | 21.88 | 14.46 | 11.89 |   -   |   -   |   -   |   -   \n| Waymo Val  | GUP Net | [run_1050](experiments/run_1050.yaml)                | [gdrive](https://drive.google.com/file/d/1wuTTuZrFVsEv4ttQ0r3X_s8D3OjYE84E/view?usp=sharing)     | APH-L1|  2.27 |  6.11 |  0.80 |  0.03 |  9.94 | 24.59 |  4.78 |  0.22\n| Waymo Val  | DEVIANT | [run_1051](experiments/run_1051.yaml)                | [gdrive](https://drive.google.com/file/d/1ixCVS85yVU9k6kuHrcYw_qJoy9Z4d0FD/view?usp=sharing)     | APH-L1|  2.67 |  6.90 |  0.98 |  0.02 | 10.89 | 26.64 |  5.08 |  0.18\n\n### Testing\n\nMake `output` folder in the `DEVIANT` directory:\n\n```bash\nmkdir output\n```\nPlace models in the `output` folder as follows:\n\n```bash\nDEVIANT\n├── output\n│      ├── config_run_201_a100_v0_1\n│      ├── run_221\n│      ├── run_250\n│      ├── run_1050\n│      └── run_1051\n│\n│ ...\n```\n\nThen, to test, run the file as:\n\n```bash\nchmod +x scripts_inference.sh\n./scripts_inference.sh\n```\n\n### Cross-Dataset Evaluation of KITTI on nuScenes Frontal Val\n\nSee [scripts_inference.sh](scripts_inference.sh)\n\n### Qualitative Plots/Visualization\n\nTo get qualitative plots and visualize the predicted+GT boxes, type the following:\n\n```bash\npython plot/plot_qualitative_output.py --dataset kitti --folder output/run_221/results_test/data\npython plot/plot_qualitative_output.py --dataset waymo --folder output/run_1051/results_test/data\n```\n\nType the following to reproduce our other plots:\n\n```bash\npython plot/plot_sesn_basis.py\npython plot/visualize_output_of_cnn_and_sesn.py\n```\n\n## FAQ\n\n- **Inference on older cuda version** For inference on older cuda version, type the following before running inference:\n\n```bash\nsource cuda_9.0_env\n```\n\n- **Correct Waymo version** You should see a 16th column in each ground truth file inside `data/waymo/validation/label/`. This corresponds to the `num_lidar_points_per_box`. If you do not see this column, run:\n\n```bash\ncd data/waymo\npython waymo_check.py \n```\n\nto see if `num_lidar_points_per_box` is printed. If nothing is printed, you are using the wrong Waymo dataset version and you should download the correct dataset version.\n\n- **Cannot convert a symbolic Tensor (strided_slice:0) to a numpy array**  This error indicates that you're trying to pass a Tensor to a NumPy call\". This means you have a wrong numpy version. Install the correct numpy as:\n\n```bash\npip install numpy==1.19.5\n```\n\n## Acknowledgements\nWe thank the authors of [GUP Net](https://github.com/SuperMHP/GUPNet), [GrooMeD-NMS](https://github.com/abhi1kumar/groomed_nms), [SiamSE](https://github.com/ISosnovik/SiamSE), [PCT](https://github.com/amazon-research/progressive-coordinate-transforms) and [patched nuscenes-devkit](https://github.com/abhi1kumar/nuscenes-devkit) for their awesome codebases. Please also consider citing them.\n\n## Contributions\nWe welcome contributions to the DEVIANT repo. Feel free to raise a pull request.\n\n### \u0026#8627; Stargazers\n[![Stargazers repo roster for @nastyox/Repo-Roster](https://reporoster.com/stars/abhi1kumar/DEVIANT)](https://github.com/abhi1kumar/DEVIANT/stargazers)\n\n### \u0026#8627; Forkers\n[![Forkers repo roster for @nastyox/Repo-Roster](https://reporoster.com/forks/abhi1kumar/DEVIANT)](https://github.com/abhi1kumar/DEVIANT/network/members)\n\n## Contact\nFor questions, feel free to post here or drop an email to this address- ```abhinav3663@gmail.com```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabhi1kumar%2FDEVIANT","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fabhi1kumar%2FDEVIANT","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabhi1kumar%2FDEVIANT/lists"}