{"id":13442919,"url":"https://github.com/cgtuebingen/SpatialDETR","last_synced_at":"2025-03-20T15:31:39.765Z","repository":{"id":47098928,"uuid":"513437461","full_name":"cgtuebingen/SpatialDETR","owner":"cgtuebingen","description":"Official implementation of SpatialDETR. The paper will be presented at ECCV 2022","archived":false,"fork":false,"pushed_at":"2023-04-03T21:17:23.000Z","size":4783,"stargazers_count":54,"open_issues_count":2,"forks_count":5,"subscribers_count":8,"default_branch":"main","last_synced_at":"2024-08-01T03:42:21.322Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cgtuebingen.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-07-13T08:19:06.000Z","updated_at":"2024-05-25T20:14:11.000Z","dependencies_parsed_at":"2022-09-23T11:11:25.790Z","dependency_job_id":null,"html_url":"https://github.com/cgtuebingen/SpatialDETR","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cgtuebingen%2FSpatialDETR","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cgtuebingen%2FSpatialDETR/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cgtuebingen%2FSpatialDETR/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cgtuebingen%2FSpatialDETR/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cgtuebingen","download_url":"https://codeload.github.com/cgtuebingen/SpatialDETR/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221772611,"owners_count":16878138,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T03:01:53.301Z","updated_at":"2024-10-28T03:31:20.372Z","avatar_url":"https://github.com/cgtuebingen.png","language":"Python","funding_links":[],"categories":["Python","3. Perception"],"sub_categories":["3.1.2 Multi Sensor Fusion"],"readme":"# SpatialDETR: Robust Scalable Transformer-Based 3D Object Detection from Multi-View Camera Images with Global Cross-Sensor Attention (ECCV 2022)\n\n![](img/overview.png \"Overview\")  \n\nThis is the official repository for [SpatialDETR](https://markus-enzweiler.de/downloads/publications/ECCV2022-spatial_detr.pdf) which will be published at ECCV 2022.   \n\n\nhttps://user-images.githubusercontent.com/46648831/190854092-9f9c7fc5-f890-4ec1-809a-9a3807b5993a.mp4  \n\n**Authors**: Simon Doll, Richard Schulz, Lukas Schneider, Viviane Benzin, Markus Enzweiler, Hendrik P.A. Lensch\n## Abstract\nBased on the key idea of DETR this paper introduces an object-centric 3D object detection framework that operates on a limited number of 3D object queries instead of dense bounding box proposals followed by non-maximum suppression. After image feature extraction a decoder-only transformer architecture is trained on a set-based loss. SpatialDETR infers the classification and bounding box estimates based on attention both spatially within each image and across the different views. To fuse the multi-view information in the attention block we introduce a novel geometric positional encoding that incorporates the view ray geometry to explicitly consider the extrinsic and intrinsic camera setup. This way, the  spatially-aware cross-view attention exploits arbitrary receptive fields to integrate cross-sensor data and therefore global context. Extensive experiments on the nuScenes benchmark demonstrate the potential of global attention and result in state-of-the-art performance.\n\nIf you find this repository useful, please cite\n```bibtex\n@inproceedings{Doll2022ECCV,\n  author = {Doll, Simon and Schulz, Richard and Schneider, Lukas and Benzin, Viviane and Enzweiler Markus and Lensch, Hendrik P.A.},\n  title = {SpatialDETR: Robust Scalable Transformer-Based 3D Object Detection from Multi-View Camera Images with Global Cross-Sensor Attention},\n  booktitle = {European Conference on Computer Vision(ECCV)},\n  year = {2022}\n}\n```\n\n[You can find the Paper here](https://markus-enzweiler.de/downloads/publications/ECCV2022-spatial_detr.pdf)\n## Setup\nTo setup the repository and run trainings we refer to [getting_started.md](getting_started.md)\n\n## Changelog\n### 06/22\n- We moved the codebase to the new coordinate conventions of `mmdetection3d rc1.0ff`\n- The performancy might slightly vary compared to the original runs on `mmdetection3d 0.17` reported in the paper\n\n\n## Experimental results\nThe baseline models have been trained on `4xV100 GPUs`, the submission models on `8xA100 GPUs`. For more details we refer to the corresponding configuration / log files. Keep in mind that the performance can vary between runs and that the current codebase uses `mmdetection3d@rc1.0`\n| Config  | Logfile | Set | #GPUs| mmdet3d | mAP | ATE | ASE | AOE | AVE | AAE | NDS|\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---| --- |\n| [query_proj_value_proj.py (baseline)](configs/submission/frozen_4/query_proj_value_proj.py)  | [log](training_logs/frozen_4/query_proj_value_proj.log) / [model](https://drive.google.com/file/d/1Tm6M0e-8QYBUeqwBwYpJk1EwxQbyZl6M/view?usp=sharing) | val | 4 | rc1.0 | 0.315 | 0.843 | 0.279 | 0.497 | 0.787 | 0.208 | 0.396 |\n| [query_proj_value_proj.py](configs/submission/frozen_4/query_proj_value_proj.py)  | [log](training_logs/frozen_4/query_proj_value_proj_0_17.log) | val | 4 | 0.17 | 0.313 | 0.850 | 0.274 | 0.494 | 0.814 | 0.213 | 0.392 |\n| [query_center_proj_no_value_proj_shared.py](configs/submission/frozen_1/query_center_proj_no_value_proj_shared.py)  | [log](training_logs/frozen_1/query_center_proj_no_value_proj_shared.log) | val | 8 | 0.17 | 0.351 | 0.772 | 0.274 | 0.395 | 0.847 | 0.217 | 0.425\n| [query_center_proj_no_value_proj_shared_cbgs_vovnet_trainval.py](configs/submission/frozen_1/query_center_proj_no_value_proj_shared_cbgs_vovnet_trainval.py)  |[log](training_logs/frozen_1/query_center_proj_no_value_proj_shared_cbgs_vovnet_trainval.log)| test | 8 | 0.17| 0.425 | 0.614 | 0.253 | 0.402 | 0.857 | 0.131 | 0.487\n\n## Qualitative results\n![](img/qualitative_results.png \"Visualization of predictions\")\n![](img/attention.png \"Attention visualization for multiple queries\")\n\n## License\nSee [license_infos.md](license_infos.md) for details.\n\n## Acknowledgement\nThis repo contains the implementations of SpatialDETR. Our implementation is a plugin to [MMDetection3D](https://github.com/open-mmlab/mmdetection3d) and also uses a [fork](https://github.com/SimonDoll/fusion_detr3d) of [DETR3D](https://github.com/WangYueFt/detr3d). Full credits belong to the contributors of those frameworks and we truly thank them for enabling our research!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcgtuebingen%2FSpatialDETR","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcgtuebingen%2FSpatialDETR","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcgtuebingen%2FSpatialDETR/lists"}