{"id":28573366,"url":"https://github.com/opendrivelab/detany3d","last_synced_at":"2025-10-09T01:40:47.012Z","repository":{"id":296670815,"uuid":"986856915","full_name":"OpenDriveLab/DetAny3D","owner":"OpenDriveLab","description":"[ICCV 2025] Detect Anything 3D in the Wild","archived":false,"fork":false,"pushed_at":"2025-07-08T19:06:22.000Z","size":603,"stargazers_count":203,"open_issues_count":4,"forks_count":11,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-10-03T08:47:45.186Z","etag":null,"topics":["object-detection"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2504.07958","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/OpenDriveLab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-05-20T08:14:58.000Z","updated_at":"2025-09-29T10:51:07.000Z","dependencies_parsed_at":"2025-06-01T16:40:15.069Z","dependency_job_id":"54aa98ee-8c14-478b-beed-05f372342c33","html_url":"https://github.com/OpenDriveLab/DetAny3D","commit_stats":null,"previous_names":["opendrivelab/detany3d"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/OpenDriveLab/DetAny3D","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenDriveLab%2FDetAny3D","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenDriveLab%2FDetAny3D/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenDriveLab%2FDetAny3D/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenDriveLab%2FDetAny3D/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/OpenDriveLab","download_url":"https://codeload.github.com/OpenDriveLab/DetAny3D/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenDriveLab%2FDetAny3D/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279000714,"owners_count":26082895,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-08T02:00:06.501Z","response_time":56,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["object-detection"],"created_at":"2025-06-10T21:17:32.675Z","updated_at":"2025-10-09T01:40:47.007Z","avatar_url":"https://github.com/OpenDriveLab.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003e [!IMPORTANT]\n\u003e 🌟 Stay up to date at [opendrivelab.com](https://opendrivelab.com/#news)!\n\n# DetAny3D\n\nThis is the official repository for the **[Detect Anything 3D in the Wild](https://arxiv.org/abs/2504.07958)**, a promptable 3D detection foundation model capable of detecting any novel object under arbitrary camera configurations using only monocular inputs\n\n\n\u003c!-- ## 🖼️ Demo Results\n\nBelow are example visualizations of DetAny3D predictions:\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/demo1.jpg\" alt=\"Demo 1\" width=\"400\"/\u003e\n  \u003cimg src=\"assets/demo2.jpg\" alt=\"Demo 2\" width=\"400\"/\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/demo3.jpg\" alt=\"Demo 3\" width=\"400\"/\u003e\n  \u003cimg src=\"assets/demo4.jpg\" alt=\"Demo 4\" width=\"400\"/\u003e\n\u003c/p\u003e --\u003e\n\n## 📖 Table of Contents\n\n- [📌 TODO](#-todo)\n- [🚀 Getting Started](#-getting-started)\n  - [Step 1: Create Environment](#step-1-create-environment)\n  - [Step 2: Install Dependencies](#step-2-install-dependencies)\n- [📦 Checkpoints](#-checkpoints)\n- [📁 Dataset Preparation](#-dataset-preparation)\n- [🏋️‍♂️ Training](#️-training)\n- [🔍 Inference](#-inference)\n- [🌐 Launch Online Demo](#-launch-online-demo)\n- [📚 Citation](#-citation)\n\n\n## 📌 TODO\n\n### ✅ Done\n- Release full code\n- Provide training and inference scripts\n- Release the model weights\n\n### 🛠️ In Progress\n- **TODO**: Provide full conversion scripts for constructing DA3D locally\n- **TODO**: Simplify the inference process\n- **TODO**: Provide a tutorial for creating customized datasets and finetuning\n\n\n## 🚀 Getting Started\n\n### Step 1: Create Environment\n\n```\nconda create -n detany3d python=3.8\nconda activate detany3d\n```\n\n---\n\n### Step 2: Install Dependencies\n\n#### ✅ (1) Install [Segment Anything (SAM)](https://github.com/facebookresearch/segment-anything)\n\nFollow the official instructions to install SAM and download its checkpoints.\n\n#### ✅ (2) Install [UniDepth](https://github.com/lpiccinelli-eth/UniDepth)\n\nFollow the UniDepth setup guide to compile and install all necessary packages.\n\n#### ✅ (3) Clone and configure [GroundingDINO](https://github.com/IDEA-Research/GroundingDINO)\n\n```\ngit clone https://github.com/IDEA-Research/GroundingDINO.git\ncd GroundingDINO\npip install -e .\n```\n\n\u003e 👉 The exact dependency versions are listed in our `requirements.txt`\n\n\n## 📦 Checkpoints\n\nPlease download third-party checkpoints from the following sources:\n\n- **SAM checkpoint**: Please download `sam_vit_h.pth` from the official [SAM GitHub Releases](https://github.com/facebookresearch/segment-anything)\n- **UniDepth / DINO checkpoints**: Available via [Google Drive](https://drive.google.com/drive/folders/17AOq5i1pCTxYzyqb1zbVevPy5jAXdNho?usp=drive_link)\n\n```\ndetany3d_private/\n├── checkpoints/\n│   ├── sam_ckpts/\n│   │   └── sam_vit_h.pth\n│   ├── unidepth_ckpts/\n│   │   └── unidepth.pth\n│   ├── dino_ckpts/\n│   │   └── dino_swin_large.pth\n│   └── detany3d_ckpts/\n│       └── detany3d.pth\n```\n\n\u003e GroundingDINO's checkpoint should be downloaded from its [official repo](https://github.com/IDEA-Research/GroundingDINO) and placed as instructed in their documentation.\n\n\n## 📁 Dataset Preparation\n\nThe `data/` directory should follow the structure below:\n\n```\ndata/\n├── DA3D_pkls/                             # DA3D processed pickle files \n├── kitti/\n│   ├── test_depth_front/\n│   ├── ImageSets/\n│   ├── training/\n│   └── testing/\n├── nuscenes/\n|   ├── nuscenes_depth/\n│   └── samples/\n├── 3RScan/\n│   └── \u003ctoken folders\u003e/             # e.g., 10b17940-3938-...\n├── hypersim/\n|   ├── depth_in_meter/\n│   └── ai_XXX_YYY/                  # e.g., ai_055_009\n├── waymo/\n│   └── kitti_format/                # KITTI-format data for Waymo\n│       ├── validation_depth_front/\n│       ├── ImageSets/\n│       ├── training/\n│       └── testing/\n├── objectron/\n│   ├── train/\n│   └── test/\n├── ARKitScenes/\n│   ├── Training/\n│   └── Validation/\n├── cityscapes3d/\n│   ├── depth/\n│   └── leftImg8bit/\n├── SUNRGBD/\n│   ├── realsense/\n│   ├── xtion/\n|   ├── kv1/\n│   └── kv2/\n```\n\n\u003e The download for `kitti`, `nuscenes`, `hypersim`, `objectron`, `arkitscenes`, and `sunrgbd` follow the [Omni3D](https://github.com/facebookresearch/omni3d) convention. Please refer to the Omni3D repository for details on how to organize and preprocess these datasets.\n\n\u003e 🗂️ The `DA3D_pkls` (minimal metadata for inference) can be downloaded from [Google Drive](https://drive.google.com/drive/folders/17AOq5i1pCTxYzyqb1zbVevPy5jAXdNho?usp=drive_link).  \n\u003e 🧩 **Note**: This release currently supports a minimal inference-only version. The conversion scripts of full dataset + all depth-related files will be provided later.\n\n\u003e ⚠️ Depth files are not required for inference. You can safely set `depth_path = None` in [detany3d_dataset.py](./detect_anything/datasets/detany3d_dataset.py) to bypass depth loading.  \n\n\n\n## 🏋️‍♂️ Training\n\n```\ntorchrun \\\n    --nproc_per_node=8 \\\n    --master_addr=${MASTER_ADDR} \\\n    --master_port=${MASTER_PORT} \\\n    --nnodes=8 \\\n    --node_rank=${RANK} \\\n    ./train.py \\\n    --config_path \\\n    ./detect_anything/configs/train.yaml\n```\n\n\n## 🔍 Inference\n\n```\ntorchrun \\\n    --nproc_per_node=8 \\\n    --master_addr=${MASTER_ADDR} \\\n    --master_port=${MASTER_PORT} \\\n    --nnodes=1 \\\n    --node_rank=${RANK} \\\n    ./train.py \\\n    --config_path \\\n    ./detect_anything/configs/inference_indomain_gt_prompt.yaml\n```\n\n\nAfter inference, a file named `{dataset}_output_results.json` will be generated in the `exps/\u003cyour_exp_dir\u003e/` directory.\n\n\u003e ⚠️ Due to compatibility issues between `pytorch3d` and the current environment, we recommend copying the output JSON file into the evaluation script of repositories like [Omni3D](https://github.com/facebookresearch/omni3d) or [OVMono3D](https://github.com/UVA-Computer-Vision-Lab/ovmono3d) for standardized metric evaluation.\n\n\u003e **TODO**: Evaluation for zero-shot datasets currently requires manual modification of the Omni3D or OVMono3D repositories and is not yet fully supported here.   \nWe plan to release a merged evaluation script in this repository to make direct evaluation more convenient in the future.\n\n\n\n## 🌐 Launch Online Demo\n\n```\npython ./deploy.py\n```\n\n\n## 📚 Citation\n\nIf you find this repository useful, please consider citing:\n\n```\n@article{zhang2025detect,\n  title={Detect Anything 3D in the Wild},\n  author={Zhang, Hanxue and Jiang, Haoran and Yao, Qingsong and Sun, Yanan and Zhang, Renrui and Zhao, Hao and Li, Hongyang and Zhu, Hongzi and Yang, Zetong},\n  journal={arXiv preprint arXiv:2504.07958},\n  year={2025}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopendrivelab%2Fdetany3d","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopendrivelab%2Fdetany3d","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopendrivelab%2Fdetany3d/lists"}