{"id":13443978,"url":"https://github.com/JonasSchult/dcm-net","last_synced_at":"2025-03-20T17:32:36.045Z","repository":{"id":44247282,"uuid":"239786343","full_name":"JonasSchult/dcm-net","owner":"JonasSchult","description":"This work is based on our paper \"DualConvMesh-Net: Joint Geodesic and Euclidean Convolutions on 3D Meshes\", which appeared at the IEEE Conference On Computer Vision And Pattern Recognition (CVPR) 2020.","archived":false,"fork":false,"pushed_at":"2022-11-27T16:30:01.000Z","size":22305,"stargazers_count":119,"open_issues_count":6,"forks_count":16,"subscribers_count":11,"default_branch":"master","last_synced_at":"2024-10-28T07:42:07.937Z","etag":null,"topics":["3d-deep-learning","3d-segmentation","cvpr","cvpr2020","dcm-net","deep-learning","scannet","semantic-segmentation"],"latest_commit_sha":null,"homepage":"https://visualcomputinginstitute.github.io/dcm-net/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/JonasSchult.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-02-11T14:47:17.000Z","updated_at":"2024-10-18T07:49:24.000Z","dependencies_parsed_at":"2023-01-22T04:03:54.744Z","dependency_job_id":null,"html_url":"https://github.com/JonasSchult/dcm-net","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JonasSchult%2Fdcm-net","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JonasSchult%2Fdcm-net/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JonasSchult%2Fdcm-net/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JonasSchult%2Fdcm-net/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/JonasSchult","download_url":"https://codeload.github.com/JonasSchult/dcm-net/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244660746,"owners_count":20489388,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-deep-learning","3d-segmentation","cvpr","cvpr2020","dcm-net","deep-learning","scannet","semantic-segmentation"],"created_at":"2024-07-31T03:02:15.620Z","updated_at":"2025-03-20T17:32:31.029Z","avatar_url":"https://github.com/JonasSchult.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# DualConvMesh-Net: Joint Geodesic and Euclidean Convolutions on 3D Meshes\nCreated by [Jonas Schult*](https://www.vision.rwth-aachen.de/person/schult), [Francis Engelmann*](https://www.vision.rwth-aachen.de/person/14/), [Theodora Kontogianni](https://www.vision.rwth-aachen.de/person/15/) and [Bastian Leibe](https://www.vision.rwth-aachen.de/person/1/) from RWTH Aachen University.\n\n![prediction example](doc/teaser.png)\n\n## Introduction\nThis work is based on our paper \n[DualConvMesh-Net: Joint Geodesic and Euclidean Convolutions on 3D Meshes](https://arxiv.org/abs/2004.01002),\nwhich appeared at the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2020. \n\nYou can also check our [project page](https://visualcomputinginstitute.github.io/dcm-net/) for further details.\n\nWe propose DualConvMesh-Nets (DCM-Net) a family of deep hierarchical convolutional networks over 3D geometric data that combines two types of convolutions. The first type, *geodesic convolutions*, defines the kernel weights over mesh surfaces or graphs. That is, the convolutional kernel weights are mapped to the local surface of a given mesh. The second type, *Euclidean convolutions*, is independent of any underlying mesh structure. The convolutional kernel is applied on a neighborhood obtained from a local affinity representation based on the Euclidean distance between 3D points. Intuitively, geodesic convolutions can easily separate objects that are spatially close but have disconnected surfaces, while Euclidean convolutions can represent interactions between nearby objects better, as they are oblivious to object surfaces. To realize a multi-resolution architecture, we borrow well-established mesh simplification methods from the geometry processing domain and adapt them to define mesh-preserving pooling and unpooling operations. We experimentally show that combining both types of convolutions in our architecture leads to significant performance gains for 3D semantic segmentation, and we report competitive results on three scene segmentation benchmarks.\n\n*In this repository, we release code for training and testing DualConvMesh-Nets on arbitrary datasets.*\n\n## Citation\nIf you find our work useful in your research, please consider citing us:\n\n    @inproceedings{Schult20CVPR,\n      author    = {Jonas Schult* and\n                   Francis Engelmann* and\n                   Theodora Kontogianni and\n                   Bastian Leibe},\n      title     = {{DualConvMesh-Net: Joint Geodesic and Euclidean Convolutions on 3D Meshes}},\n      booktitle = {{IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}},\n      year      = {2020}\n    }\n\n\n## Installation\nOur code requires **CUDA 10.0** for running correctly. Please make sure that your `$PATH`, `$CPATH` and `$LD_LIBRARBY_PATH` environment variables point to the right CUDA version.\n\n    conda deactivate\n    conda create -y -n dualmesh python=3.7\n    conda activate dualmesh\n\n    conda install -y -c open3d-admin open3d=0.6.0.0\n    conda install -y pytorch==1.1.0 torchvision==0.3.0 cudatoolkit=10.0 -c pytorch\n    conda install -y -c conda-forge tensorboardx=1.7\n    conda install -y -c conda-forge tqdm=4.31.1\n    conda install -y -c omnia termcolor=1.1.0\n\n    # Execute pip installs one after each other\n    pip install --no-cache-dir torch-scatter==1.3.1\n    pip install --no-cache-dir torch_cluster==1.4.3\n    pip install --no-cache-dir torch_sparse==0.4.0\n    pip install \"pillow\u003c7\" # torchvision bug\n\n### Batching of hierarchical meshes (PyTorch Geometric Fork)\nWe created a fork of [PyTorch Geometric](https://github.com/JonasSchult/pytorch_geometric_fork) in order to support hierarchical mesh structures interlinked with pooling trace maps.\n\n    git clone https://github.com/JonasSchult/pytorch_geometric_fork.git\n    cd pytorch-geometric-fork\n    pip install --no-cache-dir . \n\n### Mesh Simplification Preprocessing (VCGlib)\nWe adapted [VCGlib](https://github.com/JonasSchult/vcglib) to generate pooling trace maps for vertex clustering and quadric error metrics.\n\n    git clone https://github.com/JonasSchult/vcglib.git\n\n    # QUADRIC ERROR METRICS\n    cd vcglib/apps/tridecimator/\n    qmake\n    make\n\n    # VERTEX CLUSTERING\n    cd ../sample/trimesh_clustering\n    qmake\n    make\n\nAdd `vcglib/apps/tridecimator` and `vcglib/apps/sample/trimesh_clustering` to your environment path variable!\n\n## Preparation\n\n### Prepare the dataset\nPlease refer to https://github.com/ScanNet/ScanNet and https://github.com/niessner/Matterport to get access to the ScanNet and Matterport dataset. Our method relies on the .ply as well as the .labels.ply files.\nWe train on crops and we evaluate on full rooms.\nAfter inserting the paths to the dataset and deciding on the parameters, execute the scripts in `utils/preprocess/scripts/{scannet, matterport}/rooms` and *subsequently* in `utils/preprocess/scripts/scannet, matterport}/crops` to generate mesh hierarchies on rooms and crop areas for training.\nPlease note that the scripts are developed for a SLURM batch system. If your lab does not use SLURM, please consider adapting the scripts for your purposes.\nMore information about the parameters are provided in the corresponding scripts in `utils/preprocess`.\n\n### Symbolic Links pointing to the dataset\nCreate symlinks to the dataset such that our framework can find it.\nFor example:\n\n    ln -s /path/to/scannet/rooms/ data/scannet/scannet_qem_rooms\n\nAlternatively, you can also directly set the paths in the corresponding experiment files.\n\n### Model Checkpoints\nWe provide the [model checkpoints](https://omnomnom.vision.rwth-aachen.de/data/dcm_net_checkpoints/dcm_net_checkpoints.zip) on our server.\n\n## Training\nAn example training script is given in `example_scripts/train_scannet.sh` \n\n## Inference\nAn example inference script is given in `example_scripts/inference_scannet.sh` \n\n## Visualization\nAn example visualization script is given in `example_scripts/visualize_scannet.sh`.\nWe show qualitative results on the ScanNet validation set.\nPlease note that a symlink to the ScanNet mesh folder has to be in placed in `data/scannet/scans`.\nThe visualization tool is based on [open3D](http://www.open3d.org/) and handles the following key events:\n* h = RGB\n* j = prediction\n* k = ground truth\n* f = color-coded positive/negative predictions\n* l = local lighting on/off\n* s = smoothing mesh on/off\n* b = back-face culling on/off\n* d = save current meshes as .ply in `visualizations/` folder (useful, if you plan to make some decent rendering with Blender, later on :) )\n* q = quit and show next room\n\nUse your mouse to navigate in the mesh.\n\n## ToDo's\n- Preprocessing code for S3DIS data set\n\n## Acknowledgements\nThis project is based on the [PyTorch-Template](https://github.com/victoresque/pytorch-template) by [@victoresque](https://github.com/victoresque).","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FJonasSchult%2Fdcm-net","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FJonasSchult%2Fdcm-net","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FJonasSchult%2Fdcm-net/lists"}