{"id":13442313,"url":"https://github.com/Pointcept/Pointcept","last_synced_at":"2025-03-20T13:33:23.292Z","repository":{"id":144606774,"uuid":"617093311","full_name":"Pointcept/Pointcept","owner":"Pointcept","description":"Pointcept: a codebase for point cloud perception research. Latest works: PTv3 (CVPR'24 Oral), PPT (CVPR'24), OA-CNNs (CVPR'24), MSC (CVPR'23)","archived":false,"fork":false,"pushed_at":"2025-01-16T09:12:41.000Z","size":830,"stargazers_count":1897,"open_issues_count":251,"forks_count":213,"subscribers_count":19,"default_branch":"main","last_synced_at":"2025-03-13T20:04:33.533Z","etag":null,"topics":["3d-vision","point-cloud","pytorch"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Pointcept.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-03-21T17:28:09.000Z","updated_at":"2025-03-13T10:32:22.000Z","dependencies_parsed_at":"2024-12-12T07:02:28.580Z","dependency_job_id":"2d27c26a-2a59-4c54-913e-abc6023fab0f","html_url":"https://github.com/Pointcept/Pointcept","commit_stats":{"total_commits":57,"total_committers":9,"mean_commits":6.333333333333333,"dds":0.1578947368421053,"last_synced_commit":"2b247ed555e96eae11710dfde79a3fc16c8c74b3"},"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Pointcept%2FPointcept","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Pointcept%2FPointcept/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Pointcept%2FPointcept/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Pointcept%2FPointcept/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Pointcept","download_url":"https://codeload.github.com/Pointcept/Pointcept/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244619276,"owners_count":20482390,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-vision","point-cloud","pytorch"],"created_at":"2024-07-31T03:01:44.231Z","updated_at":"2025-03-20T13:33:23.285Z","avatar_url":"https://github.com/Pointcept.png","language":"Python","funding_links":[],"categories":["Uncategorized","Python","MIM for CV Downstream Tasks"],"sub_categories":["Uncategorized","3D Representation Learning"],"readme":"\u003cp align=\"center\"\u003e\n    \u003c!-- pypi-strip --\u003e\n    \u003cpicture\u003e\n    \u003csource media=\"(prefers-color-scheme: dark)\" srcset=\"https://raw.githubusercontent.com/Pointcept/Pointcept/main/docs/logo_dark.png\"\u003e\n    \u003csource media=\"(prefers-color-scheme: light)\" srcset=\"https://raw.githubusercontent.com/Pointcept/Pointcept/main/docs/logo.png\"\u003e\n    \u003c!-- /pypi-strip --\u003e\n    \u003cimg alt=\"pointcept\" src=\"https://raw.githubusercontent.com/Pointcept/Pointcept/main/docs/logo.png\" width=\"400\"\u003e\n    \u003c!-- pypi-strip --\u003e\n    \u003c/picture\u003e\u003cbr\u003e\n    \u003c!-- /pypi-strip --\u003e\n\u003c/p\u003e\n\n[![Formatter](https://github.com/pointcept/pointcept/actions/workflows/formatter.yml/badge.svg)](https://github.com/pointcept/pointcept/actions/workflows/formatter.yml)\n\n**Pointcept** is a powerful and flexible codebase for point cloud perception research. It is also an official implementation of the following paper:\n- **Point Transformer V3: Simpler, Faster, Stronger**  \n*Xiaoyang Wu, Li Jiang, Peng-Shuai Wang, Zhijian Liu, Xihui Liu, Yu Qiao, Wanli Ouyang, Tong He, Hengshuang Zhao*  \nIEEE Conference on Computer Vision and Pattern Recognition (**CVPR**) 2024 - Oral  \n[ Backbone ] [PTv3] - [ [arXiv](https://arxiv.org/abs/2312.10035) ] [ [Bib](https://xywu.me/research/ptv3/bib.txt) ] [ [Project](https://github.com/Pointcept/PointTransformerV3) ] \u0026rarr; [here](https://github.com/Pointcept/PointTransformerV3)\n\n- **OA-CNNs: Omni-Adaptive Sparse CNNs for 3D Semantic Segmentation**  \n*Bohao Peng, Xiaoyang Wu, Li Jiang, Yukang Chen, Hengshuang Zhao, Zhuotao Tian, Jiaya Jia*  \nIEEE Conference on Computer Vision and Pattern Recognition (**CVPR**) 2024  \n[ Backbone ] [ OA-CNNs ] - [ [arXiv](https://arxiv.org/abs/2403.14418) ] [ [Bib](https://xywu.me/research/oacnns/bib.txt) ] \u0026rarr; [here](#oa-cnns)\n\n- **Towards Large-scale 3D Representation Learning with Multi-dataset Point Prompt Training**  \n*Xiaoyang Wu, Zhuotao Tian, Xin Wen, Bohao Peng, Xihui Liu, Kaicheng Yu, Hengshuang Zhao*  \nIEEE Conference on Computer Vision and Pattern Recognition (**CVPR**) 2024  \n[ Pretrain ] [PPT] - [ [arXiv](https://arxiv.org/abs/2308.09718) ] [ [Bib](https://xywu.me/research/ppt/bib.txt) ] \u0026rarr; [here](#point-prompt-training-ppt)\n\n- **Masked Scene Contrast: A Scalable Framework for Unsupervised 3D Representation Learning**  \n*Xiaoyang Wu, Xin Wen, Xihui Liu, Hengshuang Zhao*  \nIEEE Conference on Computer Vision and Pattern Recognition (**CVPR**) 2023  \n[ Pretrain ] [ MSC ] - [ [arXiv](https://arxiv.org/abs/2303.14191) ] [ [Bib](https://xywu.me/research/msc/bib.txt) ] \u0026rarr; [here](#masked-scene-contrast-msc)\n\n\n- **Learning Context-aware Classifier for Semantic Segmentation** (3D Part)  \n*Zhuotao Tian, Jiequan Cui, Li Jiang, Xiaojuan Qi, Xin Lai, Yixin Chen, Shu Liu, Jiaya Jia*  \nAAAI Conference on Artificial Intelligence (**AAAI**) 2023 - Oral  \n[ SemSeg ] [ CAC ] - [ [arXiv](https://arxiv.org/abs/2303.11633) ] [ [Bib](https://xywu.me/research/cac/bib.txt) ] [ [2D Part](https://github.com/tianzhuotao/CAC) ] \u0026rarr; [here](#context-aware-classifier)\n\n\n- **Point Transformer V2: Grouped Vector Attention and Partition-based Pooling**   \n*Xiaoyang Wu, Yixing Lao, Li Jiang, Xihui Liu, Hengshuang Zhao*  \nConference on Neural Information Processing Systems (**NeurIPS**) 2022  \n[ Backbone ] [ PTv2 ] - [ [arXiv](https://arxiv.org/abs/2210.05666) ] [ [Bib](https://xywu.me/research/ptv2/bib.txt) ] \u0026rarr; [here](#point-transformers)\n\n\n- **Point Transformer**   \n*Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip Torr, Vladlen Koltun*  \nIEEE International Conference on Computer Vision (**ICCV**) 2021 - Oral  \n[ Backbone ] [ PTv1 ] - [ [arXiv](https://arxiv.org/abs/2012.09164) ] [ [Bib](https://hszhao.github.io/papers/iccv21_pointtransformer_bib.txt) ] \u0026rarr; [here](#point-transformers)\n\nAdditionally, **Pointcept** integrates the following excellent work (contain above):  \nBackbone: \n[MinkUNet](https://github.com/NVIDIA/MinkowskiEngine) ([here](#sparseunet)),\n[SpUNet](https://github.com/traveller59/spconv) ([here](#sparseunet)),\n[SPVCNN](https://github.com/mit-han-lab/spvnas) ([here](#spvcnn)),\n[OACNNs](https://arxiv.org/abs/2403.14418) ([here](#oa-cnns)),\n[PTv1](https://arxiv.org/abs/2012.09164) ([here](#point-transformers)),\n[PTv2](https://arxiv.org/abs/2210.05666) ([here](#point-transformers)),\n[PTv3](https://arxiv.org/abs/2312.10035) ([here](#point-transformers)),\n[StratifiedFormer](https://github.com/dvlab-research/Stratified-Transformer) ([here](#stratified-transformer)),\n[OctFormer](https://github.com/octree-nn/octformer) ([here](#octformer)),\n[Swin3D](https://github.com/microsoft/Swin3D) ([here](#swin3d));   \nSemantic Segmentation:\n[Mix3d](https://github.com/kumuji/mix3d) ([here](https://github.com/Pointcept/Pointcept/blob/main/configs/scannet/semseg-spunet-v1m1-0-base.py#L5)),\n[CAC](https://arxiv.org/abs/2303.11633) ([here](#context-aware-classifier));  \nInstance Segmentation: \n[PointGroup](https://github.com/dvlab-research/PointGroup) ([here](#pointgroup));  \nPre-training: \n[PointContrast](https://github.com/facebookresearch/PointContrast) ([here](#pointcontrast)), \n[Contrastive Scene Contexts](https://github.com/facebookresearch/ContrastiveSceneContexts) ([here](#contrastive-scene-contexts)),\n[Masked Scene Contrast](https://arxiv.org/abs/2303.14191) ([here](#masked-scene-contrast-msc)),\n[Point Prompt Training](https://arxiv.org/abs/2308.09718) ([here](#point-prompt-training-ppt));  \nDatasets:\n[ScanNet](http://www.scan-net.org/) ([here](#scannet-v2)), \n[ScanNet200](http://www.scan-net.org/) ([here](#scannet-v2)),\n[ScanNet++](https://kaldir.vc.in.tum.de/scannetpp/) ([here](#scannet)),\n[S3DIS](https://docs.google.com/forms/d/e/1FAIpQLScDimvNMCGhy_rmBA2gHfDu3naktRm6A8BPwAWWDv-Uhm6Shw/viewform?c=0\u0026w=1) ([here](#s3dis)),\n[Matterport3D](https://niessner.github.io/Matterport/) ([here](#matterport3d)),\n[ArkitScene](https://github.com/apple/ARKitScenes),\n[Structured3D](https://structured3d-dataset.org/) ([here](#structured3d)),\n[SemanticKITTI](http://www.semantic-kitti.org/) ([here](#semantickitti)),\n[nuScenes](https://www.nuscenes.org/nuscenes) ([here](#nuscenes)),\n[ModelNet40](https://modelnet.cs.princeton.edu/) ([here](#modelnet)),\n[Waymo](https://waymo.com/open/) ([here](#waymo)).\n\n\n## Highlights\n- *May, 2024*: In v1.5.2, we redesigned the default structure for each dataset for better performance. Please **re-preprocess** datasets or **download** our preprocessed datasets from **[here](https://huggingface.co/Pointcept)**.\n- *Apr, 2024*: **PTv3** is selected as one of the 90 **Oral** papers (3.3% accepted papers, 0.78% submissions) by CVPR'24!\n- *Mar, 2024*: We release code for **OA-CNNs**, accepted by CVPR'24. Issue related to **OA-CNNs** can @Pbihao.\n- *Feb, 2024*: **PTv3** and **PPT** are accepted by CVPR'24, another **two** papers by our Pointcept team have also been accepted by CVPR'24 🎉🎉🎉. We will make them publicly available soon!\n- *Dec, 2023*: **PTv3** is released on arXiv, and the code is available in Pointcept. PTv3 is an efficient backbone model that achieves SOTA performances across indoor and outdoor scenarios.\n- *Aug, 2023*: **PPT** is released on arXiv. PPT presents a multi-dataset pre-training framework that achieves SOTA performance in both **indoor** and **outdoor** scenarios. It is compatible with various existing pre-training frameworks and backbones.  A **pre-release** version of the code is accessible; for those interested, please feel free to contact me directly for access.\n- *Mar, 2023*: We released our codebase, **Pointcept**, a highly potent tool for point cloud representation learning and perception. We welcome new work to join the _Pointcept_ family and highly recommend reading [Quick Start](#quick-start) before starting your trail.\n- *Feb, 2023*: **MSC** and **CeCo** accepted by CVPR 2023. _MSC_ is a highly efficient and effective pretraining framework that facilitates cross-dataset large-scale pretraining, while _CeCo_ is a segmentation method specifically designed for long-tail datasets. Both approaches are compatible with all existing backbone models in our codebase, and we will soon make the code available for public use.\n- *Jan, 2023*: **CAC**, oral work of AAAI 2023, has expanded its 3D result with the incorporation of Pointcept. This addition will allow CAC to serve as a pluggable segmentor within our codebase.\n- *Sep, 2022*: **PTv2** accepted by NeurIPS 2022. It is a continuation of the Point Transformer. The proposed GVA theory can apply to most existing attention mechanisms, while Grid Pooling is also a practical addition to existing pooling methods.\n\n## Citation\nIf you find _Pointcept_ useful to your research, please cite our work as encouragement. (੭ˊ꒳​ˋ)੭✧\n```\n@misc{pointcept2023,\n    title={Pointcept: A Codebase for Point Cloud Perception Research},\n    author={Pointcept Contributors},\n    howpublished = {\\url{https://github.com/Pointcept/Pointcept}},\n    year={2023}\n}\n```\n\n## Overview\n\n- [Installation](#installation)\n- [Data Preparation](#data-preparation)\n- [Quick Start](#quick-start)\n- [Model Zoo](#model-zoo)\n- [Citation](#citation)\n- [Acknowledgement](#acknowledgement)\n\n## Installation\n\n### Requirements\n- Ubuntu: 18.04 and above.\n- CUDA: 11.3 and above.\n- PyTorch: 1.10.0 and above.\n\n### Conda Environment\n\n```bash\nconda create -n pointcept python=3.8 -y\nconda activate pointcept\nconda install ninja -y\n# Choose version you want here: https://pytorch.org/get-started/previous-versions/\nconda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=11.3 -c pytorch -y\nconda install h5py pyyaml -c anaconda -y\nconda install sharedarray tensorboard tensorboardx yapf addict einops scipy plyfile termcolor timm -c conda-forge -y\nconda install pytorch-cluster pytorch-scatter pytorch-sparse -c pyg -y\npip install torch-geometric\n\n# spconv (SparseUNet)\n# refer https://github.com/traveller59/spconv\npip install spconv-cu113\n\n# PPT (clip)\npip install ftfy regex tqdm\npip install git+https://github.com/openai/CLIP.git\n\n# PTv1 \u0026 PTv2 or precise eval\ncd libs/pointops\n# usual\npython setup.py install\n# docker \u0026 multi GPU arch\nTORCH_CUDA_ARCH_LIST=\"ARCH LIST\" python  setup.py install\n# e.g. 7.5: RTX 3000; 8.0: a100 More available in: https://developer.nvidia.com/cuda-gpus\nTORCH_CUDA_ARCH_LIST=\"7.5 8.0\" python  setup.py install\ncd ../..\n\n# Open3D (visualization, optional)\npip install open3d\n```\n\n## Data Preparation\n\n### ScanNet v2\n\nThe preprocessing supports semantic and instance segmentation for both `ScanNet20`, `ScanNet200`, and `ScanNet Data Efficient`.\n- Download the [ScanNet](http://www.scan-net.org/) v2 dataset.\n- Run preprocessing code for raw ScanNet as follows:\n\n  ```bash\n  # RAW_SCANNET_DIR: the directory of downloaded ScanNet v2 raw dataset.\n  # PROCESSED_SCANNET_DIR: the directory of the processed ScanNet dataset (output dir).\n  python pointcept/datasets/preprocessing/scannet/preprocess_scannet.py --dataset_root ${RAW_SCANNET_DIR} --output_root ${PROCESSED_SCANNET_DIR}\n  ```\n- (Optional) Download ScanNet Data Efficient files:\n  ```bash\n  # download-scannet.py is the official download script\n  # or follow instructions here: https://kaldir.vc.in.tum.de/scannet_benchmark/data_efficient/documentation#download\n  python download-scannet.py --data_efficient -o ${RAW_SCANNET_DIR}\n  # unzip downloads\n  cd ${RAW_SCANNET_DIR}/tasks\n  unzip limited-annotation-points.zip\n  unzip limited-reconstruction-scenes.zip\n  # copy files to processed dataset folder\n  mkdir ${PROCESSED_SCANNET_DIR}/tasks\n  cp -r ${RAW_SCANNET_DIR}/tasks/points ${PROCESSED_SCANNET_DIR}/tasks\n  cp -r ${RAW_SCANNET_DIR}/tasks/scenes ${PROCESSED_SCANNET_DIR}/tasks\n  ```\n- (Alternative) Our preprocess data can be directly downloaded [[here](https://huggingface.co/datasets/Pointcept/scannet-compressed)], please agree the official license before download it.\n\n- Link processed dataset to codebase:\n  ```bash\n  # PROCESSED_SCANNET_DIR: the directory of the processed ScanNet dataset.\n  mkdir data\n  ln -s ${PROCESSED_SCANNET_DIR} ${CODEBASE_DIR}/data/scannet\n  ```\n\n### ScanNet++\n- Download the [ScanNet++](https://kaldir.vc.in.tum.de/scannetpp/) dataset.\n- Run preprocessing code for raw ScanNet++ as follows:\n  ```bash\n  # RAW_SCANNETPP_DIR: the directory of downloaded ScanNet++ raw dataset.\n  # PROCESSED_SCANNETPP_DIR: the directory of the processed ScanNet++ dataset (output dir).\n  # NUM_WORKERS: the number of workers for parallel preprocessing.\n  python pointcept/datasets/preprocessing/scannetpp/preprocess_scannetpp.py --dataset_root ${RAW_SCANNETPP_DIR} --output_root ${PROCESSED_SCANNETPP_DIR} --num_workers ${NUM_WORKERS}\n  ```\n- Sampling and chunking large point cloud data in train/val split as follows (only used for training):\n  ```bash\n  # PROCESSED_SCANNETPP_DIR: the directory of the processed ScanNet++ dataset (output dir).\n  # NUM_WORKERS: the number of workers for parallel preprocessing.\n  python pointcept/datasets/preprocessing/sampling_chunking_data.py --dataset_root ${PROCESSED_SCANNETPP_DIR} --grid_size 0.01 --chunk_range 6 6 --chunk_stride 3 3 --split train --num_workers ${NUM_WORKERS}\n  python pointcept/datasets/preprocessing/sampling_chunking_data.py --dataset_root ${PROCESSED_SCANNETPP_DIR} --grid_size 0.01 --chunk_range 6 6 --chunk_stride 3 3 --split val --num_workers ${NUM_WORKERS}\n  ```\n- (Alternative) Our preprocess data can be directly downloaded [[here](https://huggingface.co/datasets/Pointcept/scannetpp-compressed)], please agree the official license before download it.\n- Link processed dataset to codebase:\n  ```bash\n  # PROCESSED_SCANNETPP_DIR: the directory of the processed ScanNet dataset.\n  mkdir data\n  ln -s ${PROCESSED_SCANNETPP_DIR} ${CODEBASE_DIR}/data/scannetpp\n  ```\n\n### S3DIS\n\n- Download S3DIS data by filling this [Google form](https://docs.google.com/forms/d/e/1FAIpQLScDimvNMCGhy_rmBA2gHfDu3naktRm6A8BPwAWWDv-Uhm6Shw/viewform?c=0\u0026w=1). Download the `Stanford3dDataset_v1.2.zip` file and unzip it.\n- Fix error in `Area_5/office_19/Annotations/ceiling` Line 323474 (103.0�0000 =\u003e 103.000000).\n- (Optional) Download Full 2D-3D S3DIS dataset (no XYZ) from [here](https://github.com/alexsax/2D-3D-Semantics) for parsing normal.\n- Run preprocessing code for S3DIS as follows:\n\n  ```bash\n  # S3DIS_DIR: the directory of downloaded Stanford3dDataset_v1.2 dataset.\n  # RAW_S3DIS_DIR: the directory of Stanford2d3dDataset_noXYZ dataset. (optional, for parsing normal)\n  # PROCESSED_S3DIS_DIR: the directory of processed S3DIS dataset (output dir).\n  \n  # S3DIS without aligned angle\n  python pointcept/datasets/preprocessing/s3dis/preprocess_s3dis.py --dataset_root ${S3DIS_DIR} --output_root ${PROCESSED_S3DIS_DIR}\n  # S3DIS with aligned angle\n  python pointcept/datasets/preprocessing/s3dis/preprocess_s3dis.py --dataset_root ${S3DIS_DIR} --output_root ${PROCESSED_S3DIS_DIR} --align_angle\n  # S3DIS with normal vector (recommended, normal is helpful)\n  python pointcept/datasets/preprocessing/s3dis/preprocess_s3dis.py --dataset_root ${S3DIS_DIR} --output_root ${PROCESSED_S3DIS_DIR} --raw_root ${RAW_S3DIS_DIR} --parse_normal\n  python pointcept/datasets/preprocessing/s3dis/preprocess_s3dis.py --dataset_root ${S3DIS_DIR} --output_root ${PROCESSED_S3DIS_DIR} --raw_root ${RAW_S3DIS_DIR} --align_angle --parse_normal\n  ```\n\n- (Alternative) Our preprocess data can also be downloaded [[here](https://huggingface.co/datasets/Pointcept/s3dis-compressed\n)] (with normal vector and aligned angle), please agree with the official license before downloading it.\n\n- Link processed dataset to codebase.\n  ```bash\n  # PROCESSED_S3DIS_DIR: the directory of processed S3DIS dataset.\n  mkdir data\n  ln -s ${PROCESSED_S3DIS_DIR} ${CODEBASE_DIR}/data/s3dis\n  ```\n### Structured3D\n\n- Download Structured3D panorama related and perspective (full) related zip files by filling this [Google form](https://docs.google.com/forms/d/e/1FAIpQLSc0qtvh4vHSoZaW6UvlXYy79MbcGdZfICjh4_t4bYofQIVIdw/viewform?pli=1) (no need to unzip them).\n- Organize all downloaded zip file in one folder (`${STRUCT3D_DIR}`).\n- Run preprocessing code for Structured3D as follows:\n  ```bash\n  # STRUCT3D_DIR: the directory of downloaded Structured3D dataset.\n  # PROCESSED_STRUCT3D_DIR: the directory of processed Structured3D dataset (output dir).\n  # NUM_WORKERS: Number for workers for preprocessing, default same as cpu count (might OOM).\n  export PYTHONPATH=./\n  python pointcept/datasets/preprocessing/structured3d/preprocess_structured3d.py --dataset_root ${STRUCT3D_DIR} --output_root ${PROCESSED_STRUCT3D_DIR} --num_workers ${NUM_WORKERS} --grid_size 0.01 --fuse_prsp --fuse_pano\n  ```\nFollowing the instruction of [Swin3D](https://arxiv.org/abs/2304.06906), we keep 25 categories with frequencies of more than 0.001, out of the original 40 categories.\n\n[//]: # (- \u0026#40;Alternative\u0026#41; Our preprocess data can also be downloaded [[here]\u0026#40;\u0026#41;], please agree the official license before download it.)\n\n- (Alternative) Our preprocess data can also be downloaded [[here](https://huggingface.co/datasets/Pointcept/structured3d-compressed\n)] (with perspective views and panorama view, 471.7G after unzipping), please agree the official license before download it.\n\n- Link processed dataset to codebase.\n  ```bash\n  # PROCESSED_STRUCT3D_DIR: the directory of processed Structured3D dataset (output dir).\n  mkdir data\n  ln -s ${PROCESSED_STRUCT3D_DIR} ${CODEBASE_DIR}/data/structured3d\n  ```\n### Matterport3D\n- Follow [this page](https://niessner.github.io/Matterport/#download) to request access to the dataset.\n- Download the \"region_segmentation\" type, which represents the division of a scene into individual rooms.\n  ```bash\n  # download-mp.py is the official download script\n  # MATTERPORT3D_DIR: the directory of downloaded Matterport3D dataset.\n  python download-mp.py -o {MATTERPORT3D_DIR} --type region_segmentations\n  ```\n- Unzip the region_segmentations data\n  ```bash\n  # MATTERPORT3D_DIR: the directory of downloaded Matterport3D dataset.\n  python pointcept/datasets/preprocessing/matterport3d/unzip_matterport3d_region_segmentation.py --dataset_root {MATTERPORT3D_DIR}\n  ```\n- Run preprocessing code for Matterport3D as follows:\n  ```bash\n  # MATTERPORT3D_DIR: the directory of downloaded Matterport3D dataset.\n  # PROCESSED_MATTERPORT3D_DIR: the directory of processed Matterport3D dataset (output dir).\n  # NUM_WORKERS: the number of workers for this preprocessing.\n  python pointcept/datasets/preprocessing/matterport3d/preprocess_matterport3d_mesh.py --dataset_root ${MATTERPORT3D_DIR} --output_root ${PROCESSED_MATTERPORT3D_DIR} --num_workers ${NUM_WORKERS}\n  ```\n- Link processed dataset to codebase.\n  ```bash\n  # PROCESSED_MATTERPORT3D_DIR: the directory of processed Matterport3D dataset (output dir).\n  mkdir data\n  ln -s ${PROCESSED_MATTERPORT3D_DIR} ${CODEBASE_DIR}/data/matterport3d\n  ```\n\nFollowing the instruction of [OpenRooms](https://github.com/ViLab-UCSD/OpenRooms), we remapped Matterport3D's categories to ScanNet 20 semantic categories with the addition of a ceiling category.\n* (Alternative) Our preprocess data can also be downloaded [here](https://huggingface.co/datasets/Pointcept/matterport3d-compressed), please agree the official license before download it.\n\n### SemanticKITTI\n- Download [SemanticKITTI](http://www.semantic-kitti.org/dataset.html#download) dataset.\n- Link dataset to codebase.\n  ```bash\n  # SEMANTIC_KITTI_DIR: the directory of SemanticKITTI dataset.\n  # |- SEMANTIC_KITTI_DIR\n  #   |- dataset\n  #     |- sequences\n  #       |- 00\n  #       |- 01\n  #       |- ...\n  \n  mkdir -p data\n  ln -s ${SEMANTIC_KITTI_DIR} ${CODEBASE_DIR}/data/semantic_kitti\n  ```\n\n### nuScenes\n- Download the official [NuScene](https://www.nuscenes.org/nuscenes#download) dataset (with Lidar Segmentation) and organize the downloaded files as follows:\n  ```bash\n  NUSCENES_DIR\n  │── samples\n  │── sweeps\n  │── lidarseg\n  ...\n  │── v1.0-trainval \n  │── v1.0-test\n  ```\n- Run information preprocessing code (modified from OpenPCDet) for nuScenes as follows:\n  ```bash\n  # NUSCENES_DIR: the directory of downloaded nuScenes dataset.\n  # PROCESSED_NUSCENES_DIR: the directory of processed nuScenes dataset (output dir).\n  # MAX_SWEEPS: Max number of sweeps. Default: 10.\n  pip install nuscenes-devkit pyquaternion\n  python pointcept/datasets/preprocessing/nuscenes/preprocess_nuscenes_info.py --dataset_root ${NUSCENES_DIR} --output_root ${PROCESSED_NUSCENES_DIR} --max_sweeps ${MAX_SWEEPS} --with_camera\n  ```\n- (Alternative) Our preprocess nuScenes information data can also be downloaded [[here](\nhttps://huggingface.co/datasets/Pointcept/nuscenes-compressed)] (only processed information, still need to download raw dataset and link to the folder), please agree the official license before download it.\n\n- Link raw dataset to processed NuScene dataset folder:\n  ```bash\n  # NUSCENES_DIR: the directory of downloaded nuScenes dataset.\n  # PROCESSED_NUSCENES_DIR: the directory of processed nuScenes dataset (output dir).\n  ln -s ${NUSCENES_DIR} {PROCESSED_NUSCENES_DIR}/raw\n  ```\n  then the processed nuscenes folder is organized as follows:\n  ```bash\n  nuscene\n  |── raw\n      │── samples\n      │── sweeps\n      │── lidarseg\n      ...\n      │── v1.0-trainval\n      │── v1.0-test\n  |── info\n  ```\n\n- Link processed dataset to codebase.\n  ```bash\n  # PROCESSED_NUSCENES_DIR: the directory of processed nuScenes dataset (output dir).\n  mkdir data\n  ln -s ${PROCESSED_NUSCENES_DIR} ${CODEBASE_DIR}/data/nuscenes\n  ```\n\n### Waymo\n- Download the official [Waymo](https://waymo.com/open/download/) dataset (v1.4.3) and organize the downloaded files as follows:\n  ```bash\n  WAYMO_RAW_DIR\n  │── training\n  │── validation\n  │── testing\n  ```\n- Install the following dependence:\n  ```bash\n  # If shows \"No matching distribution found\", download whl directly from Pypi and install the package.\n  conda create -n waymo python=3.10 -y\n  conda activate waymo\n  pip install waymo-open-dataset-tf-2-12-0\n  ```\n- Run the preprocessing code as follows:\n  ```bash\n  # WAYMO_DIR: the directory of the downloaded Waymo dataset.\n  # PROCESSED_WAYMO_DIR: the directory of the processed Waymo dataset (output dir).\n  # NUM_WORKERS: num workers for preprocessing\n  python pointcept/datasets/preprocessing/waymo/preprocess_waymo.py --dataset_root ${WAYMO_DIR} --output_root ${PROCESSED_WAYMO_DIR} --splits training validation --num_workers ${NUM_WORKERS}\n  ```\n\n- Link processed dataset to the codebase.\n  ```bash\n  # PROCESSED_WAYMO_DIR: the directory of the processed Waymo dataset (output dir).\n  mkdir data\n  ln -s ${PROCESSED_WAYMO_DIR} ${CODEBASE_DIR}/data/waymo\n  ```\n\n### ModelNet\n- Download [modelnet40_normal_resampled.zip](https://shapenet.cs.stanford.edu/media/modelnet40_normal_resampled.zip) and unzip\n- Link dataset to the codebase.\n  ```bash\n  mkdir -p data\n  ln -s ${MODELNET_DIR} ${CODEBASE_DIR}/data/modelnet40_normal_resampled\n  ```\n\n## Quick Start\n\n### Training\n**Train from scratch.** The training processing is based on configs in `configs` folder. \nThe training script will generate an experiment folder in `exp` folder and backup essential code in the experiment folder.\nTraining config, log, tensorboard, and checkpoints will also be saved into the experiment folder during the training process.\n```bash\nexport CUDA_VISIBLE_DEVICES=${CUDA_VISIBLE_DEVICES}\n# Script (Recommended)\nsh scripts/train.sh -p ${INTERPRETER_PATH} -g ${NUM_GPU} -d ${DATASET_NAME} -c ${CONFIG_NAME} -n ${EXP_NAME}\n# Direct\nexport PYTHONPATH=./\npython tools/train.py --config-file ${CONFIG_PATH} --num-gpus ${NUM_GPU} --options save_path=${SAVE_PATH}\n```\n\nFor example:\n```bash\n# By script (Recommended)\n# -p is default set as python and can be ignored\nsh scripts/train.sh -p python -d scannet -c semseg-pt-v2m2-0-base -n semseg-pt-v2m2-0-base\n# Direct\nexport PYTHONPATH=./\npython tools/train.py --config-file configs/scannet/semseg-pt-v2m2-0-base.py --options save_path=exp/scannet/semseg-pt-v2m2-0-base\n```\n**Resume training from checkpoint.** If the training process is interrupted by accident, the following script can resume training from a given checkpoint.\n```bash\nexport CUDA_VISIBLE_DEVICES=${CUDA_VISIBLE_DEVICES}\n# Script (Recommended)\n# simply add \"-r true\"\nsh scripts/train.sh -p ${INTERPRETER_PATH} -g ${NUM_GPU} -d ${DATASET_NAME} -c ${CONFIG_NAME} -n ${EXP_NAME} -r true\n# Direct\nexport PYTHONPATH=./\npython tools/train.py --config-file ${CONFIG_PATH} --num-gpus ${NUM_GPU} --options save_path=${SAVE_PATH} resume=True weight=${CHECKPOINT_PATH}\n```\n\n### Testing\nDuring training, model evaluation is performed on point clouds after grid sampling (voxelization), providing an initial assessment of model performance. However, to obtain precise evaluation results, testing is **essential**. The testing process involves subsampling a dense point cloud into a sequence of voxelized point clouds, ensuring comprehensive coverage of all points. These sub-results are then predicted and collected to form a complete prediction of the entire point cloud. This approach yields  higher evaluation results compared to simply mapping/interpolating the prediction. In addition, our testing code supports TTA (test time augmentation) testing, which further enhances the stability of evaluation performance.\n\n```bash\n# By script (Based on experiment folder created by training script)\nsh scripts/test.sh -p ${INTERPRETER_PATH} -g ${NUM_GPU} -d ${DATASET_NAME} -n ${EXP_NAME} -w ${CHECKPOINT_NAME}\n# Direct\nexport PYTHONPATH=./\npython tools/test.py --config-file ${CONFIG_PATH} --num-gpus ${NUM_GPU} --options save_path=${SAVE_PATH} weight=${CHECKPOINT_PATH}\n```\nFor example:\n```bash\n# By script (Based on experiment folder created by training script)\n# -p is default set as python and can be ignored\n# -w is default set as model_best and can be ignored\nsh scripts/test.sh -p python -d scannet -n semseg-pt-v2m2-0-base -w model_best\n# Direct\nexport PYTHONPATH=./\npython tools/test.py --config-file configs/scannet/semseg-pt-v2m2-0-base.py --options save_path=exp/scannet/semseg-pt-v2m2-0-base weight=exp/scannet/semseg-pt-v2m2-0-base/model/model_best.pth\n```\n\nThe TTA can be disabled by replace `data.test.test_cfg.aug_transform = [...]` with:\n\n```python\ndata = dict(\n    train = dict(...),\n    val = dict(...),\n    test = dict(\n        ...,\n        test_cfg = dict(\n            ...,\n            aug_transform = [\n                [dict(type=\"RandomRotateTargetAngle\", angle=[0], axis=\"z\", center=[0, 0, 0], p=1)]\n            ]\n        )\n    )\n)\n```\n\n### Offset\n`Offset` is the separator of point clouds in batch data, and it is similar to the concept of `Batch` in PyG. \nA visual illustration of batch and offset is as follows:\n\u003cp align=\"center\"\u003e\n    \u003c!-- pypi-strip --\u003e\n    \u003cpicture\u003e\n    \u003csource media=\"(prefers-color-scheme: dark)\" srcset=\"https://raw.githubusercontent.com/Pointcept/Pointcept/main/docs/offset_dark.png\"\u003e\n    \u003csource media=\"(prefers-color-scheme: light)\" srcset=\"https://raw.githubusercontent.com/Pointcept/Pointcept/main/docs/offset.png\"\u003e\n    \u003c!-- /pypi-strip --\u003e\n    \u003cimg alt=\"pointcept\" src=\"https://raw.githubusercontent.com/Pointcept/Pointcept/main/docs/offset.png\" width=\"480\"\u003e\n    \u003c!-- pypi-strip --\u003e\n    \u003c/picture\u003e\u003cbr\u003e\n    \u003c!-- /pypi-strip --\u003e\n\u003c/p\u003e\n\n## Model Zoo\n### 1. Backbones and Semantic Segmentation\n#### SparseUNet\n\n_Pointcept_ provides `SparseUNet` implemented by `SpConv` and `MinkowskiEngine`. The SpConv version is recommended since SpConv is easy to install and faster than MinkowskiEngine. Meanwhile, SpConv is also widely applied in outdoor perception.\n\n- **SpConv (recommend)**\n\nThe SpConv version `SparseUNet` in the codebase was fully rewrite from `MinkowskiEngine` version, example running script is as follows:\n\n```bash\n# ScanNet val\nsh scripts/train.sh -g 4 -d scannet -c semseg-spunet-v1m1-0-base -n semseg-spunet-v1m1-0-base\n# ScanNet200\nsh scripts/train.sh -g 4 -d scannet200 -c semseg-spunet-v1m1-0-base -n semseg-spunet-v1m1-0-base\n# S3DIS\nsh scripts/train.sh -g 4 -d s3dis -c semseg-spunet-v1m1-0-base -n semseg-spunet-v1m1-0-base\n# S3DIS (with normal)\nsh scripts/train.sh -g 4 -d s3dis -c semseg-spunet-v1m1-0-cn-base -n semseg-spunet-v1m1-0-cn-base\n# SemanticKITTI\nsh scripts/train.sh -g 4 -d semantic_kitti -c semseg-spunet-v1m1-0-base -n semseg-spunet-v1m1-0-base\n# nuScenes\nsh scripts/train.sh -g 4 -d nuscenes -c semseg-spunet-v1m1-0-base -n semseg-spunet-v1m1-0-base\n# ModelNet40\nsh scripts/train.sh -g 2 -d modelnet40 -c cls-spunet-v1m1-0-base -n cls-spunet-v1m1-0-base\n\n# ScanNet Data Efficient\nsh scripts/train.sh -g 4 -d scannet -c semseg-spunet-v1m1-2-efficient-la20 -n semseg-spunet-v1m1-2-efficient-la20\nsh scripts/train.sh -g 4 -d scannet -c semseg-spunet-v1m1-2-efficient-la50 -n semseg-spunet-v1m1-2-efficient-la50\nsh scripts/train.sh -g 4 -d scannet -c semseg-spunet-v1m1-2-efficient-la100 -n semseg-spunet-v1m1-2-efficient-la100\nsh scripts/train.sh -g 4 -d scannet -c semseg-spunet-v1m1-2-efficient-la200 -n semseg-spunet-v1m1-2-efficient-la200\nsh scripts/train.sh -g 4 -d scannet -c semseg-spunet-v1m1-2-efficient-lr1 -n semseg-spunet-v1m1-2-efficient-lr1\nsh scripts/train.sh -g 4 -d scannet -c semseg-spunet-v1m1-2-efficient-lr5 -n semseg-spunet-v1m1-2-efficient-lr5\nsh scripts/train.sh -g 4 -d scannet -c semseg-spunet-v1m1-2-efficient-lr10 -n semseg-spunet-v1m1-2-efficient-lr10\nsh scripts/train.sh -g 4 -d scannet -c semseg-spunet-v1m1-2-efficient-lr20 -n semseg-spunet-v1m1-2-efficient-lr20\n\n# Profile model run time\nsh scripts/train.sh -g 4 -d scannet -c semseg-spunet-v1m1-0-enable-profiler -n semseg-spunet-v1m1-0-enable-profiler\n```\n\n- **MinkowskiEngine**\n\nThe MinkowskiEngine version `SparseUNet` in the codebase was modified from the original MinkowskiEngine repo, and example running scripts are as follows:\n1. Install MinkowskiEngine, refer https://github.com/NVIDIA/MinkowskiEngine\n2. Training with the following example scripts:\n```bash\n# Uncomment \"# from .sparse_unet import *\" in \"pointcept/models/__init__.py\"\n# Uncomment \"# from .mink_unet import *\" in \"pointcept/models/sparse_unet/__init__.py\"\n# ScanNet\nsh scripts/train.sh -g 4 -d scannet -c semseg-minkunet34c-0-base -n semseg-minkunet34c-0-base\n# ScanNet200\nsh scripts/train.sh -g 4 -d scannet200 -c semseg-minkunet34c-0-base -n semseg-minkunet34c-0-base\n# S3DIS\nsh scripts/train.sh -g 4 -d s3dis -c semseg-minkunet34c-0-base -n semseg-minkunet34c-0-base\n# SemanticKITTI\nsh scripts/train.sh -g 2 -d semantic_kitti -c semseg-minkunet34c-0-base -n semseg-minkunet34c-0-base\n```\n\n#### OA-CNNs\nIntroducing Omni-Adaptive 3D CNNs (**OA-CNNs**), a family of networks that integrates a lightweight module to greatly enhance the adaptivity of sparse CNNs at minimal computational cost. Without any self-attention modules, **OA-CNNs** favorably surpass point transformers in terms of accuracy in both indoor and outdoor scenes, with much less latency and memory cost. Issue related to **OA-CNNs** can @Pbihao.\n```bash\n# ScanNet\nsh scripts/train.sh -g 4 -d scannet -c semseg-oacnns-v1m1-0-base -n semseg-oacnns-v1m1-0-base\n```\n\n#### Point Transformers\n- **PTv3**\n\n[PTv3](https://arxiv.org/abs/2312.10035) is an efficient backbone model that achieves SOTA performances across indoor and outdoor scenarios. The full PTv3 relies on FlashAttention, while FlashAttention relies on CUDA 11.6 and above, make sure your local Pointcept environment satisfies the requirements.\n\nIf you can not upgrade your local environment to satisfy the requirements (CUDA \u003e= 11.6), then you can disable FlashAttention by setting the model parameter `enable_flash` to `false` and reducing the `enc_patch_size` and `dec_patch_size` to a level (e.g. 128).\n\nFlashAttention force disables RPE and forces the accuracy reduced to fp16. If you require these features, please disable `enable_flash` and adjust `enable_rpe`, `upcast_attention` and`upcast_softmax`.\n\nDetailed instructions and experiment records (containing weights) are available on the [project repository](https://github.com/Pointcept/PointTransformerV3). Example running scripts are as follows:\n```bash\n# Scratched ScanNet\nsh scripts/train.sh -g 4 -d scannet -c semseg-pt-v3m1-0-base -n semseg-pt-v3m1-0-base\n# PPT joint training (ScanNet + Structured3D) and evaluate in ScanNet\nsh scripts/train.sh -g 8 -d scannet -c semseg-pt-v3m1-1-ppt-extreme -n semseg-pt-v3m1-1-ppt-extreme\n\n# Scratched ScanNet200\nsh scripts/train.sh -g 4 -d scannet200 -c semseg-pt-v3m1-0-base -n semseg-pt-v3m1-0-base\n# Fine-tuning from  PPT joint training (ScanNet + Structured3D) with ScanNet200\n# PTV3_PPT_WEIGHT_PATH: Path to model weight trained by PPT multi-dataset joint training\n# e.g. exp/scannet/semseg-pt-v3m1-1-ppt-extreme/model/model_best.pth\nsh scripts/train.sh -g 4 -d scannet200 -c semseg-pt-v3m1-1-ppt-ft -n semseg-pt-v3m1-1-ppt-ft -w ${PTV3_PPT_WEIGHT_PATH}\n\n# Scratched ScanNet++\nsh scripts/train.sh -g 4 -d scannetpp -c semseg-pt-v3m1-0-base -n semseg-pt-v3m1-0-base\n# Scratched ScanNet++ test\nsh scripts/train.sh -g 4 -d scannetpp -c semseg-pt-v3m1-1-submit -n semseg-pt-v3m1-1-submit\n\n\n# Scratched S3DIS\nsh scripts/train.sh -g 4 -d s3dis -c semseg-pt-v3m1-0-base -n semseg-pt-v3m1-0-base\n# an example for disbale flash_attention and enable rpe.\nsh scripts/train.sh -g 4 -d s3dis -c semseg-pt-v3m1-1-rpe -n semseg-pt-v3m1-0-rpe\n# PPT joint training (ScanNet + S3DIS + Structured3D) and evaluate in ScanNet\nsh scripts/train.sh -g 8 -d s3dis -c semseg-pt-v3m1-1-ppt-extreme -n semseg-pt-v3m1-1-ppt-extreme\n# S3DIS 6-fold cross validation\n# 1. The default configs are evaluated on Area_5, modify the \"data.train.split\", \"data.val.split\", and \"data.test.split\" to make the config evaluated on Area_1 ~ Area_6 respectively.\n# 2. Train and evaluate the model on each split of areas and gather result files located in \"exp/s3dis/EXP_NAME/result/Area_x.pth\" in one single folder, noted as RECORD_FOLDER.\n# 3. Run the following script to get S3DIS 6-fold cross validation performance:\nexport PYTHONPATH=./\npython tools/test_s3dis_6fold.py --record_root ${RECORD_FOLDER}\n\n# Scratched nuScenes\nsh scripts/train.sh -g 4 -d nuscenes -c semseg-pt-v3m1-0-base -n semseg-pt-v3m1-0-base\n# Scratched Waymo\nsh scripts/train.sh -g 4 -d waymo -c semseg-pt-v3m1-0-base -n semseg-pt-v3m1-0-base\n\n# More configs and exp records for PTv3 will be available soon.\n```\n\nIndoor semantic segmentation  \n| Model | Benchmark | Additional Data | Num GPUs | Val mIoU | Config | Tensorboard | Exp Record |\n| :---: | :---: |:---------------:| :---: | :---: | :---: | :---: | :---: |\n| PTv3 | ScanNet |     \u0026cross;     | 4 | 77.6% | [link](https://github.com/Pointcept/Pointcept/blob/main/configs/scannet/semseg-pt-v3m1-0-base.py) | [link](https://huggingface.co/Pointcept/PointTransformerV3/tensorboard) | [link](https://huggingface.co/Pointcept/PointTransformerV3/tree/main/scannet-semseg-pt-v3m1-0-base) |\n| PTv3 + PPT | ScanNet |     \u0026check;     | 8 | 78.5% | [link](https://github.com/Pointcept/Pointcept/blob/main/configs/scannet/semseg-pt-v3m1-1-ppt-extreme.py) | [link](https://huggingface.co/Pointcept/PointTransformerV3/tensorboard) | [link](https://huggingface.co/Pointcept/PointTransformerV3/tree/main/scannet-semseg-pt-v3m1-1-ppt-extreme) |\n| PTv3 | ScanNet200 |     \u0026cross;     | 4 | 35.3% | [link](https://github.com/Pointcept/Pointcept/blob/main/configs/scannet200/semseg-pt-v3m1-0-base.py) | [link](https://huggingface.co/Pointcept/PointTransformerV3/tensorboard) |[link](https://huggingface.co/Pointcept/PointTransformerV3/tree/main/scannet200-semseg-pt-v3m1-0-base)|\n| PTv3 + PPT | ScanNet200 | \u0026check; (f.t.)  | 4 |  |  |  |  |\n| PTv3 | S3DIS (Area5) |     \u0026cross;     | 4 | 73.6% | [link](https://github.com/Pointcept/Pointcept/blob/main/configs/s3dis/semseg-pt-v3m1-0-rpe.py) | [link](https://huggingface.co/Pointcept/PointTransformerV3/tensorboard) | [link](https://huggingface.co/Pointcept/PointTransformerV3/tree/main/s3dis-semseg-pt-v3m1-0-rpe) |\n| PTv3 + PPT | S3DIS (Area5) |     \u0026check;     | 8 | 75.4% | [link](https://github.com/Pointcept/Pointcept/blob/main/configs/s3dis/semseg-pt-v3m1-1-ppt-extreme.py) | [link](https://huggingface.co/Pointcept/PointTransformerV3/tensorboard) | [link](https://huggingface.co/Pointcept/PointTransformerV3/tree/main/s3dis-semseg-pt-v3m1-1-ppt-extreme) |\n\nOutdoor semantic segmentation  \n| Model | Benchmark | Additional Data | Num GPUs | Val mIoU | Config | Tensorboard | Exp Record |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| PTv3 | nuScenes | \u0026cross; | 4 | 80.3 | [link](https://github.com/Pointcept/Pointcept/blob/main/configs/nuscenes/semseg-pt-v3m1-0-base.py) | [link](https://huggingface.co/Pointcept/PointTransformerV3/tensorboard)|[link](https://huggingface.co/Pointcept/PointTransformerV3/tree/main/nuscenes-semseg-pt-v3m1-0-base) |\n| PTv3 + PPT | nuScenes | \u0026check; | 8 | | | | |\n| PTv3 | SemanticKITTI | \u0026cross; | 4 | | | | |\n| PTv3 + PPT | SemanticKITTI | \u0026check; | 8 | | | | |\n| PTv3 | Waymo | \u0026cross; | 4 | 71.2 | [link](https://github.com/Pointcept/Pointcept/blob/main/configs/waymo/semseg-pt-v3m1-0-base.py) | [link](https://huggingface.co/Pointcept/PointTransformerV3/tensorboard) | [link](https://huggingface.co/Pointcept/PointTransformerV3/tree/main/waymo-semseg-pt-v3m1-0-base) (log only) |\n| PTv3 + PPT | Waymo | \u0026check; | 8 | | | | |\n\n_**\\*Released model weights are trained for v1.5.1, weights for v1.5.2 and later is still ongoing.**_\n\n- **PTv2 mode2**\n\nThe original PTv2 was trained on 4 * RTX a6000 (48G memory). Even enabling AMP, the memory cost of the original PTv2 is slightly larger than 24G. Considering GPUs with 24G memory are much more accessible, I tuned the PTv2 on the latest Pointcept and made it runnable on 4 * RTX 3090 machines.\n\n`PTv2 Mode2` enables AMP and disables _Position Encoding Multiplier_ \u0026 _Grouped Linear_. During our further research, we found that precise coordinates are not necessary for point cloud understanding (Replacing precise coordinates with grid coordinates doesn't influence the performance. Also, SparseUNet is an example). As for Grouped Linear, my implementation of Grouped Linear seems to cost more memory than the Linear layer provided by PyTorch. Benefiting from the codebase and better parameter tuning, we also relieve the overfitting problem. The reproducing performance is even better than the results reported in our paper.\n\nExample running scripts are as follows:\n\n```bash\n# ptv2m2: PTv2 mode2, disable PEM \u0026 Grouped Linear, GPU memory cost \u003c 24G (recommend)\n# ScanNet\nsh scripts/train.sh -g 4 -d scannet -c semseg-pt-v2m2-0-base -n semseg-pt-v2m2-0-base\nsh scripts/train.sh -g 4 -d scannet -c semseg-pt-v2m2-3-lovasz -n semseg-pt-v2m2-3-lovasz\n\n# ScanNet test\nsh scripts/train.sh -g 4 -d scannet -c semseg-pt-v2m2-1-submit -n semseg-pt-v2m2-1-submit\n# ScanNet200\nsh scripts/train.sh -g 4 -d scannet200 -c semseg-pt-v2m2-0-base -n semseg-pt-v2m2-0-base\n# ScanNet++\nsh scripts/train.sh -g 4 -d scannetpp -c semseg-pt-v2m2-0-base -n semseg-pt-v2m2-0-base\n# ScanNet++ test\nsh scripts/train.sh -g 4 -d scannetpp -c semseg-pt-v2m2-1-submit -n semseg-pt-v2m2-1-submit\n# S3DIS\nsh scripts/train.sh -g 4 -d s3dis -c semseg-pt-v2m2-0-base -n semseg-pt-v2m2-0-base\n# SemanticKITTI\nsh scripts/train.sh -g 4 -d semantic_kitti -c semseg-pt-v2m2-0-base -n semseg-pt-v2m2-0-base\n# nuScenes\nsh scripts/train.sh -g 4 -d nuscenes -c semseg-pt-v2m2-0-base -n semseg-pt-v2m2-0-base\n```\n\n- **PTv2 mode1**\n\n`PTv2 mode1` is the original PTv2 we reported in our paper, example running scripts are as follows:\n\n```bash\n# ptv2m1: PTv2 mode1, Original PTv2, GPU memory cost \u003e 24G\n# ScanNet\nsh scripts/train.sh -g 4 -d scannet -c semseg-pt-v2m1-0-base -n semseg-pt-v2m1-0-base\n# ScanNet200\nsh scripts/train.sh -g 4 -d scannet200 -c semseg-pt-v2m1-0-base -n semseg-pt-v2m1-0-base\n# S3DIS\nsh scripts/train.sh -g 4 -d s3dis -c semseg-pt-v2m1-0-base -n semseg-pt-v2m1-0-base\n```\n\n- **PTv1**\n\nThe original PTv1 is also available in our Pointcept codebase. I haven't run PTv1 for a long time, but I have ensured that the example running script works well. \n\n```bash\n# ScanNet\nsh scripts/train.sh -g 4 -d scannet -c semseg-pt-v1-0-base -n semseg-pt-v1-0-base\n# ScanNet200\nsh scripts/train.sh -g 4 -d scannet200 -c semseg-pt-v1-0-base -n semseg-pt-v1-0-base\n# S3DIS\nsh scripts/train.sh -g 4 -d s3dis -c semseg-pt-v1-0-base -n semseg-pt-v1-0-base\n```\n\n\n#### Stratified Transformer\n1. Additional requirements:\n```bash\npip install torch-points3d\n# Fix dependence, caused by installing torch-points3d \npip uninstall SharedArray\npip install SharedArray==3.2.1\n\ncd libs/pointops2\npython setup.py install\ncd ../..\n```\n2. Uncomment `# from .stratified_transformer import *` in `pointcept/models/__init__.py`.\n3. Refer [Optional Installation](installation) to install dependence.\n4. Training with the following example scripts:\n```bash\n# stv1m1: Stratified Transformer mode1, Modified from the original Stratified Transformer code.\n# PTv2m2: Stratified Transformer mode2, My rewrite version (recommend).\n\n# ScanNet\nsh scripts/train.sh -g 4 -d scannet -c semseg-st-v1m2-0-refined -n semseg-st-v1m2-0-refined\nsh scripts/train.sh -g 4 -d scannet -c semseg-st-v1m1-0-origin -n semseg-st-v1m1-0-origin\n# ScanNet200\nsh scripts/train.sh -g 4 -d scannet200 -c semseg-st-v1m2-0-refined -n semseg-st-v1m2-0-refined\n# S3DIS\nsh scripts/train.sh -g 4 -d s3dis -c semseg-st-v1m2-0-refined -n semseg-st-v1m2-0-refined\n```\n\n#### SPVCNN\n`SPVCNN` is a baseline model of [SPVNAS](https://github.com/mit-han-lab/spvnas), it is also a practical baseline for outdoor datasets.\n1. Install torchsparse:\n```bash\n# refer https://github.com/mit-han-lab/torchsparse\n# install method without sudo apt install\nconda install google-sparsehash -c bioconda\nexport C_INCLUDE_PATH=${CONDA_PREFIX}/include:$C_INCLUDE_PATH\nexport CPLUS_INCLUDE_PATH=${CONDA_PREFIX}/include:CPLUS_INCLUDE_PATH\npip install --upgrade git+https://github.com/mit-han-lab/torchsparse.git\n```\n2. Training with the following example scripts:\n```bash\n# SemanticKITTI\nsh scripts/train.sh -g 2 -d semantic_kitti -c semseg-spvcnn-v1m1-0-base -n semseg-spvcnn-v1m1-0-base\n```\n\n#### OctFormer\nOctFormer from _OctFormer: Octree-based Transformers for 3D Point Clouds_.\n1. Additional requirements:\n```bash\ncd libs\ngit clone https://github.com/octree-nn/dwconv.git\npip install ./dwconv\npip install ocnn\n```\n2. Uncomment `# from .octformer import *` in `pointcept/models/__init__.py`.\n2. Training with the following example scripts:\n```bash\n# ScanNet\nsh scripts/train.sh -g 4 -d scannet -c semseg-octformer-v1m1-0-base -n semseg-octformer-v1m1-0-base\n```\n\n#### Swin3D\nSwin3D from _Swin3D: A Pretrained Transformer Backbone for 3D Indoor Scene Understanding_. \n1. Additional requirements:\n```bash\n# 1. Install MinkEngine v0.5.4, follow readme in https://github.com/NVIDIA/MinkowskiEngine;\n# 2. Install Swin3D, mainly for cuda operation:\ncd libs\ngit clone https://github.com/microsoft/Swin3D.git\ncd Swin3D\npip install ./\n```\n2. Uncomment `# from .swin3d import *` in `pointcept/models/__init__.py`.\n3. Pre-Training with the following example scripts (Structured3D preprocessing refer [here](#structured3d)):\n```bash\n# Structured3D + Swin-S\nsh scripts/train.sh -g 4 -d structured3d -c semseg-swin3d-v1m1-0-small -n semseg-swin3d-v1m1-0-small\n# Structured3D + Swin-L\nsh scripts/train.sh -g 4 -d structured3d -c semseg-swin3d-v1m1-1-large -n semseg-swin3d-v1m1-1-large\n\n# Addition\n# Structured3D + SpUNet\nsh scripts/train.sh -g 4 -d structured3d -c semseg-spunet-v1m1-0-base -n semseg-spunet-v1m1-0-base\n# Structured3D + PTv2\nsh scripts/train.sh -g 4 -d structured3d -c semseg-pt-v2m2-0-base -n semseg-pt-v2m2-0-base\n```\n4. Fine-tuning with the following example scripts:\n```bash\n# ScanNet + Swin-S\nsh scripts/train.sh -g 4 -d scannet -w exp/structured3d/semseg-swin3d-v1m1-1-large/model/model_last.pth -c semseg-swin3d-v1m1-0-small -n semseg-swin3d-v1m1-0-small\n# ScanNet + Swin-L\nsh scripts/train.sh -g 4 -d scannet -w exp/structured3d/semseg-swin3d-v1m1-1-large/model/model_last.pth -c semseg-swin3d-v1m1-1-large -n semseg-swin3d-v1m1-1-large\n\n# S3DIS + Swin-S (here we provide config support S3DIS normal vector)\nsh scripts/train.sh -g 4 -d s3dis -w exp/structured3d/semseg-swin3d-v1m1-1-large/model/model_last.pth -c semseg-swin3d-v1m1-0-small -n semseg-swin3d-v1m1-0-small\n# S3DIS + Swin-L (here we provide config support S3DIS normal vector)\nsh scripts/train.sh -g 4 -d s3dis -w exp/structured3d/semseg-swin3d-v1m1-1-large/model/model_last.pth -c semseg-swin3d-v1m1-1-large -n semseg-swin3d-v1m1-1-large\n```\n\n#### Context-Aware Classifier\n`Context-Aware Classifier` is a segmentor that can further boost the performance of each backbone, as a replacement for `Default Segmentor`.  Training with the following example scripts:\n```bash\n# ScanNet\nsh scripts/train.sh -g 4 -d scannet -c semseg-cac-v1m1-0-spunet-base -n semseg-cac-v1m1-0-spunet-base\nsh scripts/train.sh -g 4 -d scannet -c semseg-cac-v1m1-1-spunet-lovasz -n semseg-cac-v1m1-1-spunet-lovasz\nsh scripts/train.sh -g 4 -d scannet -c semseg-cac-v1m1-2-ptv2-lovasz -n semseg-cac-v1m1-2-ptv2-lovasz\n\n# ScanNet200\nsh scripts/train.sh -g 4 -d scannet200 -c semseg-cac-v1m1-0-spunet-base -n semseg-cac-v1m1-0-spunet-base\nsh scripts/train.sh -g 4 -d scannet200 -c semseg-cac-v1m1-1-spunet-lovasz -n semseg-cac-v1m1-1-spunet-lovasz\nsh scripts/train.sh -g 4 -d scannet200 -c semseg-cac-v1m1-2-ptv2-lovasz -n semseg-cac-v1m1-2-ptv2-lovasz\n```\n\n\n### 2. Instance Segmentation\n#### PointGroup\n[PointGroup](https://github.com/dvlab-research/PointGroup) is a baseline framework for point cloud instance segmentation.\n1. Additional requirements:\n```bash\nconda install -c bioconda google-sparsehash \ncd libs/pointgroup_ops\npython setup.py install --include_dirs=${CONDA_PREFIX}/include\ncd ../..\n```\n2. Uncomment `# from .point_group import *` in `pointcept/models/__init__.py`.\n3. Training with the following example scripts:\n```bash\n# ScanNet\nsh scripts/train.sh -g 4 -d scannet -c insseg-pointgroup-v1m1-0-spunet-base -n insseg-pointgroup-v1m1-0-spunet-base\n# S3DIS\nsh scripts/train.sh -g 4 -d scannet -c insseg-pointgroup-v1m1-0-spunet-base -n insseg-pointgroup-v1m1-0-spunet-base\n```\n\n### 3. Pre-training\n#### Masked Scene Contrast (MSC)\n1. Pre-training with the following example scripts:\n```bash\n# ScanNet\nsh scripts/train.sh -g 8 -d scannet -c pretrain-msc-v1m1-0-spunet-base -n pretrain-msc-v1m1-0-spunet-base\n```\n\n2. Fine-tuning with the following example scripts:  \nenable PointGroup ([here](#pointgroup)) before fine-tuning on instance segmentation task.\n```bash\n# ScanNet20 Semantic Segmentation\nsh scripts/train.sh -g 8 -d scannet -w exp/scannet/pretrain-msc-v1m1-0-spunet-base/model/model_last.pth -c semseg-spunet-v1m1-4-ft -n semseg-msc-v1m1-0f-spunet-base\n# ScanNet20 Instance Segmentation (enable PointGroup before running the script)\nsh scripts/train.sh -g 4 -d scannet -w exp/scannet/pretrain-msc-v1m1-0-spunet-base/model/model_last.pth -c insseg-pointgroup-v1m1-0-spunet-base -n insseg-msc-v1m1-0f-pointgroup-spunet-base\n```\n3. Example log and weight: [[Pretrain](https://connecthkuhk-my.sharepoint.com/:u:/g/personal/wuxy_connect_hku_hk/EYvNV4XUJ_5Mlk-g15RelN4BW_P8lVBfC_zhjC_BlBDARg?e=UoGFWH)] [[Semseg](https://connecthkuhk-my.sharepoint.com/:u:/g/personal/wuxy_connect_hku_hk/EQkDiv5xkOFKgCpGiGtAlLwBon7i8W6my3TIbGVxuiTttQ?e=tQFnbr)]\n\n#### Point Prompt Training (PPT)\nPPT presents a multi-dataset pre-training framework, and it is compatible with various existing pre-training frameworks and backbones.\n1. PPT supervised joint training with the following example scripts:\n```bash\n# ScanNet + Structured3d, validate on ScanNet (S3DIS might cause long data time, w/o S3DIS for a quick validation) \u003e= 3090 * 8 \nsh scripts/train.sh -g 8 -d scannet -c semseg-ppt-v1m1-0-sc-st-spunet -n semseg-ppt-v1m1-0-sc-st-spunet\nsh scripts/train.sh -g 8 -d scannet -c semseg-ppt-v1m1-1-sc-st-spunet-submit -n semseg-ppt-v1m1-1-sc-st-spunet-submit\n# ScanNet + S3DIS + Structured3d, validate on S3DIS (\u003e= a100 * 8)\nsh scripts/train.sh -g 8 -d s3dis -c semseg-ppt-v1m1-0-s3-sc-st-spunet -n semseg-ppt-v1m1-0-s3-sc-st-spunet\n# SemanticKITTI + nuScenes + Waymo, validate on SemanticKITTI (bs12 \u003e= 3090 * 4 \u003e= 3090 * 8, v1m1-0 is still on tuning)\nsh scripts/train.sh -g 4 -d semantic_kitti -c semseg-ppt-v1m1-0-nu-sk-wa-spunet -n semseg-ppt-v1m1-0-nu-sk-wa-spunet\nsh scripts/train.sh -g 4 -d semantic_kitti -c semseg-ppt-v1m2-0-sk-nu-wa-spunet -n semseg-ppt-v1m2-0-sk-nu-wa-spunet\nsh scripts/train.sh -g 4 -d semantic_kitti -c semseg-ppt-v1m2-1-sk-nu-wa-spunet-submit -n semseg-ppt-v1m2-1-sk-nu-wa-spunet-submit\n# SemanticKITTI + nuScenes + Waymo, validate on nuScenes (bs12 \u003e= 3090 * 4; bs24 \u003e= 3090 * 8, v1m1-0 is still on tuning))\nsh scripts/train.sh -g 4 -d nuscenes -c semseg-ppt-v1m1-0-nu-sk-wa-spunet -n semseg-ppt-v1m1-0-nu-sk-wa-spunet\nsh scripts/train.sh -g 4 -d nuscenes -c semseg-ppt-v1m2-0-nu-sk-wa-spunet -n semseg-ppt-v1m2-0-nu-sk-wa-spunet\nsh scripts/train.sh -g 4 -d nuscenes -c semseg-ppt-v1m2-1-nu-sk-wa-spunet-submit -n semseg-ppt-v1m2-1-nu-sk-wa-spunet-submit\n```\n\n#### PointContrast\n1. Preprocess and link ScanNet-Pair dataset (pair-wise matching with ScanNet raw RGB-D frame, ~1.5T):\n```bash\n# RAW_SCANNET_DIR: the directory of downloaded ScanNet v2 raw dataset.\n# PROCESSED_SCANNET_PAIR_DIR: the directory of processed ScanNet pair dataset (output dir).\npython pointcept/datasets/preprocessing/scannet/scannet_pair/preprocess.py --dataset_root ${RAW_SCANNET_DIR} --output_root ${PROCESSED_SCANNET_PAIR_DIR}\nln -s ${PROCESSED_SCANNET_PAIR_DIR} ${CODEBASE_DIR}/data/scannet\n```\n2. Pre-training with the following example scripts:\n```bash\n# ScanNet\nsh scripts/train.sh -g 8 -d scannet -c pretrain-msc-v1m1-1-spunet-pointcontrast -n pretrain-msc-v1m1-1-spunet-pointcontrast\n```\n3. Fine-tuning refer [MSC](#masked-scene-contrast-msc).\n\n#### Contrastive Scene Contexts\n1. Preprocess and link ScanNet-Pair dataset (refer [PointContrast](#pointcontrast)):\n2. Pre-training with the following example scripts:\n```bash\n# ScanNet\nsh scripts/train.sh -g 8 -d scannet -c pretrain-msc-v1m2-0-spunet-csc -n pretrain-msc-v1m2-0-spunet-csc\n```\n3. Fine-tuning refer [MSC](#masked-scene-contrast-msc).\n\n## Acknowledgement\n_Pointcept_ is designed by [Xiaoyang](https://xywu.me/), named by [Yixing](https://github.com/yxlao) and the logo is created by [Yuechen](https://julianjuaner.github.io/). It is derived from [Hengshuang](https://hszhao.github.io/)'s [Semseg](https://github.com/hszhao/semseg) and inspirited by several repos, e.g., [MinkowskiEngine](https://github.com/NVIDIA/MinkowskiEngine), [pointnet2](https://github.com/charlesq34/pointnet2), [mmcv](https://github.com/open-mmlab/mmcv/tree/master/mmcv), and [Detectron2](https://github.com/facebookresearch/detectron2).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FPointcept%2FPointcept","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FPointcept%2FPointcept","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FPointcept%2FPointcept/lists"}