{"id":13442943,"url":"https://github.com/vLAR-group/OGC","last_synced_at":"2025-03-20T15:31:41.690Z","repository":{"id":61034072,"uuid":"544919689","full_name":"vLAR-group/OGC","owner":"vLAR-group","description":"🔥OGC in PyTorch (NeurIPS 2022)","archived":false,"fork":false,"pushed_at":"2024-03-11T11:45:15.000Z","size":111553,"stargazers_count":97,"open_issues_count":4,"forks_count":7,"subscribers_count":4,"default_branch":"main","last_synced_at":"2024-08-01T03:42:22.645Z","etag":null,"topics":["autonomous-vehicles","geometric-deep-learning","object-detection","object-segmentation","point-cloud","scene-flow-estimation","scene-understanding","unsupervised-learning"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vLAR-group.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-10-03T13:18:49.000Z","updated_at":"2024-07-16T03:09:46.000Z","dependencies_parsed_at":"2023-01-22T03:46:01.705Z","dependency_job_id":null,"html_url":"https://github.com/vLAR-group/OGC","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vLAR-group%2FOGC","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vLAR-group%2FOGC/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vLAR-group%2FOGC/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vLAR-group%2FOGC/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vLAR-group","download_url":"https://codeload.github.com/vLAR-group/OGC/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221772622,"owners_count":16878142,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["autonomous-vehicles","geometric-deep-learning","object-detection","object-segmentation","point-cloud","scene-flow-estimation","scene-understanding","unsupervised-learning"],"created_at":"2024-07-31T03:01:53.647Z","updated_at":"2024-10-28T03:31:27.953Z","avatar_url":"https://github.com/vLAR-group.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"[![arXiv](https://img.shields.io/badge/arXiv-2210.04458-b31b1b.svg)](https://arxiv.org/abs/2210.04458)\n![code visitors](https://visitor-badge.glitch.me/badge?page_id=vLAR-group/OGC)\n[![License CC BY-NC-SA 4.0](https://img.shields.io/badge/license-CC4.0-blue.svg)](https://creativecommons.org/licenses/by-nc-sa/4.0/legalcode)\n[![Twitter Follow](https://img.shields.io/twitter/follow/vLAR_Group?style=social)](https://twitter.com/vLAR_Group)\n\n## OGC: Unsupervised 3D Object Segmentation from Rigid Dynamics of Point Clouds (NeurIPS 2022)\n[Ziyang Song](https://szy-young.github.io/), [Bo Yang](https://yang7879.github.io/)\n\n\n### !Updates:\n\nNew contents in the extension of OGC have been included in this repo:\n- [x] Support for Waymo Open dataset.\n- [x] A multi-frame co-segmentation algorithm to leverage multi-frame inputs in testing.\n- [ ] Support for SparseConv \u0026 Point Transformer backbones in the object segmentation network.\n\n\n### Overview\n\nWe propose the first unsupervised 3D object segmentation method, learning from dynamic motion patterns in point cloud sequences.\n\n\u003cimg src=\"figures/overview.png\" alt=\"drawing\" width=800/\u003e\n\nOur method demonstrates promising results on various scenarios:\n- **Object part instance segmentation**\n\n\u003cimg src=\"figures/sapien.gif\" alt=\"drawing\" width=1000/\u003e\n\n- **Object segmentation in indoor scenes**\n\n\u003cimg src=\"figures/indoor.gif\" alt=\"drawing\" width=1000/\u003e\n\n- **Object segmentation in outdoor scenes**\n\n\u003cimg src=\"figures/outdoor.gif\" alt=\"drawing\" width=1000/\u003e\n\n### [Full demo (Youtube)](https://www.youtube.com/watch?v=dZBjvKWJ4K0)\n\n\u003cp align=\"center\"\u003e \u003ca href=\"https://www.youtube.com/watch?v=dZBjvKWJ4K0\"\u003e\u003cimg src=\"figures/OGC_thumbnail.png\" width=500\u003e\u003c/a\u003e \u003c/p\u003e\n\n## 1. Environment\n\nPlease first install a **GPU-supported pytorch** version which fits your machine.\nWe have tested with pytorch 1.9.0.\n\nInstall PointNet2 CPP lib:\n```shell script\ncd pointnet2\npython setup.py install\ncd ..\n```\n\nInstall other dependencies:\n```shell script\npip install -r requirements\n```\n\n**(Optional)** Install Open3D for the visualization of point cloud segmentation:\n```shell script\npip install open3d\n```\n\n\n## 2. Data preparation\n\n### (1) SAPIEN\n\nPlease download from links provided by [MultibodySync](https://github.com/huangjh-pub/multibody-sync):\n- Train+Val (`mbs-shapepart`): [Google Drive](https://drive.google.com/file/d/1aGTn-PYxLjnhj9UKlv4YFV3Mt1E3ftci/view?usp=sharing)\n- Test (`mbs-sapien`): [Google Drive](https://drive.google.com/file/d/1HR2X0DjgXLwp8K5n2nsvfGTcDMSckX5Z/view?usp=sharing)\n\nThen put them into your `${SAPIEN}` path.\n\n### (2) OGC-DR (Dynamic Room) \u0026 OGC-DRSV (Single-View Dynamic Room)\n\nPlease download the complete datasets from links below:\n- OGC-DR: [Dropbox](https://www.dropbox.com/s/guni0d87phls4sd/OGC_DynamicRoom.zip?dl=0)\n- OGC-DRSV: [Dropbox](https://www.dropbox.com/s/67rzl2ku37e2urt/OGC_DynamicRoom_SingleView.zip?dl=0)\n\nAlternatively, you can generate the dataset by yourself.\n\n**OGC-DR:** Please first download the [ShapeNet Core v1](https://shapenet.cs.stanford.edu/shapenet/obj-zip/ShapeNetCore.v1.zip).\nSelect the archives according to object categories specified in `data_prepare/ogcdr/meta.yaml` and unzip them into your `${OGC_DR}/ShapeNet_mesh` path.\nThen run the following script to generate the dataset.\n```shell script\npython data_prepare/ogcdr/build_ogcdr.py ${OGC_DR}\n```\n\n**OGC-DRSV:** Run the following script to collect single depth scans on OGC-DR mesh models and generate incomplete point cloud dataset OGC-DRSV.\n```shell script\npython data_prepare/ogcdrsv/build_ogcdrsv.py --src_root ${OGC_DR} --dest_root ${OGC_DRSV}\n```\nCollect groundtruth segmentation for OGC-DRSV and downsample the point clouds:\n```shell script\npython data_prepare/ogcdrsv/collect_segm.py --src_root ${OGC_DR} --dest_root ${OGC_DRSV}\n```\n\n### (3) KITTI-SF (Scene Flow)\n\nPlease first download:\n- [scene flow](https://s3.eu-central-1.amazonaws.com/avg-kitti/data_scene_flow.zip) and [calibration files](https://s3.eu-central-1.amazonaws.com/avg-kitti/data_scene_flow_calib.zip), from [KITTI Scene Flow Evaluation 2015](http://www.cvlibs.net/datasets/kitti/eval_scene_flow.php)\n- [semantic instance segmentation label](https://s3.eu-central-1.amazonaws.com/avg-kitti/data_semantics.zip), from [KITTI Semantic Instance Segmentation Evaluation](http://www.cvlibs.net/datasets/kitti/eval_instance_seg.php?benchmark=instanceSeg2015)\n\nMerge the `training` folder of them in your `${KITTI_SF}` path.\nThen run the following script to unproject **disparity, optical flow, 2D segmentation** into **point cloud, scene flow, 3D segmentation**:\n```shell script\npython data_prepare/kittisf/process_kittisf.py ${KITTI_SF}\n```\n\nFinally, downsample all point clouds to 8192-point:\n```shell script\npython data_prepare/kittisf/downsample_kittisf.py ${KITTI_SF} --save_root ${KITTI_SF}_downsampled\n# After extracting flow estimations in the following, come back here to downsample flow estimations\npython data_prepare/kittisf/downsample_kittisf.py ${KITTI_SF} --save_root ${KITTI_SF}_downsampled --predflow_path flowstep3d\n```\n`${KITTI_SF}_downsampled` will be the path for the downsampled dataset.\n\n### (4) KITTI-Det (Detection)\n\nPlease first download the following items from [KITTI 3D Object Detection Evaluation 2017](http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=3d):\n- [velodyne point clouds](https://s3.eu-central-1.amazonaws.com/avg-kitti/data_object_velodyne.zip)\n- [left color image](https://s3.eu-central-1.amazonaws.com/avg-kitti/data_object_image_2.zip)\n- [calibration files](https://s3.eu-central-1.amazonaws.com/avg-kitti/data_object_calib.zip)\n- [object bounding box label](https://s3.eu-central-1.amazonaws.com/avg-kitti/data_object_label_2.zip)\n\nMerge the `training` folder of them in your `${KITTI_DET}` path.\nThen run the following script to extract 8192-point **front-view** point cloud, and obtain segmentation from bounding box annotations.\n```shell script\npython data_prepare/kittidet/process_kittidet.py ${KITTI_DET}\n```\n\n### (5) SemanticKITTI\n\nPlease first download the following iterms from [SemanticKITTI](http://www.semantic-kitti.org/dataset.html#download):\n- [velodyne point clouds](http://www.cvlibs.net/download.php?file=data_odometry_velodyne.zip)\n- [calibration files](http://www.cvlibs.net/download.php?file=data_odometry_calib.zip)\n- [segmentation label](www.semantic-kitti.org/assets/data_odometry_labels.zip)\n\nMerge the `velodyne`, `labels` and `calib.txt` of each sequence. The organized dataset should be as follows:\n```shell script\nSemanticKITTI\n└── sequences\n    └── 00\n    │   ├── velodyne\n    │   ├── labels\n    │   └── calib.txt\n    └── 01\n    ...\n```\nThen run the following script to extract 8192-point **front-view** point cloud, and obtain segmentation from panoptic annotations.\n```shell script\npython data_prepare/semantickitti/process_semantickitti.py ${SEMANTIC_KITTI}\n```\n\n### (6) Waymo Open\n\nPlease first register at the [Waymo Open Dataset](https://waymo.com/open/download/), then download the [raw data](https://console.cloud.google.com/storage/browser/waymo_open_dataset_scene_flow;tab=objects?prefix=\u0026forceOnObjectsSortingFiltering=false).\n1. Process raw data into point clouds. Please carefully follow the instructions in [OpenPCDet](https://github.com/open-mmlab/OpenPCDet/blob/master/docs/GETTING_STARTED.md#waymo-open-dataset). \nYou need to install [waymo-open-dataset](https://github.com/waymo-research/waymo-open-dataset) and [OpenPCDet](https://github.com/open-mmlab/OpenPCDet) here.\n2. Extract groundtruth scene flow. Please follow the instructions in [DCA-SRSFE](https://github.com/leolyj/DCA-SRSFE).\n3. Generate groundtruth segmentations from bounding box annotations:\n```shell script\npython data_prepare/waymo/process_waymo.py --data_root $WAYMO_RAW --save_root $WAYMO --split $SPLIT\n```\n4. Downsample all point clouds to 8192-point:\n```shell script\npython data_prepare/waymo/downsample_waymo.py --data_root $WAYMO --save_root ${WAYMO}_downsampled --split $SPLIT\n# After extracting flow estimations in the following, come back here to downsample flow estimations\npython data_prepare/waymo/downsample_waymo.py --data_root $WAYMO --save_root ${WAYMO}_downsampled --split $SPLIT --predflow_path flowstep3d\n```\n\n\n## 3. Pre-trained models\n\nYou can download all our pre-trained models from [Dropbox](https://www.dropbox.com/s/k4hv71952i0yrye/OGC_ckpt.zip?dl=0) (including self-supervised scene flow networks, and unsupervised/supervised segmentation networks) and extract them to `./ckpt`.\n\n\n## 4. Scene flow estimation\n\n### Train\n\nTrain the self-supervised scene flow networks:\n```shell script\n# SAPIEN \npython train_flow.py config/flow/sapien/sapien_unsup.yaml\n# OGC-DR \npython train_flow.py config/flow/ogcdr/ogcdr_unsup.yaml\n# OGC-DRSV \npython train_flow.py config/flow/ogcdrsv/ogcdrsv_unsup.yaml\n```\nFor KITTI-SF dataset, we directly employ the pre-trained model released by [FlowStep3D](https://github.com/yairkit/flowstep3d).\n\n### Test\n\nEvaluate and save the scene flow estimations.\n```shell script\n# SAPIEN \npython test_flow.py config/flow/sapien/sapien_unsup.yaml --split ${SPLIT} --save\n# OGC-DR \npython test_flow.py config/flow/ogcdr/ogcdr_unsup.yaml --split ${SPLIT} --test_batch_size 12 --test_model_iters 5 --save\n# OGC-DRSV \npython test_flow.py config/flow/ogcdrsv/ogcdrsv_unsup.yaml --split ${SPLIT} --test_batch_size 12 --test_model_iters 5 --save\n# KITTI-SF \npython test_flow_kittisf.py config/flow/kittisf/kittisf_unsup.yaml --split ${SPLIT} --test_model_iters 5 --save\n# Waymo Open\npython test_flow_waymo.py config/flow/waymo/kittisf_unsup.yaml --split ${SPLIT} --use_odometry --bound --test_model_iters 5 --save\n```\n`${SPLIT}` can be train/val/test for SAPIEN \u0026 OGC-DR/OGC-DRSV, train/val for KITTI-SF \u0026 Waymo Open.\n\n\n## 5. Unsupervised segmentation\n\n### Train\n\nAlternate the segmentation network training and scene flow improvement for *R* rounds. \nIn each `${ROUND}` (starting from 1):\n```shell script\n# SAPIEN: first R-1 rounds\npython train_seg.py config/seg/sapien/sapien_unsup_woinv.yaml --round ${ROUND}\npython oa_icp.py config/seg/sapien/sapien_unsup_woinv.yaml --split ${SPLIT} --round ${ROUND} --save\n# SAPIEN: the last round\npython train_seg.py config/seg/sapien/sapien_unsup.yaml --round ${ROUND}\n\n# OGC-DR: first R-1 rounds\npython train_seg.py config/seg/ogcdr/ogcdr_unsup_woinv.yaml --round ${ROUND}\npython oa_icp.py config/seg/ogcdr/ogcdr_unsup_woinv.yaml --split ${SPLIT} --round ${ROUND} --test_batch_size 24 --save\n# OGC-DR: the last round\npython train_seg.py config/seg/ogcdr/ogcdr_unsup.yaml --round ${ROUND}\n\n# OGC-DRSV: first R-1 rounds\npython train_seg.py config/seg/ogcdrsv/ogcdrsv_unsup_woinv.yaml --round ${ROUND}\npython oa_icp.py config/seg/ogcdrsv/ogcdrsv_unsup_woinv.yaml --split ${SPLIT} --round ${ROUND} --test_batch_size 24 --save\n# OGC-DRSV: the last round\npython train_seg.py config/seg/ogcdrsv/ogcdrsv_unsup.yaml --round ${ROUND}\n\n# KITTI-SF: first R-1 rounds\npython train_seg.py config/seg/kittisf/kittisf_unsup_woinv.yaml --round ${ROUND}\npython oa_icp.py config/seg/kittisf/kittisf_unsup_woinv.yaml --split ${SPLIT} --round ${ROUND} --test_batch_size 4 --save\n# KITTI-SF: the last round\npython train_seg.py config/seg/kittisf/kittisf_unsup.yaml --round ${ROUND}\n\n# Waymo Open: only 1 round\npython train_seg_waymo.py config/seg/waymo/waymo_unsup.yaml --round 1\n```\nWhen performing scene flow improvement, `${SPLIT}` needs to traverse train/val/test for SAPIEN \u0026 OGC-DR/OGC-DRSV, train/val for KITTI-SF.\n\n### Test\n\n```shell script\n# SAPIEN \npython test_seg.py config/seg/sapien/sapien_unsup.yaml --split test --round ${ROUND}\n# OGC-DR \npython test_seg.py config/seg/ogcdr/ogcdr_unsup.yaml --split test --round ${ROUND} --test_batch_size 16\n# OGC-DRSV \npython test_seg.py config/seg/ogcdrsv/ogcdrsv_unsup.yaml --split test --round ${ROUND} --test_batch_size 16\n# KITTI-SF \npython test_seg.py config/seg/kittisf/kittisf_unsup.yaml --split val --round ${ROUND} --test_batch_size 8\n# KITTI-Det \npython test_seg.py config/seg/kittidet/kittisf_unsup.yaml --split val --round ${ROUND} --test_batch_size 8\n# SemanticKITTI \npython test_seg.py config/seg/semantickitti/kittisf_unsup.yaml --round ${ROUND} --test_batch_size 8\n# Waymo Open\npython test_seg_waymo.py config/seg/waymo/waymo_unsup.yaml --split val --round 1 --test_batch_size 8\n```\n`${ROUND}` can be 1/2/3/..., and we take **2 rounds** as default in our experiments.\nSpecify `--save` to save the estimations. \nSpecify `--visualize` for qualitative evaluation mode.\n\n### Test the scene flow improvement\n\nYour can follow the evaluation settings of FlowStep3D to test the improved flow, and see how our method push the boundaries of unsupervised scene flow estimation:\n```shell script\n# Refine the scene flow estimations\npython oa_icp.py config/seg/kittisf/kittisf_unsup.yaml --split train --round 2 --test_batch_size 4 --save --saveflow_path flowstep3d_for-benchmark\npython oa_icp.py config/seg/kittisf/kittisf_unsup.yaml --split val --round 2 --test_batch_size 4 --save --saveflow_path flowstep3d_for-benchmark\n# Evaluate\npython test_flow_kittisf_benchmark.py config/flow/kittisf/kittisf_unsup.yaml\n```\n\n\n## 6. Supervised segmentation\n\nYou can train the segmentation network with full annotations.\n\n### Train\n\n```shell script\n# SAPIEN \npython train_seg_sup.py config/seg/sapien/sapien_sup.yaml\n# OGC-DR\npython train_seg_sup.py config/seg/ogcdr/ogcdr_sup.yaml\n# OGC-DRSV\npython train_seg_sup.py config/seg/ogcdrsv/ogcdrsv_sup.yaml\n# KITTI-SF \npython train_seg_sup.py config/seg/kittisf/kittisf_sup.yaml\n# KITTI-Det \npython train_seg_sup.py config/seg/kittidet/kittidet_sup.yaml\n# Waymo Open\npython train_seg_waymo_sup.py config/seg/waymo/waymo_sup.yaml \n```\n\n### Test \n\n```shell script\n# SAPIEN \npython test_seg.py config/seg/sapien/sapien_sup.yaml --split test\n# OGC-DR\npython test_seg.py config/seg/ogcdr/ogcdr_sup.yaml --split test --test_batch_size 16\n# OGC-DRSV\npython test_seg.py config/seg/ogcdrsv/ogcdrsv_sup.yaml --split test --test_batch_size 16\n# KITTI-SF \npython test_seg.py config/seg/kittisf/kittisf_sup.yaml --split val --test_batch_size 8\n# KITTI-Det \npython test_seg.py config/seg/kittidet/kittisf_sup.yaml --split val --test_batch_size 8\npython test_seg.py config/seg/kittidet/kittidet_sup.yaml --split val --test_batch_size 8\n# SemanticKITTI \npython test_seg.py config/seg/semantickitti/kittisf_sup.yaml --test_batch_size 8\n# Waymo Open\npython test_seg_waymo.py config/seg/waymo/waymo_sup.yaml --split val --test_batch_size 8\n```\n\n\n## 7. Multi-frame co-segmentation\n\n```shell script\n# SAPIEN dataset\npython vote.py config/seg/sapien/sapien_unsup.yaml --split test --round $ROUND\n# OGC-DR dataset\npython vote.py config/seg/ogcdr/ogcdr_unsup.yaml --split test --round $ROUND --test_batch_size 16\n# OGC-DRSV dataset\npython vote.py config/seg/ogcdrsv/ogcdrsv_unsup.yaml --split test --round $ROUND --test_batch_size 16\n```\nUse `--time_window_size` to control the frame range used for multi-frame co-segmentation. \n\n\n## Citation\nIf you find our work useful in your research, please consider citing:\n    \n    @article{song2022,\n      title={{OGC: Unsupervised 3D Object Segmentation from Rigid Dynamics of Point Clouds}},\n      author={Song, Ziyang and Yang, Bo},\n      journal={NeurIPS},\n      year={2022}\n    }\n\n\n## Acknowledgements\n\nSome code is borrowed from:\n- [MultibodySync](https://github.com/huangjh-pub/multibody-sync) \n- [FlowStep3D](https://github.com/yairkit/flowstep3d)\n- [Convolutional Occupancy Networks](https://github.com/autonomousvision/convolutional_occupancy_networks)\n- [KITTI Object data transformation and visualization](https://github.com/kuixu/kitti_object_vis)\n- [ICP](https://github.com/ClayFlannigan/icp)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FvLAR-group%2FOGC","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FvLAR-group%2FOGC","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FvLAR-group%2FOGC/lists"}