{"id":13442249,"url":"https://github.com/kabouzeid/point2vec","last_synced_at":"2025-03-16T21:30:34.116Z","repository":{"id":154483524,"uuid":"621462601","full_name":"kabouzeid/point2vec","owner":"kabouzeid","description":"[GCPR 2023 | CVPR 2023 Workshop] Self-Supervised Representation Learning on Point Clouds","archived":false,"fork":false,"pushed_at":"2023-10-11T11:29:35.000Z","size":1380,"stargazers_count":84,"open_issues_count":0,"forks_count":8,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-03-16T00:27:17.057Z","etag":null,"topics":["lightning","machine-learning","point-cloud","pytorch","self-supervised-learning","transformer","vision","vision-transformer"],"latest_commit_sha":null,"homepage":"https://point2vec.ka.codes","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kabouzeid.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-03-30T18:00:50.000Z","updated_at":"2025-02-27T22:54:49.000Z","dependencies_parsed_at":"2023-10-11T13:39:07.935Z","dependency_job_id":null,"html_url":"https://github.com/kabouzeid/point2vec","commit_stats":{"total_commits":6,"total_committers":1,"mean_commits":6.0,"dds":0.0,"last_synced_commit":"17f8ad80b78017f9fc74986a5b20f453abfca9b5"},"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kabouzeid%2Fpoint2vec","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kabouzeid%2Fpoint2vec/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kabouzeid%2Fpoint2vec/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kabouzeid%2Fpoint2vec/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kabouzeid","download_url":"https://codeload.github.com/kabouzeid/point2vec/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243830915,"owners_count":20354850,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["lightning","machine-learning","point-cloud","pytorch","self-supervised-learning","transformer","vision","vision-transformer"],"created_at":"2024-07-31T03:01:43.512Z","updated_at":"2025-03-16T21:30:33.203Z","avatar_url":"https://github.com/kabouzeid.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# point2vec\n\nSelf-Supervised Representation Learning on Point Clouds\n\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/point2vec-for-self-supervised-representation/3d-point-cloud-classification-on-modelnet40)](https://paperswithcode.com/sota/3d-point-cloud-classification-on-modelnet40?p=point2vec-for-self-supervised-representation)\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/point2vec-for-self-supervised-representation/3d-point-cloud-classification-on-scanobjectnn)](https://paperswithcode.com/sota/3d-point-cloud-classification-on-scanobjectnn?p=point2vec-for-self-supervised-representation)\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/point2vec-for-self-supervised-representation/3d-part-segmentation-on-shapenet-part)](https://paperswithcode.com/sota/3d-part-segmentation-on-shapenet-part?p=point2vec-for-self-supervised-representation)\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/point2vec-for-self-supervised-representation/few-shot-3d-point-cloud-classification-on-3)](https://paperswithcode.com/sota/few-shot-3d-point-cloud-classification-on-3?p=point2vec-for-self-supervised-representation)\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/point2vec-for-self-supervised-representation/few-shot-3d-point-cloud-classification-on-4)](https://paperswithcode.com/sota/few-shot-3d-point-cloud-classification-on-4?p=point2vec-for-self-supervised-representation)\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/point2vec-for-self-supervised-representation/few-shot-3d-point-cloud-classification-on-1)](https://paperswithcode.com/sota/few-shot-3d-point-cloud-classification-on-1?p=point2vec-for-self-supervised-representation)\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/point2vec-for-self-supervised-representation/few-shot-3d-point-cloud-classification-on-2)](https://paperswithcode.com/sota/few-shot-3d-point-cloud-classification-on-2?p=point2vec-for-self-supervised-representation)\n\n[[`Paper`](https://arxiv.org/abs/2303.16570)] [[`Project`](https://point2vec.ka.codes)] [[`BibTeX`](#citing-point2vec)]\n\n![architecture](https://user-images.githubusercontent.com/7303830/229210206-3df19b2c-f0bf-46ee-b0c2-d1ec1a3123d4.png)\n\n## Installation\n\n### 1. Dependencies\n\n- Python 3.10.4\n- CUDA 11.6\n- cuDNN 8.4.0\n- GCC \u003e= 6 and \u003c= 11.2.1\n\n```bash\npip install -U pip wheel\npip install torch torchvision -c requirements.txt --extra-index-url https://download.pytorch.org/whl/cu116\npip install -r requirements.txt\n```\n\n### 2. Datasets\n\nSee [DATASETS.md](DATASETS.md) for download instructions.\n\n### 3. Check (optional)\n\n```bash\npython -m point2vec.datasets.process.check # check if datasets are complete\n./scripts/test.sh # check if training works\n```\n\n## Model Zoo\n\n| Type                         | Dataset      | Evaluation                          | Config                                                                                       | Checkpoint                                                                                                                                    |\n| ---------------------------- | ------------ | ----------------------------------- | -------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |\n| Point2vec pre-trained        | ShapeNet     | -                                   | [config](configs/pretraining/shapenet.yaml)                                                  | [checkpoint](https://github.com/kabouzeid/point2vec/releases/download/paper/pre_point2vec-epoch.799-step.64800.ckpt)                          |\n| Classification fine-tuned    | ModelNet40   | **94.65** / **94.77** (OA / Voting) | [A](configs/classification/modelnet40.yaml) \u0026 [B](configs/classification/_pretrained.yaml)   | [checkpoint](https://github.com/kabouzeid/point2vec/releases/download/paper/fine_modelnet40-epoch.125-step.38682-val_acc.0.9465.ckpt)         |\n| Classification fine-tuned    | ScanObjectNN | **87.47** (OA)                      | [A](configs/classification/scanobjectnn.yaml) \u0026 [B](configs/classification/_pretrained.yaml) | [checkpoint](https://github.com/kabouzeid/point2vec/releases/download/paper/fine_scanobjectnn-epoch.146-step.52332-val_acc.0.8747.ckpt)       |\n| Part segmentation fine-tuned | ShapeNetPart | **84.59** (Cat. mIoU)               | [config](configs/part_segmentation/shapenetpart.yaml)                                        | [checkpoint](https://github.com/kabouzeid/point2vec/releases/download/paper/fine_shapenetpart-epoch.288-step.252586-val_cat_miou.0.8459.ckpt) |\n\n## Reproducing the results from the paper\n\nThe scripts in this section use Weights \u0026 Biases for logging, so it's important to log in once with `wandb login` before running them.\nCheckpoints will be saved to the `artifacts` directory.\n\n**A note on reproducibility:**\nWhile reproducing our results on most datasets is straightforward, achieving the same test accuracy on ModelNet40 is more complicated due to the high variance between runs (see also https://github.com/Pang-Yatian/Point-MAE/issues/5#issuecomment-1074886349, https://github.com/ma-xu/pointMLP-pytorch/issues/1#issuecomment-1062563404, https://github.com/CVMI-Lab/PAConv/issues/9#issuecomment-886612074).\nTo obtain comparable results on ModelNet40, you will likely need to experiment with a few different seeds.\nHowever, if you can precisely replicate our test environment, including installing CUDA 11.6, cuDNN 8.4.0, Python 3.10.4, and the dependencies listed in the `requirements.txt` file, as well as using a Volta GPU (e.g. Nvidia V100), you should be able to replicate our experiments exactly.\nUsing our _exact_ environment is necessary to ensure that you obtain the same random state during training, as a seed alone does _not_ guarantee reproducibility across different environments.\n\n### Point2vec pre-training on ShapeNet\n\n```bash\n./scripts/pretraining_shapenet.bash --data.in_memory true\n```\n\n\u003cdetails\u003e\n\u003csummary\u003eTraining curve\u003c/summary\u003e\n\n![pre-training](https://user-images.githubusercontent.com/7303830/230649527-8ccd2af9-6c3f-4495-8c58-5854aa2a3304.png)\n\n\u003c/details\u003e\n\n[[`Checkpoint`](https://github.com/kabouzeid/point2vec/releases/download/paper/pre_point2vec-epoch.799-step.64800.ckpt)]\n\n### Classification fine-tuning on ScanObjectNN\n\nReplace `XXXXXXXX` with the `WANDB_RUN_ID` from the pre-training run, or use the checkpoint from the model zoo.\n\n```bash\n./scripts/classification_scanobjectnn.bash --config configs/classification/_pretrained.yaml --model.pretrained_ckpt_path artifacts/point2vec-Pretraining-ShapeNet/XXXXXXXX/checkpoints/epoch=799-step=64800.ckpt\n```\n\n\u003cdetails\u003e\n\u003csummary\u003eTraining curve\u003c/summary\u003e\n\n![scanobjectnn](https://user-images.githubusercontent.com/7303830/230649569-b68d5389-3164-4b51-a31c-e512c3d9234b.png)\n\n\u003c/details\u003e\n\n[[`Checkpoint`](https://github.com/kabouzeid/point2vec/releases/download/paper/fine_scanobjectnn-epoch.146-step.52332-val_acc.0.8747.ckpt)]\n\n### Classification fine-tuning on ModelNet40\n\nReplace `XXXXXXXX` with the `WANDB_RUN_ID` from the pre-training run, or use the checkpoint from the model zoo.\n\n```bash\n./scripts/classification_modelnet40.bash --config configs/classification/_pretrained.yaml --model.pretrained_ckpt_path artifacts/point2vec-Pretraining-ShapeNet/XXXXXXXX/checkpoints/epoch=799-step=64800.ckpt --seed_everything 1\n```\n\n\u003cdetails\u003e\n\u003csummary\u003eTraining curve\u003c/summary\u003e\n\n![modelnet40](https://user-images.githubusercontent.com/7303830/230651790-3fa5c959-5a7f-4e35-b219-1424a21b9c2d.png)\n\n\u003c/details\u003e\n\n[[`Checkpoint`](https://github.com/kabouzeid/point2vec/releases/download/paper/fine_modelnet40-epoch.125-step.38682-val_acc.0.9465.ckpt)]\n\n### Voting on ModelNet40\n\nReplace `XXXXXXXX` with the `WANDB_RUN_ID` from the fine-tuning run, and `epoch=XXX-step=XXXXX-val_acc=0.XXXX.ckpt` with the best checkpoint from that run, or use the checkpoint from the model zoo.\n\n```bash\n./scripts/voting_modelnet40.bash --finetuned_ckpt_path artifacts/point2vec-Pretraining-ShapeNet/XXXXXXXX/checkpoints/epoch=XXX-step=XXXXX-val_acc=0.XXXX.ckpt\n```\n\n\u003cdetails\u003e\n\u003csummary\u003eVoting Process\u003c/summary\u003e\n\n![voting](https://user-images.githubusercontent.com/7303830/230651975-d356919d-b1ba-40a8-9d92-60b107f10e8d.png)\n\n\u003c/details\u003e\n\n### Classification fine-tuning on ModelNet Few-Shot\n\nReplace `XXXXXXXX` with the `WANDB_RUN_ID` from the pre-training run, or use the checkpoint from the model zoo.\nYou may also pass e.g. `--data.way 5` or `--data.shot 20` to select the desired m-way\u0026ndash;n-shot setting.\n\n```bash\nfor i in $(seq 0 9);\ndo\n    SLURM_ARRAY_TASK_ID=$i ./scripts/classification_modelnet_fewshot.bash --model.pretrained_ckpt_path artifacts/point2vec-Pretraining-ShapeNet/XXXXXXXX/checkpoints/epoch=799-step=64800.ckpt\ndone\n```\n\n### Part segmentation fine-tuning on ShapeNetPart\n\nReplace `XXXXXXXX` with the `WANDB_RUN_ID` from the pre-training run, or use the checkpoint from the model zoo.\n\n```bash\n./scripts/part_segmentation_shapenetpart.bash --model.pretrained_ckpt_path artifacts/point2vec-Pretraining-ShapeNet/XXXXXXXX/checkpoints/epoch=799-step=64800.ckpt\n```\n\n\u003cdetails\u003e\n\u003csummary\u003eTraining curve\u003c/summary\u003e\n\n![shapenetpart](https://user-images.githubusercontent.com/7303830/230651811-bd029146-aa30-4903-b011-5a0a8475cdda.png)\n\n\u003c/details\u003e\n\n[[`Checkpoint`](https://github.com/kabouzeid/point2vec/releases/download/paper/fine_shapenetpart-epoch.288-step.252586-val_cat_miou.0.8459.ckpt)]\n\n### Baselines\n\n\u003cdetails\u003e\n\n\u003csummary\u003eExpand\u003c/summary\u003e\n\n#### Data2vec\u0026ndash;pc\n\nReplace the pre-training step with:\n\n```bash\n./scripts/pretraining_shapenet.bash --data.in_memory true --model.learning_rate 2e-3 --model.decoder false --trainer.devices 2 --data.batch_size 1024 --model.fix_estimated_stepping_batches 16000\n```\n\nIf you only have a single GPU (and enough VRAM), you may replace `--trainer.devices 2 --data.batch_size 1024 --model.fix_estimated_stepping_batches 16000` with `--data.batch_size 2048`.\n\n[[`Checkpoint`](https://github.com/kabouzeid/point2vec/releases/download/paper/pre_data2vec-epoch.799-step.16000.ckpt)]\n\n#### From scratch\n\nSkip the pre-training step, and omit all occurences of `--config configs/classification/_pretrained.yaml` and `--model.pretrained_ckpt_path ...`.\n\n\u003c/details\u003e\n\n## Visualization\n\n![representations](https://user-images.githubusercontent.com/7303830/230671271-b8bb7dbb-1d21-4391-93f9-631d645587c0.png)\n\nWe use PCA to project the learned representations into RGB space.\nBoth a random initialization and data2vec\u0026ndash;pc pre-training show a fairly strong positional bias, whereas point2vec exhibits a stronger semantic grouping without being trained on downstream dense prediction tasks.\n\n## Citing point2vec\n\nIf you use point2vec in your research, please use the following BibTeX entry.\n\n```\n@inproceedings{abouzeid2023point2vec,\n  title={Point2Vec for Self-Supervised Representation Learning on Point Clouds},\n  author={Abou Zeid, Karim and Schult, Jonas and Hermans, Alexander and Leibe, Bastian},\n  journal={German Conference on Pattern Recognition (GCPR)},\n  year={2023},\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkabouzeid%2Fpoint2vec","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkabouzeid%2Fpoint2vec","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkabouzeid%2Fpoint2vec/lists"}