{"id":18600799,"url":"https://github.com/autonomousvision/shape_as_points","last_synced_at":"2025-04-05T07:06:41.097Z","repository":{"id":38223076,"uuid":"418426267","full_name":"autonomousvision/shape_as_points","owner":"autonomousvision","description":"[NeurIPS'21] Shape As Points: A Differentiable Poisson Solver","archived":false,"fork":false,"pushed_at":"2022-03-25T12:29:55.000Z","size":25351,"stargazers_count":601,"open_issues_count":7,"forks_count":42,"subscribers_count":21,"default_branch":"main","last_synced_at":"2025-03-29T06:07:41.575Z","etag":null,"topics":["3d-reconstruction","hybrid-representation","implicit-representation","neural-fields","points","poisson-solver","spectral-methods","surface-reconstruction"],"latest_commit_sha":null,"homepage":"https://pengsongyou.github.io/sap","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/autonomousvision.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-10-18T09:14:18.000Z","updated_at":"2025-03-27T16:59:42.000Z","dependencies_parsed_at":"2022-07-12T01:31:04.233Z","dependency_job_id":null,"html_url":"https://github.com/autonomousvision/shape_as_points","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/autonomousvision%2Fshape_as_points","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/autonomousvision%2Fshape_as_points/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/autonomousvision%2Fshape_as_points/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/autonomousvision%2Fshape_as_points/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/autonomousvision","download_url":"https://codeload.github.com/autonomousvision/shape_as_points/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247299832,"owners_count":20916190,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-reconstruction","hybrid-representation","implicit-representation","neural-fields","points","poisson-solver","spectral-methods","surface-reconstruction"],"created_at":"2024-11-07T02:05:39.464Z","updated_at":"2025-04-05T07:06:41.078Z","avatar_url":"https://github.com/autonomousvision.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Shape As Points (SAP)\n\n### [**Paper**](https://arxiv.org/abs/2106.03452) | [**Project Page**](https://pengsongyou.github.io/sap) | [**Short Video (6 min)**](https://youtu.be/FL8LMk_qWb4) | [**Long Video (12 min)**](https://youtu.be/TgR0NvYty0A)  \u003cbr\u003e\n\n![](./media/teaser_wheel.gif)\n\nThis repository contains the implementation of the paper:\n\nShape As Points: A Differentiable Poisson Solver  \n[Songyou Peng](https://pengsongyou.github.io/), [Chiyu \"Max\" Jiang](https://www.maxjiang.ml/), [Yiyi Liao](https://yiyiliao.github.io/), [Michael Niemeyer](https://m-niemeyer.github.io/), [Marc Pollefeys](https://www.inf.ethz.ch/personal/pomarc/) and [Andreas Geiger](http://www.cvlibs.net/)  \n**NeurIPS 2021 (Oral)**  \n\n\nIf you find our code or paper useful, please consider citing\n```bibtex\n@inproceedings{Peng2021SAP,\n author    = {Peng, Songyou and Jiang, Chiyu \"Max\" and Liao, Yiyi and Niemeyer, Michael and Pollefeys, Marc and Geiger, Andreas},\n title     = {Shape As Points: A Differentiable Poisson Solver},\n booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},\n year      = {2021}}\n```\n\n\n## Installation\nFirst you have to make sure that you have all dependencies in place.\nThe simplest way to do so, is to use [anaconda](https://www.anaconda.com/). \n\nYou can create an anaconda environment called `sap` using\n```\nconda env create -f environment.yaml\nconda activate sap\n```\n\nNext, you should install [PyTorch3D](https://pytorch3d.org/) (**\u003e=0.5**) yourself from the [official instruction](https://github.com/facebookresearch/pytorch3d/blob/master/INSTALL.md#3-install-wheels-for-linux).  \n\nAnd install [PyTorch Scatter](https://github.com/rusty1s/pytorch_scatter):\n```sh\nconda install pytorch-scatter -c pyg\n```\n\n\n## Demo - Quick Start\n\nFirst, run the script to get the demo data:\n\n```bash\nbash scripts/download_demo_data.sh\n```\n\n### Optimization-based 3D Surface Reconstruction\n\nYou can now quickly test our code on the data shown in the teaser. To this end, simply run:\n\n```python\npython optim_hierarchy.py configs/optim_based/teaser.yaml\n```\nThis script should create a folder `out/demo_optim` where the output meshes and the optimized oriented point clouds under different grid resolution are stored.\n\nTo visualize the optimization process on the fly, you can set `o3d_show: Frue` in [`configs/optim_based/teaser.yaml`](https://github.com/autonomousvision/shape_as_points/tree/main/configs/optim_based/teaser.yaml).\n\n### Learning-based 3D Surface Reconstruction\nYou can also test SAP on another application where we can reconstruct from unoriented point clouds with either **large noises** or **outliers** with a learned network.\n\n![](./media/results_large_noise.gif)\n\nFor the point clouds with large noise as shown above, you can run:\n```python\npython generate.py configs/learning_based/demo_large_noise.yaml\n```\nThe results can been found at `out/demo_shapenet_large_noise/generation/vis`.\n\n![](./media/results_outliers.gif)\nAs for the point clouds with outliers, you can run:\n```python\npython generate.py configs/learning_based/demo_outlier.yaml\n```\nYou can find the reconstrution on `out/demo_shapenet_outlier/generation/vis`.\n\n\n## Dataset\n\nWe have different dataset for our optimization-based and learning-based settings.\n\n### Dataset for Optimization-based Reconstruction\nHere we consider the following dataset: \n- [Thingi10K](https://arxiv.org/abs/1605.04797) (synthetic)\n- [Surface Reconstruction Benchmark (SRB)](https://github.com/fwilliams/deep-geometric-prior) (real scans)\n- [MPI Dynamic FAUST](https://dfaust.is.tue.mpg.de/) (real scans)\n\nPlease cite the corresponding papers if you use the data.\n\nYou can download the processed dataset (~200 MB) by running:\n```bash\nbash scripts/download_optim_data.sh\n```\n\n### Dataset for Learning-based Reconstruction  \nWe train and evaluate on [ShapeNet](https://shapenet.org/).\nYou can download the processed dataset (~220 GB) by running:\n```bash\nbash scripts/download_shapenet.sh\n``` \nAfter, you should have the dataset in `data/shapenet_psr` folder.\n\nAlternatively, you can also preprocess the dataset yourself. To this end, you can: \n* first download the preprocessed dataset (73.4 GB) by running [the script](https://github.com/autonomousvision/occupancy_networks#preprocessed-data) from Occupancy Networks.\n* check [`scripts/process_shapenet.py`](https://github.com/autonomousvision/shape_as_points/tree/main/scripts/process_shapenet.py), modify the base path and run the code\n\n\n## Usage for Optimization-based 3D Reconstruction  \n\nFor our optimization-based setting, you can consider running with a coarse-to-fine strategy:\n```python\npython optim_hierarchy.py configs/optim_based/CONFIG.yaml\n```\nWe start from a grid resolution of 32^3, and increase to 64^3, 128^3 and finally 256^3.\n\nAlternatively, you can also run on a single resolution with:\n\n```python\npython optim.py configs/optim_based/CONFIG.yaml\n```\nYou might need to modify the `CONFIG.yaml` accordingly.\n\n## Usage for Learning-based 3D Reconstruction\n\n### Mesh Generation \nTo generate meshes using a trained model, use\n```python\npython generate.py configs/learning_based/CONFIG.yaml\n```\nwhere you replace `CONFIG.yaml` with the correct config file.\n\n#### Use a pre-trained model\nThe easiest way is to use a pre-trained model. You can do this by using one of the config files with postfix `_pretrained`.\n\nFor example, for 3D reconstruction from point clouds with outliers using our model with 7x offsets, you can simply run:\n```python\npython generate.py configs/learning_based/outlier/ours_7x_pretrained.yaml\n```\n\nThe script will automatically download the pretrained model and run the generation. You can find the outputs in the `out/.../generation_pretrained` folders.\n\n**Note** config files are only for generation, not for training new models: when these configs are used for training, the model will be trained from scratch, but during inference our code will still use the pretrained model.\n\nWe provide the following pretrained models:\n```\nnoise_small/ours.pt\nnoise_large/ours.pt\noutlier/ours_1x.pt\noutlier/ours_3x.pt\noutlier/ours_5x.pt\noutlier/ours_7x.pt\noutlier/ours_3plane.pt\n```\n\n\n### Evaluation\nTo evaluate a trained model, we provide the script [`eval_meshes.py`](https://github.com/autonomousvision/shape_as_points/blob/main/eval_meshes.py). You can run it using:\n```python\npython eval_meshes.py configs/learning_based/CONFIG.yaml\n```\nThe script takes the meshes generated in the previous step and evaluates them using a standardized protocol. The output will be written to `.pkl` and `.csv` files in the corresponding generation folder that can be processed using [pandas](https://pandas.pydata.org/).\n\n### Training\n\nFinally, to train a new network from scratch, simply run:\n```python\npython train.py configs/learning_based/CONFIG.yaml\n```\nFor available training options, please take a look at `configs/default.yaml`.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fautonomousvision%2Fshape_as_points","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fautonomousvision%2Fshape_as_points","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fautonomousvision%2Fshape_as_points/lists"}