{"id":20862433,"url":"https://github.com/nmwsharp/diffusion-net","last_synced_at":"2025-04-05T23:06:55.013Z","repository":{"id":41403858,"uuid":"338396392","full_name":"nmwsharp/diffusion-net","owner":"nmwsharp","description":"Pytorch implementation of DiffusionNet for fast and robust learning on 3D surfaces like meshes or point clouds.","archived":false,"fork":false,"pushed_at":"2022-08-08T09:09:23.000Z","size":28713,"stargazers_count":442,"open_issues_count":24,"forks_count":61,"subscribers_count":11,"default_branch":"master","last_synced_at":"2025-03-29T22:09:45.737Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2012.00888","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nmwsharp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-02-12T18:10:48.000Z","updated_at":"2025-03-22T02:14:41.000Z","dependencies_parsed_at":"2022-08-10T02:07:24.636Z","dependency_job_id":null,"html_url":"https://github.com/nmwsharp/diffusion-net","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nmwsharp%2Fdiffusion-net","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nmwsharp%2Fdiffusion-net/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nmwsharp%2Fdiffusion-net/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nmwsharp%2Fdiffusion-net/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nmwsharp","download_url":"https://codeload.github.com/nmwsharp/diffusion-net/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247411231,"owners_count":20934653,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-18T05:23:53.077Z","updated_at":"2025-04-05T23:06:54.982Z","avatar_url":"https://github.com/nmwsharp.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"**DiffusionNet** is a general-purpose method for deep learning on surfaces such as 3D triangle meshes and point clouds. It is well-suited for tasks like segmentation, classification, feature extraction, etc.\n\nWhy try DiffusionNet?\n- It is _efficient_ and _scalable_. On a single GPU, we can easily train on meshes of 20k vertices, and infer on meshes with 200k vertices. One-time preprocessing takes a few seconds in the former case, and about a minute in the latter.\n- It is _sampling agnostic_. Many graph-based mesh learning approaches tend to overfit to mesh connectivity, and can output nonsense when you run them on meshes that are triangulated differently from the training set. With DiffusionNet we can intermingle different triangulations and very coarse or fine meshes without issue. No special regularization or data augmentation needed!\n- It is _representation agnostic_. For instance, you can train on a mesh and infer on a point cloud, or mix meshes and point clouds in the training set.\n- It is _robust_. DiffusionNet avoids potentially-brittle geometric operations, and does not impose any assumptions such as manifoldness, etc.\n- It is _data efficient_. DiffusionNet can learn from 10s of models, even without any data augmentation.\n\nDiffusionNet is described in the paper [\"DiffusionNet: Discretization Agnostic Learning on Surfaces\"](https://arxiv.org/abs/2012.00888), by \n- [Nicholas Sharp](https://nmwsharp.com/)\n- Souhaib Attaiki\n- [Keenan Crane](http://keenan.is/here)\n- [Maks Ovsjanikov](http://www.lix.polytechnique.fr/~maks/)\n\n![network diagram](https://github.com/nmwsharp/diffusion-net/blob/master/media/diagram.jpg)\n\n## Outline\n\n  - `diffusion_net/src` implementation of the method, including preprocessing, layers, etc\n  - `experiments` examples and scripts to reproduce experiments from the DiffusionNet paper\n  - `environment.yml` A conda environment file which can be used to install packages.\n\n\n## Prerequisites\n\nDiffusionNet depends on pytorch, as well as a handful of other fairly typical numerical packages. These can usually be installed manually without much trouble, but alternately a conda environment file is also provided (see conda documentation for additional instructions). These package versions were tested with CUDA 10.1 and 11.1. \n\n```\nconda env create --name diffusion_net -f environment.yml\n```\n\nThe code assumes a GPU with CUDA support. DiffusionNet has minimal memory requirements; \u003e4GB GPU memory should be sufficient. \n\n## Applying DiffusionNet to your task\n\nThe `DiffusionNet` class can be applied to meshes or point clouds. The basic recipe looks like:\n\n```python\nimport diffusion_net\n\n# Here we use Nx3 positions as features. Any other features you might have will work!\n# See our experiments for the use of of HKS features, which are naturally \n# invariant to (isometric) deformations.\nC_in = 3\n\n# Output dimension (e.g., for a 10-class segmentation problem)\nC_out = 10 \n\n# Create the model\nmodel = diffusion_net.layers.DiffusionNet(\n            C_in=C_in,\n            C_out=n_class,\n            C_width=128, # internal size of the diffusion net. 32 -- 512 is a reasonable range\n            last_activation=lambda x : torch.nn.functional.log_softmax(x,dim=-1), # apply a last softmax to outputs \n                                                                                  # (set to default None to output general values in R^{N x C_out})\n            outputs_at='vertices')\n\n# An example epoch loop.\n# For a dataloader example see experiments/human_segmentation_original/human_segmentation_original_dataset.py\nfor sample in your_dataset:\n    \n    verts = sample.vertices  # (Vx3 array of vertices)\n    faces = sample.faces     # (Fx3 array of faces, None for point cloud) \n    \n    # center and unit scale\n    verts = diffusion_net.geometry.normalize_positions(verts)\n    \n    # Get the geometric operators needed to evaluate DiffusionNet. This routine \n    # automatically populates a cache, precomputing only if needed.\n    # TIP: Do this once in a dataloader and store in memory to further improve \n    # performance; see examples.\n    frames, mass, L, evals, evecs, gradX, gradY = \\\n        get_operators(verts, faces, op_cache_dir='my/cache/directory/')\n    \n    # this example uses vertex positions as features \n    features = verts\n    \n    # Forward-evaluate the model\n    # preds is a NxC_out array of values\n    outputs = model(features, mass, L=L, evals=evals, evecs=evecs, gradX=gradX, gradY=gradY, faces=faces)\n    \n    # Now do whatever you want! Apply your favorite loss function, \n    # backpropgate with loss.backward() to train the DiffusionNet, etc. \n```\n\nSee the examples in `experiments/` for complete examples, including dataloaders, other features, optimizers, etc. Please feel free to file an issue to discuss applying DiffusionNet to your problem!\n\n### Tips and Tricks\n\nBy default, DiffusionNet uses _spectral acceleration_ for fast performance, which requires some CPU-based precomputation to compute operators \u0026 eigendecompositions for each input, which can take a few seconds for moderately sized inputs. DiffusionNet will be fastest if this precomputation only needs to be performed once for the dataset, rather than for each input. \n\n- If you are learning on a **template mesh**, consider precomputing operators for the _reference pose_ of the template, but then using xyz the coordinates of the _deformed pose_ as inputs to the network. This is a slight approximation, but will make DiffusionNet very fast, since the precomputed operators are shared among all poses.\n- If  you need **data augmentation**, try to apply augmentations _after_ computing operators whenever possible. For instance, in our examples, we apply random rotation to positions, but only _after_ computing operators. Note that we find common augmentations such as slightly skewing/scaling/subsampling inputs are generally unnecessary with DiffusionNet.\n\n### Thanks\n\nParts of this work were generously supported by the Fields Institute for Mathematics, the Vector Institute, ERC Starting Grant No. 758800 (EXPROTEA) the ANR AI Chair AIGRETTE, a Packard Fellowship, NSF CAREER Award 1943123, an NSF Graduate Research Fellowship, and gifts from Activision Blizzard, Adobe, Disney, Facebook, and nTopology. The dataset loaders mimic code from [HSN](https://github.com/rubenwiersma/hsn), [pytorch-geometric](https://github.com/rusty1s/pytorch_geometric), and probably indirectly from other sources too. Thank you!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnmwsharp%2Fdiffusion-net","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnmwsharp%2Fdiffusion-net","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnmwsharp%2Fdiffusion-net/lists"}