{"id":13668631,"url":"https://github.com/google/ldif","last_synced_at":"2025-04-27T01:31:44.774Z","repository":{"id":37645481,"uuid":"272524936","full_name":"google/ldif","owner":"google","description":"3D Shape Representation with Local Deep Implicit Functions.","archived":false,"fork":false,"pushed_at":"2024-05-06T15:42:40.000Z","size":19851,"stargazers_count":319,"open_issues_count":18,"forks_count":33,"subscribers_count":14,"default_branch":"master","last_synced_at":"2025-04-19T22:27:27.355Z","etag":null,"topics":["3d-vision","cvpr2020","graphics","iccv2019","ldif","shape-reconstruction","sif"],"latest_commit_sha":null,"homepage":"https://ldif.cs.princeton.edu","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/google.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-06-15T19:22:51.000Z","updated_at":"2025-04-16T17:27:32.000Z","dependencies_parsed_at":"2024-08-02T08:04:37.673Z","dependency_job_id":"d3749e89-a617-4085-97a6-2b9e6424a6a0","html_url":"https://github.com/google/ldif","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google%2Fldif","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google%2Fldif/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google%2Fldif/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google%2Fldif/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/google","download_url":"https://codeload.github.com/google/ldif/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251077065,"owners_count":21532606,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-vision","cvpr2020","graphics","iccv2019","ldif","shape-reconstruction","sif"],"created_at":"2024-08-02T08:00:44.265Z","updated_at":"2025-04-27T01:31:39.763Z","avatar_url":"https://github.com/google.png","language":"Python","funding_links":[],"categories":[":open_hands: Contributing"],"sub_categories":[],"readme":"# Overview\n\n![alt text](assets/sif-teaser.png)\n![alt text](assets/ldif-teaser.png)\n\nThis is a joint codebase for LDIF ([Local Deep Implicit Functions for 3D\nShape](https://arxiv.org/abs/1912.06126)) and SIF ([Learning Shape Templates\nwith Structured Implicit Functions](https://arxiv.org/abs/1904.06447)). Note\nthat LDIF was previously called Deep Structured Implicit Functions. It contains\ncode to reproduce the results of those papers, convert input meshes into the LDIF\nand SIF representations, and visualize and extract meshes.\nrepresentations.\n\nAll .py and .sh files in the top-level\n`ldif/` directory are entry points into the code (`train.py`, `eval.py`,\n`meshes2dataset.py`, `unit_test.sh`, and `reproduce_shapenet_autoencoder.sh`).\nThe rest of this README provides information on initial setup and basic\ndocumentation for those files. For additional documentation, please see each file.\n\n## Environment\n\nTo set up the LDIF/SIF environment, follow these steps:\n\n#### 1. Set up the python environment\n\nThe code was tested with python 3.6 and tensorflow 1.15 on linux. There is a\nrequirements.txt containing all dependencies.\n\nIf you use anaconda, run the following:\n\n```\nconda env create --name ldif -f environment.yml\nconda activate ldif\n```\n\nIf you use a system pip installation, run `pip install -r requirements.txt`\n\nAfter this, the python environment should be ready to go. Please activate the\nenvironment before proceeding. The build scripts include some python.\n\n#### 2. Build GAPS\n\n```./build_gaps.sh```\n\nGAPS is a geometry processing library used by this package to generate the data\nand create interactive visualizations. The script `build_gaps.sh` does the\nfollowing. One, it installs the necessary dependencies with apt. If sudo is\nnot available on the system, the requirements are that GAPS have include access\nto standard OpenGL and GLu library headers (`GL/gl.h`, `GL/glu.h`) (on both linux and\nmacos), and that OSMesa static libraries can be linked (on linux). If these are\nsatisfied, the sudo line can be commented out. Two, it clones the\n[GAPS](https://github.com/tomfunkhouser/gaps) repository from GitHub, make some\nchanges, and builds it. It also moves the qview folder into the gaps repository\nand modifies the makefiles. The `qview` executable is a C++ program written using\nGAPS to visualize SIF and LDIF representations. Finally, the script compiles all\nnecessary GAPS C++ executables, which are called by the python code. If this step\nwas successful, running `./gaps_is_installed.sh` should echo `Ready to go!`\n\nGAPS should compile with no warnings. Please report any warnings by opening a\nGitHub issue- the information would be greatly appreciated.\n\n#### 3. Build the inference kernel (Optional, but highly recommended)\n\n```./build_kernel.sh```\n\nIf successful, there should be a binary ldif2mesh in the ldif/ldif2mesh/\nsubdirectory. Note that the inference kernel assumes the CUDA toolkit is\ninstalled and that a gpu supporting compute 6.1 (Pascal, so 10-series or newer)\nis available. The nvcc command is part of the CUDA toolkit. If you have an older\ngpu, you can try older compute versions for `--gpu-architecture` and `--gpu-code`,\nbut performance may be reduced and some newer features are used, so it might not\ncompile.\n\nIf you do not want to use the inference kernel or don't have a GPU, then you can\npass `--nouse_inference_kernel` to `eval.py`, which is the only script that\ntypically calls the kernel. It will then use pure tensorflow ops for evaluating\nLDIF, as is done during training (for autodiff support). However, it would be\norders of magnitude slower, so it is really not recommended if more than ~20\nmeshes need to be evaluated.\n\nThe kernel should compile with no warnings. Please report any warnings by\nopening a GitHub issue- this information would be greatly appreciated.\n\n## Datasets\n\nTo run LDIF/SIF, first a dataset should be made. The input to this step is a\ndirectory of watertight meshes, and the output is a directory containing the\nfiles needed to train and evaluate LDIF/SIF.\n\nCreate an input directory somewhere on disk, with the following structure:\n\n```[path/to/root]/{train/val/test}/{class names}/{.ply files}```\n\nThe properties of the dataset (# and name of classes, size of the splits, name\nof examples, etc.) are determined from the directory structure. If you want to\nreproduce the shapenet results, then see `./reproduce_shapenet_autoencoder.sh`. A\ndataset doesn't need to have a train, test, and val split, only whichever splits\nyou want to use. You could make a dataset with just a test split for a\ncomparison, for example. Note that for convenience the code tries to check if\nthe class names are wordnet synsets and will convert them to shapenet names\n(i.e. 02691156 -\u003e airplane) if they are-- if it can't detect a synset it will\njust use the folder name as the class name.\n\nNote that .ply files are required, but the GAPS library provides a shell utility\nfor converting between file formats. You can do\n`./ldif/gaps/bin/x86_64/msh2msh mesh.obj mesh.ply` as an example\nconversion, which will read mesh.obj and write a new file mesh.ply to disk.\n\nIt is very important that the input meshes be watertight at training time.\nGAPS provides a program msh2df that can do the conversion, if you are not\ninterested in exactly replicating the OccNet experiment's process. Here is an\nexample command that will make a unit-cube sized mesh watertight:\n\n```\n./ldif/gaps/bin/x86_64/msh2df input.ply tmp.grd -estimate_sign -spacing 0.002 -v\n./ldif/gaps/bin/x86_64/grd2msh tmp.grd output.ply\nrm tmp.grd\n```\nMsh2df outputs an SDF voxel grid, while grd2msh runs marching cubes to extract a\nmesh from the generated SDF grid. The msh2df algorithm rasterizes the mesh to a\nvoxel grid and then floodfills at a resolution determined by the `-spacing`\nparameter in order to determine the sign. The smaller the value, the higher the\nresolution, the smaller the smallest allowable hole in the mesh, and the slower\nthe algorithm. The bigger the value, the lower the resolution, the bigger the\nsmallest allowable hole in the mesh, and the faster the algorithm. The run time\nof both msh2df and of the rest of the dataset creation pipeline will vary greatly\ndepending on the `-spacing` parameter. The default value of 0.002 is quite high\nresolution for a mesh the size of a unit cube.\n\nWhile msh2df is provided as a utility, it was not used to generate the data for\nthe trained LDIF+SIF models. For reproducing the shapenet results, please use\nthe TSDF fusion package used by the\n[OccNet repository](https://github.com/autonomousvision/occupancy_networks), not\nmsh2df.\n\nTo actually make a dataset once watertight meshes are available, run:\n\n```\npython meshes2dataset.py --mesh_directory [path/to/dataset_root] \\\n  --dataset_directory [path/to/nonexistent_output_directory]\n```\n\nPlease see `meshes2dataset.py` for more flags and documentation. To avoid excess\ndisk usage (and avoid having to pass in the input directory path to all\nsubsequent scripts), symlinks are created during this process that point to the\nmeshes in the input directory. Please do not delete or move the input directory\nafter dataset creation, or the code won't have access to the ground truth meshes\nfor evaluation.\n\nThe dataset generation code writes 7-9mb of data per mesh (about 330GB for\nshapenet-13).\n\n## Training\n\nTo train a SIF or LDIF, run the following:\n\n```\npython train.py --dataset_directory [path/to/dataset_root] \\\n  --experiment_name [name] --model_type {ldif, sif, or sif++}\n```\n\nThe dataset directory should be whatever it was set to when running\nmeshes2dataset.py. The experiment name can be arbitrary, it is a tag used to\nload the model during inference/eval/interactive sessions. The `model_type`\ndetermines what hyperparameters to use. ``ldif`` will train a 32x32 LDIF\nwith 16 symmetric and 16 asymmetric elements. ``sif`` will replicate the\nSIF representation proposed in the SIF paper. ``sif++`` will train\nan improved version of SIF using the loss and network from LDIF, as well\nas gaussians that support rotation, but without any latent codes per element.\nBy default trained models are stored under `{root}/trained_models/`, but\nthis can be changed with the ``--model_directory`` flag. For more flags and\ndocumentation, please see `train.py`.\n\nIt is also possible to make model types besides the paper version of LDIF/SIF.\nFor details, please see `ldif/model/hparams.py`. Both LDIF and SIF are stored as\nspecific hparam combos. Adding a new combo and/or new hyperparameters would be\nthe easiest way to evaluate how a modification to LDIF/SIF would change the\nperformance. It would also be how to turn off partial symmetry, or adjust\nthe number of shape elements or size of the latent codes. The only\nspecial hyperparameter is batch size, which is read directly by the `train.py`\nscript, and always set to 1 during inference.\n\nWhile training, the model write tensorboard summaries. If you don't have\ntensorboard, you can install it with `conda install tensorboard` or\n`pip install tensorboard`. Then you can run\n\n```\ntensorboard --logdir [ldif_root]/trained_models/sif-transcoder-[experiment_name]/log\n```\n\nassuming that `--model_root` was set to the default `ldif_root]/trained_models/`\n\nWarning: Training an LDIF from scratch takes a long time. SIF also takes a while, though\nnot nearly as long. The expected performance with a V100 and a batch size of 24 is\n3.5 steps per second for LDIF, 6 steps per second for SIF. LDIF takes about 3.5M steps\nto fully converge on ShapeNet, while SIF takes about 700K. So that is about 10 days to\ntrain an LDIF from scratch, and about 32 hours for SIF. Note that LDIF performance is\npretty reasonable after 3-4 days, so depending on your uses it may not be necessary to\nwait the whole time. The plan is to 1) add pretrained checkpoints (the most pressing\nTODO) and 2) add multi-gpu support, later on, to help mitigate this issue. Another\npractical option might be switching out the encoder for a smaller one, because most\nof the training time is the forward+backward pass on the ResNet50.\n\n## Evaluation and Inference\n\nTo evaluate a fully trained LDIF or SIF network, run the following:\n\n```\npython eval.py --dataset_directory [path/to/dataset_root] \\\n  --experiment_name [name] --split {test, val, or train}\n```\n\nThis will compute metrics over the dataset and then print out the result to the\nterminal. By default, it will print out a table of results and LaTeX code. In\naddition, there are flags `--save_results`, `--save_meshes`, and `--save_ldifs`,\nwhich can be set to true. If they are set, the code will also write\n1) pandas-readable CSV files containing the metrics for each mesh and class,\n2) a directory of result meshes generated by the algorithm, and/or\n3) a directory of txt files containing the actual LDIF/SIF representation\n(parseable by qview, ldif2mesh, and the Decoder class in the ipynb). If these\nflags are set, then `--result_directory` must also be provided indicating where\nthey should be written.\n\n## Interactive Sessions\n\nYou can run the code interactively in an Jupyter notebook. To do so, open the\nprovided file `ldif_example_inference.ipynb` with Jupyter and attach\nit to a python 3.6 kernel with the requirements.txt installed. Then follow the\nprompts in the notebook. The notebook has a demo of loading a mesh, creating an\nexample, running inference, visualizing the underlying SIF elements, extracting\na mesh, and computing metrics. There is additional documentation in the .ipynb.\n\n## Unit Tests\n\nThere is a script `unit_test.sh`. If you want to check whether the code is\ninstalled correctly and works, run it with no arguments. It will make a small\ndataset using open source models and train/evaluate an LDIF network. Note that\nit doesn't train to convergence so that it doesn't take very long to run. As a\nresult, the final outputs don't look very good. You can fix this by setting the\nstep count in `unit_test.sh` higher (around 50K steps is definitely sufficient).\n\nThe code also has some unit tests for various pieces of functionality. To run a\ntest, cd into the directory of the `*_test.py` file, and run it with no arguments.\nPlease be aware that not all of the code is tested, and that the unit tests\naren't well documented. The easiest way to check if the code still works is by\nrunning `./unit_test.sh`.\n\n## Other code and PyTorch\n\nIn addition to the scripts described above, there are also model definitions and\nbeam pipelines provided for generating datasets and running inference on a\nlarger scale. To use these scripts, it would be necessary to hook up your own\nbeam backend.\n\nThere is also very limited PyTorch support in the `ldif/torch` directory. This code\nis a basic implementation of SIF that can't train new SIF models, but can load\nand evaluate SIFs generated by the tensorflow training+evaluation code. It is mainly\nintended for using SIF correspondences as a building block of another unrelated\nproject in PyTorch. Note that PyTorch is not included in the `requirements.txt`, and\nthe `torch/` subdirectory is independent from the rest of the code base (it interacts\nonly through the `.txt` files written by the tensorflow code and takes no dependencies\non the rest of this codebase). To use it, it is probably easiest to just download\nthe `torch/` folder and import the `sif.py` file as a module.\n\n## Updates to the code\n\n* The code now supports tfrecords dataset generation and usage. This reduces\n  the IO workload done during training. It is enabled by default. Existing\n  users can git pull, rerun `meshes2dataset.py` with `--optimize` and \n  `--optimize_only`, and then resume training where they left off with the\n  new dataset improvements. If you currently experience less than 100% GPU\n  utilization, it is highly recommended. Note it increases dataset size by\n  3mb per example (and can be disabled with `--nooptimize`).\n  \n* Support for the inference kernel on Volta, Turing and CC 6.0 Pascal cards\n  should now work as intended. If you had trouble with the inference kernel,\n  please git pull and rerun `./build_kernel.sh`.\n  \n\n## TODOS\n\nThis is a preliminary release of the code, and there are a few steps left:\n\n* Pretrained model checkpoints are on the way. In the mean-time, please see\n`reproduce_shapenet_autoencoder.sh` for shapenet results.\n* This code base does not yet support training a single-view network. In the\n  mean-time, the single-view network architecture has been provided (see\n  `ldif/model/hparams.py` for additional information).\n* While the eval code is fast enough for shapenet, the post-kernel\neval code is written in numpy and is an unnecessary bottleneck. So inference at\n  256^3 takes a few seconds per mesh, even though the kernel completes in\n  ~300ms.\n* Pressing the 'f' key in a qview visualization session will extract a mesh\nand show it alongside the SIF elements; however it only considers the analytic\nparameters. Therefore, it shows a confusing result for LDIF representations,\n  which also have neural features.\n* To make setup easier, we would like to provide a docker container\nthat is ready to go.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle%2Fldif","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgoogle%2Fldif","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle%2Fldif/lists"}