{"id":15735655,"url":"https://github.com/roym899/sdfest","last_synced_at":"2025-04-15T05:56:13.382Z","repository":{"id":44435233,"uuid":"391901247","full_name":"roym899/sdfest","owner":"roym899","description":"Shape and pose estimation using discretized signed distance fields","archived":false,"fork":false,"pushed_at":"2024-06-11T18:49:25.000Z","size":203056,"stargazers_count":16,"open_issues_count":2,"forks_count":3,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-15T05:56:02.908Z","etag":null,"topics":["analysis-by-synthesis","deep-learning","object-pose-estimation","pose-estimation","shape-estimation"],"latest_commit_sha":null,"homepage":"https://roym899.github.io/sdfest/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/roym899.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-08-02T10:12:39.000Z","updated_at":"2025-03-12T03:19:42.000Z","dependencies_parsed_at":"2024-06-11T22:35:06.949Z","dependency_job_id":"98e4ad89-2307-4058-94a0-d200076e7007","html_url":"https://github.com/roym899/sdfest","commit_stats":{"total_commits":278,"total_committers":4,"mean_commits":69.5,"dds":"0.12230215827338131","last_synced_commit":"787e4ede0f3471c04fde8b27220f79ebf30da02b"},"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/roym899%2Fsdfest","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/roym899%2Fsdfest/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/roym899%2Fsdfest/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/roym899%2Fsdfest/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/roym899","download_url":"https://codeload.github.com/roym899/sdfest/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249016320,"owners_count":21198832,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["analysis-by-synthesis","deep-learning","object-pose-estimation","pose-estimation","shape-estimation"],"created_at":"2024-10-04T01:14:20.796Z","updated_at":"2025-04-15T05:56:13.363Z","avatar_url":"https://github.com/roym899.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# SDFEst\n[![PyPI Release](https://github.com/roym899/sdfest/actions/workflows/publish_release.yaml/badge.svg)](https://github.com/roym899/sdfest/actions/workflows/publish_release.yaml) [![Docs](https://github.com/roym899/sdfest/actions/workflows/build_docs.yaml/badge.svg)](https://github.com/roym899/sdfest/actions/workflows/build_docs.yaml)\n\nSDFEst is a package for pose, scale, and shape estimation using discretized signed distance fields. It includes and combines three main components (generative model, discriminative initialization, and differentiable renderer) to enable pose and shape estimation in an analysis-by-synthesis framework.\n\n[Installation](#installation) | [Dataset Preparation](#dataset-preparation)  | [Paper Reproduction](#paper-reproduction) | [Code Structure](#code-structure) | [Development](#development) | [Citation](#citation) | [Docs](https://roym899.github.io/sdfest/)\n\n![Animation of differentiable renderer.](resources/sdfest_overview.gif)\n\n## Installation\n\nYou can install everything with pip (either in your global Python environment, or a virtual environment).\n\n### Non-editable\n```\npip install sdfest\n```\nIn principle, this allows to train novel networks, apply the pipeline, use the individual modules, and reproduce our experiments.\n\n### Editable (i.e., for modifying / experimenting with the code):\n```\ngit clone git@github.com:roym899/sdfest.git\ncd sdfest\npip install -e .\n```\nThis will get the latest version of all dependencies, which might break if a breaking change was introduced in a dependency.\nTo reproduce our tested environment you can use the provided requirements.txt.\n\nExecute the following command to install the packages and their dependencies. Note that this will likely downgrade / upgrade versions in your current environment, so it's better to use this in a virtual environment only.\n```bash\ngit clone git@github.com:roym899/sdfest.git\ncd sdfest\npip install -r requirements.txt -e .\n```\n\n### Optional: Detectron2\nYou need to install Detectron2 manually to run the pipeline with automatic instance segmentation.\n\nFollow the [detectron2 installation guide](https://detectron2.readthedocs.io/en/latest/tutorials/install.html) from there.\nTested with detectron2 0.6 + torch 1.12.0 (and various older versions).\n\n## Dataset Preparation\nSee below for expected folder structure for each dataset.\n\n#### ShapeNet ([Website](https://shapenet.org/))\n```\n./data/shapenet/02876657/...\n./data/shapenet/02880940/...\n./data/shapenet/03797390/...\n```\n\n#### ModelNet ([Website](https://modelnet.cs.princeton.edu/))\n```\n./data/modelnet/bottle/...\n./data/modelnet/bowl/...\n./data/modelnet/cup/...\n```\n\n#### Redwood ([Website](http://redwood-data.org/3dscan/dataset.html))\n```\n./data/redwood/bottle/rgbd/...\n./data/redwood/bowl/rgbd/...\n./data/redwood/mug/rgbd/...\n```\n\n#### RGB-D Object Dataset ([Website](https://rgbd-dataset.cs.washington.edu/index.html))\n```\n./data/rgbd_objects_uw/{bowl,coffee_mug,shampoo,water_bottle}/{bowl,coffee_mug,shampoo,water_bottle}_1/\n./data/rgbd_objects_uw/{bowl,coffee_mug,shampoo,water_bottle}/{bowl,cofee_mug,shampoo,water_bottle}_2/\n...\n```\n\n## Paper Reproduction\n\nFirst, make sure the datasets are in the right format.\n\n### Full paper (training and experiments)\nSee the bash script in `reproduction_scripts/reproduce_paper.sh`.\n\nEvaluation is integrated in [cpas_toolbox](https://github.com/roym899/pose_and_shape_evaluation).\n\n### Train Models Only\nTo train a network for a specific category you need to first train a per-category VAE, and *afterwards* an initialization network.\n#### VAE\nFirst we need to convert the ShapeNet meshes to SDFs and optionally filter the dataset. To reproduce the preprocessing of the paper run\n```bash\npip install git+https://github.com/marian42/mesh_to_sdf.git\nsource preprocess_shapenet.sh\n```\nThen run\n```bash\nsource train_vaes.sh\n```\nto train the models using the same configuration as used for the paper.\n\n#### Initialization Network\nTo train the initialization network we used in our paper, run\n```bash\nsource train_init_networks.sh\n```\nIf you want to train the initialization network based on a previously trained object model, you need to create a new config linking to the newly trained VAE. \nSee, for example, `sdfest/initialization/configs/discretized_mug.yaml`, which links to `sdfest/initialization/vae_models/mug.yaml`).\n\n## Code Structure\nCode is structured into 4 sub-packages:\n- *vae*: variational auto-encoder for shapes\n- *initialization*: pose and shape estimation from partial pointset\n- *differentiable_renderer*: differentiable renderer for discretized SDF\n- *estimation*: integration of VAE, differentiable renderer and single shot network into initialization + render and compare pipeline\n\n\n### `sdfest.differentiable_renderer`\n\n![Animation of differentiable renderer.](resources/dr_test_script.gif)\n\nDifferentiable rendering of depth image for signed distance fields.\n\nThe signed distance field is assumed to be voxelized and its pose is given by a x, y, z in the camera frame, a quaternion describing its orientation and a scale parameter describing its size. This module provides the derivative with respect to the signed distance values, and the full pose description (position, orientation, scale).\n\n#### Generating compile_commands.json\n\u003csup\u003eGeneral workflow for PyTorch extensions (only tested for JIT, probably similar otherwise)\u003c/sup\u003e\n\nIf you develop PyTorch extensions and want to get correct code checking with ccls / etc. you can do so by going to the ninja build directory (normally `home_directory/.cache/torch_extensions/sdf_renderer_cpp`, or set `load(..., verbose=True)` in `sdf_renderer.py` and check the output), running\n```\nninja -t compdb \u003e compile_commands.json\n```\nand moving `compile_commands.json` to the projects root directory.\n\n\n### `sdfest.vae`\n\n![Animation of differentiable renderer.](resources/vae_visualizer.gif)\n\nArchitecture to learn a low-dimensional representation of signed-distance fields (i.e., an explicit voxel representation of a signed distance function)\n\n#### Preprocessing and training\n\nYou need to preprocess the mesh data prior to running the training script, like this:\n```bash\npip install git+https://github.com/marian42/mesh_to_sdf.git\npython -m sdfest.vae.scripts.process_shapenet --inpath shapenet_subfolder --outpath output_path --resolution 64 --padding 2\n```\nYou can control the resolution and added padding so that there is some empty space left in the signed distance field on all borders. If you are running this script via ssh you need to run `export PYOPENGL_PLATFORM=egl` prior to running the script and use the `--all` option which will disable any filtering. Otherwise mesh selection will proceed in two steps: first you see one mesh after another and need to decide which to keep. Pressing left will remove a mesh, pressing right will keep it. After a decision has been made, the conversion will run. Finally another manual selection process is started, where you can remove SDFs in which the mesh to SDF conversion has failed. \n\nTo train the network you can now use either `python -m sdfest.vae.scripts.train` or `python sdfest/vae/scripts/train.py`.\n\n#### config\nConfiguration can be provided through command-line arguments and hierarchical yaml files. The config files are read in depth first order and later specifications will overwrite previous specifications. \nTo summarize:\n- command line parameters will take precedence over all config files\n- when specifying multiple config files, the second config file will overwrite params set by the first config file\n- parent config files overwrite the params set in the contained config files\n\n### `sdfest.initialization`\nArchitectures for single-shot SDF shape and pose estimation from a single (in the future possibly also multiple) depth views.\n\nTo train a new model run\n```\npython -m sdfest.initialization.scripts.train --config CONFIG_FILE\n```\nSee the `sdfest/initialization/configs` or `reproduction_scripts` folder.\n\n### `sdfest.estimation`\nModular architecture and experiments for SDF shape and pose estimation.\n\nCombines initialization, VAE, and differentiable renderer for full analysis-by-synthesis.\n\n\n## Development\n- Use `pip install -e .` to install the package in editable mode\n- Use `pip install -r requirements-dev.txt` to install dev tools\n- Use `pytest --cov=sdfest --cov-report term-missing tests/` to run tests and check code coverage\n\n## Citation\n\nIf you find this library useful in your research, consider citing our publication:\n\n```\n@article{bruns2022sdfest,\n  title={{SDFEst}: Categorical Pose and Shape Estimation of Objects From {RGB-D} Using Signed Distance Fields},\n  author={Bruns, Leonard and Jensfelt, Patric},\n  journal={IEEE Robotics and Automation Letters},\n  volume={7},\n  number={4},\n  pages={9597--9604},\n  year={2022}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Froym899%2Fsdfest","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Froym899%2Fsdfest","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Froym899%2Fsdfest/lists"}