{"id":13449295,"url":"https://muelea.github.io/buddi/","last_synced_at":"2025-03-22T22:32:49.725Z","repository":{"id":175771317,"uuid":"651885174","full_name":"muelea/buddi","owner":"muelea","description":"Our model BUDDI learns the joint distribution of interacting people","archived":false,"fork":false,"pushed_at":"2023-12-14T10:11:35.000Z","size":309767,"stargazers_count":67,"open_issues_count":3,"forks_count":1,"subscribers_count":5,"default_branch":"main","last_synced_at":"2023-12-14T23:30:00.535Z","etag":null,"topics":["3d-human-mesh","3d-human-reconstruction","3d-human-shape-and-pose-estimation","diffusion-model","human-interactions","smplx"],"latest_commit_sha":null,"homepage":"https://muelea.github.io/buddi/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/muelea.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-06-10T11:44:02.000Z","updated_at":"2023-12-20T16:28:16.490Z","dependencies_parsed_at":"2023-12-20T16:41:51.061Z","dependency_job_id":null,"html_url":"https://github.com/muelea/buddi","commit_stats":null,"previous_names":["muelea/buddi"],"tags_count":0,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/muelea%2Fbuddi","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/muelea%2Fbuddi/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/muelea%2Fbuddi/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/muelea%2Fbuddi/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/muelea","download_url":"https://codeload.github.com/muelea/buddi/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221840629,"owners_count":16889834,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-human-mesh","3d-human-reconstruction","3d-human-shape-and-pose-estimation","diffusion-model","human-interactions","smplx"],"created_at":"2024-07-31T06:00:35.131Z","updated_at":"2024-10-28T14:30:45.525Z","avatar_url":"https://github.com/muelea.png","language":"Python","funding_links":[],"categories":["2D Diffusion with Pretraining"],"sub_categories":["Human and Animal"],"readme":"# BUDDI\n\u003cb\u003e Generative Proxemics: A Prior for 3D Social Interaction from Images \u003c/b\u003e\\\n[![Website shields.io](https://img.shields.io/website-up-down-green-red/http/shields.io.svg)](https://muelea.github.io/buddi/) [![arXiv](https://img.shields.io/badge/arXiv-2305.20091-00ff00.svg)](https://arxiv.org/abs/2306.09337) [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1P7x2gY_VuFz5yjZHTRfEBzsEmA-JYGlE?usp=sharing)\n\nhttps://github.com/muelea/buddi/assets/13314980/b0de0db7-e24f-4c74-8f4d-5029b7d320a2\n\n## In this repo you will find ...\n\n... [BUDDI](#unconditional-sampling), a diffusion model that learned the joint distribution of two people in close proxeminty -- thus a a \u003cb\u003eBUD\u003c/b\u003edies \u003cb\u003eDI\u003c/b\u003effusion model. BUDDI directly generates [SMPL-X](https://smpl-x.is.tue.mpg.de) body model parameters for two people.\n\n... [Optimization with BUDDI](#optimization-with-buddi), we use BUDDI as a prior during optimization via an SDS loss inspired by [DreamFusion](https://arxiv.org/pdf/2209.14988.pdf). This approach does not require ground-truth contact annotations.\n\n... [Flickr Fits](#flickr-fits), we create SMPL-X fits for [FlickrCI3D](https://ci3d.imar.ro) via an optimization method that takes ground-truth contact annotations between two people into account.\n\n## NEWS!!\n:boom: [Google Colab](https://colab.research.google.com/drive/1P7x2gY_VuFz5yjZHTRfEBzsEmA-JYGlE?usp=sharing) with custom images is available :boom:\n\n:boom: Demo code available to run the optimization with BUDDI on your own images :boom:\n\n:boom: Improved installation scripts (with ViTPose and BEV included) :boom:\n\nWe have a new version of BUDDI with BEV conditioning \n\n## Release status\n\n| BUDDI inference | BUDDI training | Optimization with BUDDI | Optimization with BUDDI conditional | Training Data / Flickr Fits |\n| :----: | :----: | :----: | :----: | :----: |\n| \u0026check; | \u0026check;  | \u0026check; | \u0026check; | \u0026#x2717; |\n\n\n## Installation and Quick Start\nPlease see [Installation](./documentation/INSTALL.md) for details. \n\n```\n# install conda environment\n./install_conda_env.sh\n\n# download essentials and models\n./fetch_data.sh\n\n# download body models (SMPL-X, SMPL, SMIL). The script will ask for you username\n# and password for the SMPL-X and SMPL website. If you don't have an account, please\n# register under https://smpl-x.is.tue.mpg.de/ and https://smpl.is.tue.mpg.de/.\n./fetch_bodymodels.sh\n\n# Install BEV and ViTPose and convert body models to BEV format \n./install_thirdparty.sh\n\n# Run optimization with BUDDI on your own images\n# We have some internet images in [this](./demo/data/FlickrCI3D_Signatures/demo/images_live) folder.\n# The script will first run BEV and ViTPose and then start the optimization with BUDDI.\n# To run the demo with OpenPose on top, please read the comments in demo.sh\n./demo.sh\n```\n\n## Datasets\nPlease see [Dataset](./documentation/DATA.md) for details.\n\n## Demo \n\n### Unconditional sampling\n\n\n\nhttps://github.com/muelea/buddi/assets/13314980/ac93baf2-750e-4223-b9eb-2422004e972c\n\n\n\nUnconditional generation stating from random noise using different sampling schedules\n\n```\n# linear schedule starting from max-t and skipping every skip-steps step. Here, it's 1000 990 980 ... 20 10. \npython llib/methods/hhc_diffusion/evaluation/sample.py --exp-cfg essentials/buddi/buddi_unconditional.yaml --output-folder demo/diffusion/samples/ --checkpoint-name essentials/buddi/buddi_unconditional.pt --max-images-render=100 --num-samples 100 --max-t 1000 --skip-steps 10 --log-steps=100 --save-vis \n```\n\n\n\n### Optimization with BUDDI trained with BEV conditioning\n\n\n\n\nhttps://github.com/muelea/buddi/assets/13314980/89d6a7de-e907-46ac-83c5-321174ca0eba\n\n\n\n\nRun optimization using BUDDI as prior. This script will find all OpenPose Bounding boxes on a photo and run Optimization with BUDDI for all pairs of people who overlap on the picture.\n```\npython llib/methods/hhcs_optimization/main.py --exp-cfg llib/methods/hhcs_optimization/configs/buddi_cond_bev_demo.yaml --exp-opts logging.base_folder=demo/optimization/buddi_cond_bev_demo datasets.train_names=['demo'] datasets.train_composition=[1.0] datasets.demo.original_data_folder=demo/data/FlickrCI3D_Signatures/demo datasets.demo.image_folder=images model.optimization.pretrained_diffusion_model_ckpt=essentials/buddi/buddi_cond_bev.pt model.optimization.pretrained_diffusion_model_cfg=essentials/buddi/buddi_cond_bev.yaml logging.run=fit_buddi_cond_bev_flickrci3ds\n```\n\nRun optimization with BUDDI on FlickrCI3D. First follow the data and install instructions, then run the commands below. You can use --cluster_pid and --cluster_bs flags to process only a few images or distribute batches of data on a cluster.\n```\n# run optimization for training split\npython llib/methods/hhcs_optimization/main.py --exp-cfg llib/methods/hhcs_optimization/configs/buddi_cond_bev.yaml --exp-opts logging.base_folder=demo/optimization/buddi_cond_bev logging.run=fit_buddi_cond_bev_flickrci3ds datasets.train_names=['flickrci3ds'] datasets.train_composition=[1.0] datasets.val_names=[] datasets.test_names=[] model.optimization.pretrained_diffusion_model_ckpt=essentials/buddi/buddi_cond_bev.pt model.optimization.pretrained_diffusion_model_cfg=essentials/buddi/buddi_cond_bev.yaml\n\n# to run optimization on the validation split set\ndatasets.train_names=[] datasets.train_composition=[] datasets.val_names=['flickrci3ds'] datasets.test_names=[]\n\n# to run optimization on the test split set\ndatasets.train_names=[] datasets.train_composition=[] datasets.val_names=[] datasets.test_names=['flickrci3ds']\n```\n\n## Training\n\n### Flickr Fits\n\nTo create training data, we fit SMPL-X to images from [FlickrCI3D Signatures](https://ci3d.imar.ro/flickrci3d) via an optimization method that takes ground-truth contact annotations between two people on the human body into account. First follow the data and install instructions, then run the commands below. You can use --cluster_pid and --cluster_bs flags to process only a few images or distribute batches of data on a cluster.\n```\n# run optimization on FlickrCI3D Signatures training split\npython llib/methods/hhcs_optimization/main.py --exp-cfg llib/methods/hhcs_optimization/configs/flickr_fits.yaml --exp-opts logging.base_folder=demo/optimization logging.run=flickr_fits datasets.train_names=['flickrci3ds'] datasets.train_composition=[1.0] datasets.val_names=[] datasets.test_names=[]\n\n# to run optimization on the validation split set\ndatasets.train_names=[] datasets.train_composition=[] datasets.val_names=['flickrci3ds'] datasets.test_names=[]\n\n# to run optimization on the test split set\ndatasets.train_names=[] datasets.train_composition=[] datasets.val_names=[] datasets.test_names=['flickrci3ds']\n```\n\n### BUDDI Training\nFollow the data download and processing steps in [Dataset](./documentation/DATA.md). Then run: \n```\n# conditional model\npython llib/methods/hhc_diffusion/main.py --exp-cfg llib/methods/hhc_diffusion/configs/config_buddi_v02_cond_bev.yaml --exp-opts logging.base_folder=demo/diffusion/training logging.run=buddi_cond_bev datasets.augmentation.use=True model.regressor.losses.pseudogt_v2v.weight=[1000.0] logging.logger='tensorboard'\n\n# unconditional model \npython llib/methods/hhc_diffusion/main.py --exp-cfg llib/methods/hhc_diffusion/configs/config_buddi_v02.yaml --exp-opts logging.base_folder=demo/diffusion/training logging.run=buddi datasets.augmentation.use=True datasets.chi3d.load_unit_glob_and_transl=True datasets.hi4d.load_unit_glob_and_transl=True model.regressor.losses.pseudogt_v2v.weight=[100.0] logging.logger='tensorboard'\n```\n\n\n## Evaluation \n\nTo compare generated meshes against the training data on SMPL-X parameter FID you can use the evaluation script. To generate a x_starts_smplx.pkl file, see [here](#unconditional-sampling).\n```\npython llib/methods/hhc_diffusion/evaluation/eval.py --exp-cfg llib/methods/hhc_diffusion/evaluation/config_eval.yaml --buddi demo/diffusion/samples/generate_1000_10_v0/x_starts_smplx.pkl --load-training-data\n```\n\nWe evaluate BUDDI against pseudo-ground truth fits and ground-truth contact labels of FlickrCI3D.\n```\npython llib/methods/hhcs_optimization/evaluation/flickrci3ds_eval.py --exp-cfg llib/methods/hhcs_optimization/evaluation/flickrci3ds_eval.yaml -gt \u003cbase_folder\u003e/fit_pseudogt_flickrci3ds_test -p \u003cbase_folder\u003e/\u003crun_folder\u003e --flickrci3ds-split test\n\npython llib/methods/hhcs_optimization/evaluation/chi3d_eval.py --exp-cfg llib/methods/hhcs_optimization/evaluation/chi3d_eval.yaml --predictions-folder \u003cbase_folder\u003e/\u003crun_folder\u003e --eval-split test\n\npython llib/methods/hhcs_optimization/evaluation/hi4d_eval.py --exp-cfg llib/methods/hhcs_optimization/evaluation/hi4d_eval.yaml --predictions-folder \u003cbase_folder\u003e/\u003crun_folder\u003e --eval-split test\n```\n\n\n## Acknowledgments\nWe thank our colleagues for their feedback, in particular, we thank Aleksander Holynski, Ethan Weber, and Frederik Warburg for their discussions about diffusion and the SDS loss, Jathushan Rajasegaran, Karttikeya Mangalam and Nikos Athanasiou for their discussion about transformers, and Alpar Cseke, Taylor McConnell and Tsvetelina Alexiadis for running the user study.\n\nPrevious work on human pose and shape estimation has made this project possible: we use [BEV](https://github.com/Arthur151/ROMP) to initialize the optimization method, the Flickr and mocap data provided in [Close interactions 3D](https://ci3d.imar.ro/index.php/). We also use previous workon diffusion models and their code bases, [diffusion](https://github.com/hojonathanho/diffusion) and [guided-diffusion](https://github.com/openai/guided-diffusion/tree/main/guided_diffusion).\n\n\n\n## Citation\n```\n@article{mueller2023buddi,\n    title={Generative Proxemics: A Prior for {3D} Social Interaction from Images},\n    author={M{\\“u}ller, Lea and Ye, Vickie and Pavlakos, Georgios and Black, Michael J. and Kanazawa, Angjoo},\n    booktitle = {{Computer Vision and Pattern Recognition (CVPR)}},\n    year={2024}}\n```\n\n## License\nSee [License](./LICENSE).\n\n\n## Disclosure\nMJB has received research gift funds from Adobe, Intel, Nvidia, Meta/Facebook, and Amazon. MJB has financial interests in Amazon, Datagen Technologies, and Meshcapade GmbH. While MJB is a part-time employee of Meshcapade, his research was performed solely at, and funded solely by, the Max Planck Society.\n\n\n## Contact\nPlease contact lea.mueller@tuebingen.mpg.de for technical questions.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/muelea.github.io%2Fbuddi%2F","html_url":"https://awesome.ecosyste.ms/projects/muelea.github.io%2Fbuddi%2F","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/muelea.github.io%2Fbuddi%2F/lists"}