{"id":13656171,"url":"https://github.com/BarqueroGerman/BeLFusion","last_synced_at":"2025-04-23T17:31:32.726Z","repository":{"id":185351139,"uuid":"570130753","full_name":"BarqueroGerman/BeLFusion","owner":"BarqueroGerman","description":"[ICCV2023] Official PyTorch Implementation of \"BeLFusion: Latent Diffusion for Behavior-Driven Human Motion Prediction\". ICCV 2023","archived":false,"fork":false,"pushed_at":"2023-10-09T13:03:23.000Z","size":8353,"stargazers_count":113,"open_issues_count":0,"forks_count":8,"subscribers_count":6,"default_branch":"main","last_synced_at":"2024-11-10T08:37:27.960Z","etag":null,"topics":["belfusion","ddim","ddpm","deep-learning","diffusion","generative-models","human-motion-generation","human-motion-prediction","iccv2023","latent-diffusion","ldm","motion-forecasting","motion-prediction","pose-forecasting","pose-prediction","pytorch"],"latest_commit_sha":null,"homepage":"https://barquerogerman.github.io/BeLFusion/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/BarqueroGerman.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2022-11-24T12:08:31.000Z","updated_at":"2024-11-01T20:39:08.000Z","dependencies_parsed_at":"2023-08-01T16:06:11.389Z","dependency_job_id":null,"html_url":"https://github.com/BarqueroGerman/BeLFusion","commit_stats":null,"previous_names":["barquerogerman/belfusion"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BarqueroGerman%2FBeLFusion","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BarqueroGerman%2FBeLFusion/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BarqueroGerman%2FBeLFusion/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BarqueroGerman%2FBeLFusion/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/BarqueroGerman","download_url":"https://codeload.github.com/BarqueroGerman/BeLFusion/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250480452,"owners_count":21437548,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["belfusion","ddim","ddpm","deep-learning","diffusion","generative-models","human-motion-generation","human-motion-prediction","iccv2023","latent-diffusion","ldm","motion-forecasting","motion-prediction","pose-forecasting","pose-prediction","pytorch"],"created_at":"2024-08-02T04:00:52.455Z","updated_at":"2025-04-23T17:31:28.993Z","avatar_url":"https://github.com/BarqueroGerman.png","language":"Python","funding_links":[],"categories":["Papers"],"sub_categories":["Human motion prediction"],"readme":"# BeLFusion\n### Latent Diffusion for Behavior-Driven Human Motion Prediction (ICCV'23)\n\u003ca href=\"https://pytorch.org/get-started/locally/\"\u003e\u003cimg alt=\"PyTorch\" src=\"https://img.shields.io/badge/PyTorch-ee4c2c?logo=pytorch\u0026logoColor=white\"\u003e\u003c/a\u003e \n[![arXiv](https://img.shields.io/badge/arXiv-2210.06551-b31b1b.svg)](https://arxiv.org/abs/2211.14304)\n\u003ca href=\"https://barquerogerman.github.io/BeLFusion/\"\u003e\u003cimg alt=\"Project\" src=\"https://img.shields.io/badge/-Project%20Page-lightgrey?logo=Google%20Chrome\u0026color=informational\u0026logoColor=white\"\u003e\u003c/a\u003e\n\u003cbr\u003e\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/belfusion-latent-diffusion-for-behavior/human-pose-forecasting-on-amass)](https://paperswithcode.com/sota/human-pose-forecasting-on-amass?p=belfusion-latent-diffusion-for-behavior)\n\u003cbr\u003e\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/belfusion-latent-diffusion-for-behavior/human-pose-forecasting-on-human36m)](https://paperswithcode.com/sota/human-pose-forecasting-on-human36m?p=belfusion-latent-diffusion-for-behavior)\n\u003cbr\u003e\n\n![BeLFusion's architecture](assets/intro.png)\n\nThis repository contains the official PyTorch implementation of the paper:\n\n**BeLFusion: Latent Diffusion for Behavior-Driven Human Motion Prediction**\u003cbr\u003e\n*German Barquero, Sergio Escalera, and Cristina Palmero*\u003cbr\u003e\n**ICCV 2023**\u003cbr\u003e\n[[website](https://barquerogerman.github.io/BeLFusion/)] [[paper](https://arxiv.org/abs/2211.14304)] [[demo](https://barquerogerman.github.io/BeLFusion/)]\n\n**Note**: our data loaders consider an extra dimension for the number of people in the scene. Since the project aims at single-human motion prediction, this dimension is always 1.\n\n## Installation\n\n\n### 1. Environment\n\n\u003cdetails\u003e \n\u003csummary\u003eOPTION 1 - Python/conda environment\u003c/summary\u003e\n\u003cp\u003e\n\n```\nconda create -n belfusion python=3.9.5\nconda activate belfusion\npip install -r requirements.txt\n```\n\u003c/p\u003e\n\u003c/details\u003e \n\n\u003cdetails\u003e \n\u003csummary\u003eOPTION 2 - Docker\u003c/summary\u003e\n\u003cp\u003e\nWe also provide a DockerFile to build a Docker image with all the required dependencies. \n\n**IMPORTANT**: This option will not let you launch the visualization script, as it requires a GUI. You will be able though to train and evaluate the models.\n\nTo build and launch the Docker image, run the following commands from the root of the repository:\n```\ndocker build . -t belfusion\ndocker run -it --gpus all --rm --name belfusion \\\n-v ${PWD}:/project \\\nbelfusion\n```\n\nYou should now be in the container, ready to run the code.\n\u003c/p\u003e\n\u003c/details\u003e \n\n\n### 2. Datasets\n\n#### [**\u003e Human3.6M**](http://vision.imar.ro/human3.6m/description.php)\nExtract the Poses-D3Positions* folders for S1, S5, S6, S7, S8, S9, S11 into `./datasets/Human36M`. Then, run:\n\n```\npython -m data_loader.parsers.h36m\n```\n\n#### [**\u003e AMASS**](https://amass.is.tue.mpg.de/)\nDownload the *SMPL+H G* files for **22 datasets**: ACCAD, BMLhandball, BMLmovi, BMLrub, CMU, DanceDB, DFaust, EKUT, EyesJapanDataset, GRAB, HDM05, HUMAN4D, HumanEva, KIT, MoSh, PosePrior (MPI_Limits), SFU, SOMA, SSM, TCDHands, TotalCapture, and Transitions. Then, move the **tar.bz2** files to `./datasets/AMASS` (DO NOT extract them). \n\nNow, download the 'DMPLs for AMASS' from [here](https://smpl.is.tue.mpg.de), and the 'Extended SMPL+H model' from [here](https://mano.is.tue.mpg.de/). Move both extracted folders (dmpls, smplh) to `./auxiliar/body_models`. Then, run:\n\n```\npython -m data_loader.parsers.amass --gpu\n```\n\n**Note 1**: remove the `--gpu` flag if you do not have a GPU.\n\n**Note 2**: this step could take a while (~2 hours in CPU, ~20-30 minutes in GPU).\n\n### 3. Checkpoints [(link)](https://ubarcelona-my.sharepoint.com/:f:/g/personal/germanbarquero_ub_edu/EhInsrgQfe5OoqxBdHS21vcBxEJRU5JJq0zzmS2l8csc-A?e=LL1Guq)\nReplace the folder 'checkpoints' in the root of the repository with the downloaded one. If you want to train the models from scratch, you can skip this step and go to the *training* section.\n\n\n## Evaluation\nRun the following scripts to evaluate BeLFusion and the other state-of-the-art methods.\n\nHuman3.6M:\n```\n# BeLFusion \npython eval_belfusion.py -c checkpoints/ours/h36m/BeLFusion/final_model/ -i 217 --ema --mode stats --batch_size 512\n\n# Baselines --\u003e {ThePoseKnows, DLow, GSPS, DiverseSampling}\npython eval_baseline.py -c checkpoints/baselines/h36m/\u003cBASELINE_NAME\u003e/exp -m stats --batch_size 512\n```\n\nAMASS:\n```\n# BeLFusion\npython eval_belfusion.py -c checkpoints/ours/amass/BeLFusion/final_model/ -i 1262 --multimodal_threshold 0.4 --ema --mode stats --batch_size 512\n\n# Baselines --\u003e {ThePoseKnows, DLow, GSPS, DiverseSampling}\npython eval_baseline.py -c checkpoints/baselines/amass/\u003cBASELINE_NAME\u003e/exp -m stats --batch_size 512 --multimodal_threshold 0.4\n```\n\n- Add `--stats_mode all` to also compute the MMADE, MMFDE (increased computation time).\n- Add `-cpu` to run the evaluation in CPU (recommended for low-memory GPUs).\n- (only for BeLFusion) Use `--dstride S` to compute the evaluation metrics every S denoising steps (increased computation time). If S=10, the metrics will be computed for step 1 (BeLFusion_D), and 10 (BeLFusion).\n\n\n## Visualization\n![BeLFusion's architecture](assets/A_103_Transitions.gif)\n\nRun the following scripts to visualize the results of BeLFusion and the other state-of-the-art methods (\\\u003cDATASET\\\u003e in {`h36m`, `amass`}).\n\n```\n# BeLFusion with Human3.6M (press '0' to visualize BeLFusion_D)\npython eval_belfusion.py -c checkpoints/ours/h36m/BeLFusion/final_model/ -i 217 --ema --mode vis --batch_size 64 --dstride 10\n\n# BeLFusion with AMASS (press '0' to visualize BeLFusion_D)\npython eval_belfusion.py -c checkpoints/ours/amass/BeLFusion/final_model/ -i 1262 --ema --mode vis --batch_size 64 --dstride 10\n\n# Baselines --\u003e {ThePoseKnows, DLow, GSPS, DiverseSampling}\npython eval_baseline.py -c checkpoints/baselines/\u003cDATASET\u003e/\u003cBASELINE_NAME\u003e/exp -m vis --batch_size 64\n```\n\n- Press `n` to navigate between the samples.\n- Set `--samples N` to generate `N` samples. Set the columns in the visualization grid with `--ncols N`.\n- During visualization, press `h` to show only the future motion (without observation).\n- (only for BeLFusion) When `--dstride S` for S != -1, you can visualize the output of BeLFusion every `S` denoising steps (press keys `0`, `1`, `2`, ..., to navigate from 1, 1+S, 1+2S, ...).\n\n**Note:** Replace `--mode vis` with `--mode gen` to generate the gif animations instead of visualizing them. In this mode, set the argument `--store_idx I` to store the gifs for denoising step `I`. For example, set `I` to 1 for BeLFusion_D's outputs.\n\n## Training\nFor training BeLFusion from scratch, you need to first train the Behavioral Latent Space (BLS) and the observation autoencoder (\\\u003cDATASET\\\u003e in {`h36m`, `amass`}). Both models can be trained in parallel:\n  \n```\n# Observation autoencoder --\u003e 500 epochs\npython train_auto.py -c checkpoints/ours/\u003cDATASET\u003e/BeLFusion/final_model/autoencoder_obs/config.json\n\n# BLS --\u003e 2x500 epochs\npython train_bls.py -c checkpoints/ours/\u003cDATASET\u003e/BeLFusion/final_model/behavioral_latent_space/config.json\n```\n\nOnce they finish, you can train the Latent Diffusion Model (LDM):\n\n```\n# BeLFusion --\u003e 217/1262 epochs for H36M/AMASS\npython train_belfusion.py -c checkpoints/ours/\u003cDATASET\u003e/BeLFusion/final_model/config.json\n```\n\n## Citation\nIf you find our work useful in your research, please consider citing our paper:\n```\n@inproceedings{barquero2023belfusion,\n  title={BeLFusion: Latent Diffusion for Behavior-Driven Human Motion Prediction},\n  author={Barquero, German and Escalera, Sergio and Palmero, Cristina},\n  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},\n  year={2023}\n}\n```\n\n## License\nThe software in this repository is freely available for free non-commercial use (see [license](https://github.com/BarqueroGerman/BeLFusion/blob/main/LICENSE) for further details).\n\n**Note 1:** project structure borrowed from @victoresque's [template](https://github.com/victoresque/pytorch-template).\n\n**Note 2:** code under `./models/sota` is based on the original implementations of the corresponding papers ([Dlow](https://github.com/Khrylx/DLow), [DiverseSampling](https://github.com/Droliven/diverse_sampling), and [GSPS](https://github.com/wei-mao-2019/gsps)).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FBarqueroGerman%2FBeLFusion","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FBarqueroGerman%2FBeLFusion","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FBarqueroGerman%2FBeLFusion/lists"}