{"id":13656114,"url":"https://github.com/mingyuan-zhang/ReMoDiffuse","last_synced_at":"2025-04-23T17:31:15.541Z","repository":{"id":162368728,"uuid":"622234196","full_name":"mingyuan-zhang/ReMoDiffuse","owner":"mingyuan-zhang","description":"ReMoDiffuse: Retrieval-Augmented Motion Diffusion Model","archived":false,"fork":false,"pushed_at":"2024-04-22T00:57:00.000Z","size":882,"stargazers_count":321,"open_issues_count":10,"forks_count":14,"subscribers_count":16,"default_branch":"main","last_synced_at":"2024-10-30T00:36:14.890Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://mingyuan-zhang.github.io/projects/ReMoDiffuse.html","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mingyuan-zhang.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-04-01T14:15:33.000Z","updated_at":"2024-10-24T07:30:04.000Z","dependencies_parsed_at":"2024-10-29T23:32:49.272Z","dependency_job_id":null,"html_url":"https://github.com/mingyuan-zhang/ReMoDiffuse","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mingyuan-zhang%2FReMoDiffuse","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mingyuan-zhang%2FReMoDiffuse/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mingyuan-zhang%2FReMoDiffuse/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mingyuan-zhang%2FReMoDiffuse/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mingyuan-zhang","download_url":"https://codeload.github.com/mingyuan-zhang/ReMoDiffuse/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223931638,"owners_count":17227256,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-02T04:00:51.272Z","updated_at":"2024-11-10T08:31:01.053Z","avatar_url":"https://github.com/mingyuan-zhang.png","language":"Python","funding_links":[],"categories":["Papers"],"sub_categories":["Text-Driven motion generation"],"readme":"\u003cdiv align=\"center\"\u003e\n\n\u003ch1\u003eReMoDiffuse: Retrieval-Augmented Motion Diffusion Model\u003c/h1\u003e\n\n\u003cdiv\u003e\n    \u003ca href='https://mingyuan-zhang.github.io/' target='_blank'\u003eMingyuan Zhang\u003c/a\u003e\u003csup\u003e1\u003c/sup\u003e\u0026emsp;\n    \u003ca href='https://gxyes.github.io/' target='_blank'\u003eXinying Guo\u003c/a\u003e\u003csup\u003e1\u003c/sup\u003e\u0026emsp;\n    \u003ca href='https://scholar.google.com/citations?user=lSDISOcAAAAJ\u0026hl=zh-CN' target='_blank'\u003eLiang Pan\u003c/a\u003e\u003csup\u003e1\u003c/sup\u003e\u0026emsp;\n    \u003ca href='https://caizhongang.github.io/' target='_blank'\u003eZhongang Cai\u003c/a\u003e\u003csup\u003e1,2\u003c/sup\u003e\u0026emsp;\n    \u003ca href='https://hongfz16.github.io/' target='_blank'\u003eFangzhou Hong\u003c/a\u003e\u003csup\u003e1\u003c/sup\u003e\u0026emsp;\n    \u003ca href='https://www.linkedin.com/in/huirong-li' target='_blank'\u003eHuirong Li\u003c/a\u003e\u003csup\u003e1\u003c/sup\u003e\u0026emsp; \u003cbr\u003e\n    \u003ca href='https://yanglei.me/' target='_blank'\u003eLei Yang\u003c/a\u003e\u003csup\u003e2\u003c/sup\u003e\u0026emsp;\n    \u003ca href='https://liuziwei7.github.io/' target='_blank'\u003eZiwei Liu\u003c/a\u003e\u003csup\u003e1+\u003c/sup\u003e\n\u003c/div\u003e\n\u003cdiv\u003e\n    \u003csup\u003e1\u003c/sup\u003eS-Lab, Nanyang Technological University\u0026emsp;\n    \u003csup\u003e2\u003c/sup\u003eSenseTime Research\u0026emsp;\n\u003c/div\u003e\n\u003cdiv\u003e\n    \u003csup\u003e+\u003c/sup\u003ecorresponding author\n\u003c/div\u003e\n\n\n---\n\n\u003ch4 align=\"center\"\u003e\n  \u003ca href=\"https://mingyuan-zhang.github.io/projects/ReMoDiffuse.html\" target='_blank'\u003e[Project Page]\u003c/a\u003e •\n  \u003ca href=\"https://arxiv.org/abs/2304.01116\" target='_blank'\u003e[arXiv]\u003c/a\u003e •\n  \u003ca href=\"https://youtu.be/wSddrIA_2p8\" target='_blank'\u003e[Video]\u003c/a\u003e •\n  \u003ca href=\"https://colab.research.google.com/drive/1jztE7c8js3P4YFbw5cGNPJAsCVrreTov?usp=sharing\" target='_blank'\u003e[Colab Demo]\u003c/a\u003e •\n  \u003ca href=\"https://huggingface.co/spaces/mingyuan/ReMoDiffuse\" target='_blank'\u003e[Hugging Face Demo]\u003c/a\u003e \n  \u003cbr\u003e \u003cbr\u003e\n  Accepted to \u003ca href=\"https://iccv2023.thecvf.com/\" target=\"_blank\"\u003e\u003cstrong\u003eICCV 2023\u003c/strong\u003e\u003c/a\u003e\u003c/h2\u003e\n  \u003cimg src=\"https://visitor-badge.laobi.icu/badge?page_id=mingyuan-zhang/ReMoDiffuse\" width=\"8%\" alt=\"visitor badge\"/\u003e\n\u003c/h4\u003e\n\n\u003c/div\u003e\n\n\n\u003e**Abstract:** 3D human motion generation is crucial for creative industry. Recent advances rely on generative models with domain knowledge for text-driven motion generation, leading to substantial progress in capturing common motions. However, the performance on more diverse motions remains unsatisfactory. In this work, we propose **ReMoDiffuse**, a diffusion-model-based motion generation framework that integrates a retrieval mechanism to refine the denoising process.\n\n\u003cdiv align=\"center\"\u003e\n\u003ctr\u003e\n    \u003cimg src=\"imgs/teaser.png\" width=\"90%\"/\u003e\n    \u003cimg src=\"imgs/pipeline.png\" width=\"90%\"/\u003e\n\u003c/tr\u003e\n\u003c/div\u003e\n\n\u003e**Pipeline Overview:** ReMoDiffuse is a retrieval-augmented 3D human motion diffusion model. Benefiting from the extra knowledge from the retrieved samples, ReMoDiffuse is able to achieve high-fidelity on the given prompts. It contains three core components: a) **Hybrid Retrieval** database stores multi-modality features of each motion sequence. b) Semantics-modulated transformer incorporates several identical decoder layers, including a **Semantics-Modulated Attention (SMA)** layer and an FFN layer. The SMA layer will adaptively absorb knowledge from both retrived samples and the given prompts. c) **Condition Mxture** technique is proposed to better mix model's outputs under different combinations of conditions.\n\n## Updates\n\n[09/2023] Add a [🤗Hugging Face Demo](https://huggingface.co/spaces/mingyuan/ReMoDiffuse)!\n\n[09/2023] Add a [Colab Demo](https://colab.research.google.com/drive/1jztE7c8js3P4YFbw5cGNPJAsCVrreTov?usp=sharing)! [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1jztE7c8js3P4YFbw5cGNPJAsCVrreTov?usp=sharing)\n\n[09/2023] Release code for [ReMoDiffuse](https://mingyuan-zhang.github.io/projects/ReMoDiffuse.html) and [MotionDiffuse](https://mingyuan-zhang.github.io/projects/MotionDiffuse.html)\n\n## Benchmark and Model Zoo\n\n#### Supported methods\n\n- [x] [MotionDiffuse](https://mingyuan-zhang.github.io/projects/ReMoDiffuse.html)\n- [x] [MDM](https://guytevet.github.io/mdm-page/)\n- [x] [ReMoDiffuse](https://mingyuan-zhang.github.io/projects/MotionDiffuse.html)\n\n\n## Citation\n\nIf you find our work useful for your research, please consider citing the paper:\n\n```\n@article{zhang2023remodiffuse,\n  title={ReMoDiffuse: Retrieval-Augmented Motion Diffusion Model},\n  author={Zhang, Mingyuan and Guo, Xinying and Pan, Liang and Cai, Zhongang and Hong, Fangzhou and Li, Huirong and Yang, Lei and Liu, Ziwei},\n  journal={arXiv preprint arXiv:2304.01116},\n  year={2023}\n}\n@article{zhang2022motiondiffuse,\n  title={MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model},\n  author={Zhang, Mingyuan and Cai, Zhongang and Pan, Liang and Hong, Fangzhou and Guo, Xinying and Yang, Lei and Liu, Ziwei},\n  journal={arXiv preprint arXiv:2208.15001},\n  year={2022}\n}\n```\n\n## Installation\n\n```shell\n# Create Conda Environment\nconda create -n mogen python=3.9 -y\nconda activate mogen\n\n# C++ Environment\nexport PATH=/mnt/lustre/share/gcc/gcc-8.5.0/bin:$PATH\nexport LD_LIBRARY_PATH=/mnt/lustre/share/gcc/gcc-8.5.0/lib:/mnt/lustre/share/gcc/gcc-8.5.0/lib64:/mnt/lustre/share/gcc/gmp-4.3.2/lib:/mnt/lustre/share/gcc/mpc-0.8.1/lib:/mnt/lustre/share/gcc/mpfr-2.4.2/lib:$LD_LIBRARY_PATH\n\n# Install Pytorch\nconda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=11.3 -c pytorch -y\n\n# Install MMCV\npip install \"mmcv-full\u003e=1.4.2,\u003c=1.9.0\" -f https://download.openmmlab.com/mmcv/dist/cu113/torch1.12.1/index.html\n\n# Install Pytorch3d\nconda install -c bottler nvidiacub -y\nconda install -c fvcore -c iopath -c conda-forge fvcore iopath -y\nconda install pytorch3d -c pytorch3d -y\n\n# Install other requirements\npip install -r requirements.txt\n```\n\n## Data Preparation\n\nDownload data files from google drive [link](https://drive.google.com/drive/folders/13kwahiktQ2GMVKfVH3WT-VGAQ6JHbvUv?usp=sharing) or Baidu Netdisk [link](https://pan.baidu.com/s/1604jks-9PtBUtqCpQQmeEg)(access code: vprc). Unzipped all files and arrange them in the following file structure:\n\n```text\nReMoDiffuse\n├── mogen\n├── tools\n├── configs\n├── logs\n│   ├── motiondiffuse\n│   ├── remodiffuse\n│   └── mdm\n└── data\n    ├── database\n    ├── datasets\n    ├── evaluators\n    └── glove\n```\n\n## Training\n\n### Training with a single / multiple GPUs\n\n```shell\nPYTHONPATH=\".\":$PYTHONPATH python tools/train.py ${CONFIG_FILE} ${WORK_DIR} --no-validate\n```\n\n**Note:** The provided config files are designed for training with 8 gpus. If you want to train on a single gpu, you can reduce the number of epochs to one-fourth of the original.\n\n### Training with Slurm\n\n```shell\n./tools/slurm_train.sh ${PARTITION} ${JOB_NAME} ${CONFIG_FILE} ${WORK_DIR} ${GPU_NUM} --no-validate\n```\n\nCommon optional arguments include:\n- `--resume-from ${CHECKPOINT_FILE}`: Resume from a previous checkpoint file.\n- `--no-validate`: Whether not to evaluate the checkpoint during training.\n\nExample: using 8 GPUs to train ReMoDiffuse on a slurm cluster.\n```shell\n./tools/slurm_train.sh my_partition my_job configs/remodiffuse/remodiffuse_kit.py logs/remodiffuse_kit 8 --no-validate\n```\n\n## Evaluation\n\n### Evaluate with a single GPU / multiple GPUs\n\n```shell\nPYTHONPATH=\".\":$PYTHONPATH python tools/test.py ${CONFIG} --work-dir=${WORK_DIR} ${CHECKPOINT}\n```\n\n### Evaluate with slurm\n\n```shell\n./tools/slurm_test.sh ${PARTITION} ${JOB_NAME} ${CONFIG} ${WORK_DIR} ${CHECKPOINT}\n```\nExample:\n```shell\n./tools/slurm_test.sh my_partition test_remodiffuse configs/remodiffuse/remodiffuse_kit.py logs/remodiffuse_kit logs/remodiffuse_kit/latest.pth\n```\n\n**Note:** Run full evaluation for HumanML3D dataset is very slow. You can change `replication_times` in [human_ml3d_bs128.py](configs/_base_/datasets/human_ml3d_bs128.py) to $1$ for a quick evaluation.\n\n## Visualization\n\n```shell\nPYTHONPATH=\".\":$PYTHONPATH python tools/visualize.py ${CONFIG} ${CHECKPOINT} \\\n    --text ${TEXT} \\\n    --motion_length ${MOTION_LENGTH} \\\n    --out ${OUTPUT_ANIMATION_PATH} \\\n    --device cpu\n```\n\nExample:\n```shell\nPYTHONPATH=\".\":$PYTHONPATH python tools/visualize.py \\\n    configs/remodiffuse/remodiffuse_t2m.py \\\n    logs/remodiffuse/remodiffuse_t2m/latest.pth \\\n    --text \"a person is running quickly\" \\\n    --motion_length 120 \\\n    --out \"test.gif\" \\\n    --device cpu\n```\n\n## Acknowledgement\n\nThis study is supported by the Ministry of Education, Singapore, under its MOE AcRF Tier 2 (MOE-T2EP20221-0012), NTU NAP, and under the RIE2020 Industry Alignment Fund – Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s).\n\nThe visualization tool is developed on top of [Generating Diverse and Natural 3D Human Motions from Text](https://github.com/EricGuo5513/text-to-motion)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmingyuan-zhang%2FReMoDiffuse","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmingyuan-zhang%2FReMoDiffuse","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmingyuan-zhang%2FReMoDiffuse/lists"}