{"id":22230633,"url":"https://github.com/Silverster98/HUMANISE","last_synced_at":"2025-07-27T20:31:16.470Z","repository":{"id":108156872,"uuid":"530065141","full_name":"Silverster98/HUMANISE","owner":"Silverster98","description":"Official implementation of the NeurIPS22 paper \"HUMANISE: Language-conditioned Human Motion Generation in 3D Scenes\"","archived":false,"fork":false,"pushed_at":"2023-11-22T06:33:04.000Z","size":31337,"stargazers_count":88,"open_issues_count":0,"forks_count":7,"subscribers_count":8,"default_branch":"main","last_synced_at":"2023-11-22T07:33:17.119Z","etag":null,"topics":["3d-scene-understanding","deep-learning","motion-generation"],"latest_commit_sha":null,"homepage":"https://silverster98.github.io/HUMANISE/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Silverster98.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2022-08-29T04:29:05.000Z","updated_at":"2023-11-21T03:44:50.000Z","dependencies_parsed_at":"2023-11-22T07:42:48.212Z","dependency_job_id":null,"html_url":"https://github.com/Silverster98/HUMANISE","commit_stats":null,"previous_names":[],"tags_count":0,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Silverster98%2FHUMANISE","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Silverster98%2FHUMANISE/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Silverster98%2FHUMANISE/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Silverster98%2FHUMANISE/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Silverster98","download_url":"https://codeload.github.com/Silverster98/HUMANISE/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":227831083,"owners_count":17826155,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-scene-understanding","deep-learning","motion-generation"],"created_at":"2024-12-03T01:15:20.578Z","updated_at":"2024-12-03T01:15:34.293Z","avatar_url":"https://github.com/Silverster98.png","language":"Python","funding_links":[],"categories":["Human-Scene Interaction"],"sub_categories":[],"readme":"# *HUMANISE*: Language-conditioned Human Motion Generation in 3D Scenes\n\n\u003cp align=\"left\"\u003e\n    \u003ca href='https://arxiv.org/abs/2210.09729'\u003e\n      \u003cimg src='https://img.shields.io/badge/Paper-arXiv-green?style=plastic\u0026logo=arXiv\u0026logoColor=green' alt='Paper arXiv'\u003e\n    \u003c/a\u003e\n    \u003ca href='https://silvester.wang/HUMANISE/paper.pdf'\u003e\n      \u003cimg src='https://img.shields.io/badge/Paper-PDF-red?style=plastic\u0026logo=adobeacrobatreader\u0026logoColor=red' alt='Paper PDF'\u003e\n    \u003c/a\u003e\n    \u003ca href='https://silverster98.github.io/HUMANISE/'\u003e\n      \u003cimg src='https://img.shields.io/badge/Project-Page-blue?style=plastic\u0026logo=Google%20chrome\u0026logoColor=blue' alt='Project Page'\u003e\n    \u003c/a\u003e\n     \u003ca href='https://docs.google.com/forms/d/e/1FAIpQLSfzhj2wrRLqAXFVOTn8K5NDN-J_5HueRTohMAlayqBuPPWA1w/viewform?usp=sf_link'\u003e\n      \u003cimg src='https://img.shields.io/badge/Dataset-Data-yellow?style=plastic\u0026logo=Databricks\u0026logoColor=yellow' alt='Dataset'\u003e\n    \u003c/a\u003e\n\u003c/p\u003e\n\n[Zan Wang](https://silvester.wang),\n[Yixin Chen](https://yixchen.github.io/),\n[Tengyu Liu](http://tengyu.ai/),\n[Yixin Zhu](https://yzhu.io/),\n[Wei Liang](https://liangwei-bit.github.io/web/),\n[Siyuan Huang](https://siyuanhuang.com/)\n\nThis repository is an official implementation of \"HUMANISE: Language-conditioned Human Motion Generation in 3D Scenes\".\n\nIn this work, we propose a large-scale and semantic-rich human-scene interaction dataset, HUMANISE. It has language description for each human-scene interaction. HUMANISE enables a new task: language-conditioned human motion generation in 3D scenes. \n\n[Paper](https://silvester.wang/HUMANISE/paper.pdf) | [arXiv](https://arxiv.org/abs/2210.09729) | [Project Page](https://silverster98.github.io/HUMANISE) | [Data](https://docs.google.com/forms/d/e/1FAIpQLSfzhj2wrRLqAXFVOTn8K5NDN-J_5HueRTohMAlayqBuPPWA1w/viewform?usp=sf_link)\n\n\u003c!-- \u003cimg src='./figure/teaser_md.png' weight=\"50%\"\u003e --\u003e\n\u003cdiv align=center\u003e\n\u003cimg src='./figure/teaser_md.png' width=60%\u003e\n\u003c/div\u003e\n\n**Update**\n\n- 2023/11/22: **We plan to refactor this repository to optimize the framework and code in recent weeks.**\n- 2023/6/5: We re-format the data for using HUMANISE easier. Please check [HUMANISE New Format](./dataset/dataset.md) for more details.\n\n## Abstract\n\nLearning to generate diverse scene-aware and goal-oriented human motions in 3D scenes remains challenging due to the mediocre characteristics of the existing datasets on Human-Scene Interaction(HSI); they only have limited scale/quality and lack semantics. To fill in the gap, we propose a large-scale and semantic-rich synthetic HSI dataset, denoted as HUMANISE, by aligning the captured human motion sequences with various 3D indoor scenes. We automatically annotate the aligned motions with language descriptions that depict the action and the unique interacting objects in the scene; e.g., sit on the armchair near the desk. HUMANISE thus enables a new generation task, language-conditioned human motion generation in 3D scenes. The proposed task is challenging as it requires joint modeling of the 3D scene, human motion, and natural language. To tackle this task, we present a novel scene-and-language conditioned generative model that can produce 3D human motions of the desirable action interacting with the specified objects. Our experiments demonstrate that our model generates diverse and semantically consistent human motions in 3D scenes.\n\n## Preparation\n\n### 1. Environment Setup\n\n- Install the following key libraries:\n  - pytorch, numpy, trimesh, smplx, pyquaternion, pyrender, scikit-learn, natsort, pillow, tqdm, ...\n\n- Make sure your machine supports on-screen/[off-screen](https://pyrender.readthedocs.io/en/latest/examples/offscreen.html) rendering.\n\nNotes: we run our code with pytorch 1.10 and cuda11.3.\n\n### 2. Data Preparation\n\n1. [ScanNet V2](http://www.scan-net.org/) Dataset\n\n    Remember to change the dataset folder configuration in `utils/configuration.py`. \n\n2. Our pre-synthesized [data](https://docs.google.com/forms/d/e/1FAIpQLSfzhj2wrRLqAXFVOTn8K5NDN-J_5HueRTohMAlayqBuPPWA1w/viewform?usp=sf_link), or you can generate your own data with our pipeline, see [HUMANISE Synthesis](./dataset/README.md) for more details.\n\n3. [SMPLX v1.1](https://smpl-x.is.tue.mpg.de/download.php)\n\n## HUMANISE Dataset\n\n### 1. Synthesis\n\nSee [HUMANISE Synthesis](./dataset/README.md) for more details.\n\n### 2. Visualization\n\nFor HUMANISE dataset visualization, we provide rendering script `visualize_dataset.py` which will render an animation video with top-down view. The result will be saved in `./tmp/`.\n\n- on-screen rendering\n\n```bash\npython visualize_dataset.py --pkl ${PKL} --index ${index} --vis\n# python visualize_dataset.py --pkl your_path/lie/scene0000_001810_c71dc702-1f1d-4381-895c-f07e9a10876b/anno.pkl --index 0 --vis\n```\n\nNotes: `--vis` will render the static human-scene interaction with [trimesh](https://trimsh.org/trimesh.html) on screen.\n\n- off-screen rendering\n\n```bash\nPYOPENGL_PLATFORM=egl python visualize_dataset.py --pkl ${PKL} --index ${index}\n# PYOPENGL_PLATFORM=egl python visualize_dataset.py --pkl your_path/lie/scene0000_001810_c71dc702-1f1d-4381-895c-f07e9a10876b/anno.pkl --index 0\n```\n\nSee more information about the [data format](./dataset/README.md#file-format).\n\n## Our Model\n\n### Preprocess ScanNet Scenes\n\nFollowing [link](https://github.com/daveredrum/Pointnet2.ScanNet#preprocess-scannet-scenes) to preprocess the ScanNet scenes; then change the `preprocess_scene_folder` configuration in `utils/configuration.py`.\n\n### Action-Specific Model\n\n- Train\n\n  ```bash\n  bash scripts/train.sh \"${ACTION}\"\n  # e.g., bash scripts/train.sh \"walk\"\n  ```\n\n- Eval (Quantitative)\n\n  - Reconstruction metrics.\n\n    ```bash\n    bash scripts/eval_metric.sh ${STAMP} \"${ACTION}\"\n    # e.g., bash scripts/eval_metric.sh 20220829_194320 \"walk\"\n    ```\n\n  - Generation metrics (goal dist.). Set `eval_rec=False` (Line 64) in `eval_metric_motion.py` to compute generation metrics. Then run the following script. It will take several hours to compute the results.\n    \n    ```bash\n    bash scripts/eval_metric.sh ${STAMP} \"${ACTION}\"\n    # e.g., bash scripts/eval_metric.sh 20220829_194320 \"walk\"\n    ```\n\n  - Generation metrics (APD).\n\n    ```bash\n    bash scripts/eval_pairwise_distance.sh ${STAMP} \"${ACTION}\"\n    # e.g., bash scripts/eval_pairwise_distance.sh 20220829_194320 \"walk\"\n    ```\n\n- Eval (Qualitative)\n\n  - Qualitative results of reconstruction.\n\n    ```bash\n    bash scripts/eval.sh ${STAMP} \"${ACTION}\"\n    # e.g., bash scripts/eval.sh 20220829_194320 \"walk\"\n    ```\n  \n  - Qualitative results of generation. Comment Line 36 and uncomment Line 35 in `eval_motion.py`. The parameter `k` in `solver.save_k_sample(k: int)` indicates the number of samples for each case.\n\n    ```bash\n    bash scripts/eval.sh ${STAMP} \"${ACTION}\"\n    # e.g., bash scripts/eval.sh 20220829_194320 \"walk\"\n    ```\n\n### Action-Agnostic Model\n\n- First, change to `agnostic` branch by executing `git checkout agnostic`.\n\n- Train\n\n  ```bash\n  bash scripts/train.sh\n  ```\n\n- Eval (Quantitative)\n\n  - Reconstruction metrics.\n\n    ```bash\n    bash scripts/eval_metric.sh ${STAMP}\n    # e.g., bash scripts/eval_metric.sh 20220831_153356\n    ```\n\n  - Generation metrics (goal dist.). Set `eval_rec=False` (Line 64) in `eval_metric_motion.py` to compute generation metrics. Then run the following script. It will take several hours to compute the results.\n    \n    ```bash\n    bash scripts/eval_metric.sh ${STAMP}\n    # e.g., bash scripts/eval_metric.sh 20220831_153356\n    ```\n\n  - Generation metrics (APD).\n\n    ```bash\n    bash scripts/eval_pairwise_distance.sh ${STAMP}\n    # e.g., bash scripts/eval_pairwise_distance.sh 20220831_153356\n    ```\n\n- Eval (Qualitative)\n\n  - Qualitative results of generation.\n\n    ```bash\n    bash scripts/eval.sh ${STAMP}\n    # e.g., bash scripts/eval.sh 20220831_153356\n    ```\n\n### Pretrained Models\n\nYou can use our pretrained [models](https://docs.google.com/forms/d/e/1FAIpQLSfzhj2wrRLqAXFVOTn8K5NDN-J_5HueRTohMAlayqBuPPWA1w/viewform?usp=sf_link). (In the `checkpoints` folder)\n\n|STAMP|Pretrained Models|\n|-|-|\n|POINTTRANS_C_32768|scene model (point transformer)|\n|20220829_194320|action-specific model (walk)|\n|20220830_203617|action-specific model (sit)|\n|20220830_203832|action-specific model (stand up)|\n|20220830_204043|action-specific model (lie)|\n|20220831_153356|action-agnostic model|\n\nPut the downloaded checkpoints into `outputs/` folder as following:\n\n```bash\n-| model/\n-| outputs/\n---| POINTTRANS_C_32768/\n---| 20220829_194320/\n---| ...\n-| scripts/\n-| ...\n```\n\n## Citation\n\nIf you find our project useful, please consider citing us:\n\n```bibtex\n@inproceedings{wang2022humanise,\n  title={HUMANISE: Language-conditioned Human Motion Generation in 3D Scenes},\n  author={Wang, Zan and Chen, Yixin and Liu, Tengyu and Zhu, Yixin and Liang, Wei and Huang, Siyuan},\n  booktitle={Advances in Neural Information Processing Systems (NeurIPS)},\n  year={2022}\n}\n```\n\n## Acknowledgements\n\nSome codes are borrowed from [PSI-release](https://github.com/yz-cnsdqz/PSI-release), [point-transformer](https://github.com/POSTECH-CVLab/point-transformer), [Pointnet2.ScanNet](https://github.com/daveredrum/Pointnet2.ScanNet), and [YouRefIt_ERU](https://github.com/yixchen/YouRefIt_ERU).\n\n### License\n\nOur code and data are released under the [MIT license](./LICENSE). The following datasets are used in our project and are subject to their respective licenses:\n\n- AMASS is under the [Dataset Copyright License for non-commercial scientific research purposes](https://amass.is.tue.mpg.de/license.html).\n- BABEL is under the [Software Copyright License for non-commercial scientific research purposes](https://babel.is.tue.mpg.de/license.htmll).\n- ScanNet V2 is under the [ScanNet Terms of Use](http://kaldir.vc.in.tum.de/scannet/ScanNet_TOS.pdf).\n- Scan2Cad is under the [Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License](https://github.com/skanti/Scan2CAD/blob/master/LICENSE.txt).\n- ReferIt3D is under the [MIT license](https://github.com/referit3d/referit3d/blob/eccv/LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FSilverster98%2FHUMANISE","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FSilverster98%2FHUMANISE","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FSilverster98%2FHUMANISE/lists"}