{"id":14961267,"url":"https://github.com/idea-research/humantomato","last_synced_at":"2025-04-06T12:08:36.127Z","repository":{"id":203010895,"uuid":"706542652","full_name":"IDEA-Research/HumanTOMATO","owner":"IDEA-Research","description":"[ICML 2024] 🍅HumanTOMATO: Text-aligned Whole-body Motion Generation","archived":false,"fork":false,"pushed_at":"2024-06-19T04:51:49.000Z","size":9325,"stargazers_count":321,"open_issues_count":20,"forks_count":10,"subscribers_count":20,"default_branch":"main","last_synced_at":"2025-04-06T12:08:33.736Z","etag":null,"topics":["generation","gpt","motion","motion-generation","smplx","whole-body","whole-body-motion"],"latest_commit_sha":null,"homepage":"https://lhchen.top/HumanTOMATO","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/IDEA-Research.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-10-18T06:48:20.000Z","updated_at":"2025-04-02T07:18:38.000Z","dependencies_parsed_at":null,"dependency_job_id":"b27d0b14-3fdb-4a39-8aa6-b5b62ab3165d","html_url":"https://github.com/IDEA-Research/HumanTOMATO","commit_stats":{"total_commits":21,"total_committers":3,"mean_commits":7.0,"dds":"0.33333333333333337","last_synced_commit":"b5ccf060465e986585618fac7461ed1674dedd92"},"previous_names":["idea-research/humantomato"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IDEA-Research%2FHumanTOMATO","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IDEA-Research%2FHumanTOMATO/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IDEA-Research%2FHumanTOMATO/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IDEA-Research%2FHumanTOMATO/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/IDEA-Research","download_url":"https://codeload.github.com/IDEA-Research/HumanTOMATO/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247478323,"owners_count":20945266,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["generation","gpt","motion","motion-generation","smplx","whole-body","whole-body-motion"],"created_at":"2024-09-24T13:24:18.306Z","updated_at":"2025-04-06T12:08:36.104Z","avatar_url":"https://github.com/IDEA-Research.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# HumanTOMATO: Text-aligned Whole-body Motion Generation\n\n![](./assets/tomato-logo.png)\n\n[Shunlin Lu](https://shunlinlu.github.io)\u003csup\u003e🍅 2, 3\u003c/sup\u003e, [Ling-Hao Chen](https://lhchen.top)\u003csup\u003e🍅 1, 2\u003c/sup\u003e, [Ailing Zeng](https://ailingzeng.site)\u003csup\u003e2\u003c/sup\u003e, [Jing Lin](https://jinglin7.github.io)\u003csup\u003e1, 2\u003c/sup\u003e, [Ruimao Zhang](http://zhangruimao.site)\u003csup\u003e3\u003c/sup\u003e, [Lei Zhang](https://leizhang.org)\u003csup\u003e2\u003c/sup\u003e, and [Heung-Yeung Shum](https://scholar.google.com/citations?user=9akH-n8AAAAJ\u0026hl=en)\u003csup\u003e1, 2\u003c/sup\u003e\n\n\u003csup\u003e🍅\u003c/sup\u003eCo-first author. Listing order is random. \n\n\u003csup\u003e1\u003c/sup\u003eTsinghua University, \u003csup\u003e2\u003c/sup\u003eInternational Digital Economy Academy (IDEA),\n\u003csup\u003e3\u003c/sup\u003eSchool of Data Science, The Chinese University of Hong Kong, Shenzhen (CUHK-SZ)\n\n\u003cp align=\"center\"\u003e\n  \u003ca href='https://arxiv.org/abs/2310.12978'\u003e\n  \u003cimg src='https://img.shields.io/badge/Arxiv-2310.12978-A42C25?style=flat\u0026logo=arXiv\u0026logoColor=A42C25'\u003e\n  \u003c/a\u003e \n  \u003ca href='https://arxiv.org/pdf/2310.12978.pdf'\u003e\n  \u003cimg src='https://img.shields.io/badge/Paper-PDF-yellow?style=flat\u0026logo=arXiv\u0026logoColor=yellow'\u003e\n  \u003c/a\u003e \n  \u003ca href='https://lhchen.top/HumanTOMATO'\u003e\n  \u003cimg src='https://img.shields.io/badge/Project-Page-%23df5b46?style=flat\u0026logo=Google%20chrome\u0026logoColor=%23df5b46'\u003e\u003c/a\u003e \n  \u003ca href='https://youtu.be/PcxUzZ1zg6o'\u003e\n  \u003cimg src='https://img.shields.io/badge/YouTube-Video-EA3323?style=flat\u0026logo=youtube\u0026logoColor=EA3323'\u003e\u003c/a\u003e \n  \u003ca href='https://www.bilibili.com/video/BV1xH4y1973x/'\u003e\n    \u003cimg src='https://img.shields.io/badge/Bilibili-Video-4EABE6?style=flat\u0026logo=Bilibili\u0026logoColor=4EABE6'\u003e\u003c/a\u003e\n  \u003ca href='https://github.com/IDEA-Research/HumanTOMATO'\u003e\n  \u003cimg src='https://img.shields.io/badge/GitHub-Code-black?style=flat\u0026logo=github\u0026logoColor=white'\u003e\u003c/a\u003e \n  \u003ca href='LICENSE'\u003e\n  \u003cimg src='https://img.shields.io/badge/License-IDEA-blue.svg'\u003e\n  \u003c/a\u003e \n  \u003ca href=\"\" target='_blank'\u003e\n  \u003cimg src=\"https://visitor-badge.laobi.icu/badge?page_id=IDEA-Research.HumanTOMATO\u0026left_color=gray\u0026right_color=%2342b983\"\u003e\n  \u003c/a\u003e \n\u003c/p\u003e\n\n# 🤩 Abstract\n\u003e This work targets a novel text-driven whole-body motion generation task, which takes a given textual description as input and aims at generating high-quality, diverse, and coherent facial expressions, hand gestures, and body motions simultaneously. Previous works on text-driven motion generation tasks mainly have two limitations: they ignore the key role of fine-grained hand and face controlling in vivid whole-body motion generation, and lack a good alignment between text and motion. To address such limitations, we propose a \u003cu\u003eT\u003c/u\u003eext-aligned wh\u003cu\u003eO\u003c/u\u003ele-body \u003cu\u003eM\u003c/u\u003eotion gener\u003cu\u003eAT\u003c/u\u003ei\u003cu\u003eO\u003c/u\u003en framework, named HumanTOMATO, which is the first attempt to our knowledge towards applicable holistic motion generation in this research area. To tackle this challenging task, our solution includes two key designs: (1) a \u003cu\u003eH\u003c/u\u003eolistic \u003cu\u003eH\u003c/u\u003eierarchical VQ-VAE (aka H²VQ) and a Hierarchical-GPT for fine-grained body and hand motion reconstruction and generation with two structured codebooks; and (2) a pre-trained text-motion-alignment model to help generated motion align with the input textual description explicitly. Comprehensive experiments verify that our model has significant advantages in both the quality of generated motions and their alignment with text.\n\n\n\n\n# 📢 News\n\n- **[2024/05/13] Release OpenTMA project. It is exactly the text-motion alignment used in HumanTOMATO.**\n- **[2024/05/02] HumanTOMATO is accepted by ICML-2024. See you in Vienna!**\n- **[2023/11/15] Publish HumanTOMATO Motion Representation (`tomato` representation) processing code.**\n- **[2023/10/22] Publish project!**\n\n# 🎬 Highlight Whole-body Motions\n\n![](./assets/highlight.png)\nThe proposed HumanTOMATO model can generate text-aligned whole-body motions with vivid and harmonious face, hand, and body motion. We show two generated qualitative results.\n\n\n# 🔍 System Overview\n\n![](./assets/system.png)\nThe framework overview of the proposed text-driven whole-body motion generation. (a) Holistic Hierarchical Vector Quantization (H²VQ) to compress fine-grained body-hand motion into two discrete codebooks with hierarchical structure relations. (b) Hierarchical-GPT using motion-aware textual embedding as the input to hierarchically generate body-hand motions. (c) Facial text-conditional VAE (cVAE) to generate the corresponding facial motions. The outputs of body, hand, and face motions comprise a vivid and text-aligned whole-body motion.\n\n\n# 🚀 Quick Start\n\n\n# 🚅 Model Training\n\n\n# 📸 Visualization\n\n\n# 🤝🏼 Citation\nIf you find the code is useful in your research, please cite us: \n```bash\n@article{humantomato,\n  title={HumanTOMATO: Text-aligned Whole-body Motion Generation},\n  author={Lu, Shunlin and Chen, Ling-Hao and Zeng, Ailing and Lin, Jing and Zhang, Ruimao and Zhang, Lei and Shum, Heung-Yeung},\n  journal={arxiv:2310.12978},\n  year={2023}\n}\n```\n\n## 📚 License\n\nThis code is distributed under an [IDEA LICENSE](LICENSE). Note that our code depends on other libraries and datasets which each have their own respective licenses that must also be followed.\n\n## 💋 Acknowledgement\n\nThe code is on the basis of [TMR](https://github.com/Mathux/TMR), [MLD](https://github.com/ChenFengYe/motion-latent-diffusion), [T2M-GPT](https://github.com/Mael-zys/T2M-GPT), and [HumanML3D](https://github.com/EricGuo5513/HumanML3D). Thanks to all contributors!\n\n## 🌟 Star History\n\n\u003cp align=\"center\"\u003e\n    \u003ca href=\"https://star-history.com/#IDEA-Research/HumanTOMATO\u0026Date\" target=\"_blank\"\u003e\n        \u003cimg width=\"500\" src=\"https://api.star-history.com/svg?repos=IDEA-Research/HumanTOMATO\u0026type=Date\" alt=\"Star History Chart\"\u003e\n    \u003c/a\u003e\n\u003cp\u003e\n\nIf you have any question, please contact at: shunlinlu0803 [AT] gmail [DOT] com AND thu [DOT] lhchen [AT] gmail [DOT] com.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fidea-research%2Fhumantomato","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fidea-research%2Fhumantomato","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fidea-research%2Fhumantomato/lists"}