{"id":13563621,"url":"https://github.com/Mathux/TEMOS","last_synced_at":"2025-04-03T20:31:20.119Z","repository":{"id":41824941,"uuid":"487791649","full_name":"Mathux/TEMOS","owner":"Mathux","description":"Official PyTorch implementation of the paper \"TEMOS: Generating diverse human motions from textual descriptions\", ECCV 2022 (Oral)","archived":false,"fork":false,"pushed_at":"2023-12-13T18:39:05.000Z","size":4113,"stargazers_count":377,"open_issues_count":2,"forks_count":25,"subscribers_count":12,"default_branch":"master","last_synced_at":"2024-11-04T16:44:56.466Z","etag":null,"topics":["human-motion","motion-generation","text-analysis"],"latest_commit_sha":null,"homepage":"https://mathis.petrovich.fr/temos/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Mathux.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2022-05-02T09:46:21.000Z","updated_at":"2024-11-01T03:14:35.000Z","dependencies_parsed_at":"2024-01-20T19:07:10.625Z","dependency_job_id":"ab909520-8603-40c2-a057-0032d7c2be4c","html_url":"https://github.com/Mathux/TEMOS","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Mathux%2FTEMOS","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Mathux%2FTEMOS/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Mathux%2FTEMOS/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Mathux%2FTEMOS/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Mathux","download_url":"https://codeload.github.com/Mathux/TEMOS/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247074616,"owners_count":20879282,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["human-motion","motion-generation","text-analysis"],"created_at":"2024-08-01T13:01:21.431Z","updated_at":"2025-04-03T20:31:19.642Z","avatar_url":"https://github.com/Mathux.png","language":"Python","funding_links":[],"categories":["Python","Papers"],"sub_categories":["Text-Driven motion generation"],"readme":"\u003cdiv align=\"center\"\u003e\n\n# TEMOS: TExt to MOtionS\n## Generating diverse human motions from textual descriptions\n\n\u003c/div\u003e\n\n## Description\nOfficial PyTorch implementation of the paper [**\"TEMOS: Generating diverse human motions from textual descriptions\"**](http://arxiv.org/abs/2204.14109), [ECCV 2022 (Oral)](https://eccv2022.ecva.net).\n\nPlease visit our [**webpage**](https://mathis.petrovich.fr/temos/) for more details.\n\n![teaser_light](visuals/teaser_white.png#gh-light-mode-only)![teaser_dark](visuals/teaser_black.png#gh-dark-mode-only)\n\n\n### Bibtex\nIf you find this code useful in your research, please cite:\n\n```\n@inproceedings{petrovich22temos,\n  title     = {{TEMOS}: Generating diverse human motions from textual descriptions},\n  author    = {Petrovich, Mathis and Black, Michael J. and Varol, G{\\\"u}l},\n  booktitle = {European Conference on Computer Vision ({ECCV})},\n  year      = {2022}\n}\n```\n\nYou can also put a star :star:, if the code is useful to you.\n\n\n## Installation :construction_worker:\n\n\u003cdetails\u003e\u003csummary\u003eClick to expand\u003c/summary\u003e\n\n### 1. Create conda environment\n\n\u003cdetails\u003e\u003csummary\u003eInstructions\u003c/summary\u003e\n\n```\nconda create python=3.9 --name temos\nconda activate temos\n```\n\nInstall [PyTorch 1.10](https://pytorch.org/) inside the conda environment, and install the following packages:\n```bash\npip install pytorch_lightning --upgrade\npip install torchmetrics==0.7\npip install hydra-core --upgrade\npip install hydra_colorlog --upgrade\npip install shortuuid\npip install rich\npip install pandas\npip install transformers\npip install psutil\npip install einops\n```\nThe code was tested on Python 3.9.7 and PyTorch 1.10.0.\n\n\u003c/details\u003e\n\n### 2. Download the datasets\n\n\u003cdetails\u003e\u003csummary\u003eInstructions\u003c/summary\u003e\n\n#### KIT Motion-Language dataset\n**Be sure to read and follow their license agreements, and cite accordingly.**\n\nUse the code from [Ghosh et al.](https://github.com/anindita127/Complextext2animation) to download and prepare the kit dataset (extraction of xyz joints coodinates data from axis-angle Master Motor Map). Move or copy all the files which ends with \"_meta.json\", \"_annotations.json\" and \"_fke.csv\" inside the ``datasets/kit`` folder.\n\"\nThese motions are process by the Master Motor Map (MMM) framework. To be able to generate motions with SMPL body model, please look at the next section.\n\n#### (Optional) Motion processed with MoSh++ (in AMASS)\n**Be sure to read and follow their license agreements, and cite accordingly.**\n\nCreate this folder:\n```bash\nmkdir datasets/AMASS/\n```\n\nGo to the [AMASS website](https://amass.is.tuebingen.mpg.de/download.php), register and go to the Download tab. Then download the \"SMPL+H G\" files corresponding to the datasets [KIT, CMU, EKUT] into the ``datasets/AMASS`` directory and uncompress the archives:\n\n```bash\ncd datasets/AMASS/\ntar xfv CMU.tar.bz2\ntar xfv KIT.tar.bz2\ntar xfv EKUT.tar.bz2\ncd ../../\n```\n\n\u003c/details\u003e\n\n### 3. Download text model dependencies\n\n\u003cdetails\u003e\u003csummary\u003eInstructions\u003c/summary\u003e\n\n#### Download distilbert from __Hugging Face__\n```bash\ncd deps/\ngit lfs install\ngit clone https://huggingface.co/distilbert-base-uncased\ncd ..\n```\n\n\u003c/details\u003e\n\n### 4. (Optional) SMPL body model\n\n\u003cdetails\u003e\u003csummary\u003eInstructions\u003c/summary\u003e\n\nThis is only useful if you want to use generate 3D human meshes like in the teaser. In this case, you also need a subset of the AMASS dataset (see instructions below).\n\nGo to the [MANO website](https://mano.is.tue.mpg.de/download.php), register and go to the Download tab.\n\n- Click on \"Models \u0026 Code\" to download ``mano_v1_2.zip`` and place it in the folder ``deps/smplh/``.\n- Click on \"Extended SMPL+H model\" to download ``smplh.tar.xz`` and place it in the folder ``deps/smplh/``.\n\nThe next step is to extract the archives, merge the hands from ``mano_v1_2`` into the ``Extended SMPL+H models``, and remove any chumpy dependency.\nAll of this can be done using with the following commands. (I forked both scripts from this repo [SMPLX repo](https://github.com/vchoutas/smplx/tree/master/tools), updated them to Python 3, merged them, and made it compatible with ``.npz`` files).\n\n\n```bash\npip install scipy chumpy\nbash prepare/smplh.sh\n```\n\nThis will create ``SMPLH_FEMALE.npz``, ``SMPLH_MALE.npz``, ``SMPLH_NEUTRAL.npz`` inside the ``deps/smplh`` folder.\n\n\u003c/details\u003e\n\n### 5. (Optional) Download pre-trained models\n\n\u003cdetails\u003e\u003csummary\u003eInstructions\u003c/summary\u003e\n\nMake sure to have gdown installed\n\n```bash\npip install --user gdown\n```\n\nThen, please run this command line:\n\n```bash\nbash prepare/download_pretrained_models.sh\n```\n\nInside the ``pretrained models`` folder, you will find one for each type of data (see Section [datasets](#datasets) below for more information).\n```\npretrained_models\n├── kit-amass-rot\n│   └── 1cp6dwpa\n├── kit-amass-xyz\n│   └── 5xp9647f\n└── kit-mmm-xyz\n    └── 3l49g7hv\n```\n\n\u003c/details\u003e\n\n\u003c/details\u003e\n\n## How to train TEMOS :rocket:\n\n\u003cdetails\u003e\u003csummary\u003eClick to expand\u003c/summary\u003e\n\nThe command to launch a training experiment is the folowing:\n```bash\npython train.py [OPTIONS]\n```\n\nThe parsing is done by using the powerful [Hydra](https://github.com/facebookresearch/hydra) library. You can override anything in the configuration by passing arguments like ``foo=value`` or ``foo.bar=value``.\n\n\n### Experiment path\nEach training will create a unique output directory (referred to as ``FOLDER`` below), where logs, configuations and checkpoints are stored.\n\nBy default it is defined as ``outputs/${data.dataname}/${experiment}/${run_id}`` with ``data.dataname`` the name of the dataset (see examples below), ``experiment=baseline`` and ```run_id``` a 8 unique random alpha-numeric identifier for the run (everything can be overridden if needed).\n\nThis folder is printed during logging, it should look like ``outputs/kit-mmm-xyz/baseline/3gn7h7v6/``.\n\n\n### Some optional parameters\n#### Datasets\n- ``data=kit-mmm-xyz``: KIT-ML motions processed by the [MMM](https://mmm.humanoids.kit.edu/) framework (as in the [original data](https://motion-annotation.humanoids.kit.edu/dataset/)) loaded as xyz joint coordinates (after axis-angle transformation → xyz) (by default)\n- ``data=kit-amass-rot``: KIT-ML motions loaded as [SMPL](https://smpl.is.tue.mpg.de/) rotations and translations, from [AMASS](https://amass.is.tue.mpg.de/) (processed with [MoSh++](https://github.com/nghorbani/moshpp))\n- ``data=kit-amass-xyz``: KIT-ML motions loaded as xyz joint coordinates, from [AMASS](https://amass.is.tue.mpg.de/) (processed with [MoSh++](https://github.com/nghorbani/moshpp)) after passing through a [SMPL](https://smpl.is.tue.mpg.de/) layer and regressing the correct joints.\n\n\n#### Training\n- ``trainer=gpu``: training with CUDA, on an automatically selected GPU (default)\n- ``trainer=cpu``: training on the CPU (not recommended)\n\n\n\u003c/details\u003e\n\n## How to generate motions with TEMOS :walking:\n\n\u003cdetails\u003e\u003csummary\u003eClick to expand\u003c/summary\u003e\n\n### Dataset splits\nTo get results comparable to previous work, we use the same splits as in [Language2Pose](https://github.com/chahuja/language2pose) and [Ghosh et al.](https://github.com/anindita127/Complextext2animation).\nTo be explicit, and not rely on random seeds, you can find the list of id-files in [datasets/kit-splits/](datasets/kit-splits/) ([train](datasets/kit-splits/train)/[val](datasets/kit-splits/val)/[test](datasets/kit-splits/test)).\n\nWhen sampling [Ghosh et al.](https://github.com/anindita127/Complextext2animation)'s motions with their code, I noticed that their dataloader is missing some sequences (see the discussion [here](https://github.com/anindita127/Complextext2animation/issues/3#issuecomment-1059566036)).\nIn order to compare all the methods with the same test set, we use the 520 sequences produced by Ghosh et al. code for the test set (instead of the 587 sequences). This split is refered as [gtest](datasets/kit-splits/gtest) (for \"Ghosh test\"). It is used per default in the sampling/evaluation/rendering code. You can change this set by specifying ``split=SPLIT`` in each command line.\n\nYou can also find in [datasets/kit-splits/](datasets/kit-splits/), the split used for the human-study ([human-study](datasets/kit-splits/human-study)) and the split used for the visuals of the paper ([visu](datasets/kit-splits/visu)).\n\n\n### Sampling/generating motions\nThe command line to sample one motion per sequence is the following:\n```bash\npython sample.py folder=FOLDER [OPTIONS]\n```\n\nThis command will create the folder ``FOLDER/samples/SPLIT`` and save the motions in the npy format.\n\n### Some optional parameters\n- ``mean=false``: Take the mean value for the latent vector, instead of sampling (default is false)\n- ``number_of_samples=X``: Generate ``X`` motions (by default it generates only one)\n- ``fact=X``: Multiplies sigma by ``X`` during sampling (1.0 by default, diversity can be increased when ``fact\u003e1``)\n\n\n### Model trained on SMPL rotations\nIf your model has been trained with ``data=kit-amass-rot``, it produces [SMPL](https://smpl.is.tue.mpg.de/) rotations and translations. In this case, you can specify the type of data you want to save after passing through the [SMPL](https://smpl.is.tue.mpg.de/) layer.\n- ``jointstype=mmm``: Generate xyz joints compatible with the [MMM](https://mmm.humanoids.kit.edu/) bodies (by default). This gives skeletons comparable to ``data=kit-mmm-xyz`` (needed for evaluation).\n- ``jointstype=vertices``: Generate human body meshes (needed for rendering).\n\n\u003c/details\u003e\n\n## Evaluating TEMOS (and prior works) :bar_chart:\n\n\u003cdetails\u003e\u003csummary\u003eClick to expand\u003c/summary\u003e\n\nTo evaluate TEMOS on the metrics defined in the paper, you must generate motions first (see above), and then run:\n```bash\npython evaluate.py folder=FOLDER [OPTIONS]\n```\nThis will compute and store the metrics in the file ``FOLDER/samples/metrics_SPLIT`` in a yaml format.\n\n### Some optional parameters\nSame parameters as in ``sample.py``, it will choose the right directories for you. In the case of evaluating with ``number_of_samples\u003e1``, the script will compute two metrics ``metrics_gtest_multi_avg`` (the average of single metrics) and ``metrics_gtest_multi_best`` (chosing the best output for each motion). Please check the paper for more details.\n\n### Model trained on SMPL rotations\nCurrently, evaluation is only implemented on skeletons with [MMM](https://mmm.humanoids.kit.edu/) format. You must therefore use ``jointstype=mmm`` during sampling.\n\n\n### Evaluating prior works\n\nPlease use this command line to download the motions generated from previous work:\n\n```bash\nbash prepare/download_previous_works.sh\n```\n\nThen, to evaluate a method, you can do for example:\n\n```bash\npython evaluate.py folder=previous_work/ghosh\n```\n\nor change \"ghosh\" with \"jl2p\" or \"lin\".\n\n\nTo give an overview on how to extract their motions:\n1. Generate motions with their code (it is still in the rifke feature space)\n2. Save them in xyz format (I \"hack\" their render script, to save them in xyz npy format instead of rendering)\n3. Load them into the evaluation code, as shown above.\n\n\u003c/details\u003e\n\n## Rendering motions :high_brightness:\n\n\u003cdetails\u003e\u003csummary\u003eClick to expand\u003c/summary\u003e\n\nTo get the visuals of the paper, I use [Blender 2.93](https://www.blender.org/download/releases/2-93/). The setup is not trivial (installation + running), I do my best to explain the process but don't hesitate to tell me if you have a problem.\n\n\n### Instalation\nThe goal is to be able to install blender so that it can be used with python scripts (so we can use ``import bpy''). There seem to be many different ways to do this, I will explain the one I use and understand (feel free to use other methods or suggest an easier way). The installation of Blender will be done as a standalone package. To use my scripts, we will run blender in the background, and the python executable in blender will run the script.\n\nIn any case, after the installation, please do step 5/6. to install the dependencies in the python environment.\n\n1. Please follow the [instructions](https://www.blender.org/download/lts/2-93/) to install blender 2.93 on your operating system. Please install exactly this version.\n2. Locate the blender executable if it is not in your path. For the following commands, please replace ``blender`` with the path to your executable (or create a symbolic link or use an alias).\n   - On Linux, it could be in ``/usr/bin/blender`` or ``/snap/bin/blender`` (already in your path).\n   - On macOS, it could be in ``/Applications/Blender.app/Contents/MacOS/Blender`` (not in your path)\n3. Check that the correct version is installed:\n   - ``blender --background --version`` should return \"Blender 2.93.X\".\n   - ``blender --background --python-expr \"import sys; print('\\nThe version of python is '+sys.version.split(' ')[0])\"`` should return \"3.9.X\".\n4. Locate the python installation used by blender the following line. I will refer to this path as ``/path/to/blender/python``.\n```bash\nblender --background --python-expr \"import sys; import os; print('\\nThe path to the installation of python of blender can be:'); print('\\n'.join(['- '+x.replace('/lib/python', '/bin/python') for x in sys.path if 'python' in (file:=os.path.split(x)[-1]) and not file.endswith('.zip')]))\"\n```\n\n5. Install pip\n```bash\n/path/to/blender/python -m ensurepip --upgrade\n```\n\n6. Install these packages in the python environnement of blender:\n```bash\n/path/to/blender/python -m pip install --user numpy\n/path/to/blender/python -m pip install --user matplotlib\n/path/to/blender/python -m pip install --user hydra-core --upgrade\n/path/to/blender/python -m pip install --user hydra_colorlog --upgrade\n/path/to/blender/python -m pip install --user moviepy\n/path/to/blender/python -m pip install --user shortuuid\n```\n\n### Launch a python script (with arguments) with blender\nNow that blender is installed, if we want to run the script ``script.py`` with the blender API (the ``bpy`` module), we can use:\n```bash\nblender --background --python script.py\n```\n\nIf you need to add additional arguments, this will probably fail (as blender will interpret the arguments). Please use the double dash ``--`` to tell blender to ignore the rest of the command.\nI then only parse the last part of the command (check [temos/launch/blender.py](temos/launch/blender.py) if you are interested).\n\n\n### Rendering one sample\nTo render only one motion, please use this command line:\n```bash\nblender --background --python render.py -- npy=PATH_TO_DATA.npy [OPTIONS]\n```\n\n### Rendering all the npy of a folder\nPlease use this command line to render all the npy inside a specific folder.\n```bash\nblender --background --python render.py -- folder=FOLDER_WITH_NPYS [OPTIONS]\n```\n\n### SMPL bodies\nDon't forget to generate the data with the option ``jointstype=vertices`` before.\nThe renderer will automatically detect whether the motion is a sequence of joints or meshes.\n\n\n### Some optional parameters\n- ``downsample=true``: Render only 1 frame every 8 frames, to speed up rendering (by default)\n- ``canonicalize=true``: Make sure the first pose is oriented canonically (by translating and rotating the entire sequence) (by default)\n- ``mode=XXX``: Choose the rendering mode (default is ``mode=sequence``)\n  - ``video``: Render all the frames and generate a video (as in the supplementary video)\n  - ``sequence``: Render a single frame, with ``num=8`` bodies (sampled equally, as in the figures of the paper)\n  - ``frame``: Render a single frame, at a specific point in time (``exact_frame=0.5``, generates the frame at about 50% of the video)\n- ``quality=false``: Render to a higher resolution and denoise the output (default to false to speed up))\n\n\u003c/details\u003e\n\n## License :books:\nThis code is distributed under an [MIT LICENSE](LICENSE).\n\nNote that our code depends on other libraries, including SMPL, SMPL-X, PyTorch3D, Hugging Face, Hydra, and uses datasets which each have their own respective licenses that must also be followed.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMathux%2FTEMOS","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FMathux%2FTEMOS","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMathux%2FTEMOS/lists"}