https://github.com/TheTempAccount/Co-Speech-Motion-Generation

Freeform Body Motion Generation from Speech
https://github.com/TheTempAccount/Co-Speech-Motion-Generation

co-speech-gesture generation-algorithms gesture-controller talking-head

Last synced: 3 months ago
JSON representation

Freeform Body Motion Generation from Speech

• Host: GitHub
• URL: https://github.com/TheTempAccount/Co-Speech-Motion-Generation
• Owner: TheTempAccount
• Created: 2021-10-17T13:55:13.000Z (about 3 years ago)
• Default Branch: main
• Last Pushed: 2022-11-12T03:56:58.000Z (almost 2 years ago)
• Last Synced: 2024-05-19T16:34:41.450Z (6 months ago)
• Topics: co-speech-gesture, generation-algorithms, gesture-controller, talking-head
• Language: Python
• Homepage:
• Size: 82.4 MB
• Stars: 191
• Watchers: 8
• Forks: 26
• Open Issues: 22
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

• awesome-conditional-content-generation - [Code
• awesome-ai-talking-heads - Co-Speech-Motion-Generation

README

        
# Freeform Co-Speech Gesture Generation

The repo for work "Free-form Co-Speech Gesture Generation"

- source code 

- data preparation (partially)

### Video Demo

[![](https://res.cloudinary.com/marcomontalbano/image/upload/v1639640143/video_to_markdown/images/youtube--Wb5VYqKX_x0-c05b58ac6eb4c4700831b2b3070cd403.jpg)](https://youtu.be/Wb5VYqKX_x0 "")

### Data & Pretrained model

Avaliable through 

- [Google Drive](https://drive.google.com/drive/folders/1v8_4agLNM2jmRuSKnflkdbEcKWZkdtka?usp=sharing)

Unzip everything in *pose_dataset*, then change the *Data.data_root* in src/config/*.json. You should be seeing directory structure like this:

    pose_dataset

    |-videos

    |   |-Speaker_A

    |   |-Speaker_B

    |   |-...

    |   |-test_audios

    |-ckpt

The rest of the data will be updated after I finish checking the annotations.

### Inference

Generated gestures for an example audio clip:

    bash demo.sh ../sample_audio/clip000040_ozfGHONpdTA.wav ../sample_audio/clip000040_ozfGHONpdTA.TextGrid

Visualise the generated motions:

    bash visualse.sh

Generate gestures for a speaker in test_audios:

    cd src

    bash infer.sh  \

            pose_dataset/ckpt/ckpt-99.pth \

            pose_dataset/ckpt/freeMo.json \

             \

            

The results will be saved as "pose_dataset/videos/test_audios//*_.json", including the json file of 64 randomly generated gesture sequences for every audio. 

To visualise the results, run

    bash visualise/visualise_all.sh  

Remember to change the file path in all files.

### Training

    

    bash train.sh

For any problem, please let us know.