Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/jw09191/MNET
Official PyTorch implementation of the paper "A Brand New Dance Partner:Music-Conditioned Pluralistic Dancing Synthesized by Multiple Dance Genres", CVPR 2022
https://github.com/jw09191/MNET
Last synced: about 2 months ago
JSON representation
Official PyTorch implementation of the paper "A Brand New Dance Partner:Music-Conditioned Pluralistic Dancing Synthesized by Multiple Dance Genres", CVPR 2022
- Host: GitHub
- URL: https://github.com/jw09191/MNET
- Owner: jw09191
- Created: 2022-02-17T06:32:30.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2022-07-06T05:49:47.000Z (about 2 years ago)
- Last Synced: 2024-04-05T01:51:37.349Z (6 months ago)
- Language: Python
- Size: 62.9 MB
- Stars: 34
- Watchers: 4
- Forks: 4
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# MNET
Official PyTorch implementation of the paper ["A Brand New Dance Partner:Music-Conditioned Pluralistic Dancing Synthesized by Multiple Dance Genres", CVPR 2022](https://openaccess.thecvf.com/content/CVPR2022/html/Kim_A_Brand_New_Dance_Partner_Music-Conditioned_Pluralistic_Dancing_Controlled_by_CVPR_2022_paper.html).
Please visit our [website](https://jw09191.github.io/MNET_CVPR22/) for more details.
## Framework
We propose MNET, a novel and scalable approach that can perform music-conditioned pluralistic dance generation synthesized by multiple dance genres using only a single model. Here, we learn a dance-genre aware latent representation by training a conditional generative adversarial network leveraging Transformer architecture.
## 1. Getting started
```bash
# clone this repo:
git clone https://github.com/jw09191/MNET
cd MNET
# [OPTIONAL] create conda environment
conda create -n mnet python=3.8
conda activate mnet
# install requirements
pip install -r requirements.txt
```
## 2. Download the datasets
In our experiment, we use AIST++. For more information about the dataset, please check this [page](https://google.github.io/aistplusplus_dataset/).
Please download the annotated motion datasets from [here](https://storage.cloud.google.com/aist_plusplus_public/20210308/motions.zip),
and unzip them as ``./data/AIST++/motions/`` folder.
Please download the music pieces (wav) from [here](https://aistdancedb.ongaaccel.jp/v1.0.0/audio/wav.zip),
and unzip them as ``./data/AIST++/wav/`` folder. And please download SMPL models [here](https://smpl.is.tue.mpg.de/),
and copy the downloaded 'male' SMPL model to ``./data/AIST++/SMPL_DIR/`` folder.
## 3. How to use MNET
### Training
```bash
# train on CPU
python train.py experiment=AISTPP trainer.gpus=0
# train on single GPU
python train.py experiment=AISTPP trainer.gpus=1
# train with DDP (Distributed data Parallel) (4 GPUs)
python train.py experiment=AISTPP trainer.gpus=4 +trainer.strategy=ddp
```
### Pretrained models
You can download pretrained models from [here](https://drive.google.com/file/d/1oT8w6EtURFiC0mmre5Rrg3Ko4Qdns328/view?usp=sharing) (Google Drive),
and move them to ``./logs/pretrain/`` folder.
### Visualization
You can choose .pkl data from ``./data/AIST++/motions`` and the generated dance videos are saved in ``./logs/pretrain/demos/`` folder.
#### Generate diverse dance
```bash
# example
python demo.py -l pretrain -p gBR_sBM_cAll_d04_mBR0_ch01.pkl -t diversity
```
#### Generate dance controlled by dance genre
```bash
# example
python demo.py -l pretrain -p gBR_sBM_cAll_d04_mBR0_ch01.pkl -t synthesis
```
## 4. Citation
@InProceedings{Kim_2022_CVPR,
author = {Kim, Jinwoo and Oh, Heeseok and Kim, Seongjean and Tong, Hoseok and Lee, Sanghoon},
title = {A Brand New Dance Partner: Music-Conditioned Pluralistic Dancing Controlled by Multiple Dance Genres},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2022},
pages = {3490-3500}
}
Our code is released under MIT License.