{"id":13545336,"url":"https://github.com/MoonInTheRiver/NeuralSVB","last_synced_at":"2025-04-02T15:31:04.970Z","repository":{"id":41170420,"uuid":"464788141","full_name":"MoonInTheRiver/NeuralSVB","owner":"MoonInTheRiver","description":"Learning the Beauty in Songs: Neural Singing Voice Beautifier; ACL 2022 (Main conference); Official code","archived":false,"fork":false,"pushed_at":"2024-01-02T08:30:32.000Z","size":2040,"stargazers_count":434,"open_issues_count":14,"forks_count":52,"subscribers_count":11,"default_branch":"master","last_synced_at":"2025-03-29T12:10:04.859Z","etag":null,"topics":["acl2022","gan","singing-synthesis","singing-voice","singing-voice-conversion","singing-voice-synthesis"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MoonInTheRiver.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-03-01T07:28:19.000Z","updated_at":"2025-03-28T09:59:12.000Z","dependencies_parsed_at":"2024-11-03T12:30:37.979Z","dependency_job_id":"d07547ce-2d56-4626-9c81-30aad0fbc2e2","html_url":"https://github.com/MoonInTheRiver/NeuralSVB","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MoonInTheRiver%2FNeuralSVB","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MoonInTheRiver%2FNeuralSVB/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MoonInTheRiver%2FNeuralSVB/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MoonInTheRiver%2FNeuralSVB/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MoonInTheRiver","download_url":"https://codeload.github.com/MoonInTheRiver/NeuralSVB/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246841631,"owners_count":20842624,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["acl2022","gan","singing-synthesis","singing-voice","singing-voice-conversion","singing-voice-synthesis"],"created_at":"2024-08-01T11:01:01.114Z","updated_at":"2025-04-02T15:31:03.026Z","avatar_url":"https://github.com/MoonInTheRiver.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# Learning the Beauty in Songs: Neural Singing Voice Beautifier\n---\n[![arXiv](https://img.shields.io/badge/arXiv-Paper-\u003cCOLOR\u003e.svg)](https://arxiv.org/abs/2202.13277)\n[![GitHub Stars](https://img.shields.io/github/stars/MoonInTheRiver/NeuralSVB)](https://github.com/MoonInTheRiver/NeuralSVB)\n![visitors](https://visitor-badge.glitch.me/badge?page_id=moonintheriver/NeuralSVB)\n\n\u003cdiv align=\"center\"\u003e\n    \u003ca href=\"https://neuralsvb.github.io\" target=\"_blank\"\u003eDemo\u0026nbsp;Page\u003c/a\u003e\n\u003c/div\u003e\n\n\nThis repository is the official PyTorch implementation of our ACL-2022 [paper](https://arxiv.org/abs/2202.13277). \n\n\n## 0. Dataset (PopBuTFy) Acquirement\n### Audio samples\n- You can download the dataset from [here](https://drive.google.com/file/d/1IKFp7y1WeYGrwXgJ0HC3rdPj54WoqIsU/view?usp=sharing). Please send us an email for registration (See in [apply_form](resources/apply_form.md)).\n- Dataset [preview](https://github.com/MoonInTheRiver/NeuralSVB/releases/download/pre-release/PopBuTFy-preview.zip).\n\n### Text labels\nNeuralSVB does not need text as input, but the ASR model to extract PPG needs text. Thus we also provide the [text labels](https://github.com/MoonInTheRiver/NeuralSVB/releases/download/pre-release/text_labels.zip) of PopBuTFy. \n\u003c!-- We recommend mixing [LibriTTS](https://www.openslr.org/60/) with PopBuTFy to train the ASR model. --\u003e\n\n## 1. Preparation\n\n### Environment Preparation\nMost of the required packages are in https://github.com/NATSpeech/NATSpeech/blob/main/requirements.txt\n\nOr you can prepare environments with the Requirements.txt file in the repository directory.\n```sh\npip install Requirements.txt\n```\n### Data Preparation\n\n\n1. Extract embeddings of vocal timbre:\n    ```sh \n    CUDA_VISIBLE_DEVICES=0 python data_gen/tts/bin/binarize.py --config egs/datasets/audio/PopBuTFy/save_emb.yaml\n    ```\n2. Pack the dataset:\n    ```sh \n    CUDA_VISIBLE_DEVICES=0 python data_gen/tts/bin/binarize.py --config egs/datasets/audio/PopBuTFy/para_bin.yaml\n    ```\n\n\n### Vocoder Preparation\nWe provide the pre-trained model of [HifiGAN-Singing](https://github.com/MoonInTheRiver/NeuralSVB/releases/download/pre-release/1012_hifigan_all_songs_nsf.zip) which is specially designed for SVS with NSF mechanism.\n\nPlease unzip pre-trained vocoder into `checkpoints` before training your acoustic model.\n\nThis singing vocoder is trained on 100+ hours singing data (including Chinese and English songs). \n\n### PPG Extractor Preparation\nWe provide the pre-trained model of [PPG Extractor](https://github.com/MoonInTheRiver/NeuralSVB/releases/download/pre-release/1009_pretrain_asr_english.zip).\n\nPlease unzip pre-trained PPG extractor into `checkpoints` before training your acoustic model.\n\n\nAfter the instructions above, the directory structure should be as follows:\n\n```\n.\n|--data\n    |--processed\n        |--PopBuTFy (unzip PopBuTFy.zip)\n            |--data\n                |--directories containing wavs\n    |--binary\n        |--PopBuTFyENSpkEM\n|--checkpoints\n    |--1009_pretrain_asr_english\n        |--\n        |--config.yaml\n    |--1012_hifigan_all_songs_nsf\n        |--\n        |--config.yaml\n```\n\n\n## 2. Training Example\n\n```sh\nCUDA_VISIBLE_DEVICES=0,1 python tasks/run.py --config egs/datasets/audio/PopBuTFy/vae_global_mle_eng.yaml --exp_name exp_name --reset\n```\n\n## 3. Inference\n### Inference from packed test set\n\n```sh\nCUDA_VISIBLE_DEVICES=0,1 python tasks/run.py --config egs/datasets/audio/PopBuTFy/vae_global_mle_eng.yaml --exp_name exp_name --reset --infer\n```\nInference results will be saved in `./checkpoints/EXP_NAME/generated_` by default.\n\nWe provided:\n - the [pre-trained model](https://github.com/MoonInTheRiver/NeuralSVB/releases/download/pre-release/1030_vae_mle.zip) of NSVB (en version);\n\nRemember to put the pre-trained models in `checkpoints` directory.\n\n### Inference from raw inputs\nWIP.\n\n## Limitations\nSee Appendix D \"Limitations and Solutions\" in our [paper](https://aclanthology.org/2022.acl-long.549.pdf).\n\n## Citation\nIf this repository helps your research, please cite:\n\n    @inproceedings{liu-etal-2022-learning-beauty,\n    title = \"Learning the Beauty in Songs: Neural Singing Voice Beautifier\",\n    author = \"Liu, Jinglin  and\n      Li, Chengxi  and\n      Ren, Yi  and\n      Zhu, Zhiying  and\n      Zhao, Zhou\",\n    booktitle = \"Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)\",\n    month = may,\n    year = \"2022\",\n    address = \"Dublin, Ireland\",\n    publisher = \"Association for Computational Linguistics\",\n    url = \"https://aclanthology.org/2022.acl-long.549\",\n    pages = \"7970--7983\",}\n\n\n## Issues\n - Before raising a issue, please check our Readme and other issues for possible solutions.\n - We will try to handle your problem in time but we could not guarantee a satisfying solution.\n - Please be friendly.\n\n## Acknowledgements\n* r9y9's [wavenet_vocoder](https://github.com/r9y9/wavenet_vocoder)\n* Po-Hsun-Su's [ssim](https://github.com/Po-Hsun-Su/pytorch-ssim)\n* descriptinc's [melgan](https://github.com/descriptinc/melgan-neurips)\n* Official [espnet](https://github.com/espnet/espnet)\n* Official [PyTorch Lightning](https://github.com/PyTorchLightning/pytorch-lightning)\n\nThe framework of this repository is based on [DiffSinger](https://github.com/MoonInTheRiver/DiffSinger), \nand is a predecessor of [NATSpeech](https://github.com/NATSpeech/NATSpeech/). \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMoonInTheRiver%2FNeuralSVB","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FMoonInTheRiver%2FNeuralSVB","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMoonInTheRiver%2FNeuralSVB/lists"}