{"id":19684162,"url":"https://github.com/maxmax2016/grad-tts-chinese","last_synced_at":"2025-04-29T05:32:10.064Z","repository":{"id":195724978,"uuid":"693449863","full_name":"MaxMax2016/Grad-TTS-Chinese","owner":"MaxMax2016","description":"Huawei Grad-TTS for Chinese","archived":false,"fork":false,"pushed_at":"2023-09-26T01:55:41.000Z","size":691,"stargazers_count":49,"open_issues_count":2,"forks_count":4,"subscribers_count":3,"default_branch":"V2-CFM","last_synced_at":"2025-04-05T13:38:09.611Z","etag":null,"topics":["diffusion","grad-tts","speach","tts"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MaxMax2016.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-09-19T03:58:04.000Z","updated_at":"2025-04-02T09:53:58.000Z","dependencies_parsed_at":null,"dependency_job_id":"897094cf-e983-414e-a75b-bccc6fc2fb04","html_url":"https://github.com/MaxMax2016/Grad-TTS-Chinese","commit_stats":null,"previous_names":["playvoice/grad-tts-chinese","yuchendd/grad-tts-chinese","maxmax2016/grad-tts-chinese"],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MaxMax2016%2FGrad-TTS-Chinese","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MaxMax2016%2FGrad-TTS-Chinese/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MaxMax2016%2FGrad-TTS-Chinese/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MaxMax2016%2FGrad-TTS-Chinese/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MaxMax2016","download_url":"https://codeload.github.com/MaxMax2016/Grad-TTS-Chinese/tar.gz/refs/heads/V2-CFM","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251444695,"owners_count":21590557,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["diffusion","grad-tts","speach","tts"],"created_at":"2024-11-11T18:16:59.801Z","updated_at":"2025-04-29T05:32:09.634Z","avatar_url":"https://github.com/MaxMax2016.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Huawei Grad-TTS for Chinese, integrated Bert and BigVGAN\n\n**用于学习的TTS算法项目，推理速度比较慢，但diffusion是大趋势**\n\u003cdiv align=\"center\"\u003e\n\n![grad_tts](assets/grad_tts.jpg)\n\n![bert_grad_tts](assets/bert_grad_tts.jpg)\nGrad-TTS-CFM Framework\n\u003c/div\u003e\n\n\n## 使用已训练模型测试\n\n1. 从 [NVIDIA/BigVGAN]( https://github.com/NVIDIA/BigVGAN) 下载声码器模型 [bigvgan_base_24khz_100band](https://drive.google.com/drive/folders/1e9wdM29d-t3EHUpBb8T4dcHrkYGAXTgq) \n\n\t将 [g_05000000]() 放到 ./bigvgan_pretrain/g_05000000\n\n2. 从 [Executedone/Chinese-FastSpeech2](https://github.com/Executedone/Chinese-FastSpeech2) 下载BERT韵律模型 [prosody_model](https://github.com/Executedone/Chinese-FastSpeech2)\n\n\t将 best_model.pt 改名为 prosody_model.pt，并放到 ./bert/prosody_model.pt\n\n3. 从Release页面下载TTS模型 [grad_tts.pt](https://github.com/PlayVoice/Grad-TTS-Chinese/releases/tag/20230923) from release page\n\n\t将 [grad_tts.pt]() 放到当前目录，或者任意地方\n\n4. 安装环境依赖\n\n\t\u003e pip install -r requirements.txt\n \n\t\u003e cd ./grad/monotonic_align\n \n\t\u003e python setup.py build_ext --inplace\n \n\t\u003e cd -\n\n5. 推理测试\n\n\t\u003e python inference.py --file test.txt --checkpoint grad_tts.pt --timesteps 10 --temperature 1.015\n\n\t生成音频在文件夹`./inference_out`\n\n\t**`timesteps` 越大效果越好、推理时间越久；当被设置为0, 将跳过diffusion、输出FrameEncoder生成的mel谱**\n\n\t**`temperature` 决定diffusion推理添加的噪声量，需要调试出最佳值**\n\n## 标贝数据\n\n1. 下载 [标贝数据](https://aistudio.baidu.com/datasetdetail/36741) 官方连接: https://www.data-baker.com/data/index/TNtts/\n\n\t将 `Waves` 放到 ./data/Waves\n\n\t将 `000001-010000.txt` 放到 ./data/000001-010000.txt\n\n2. 重采样到24KHz，因为采用BigVGAN 24K模型\n\n\t\u003e python tools/preprocess_a.py -w ./data/Wave/ -o ./data/wavs -s `24000`\n\n3. 提取mel谱，替换声码器需注意，mel参数写死在代码中\n\n\t\u003e python tools/preprocess_m.py --wav data/wavs/ --out data/mels/\n\n4. 提取BERT韵律向量，同时生成训练索引文件 `train.txt` 和 `valid.txt`\n\n\t\u003e python tools/preprocess_b.py\n\n\t输出包括 `data/berts/` 和 `data/files`\n\n\t注意：打印信息，是在剔除`儿化音`（项目为算法演示，不做生产）\n\n5. 额外说明\n\n\t原始标注为\n\t``` c\n\t000001\t卡尔普#2陪外孙#1玩滑梯#4。\n\t\tka2 er2 pu3 pei2 wai4 sun1 wan2 hua2 ti1\n\t000002\t假语村言#2别再#1拥抱我#4。\n\t\tjia2 yu3 cun1 yan2 bie2 zai4 yong1 bao4 wo3\n\t```\n\n\t需要标注为，BERT需要汉字 `卡尔普陪外孙玩滑梯。` (包括标点)，TTS需要声韵母 `sil k a2 ^ er2 p u3 p ei2 ^ uai4 s uen1 ^ uan2 h ua2 t i1 sp sil`\n\t``` c\n\t000001\t卡尔普陪外孙玩滑梯。\n\t\tka2 er2 pu3 pei2 wai4 sun1 wan2 hua2 ti1\n\t\tsil k a2 ^ er2 p u3 p ei2 ^ uai4 s uen1 ^ uan2 h ua2 t i1 sp sil\n\t000002\t假语村言别再拥抱我。\n\t\tjia2 yu3 cun1 yan2 bie2 zai4 yong1 bao4 wo3\n\t\tsil j ia2 ^ v3 c uen1 ^ ian2 b ie2 z ai4 ^ iong1 b ao4 ^ uo3 sp sil\n\t```\n\n\t训练标注为\n\t```\n\t./data/wavs/000001.wav|./data/mels/000001.pt|./data/berts/000001.npy|sil k a2 ^ er2 p u3 p ei2 ^ uai4 s uen1 ^ uan2 h ua2 t i1 sp sil\n\t./data/wavs/000002.wav|./data/mels/000002.pt|./data/berts/000002.npy|sil j ia2 ^ v3 c uen1 ^ ian2 b ie2 z ai4 ^ iong1 b ao4 ^ uo3 sp sil\n\t```\n\n\t遇到这句话会出错\n\t```\n\t002365\t这图#2难不成#2是#1Ｐ过的#4？\n\t\tzhe4 tu2 nan2 bu4 cheng2 shi4 P IY1 guo4 de5\n\t```\n\n## 训练\n\n1. 调试dataset\n\n\t\u003e python tools/preprocess_d.py\n\n2. 启动训练\n\n\t\u003e python train.py\n\n3. 恢复训练\n\n\t\u003e python train.py -p logs/new_exp/grad_tts_***.pt\n\n## 推理\n\n\u003e python inference.py --file test.txt --checkpoint ./logs/new_exp/grad_tts_***.pt --timesteps 20 --temperature 1.15\n\n## Loss\n![grad_tts_loss](assets/grad_tts_loss.png)\n\n## 本项目基于以下项目\n\nhttps://github.com/huawei-noah/Speech-Backbones/blob/main/Grad-TTS\n\nhttps://github.com/shivammehta25/Matcha-TTS\n\nhttps://github.com/thuhcsi/LightGrad\n\nhttps://github.com/Executedone/Chinese-FastSpeech2\n\nhttps://github.com/PlayVoice/vits_chinese\n\nhttps://github.com/NVIDIA/BigVGAN\n\n# Grad-TTS官方信息\n\nOfficial implementation of the Grad-TTS model based on Diffusion Probabilistic Modelling. For all details check out our paper accepted to ICML 2021 via [this](https://arxiv.org/abs/2105.06337) link.\n\n**Authors**: Vadim Popov\\*, Ivan Vovk\\*, Vladimir Gogoryan, Tasnima Sadekova, Mikhail Kudinov.\n\n\u003csup\u003e\\*Equal contribution.\u003c/sup\u003e\n\n## Abstract\n\n**Demo page** with voiced abstract: [link](https://grad-tts.github.io/).\n\nRecently, denoising diffusion probabilistic models and generative score matching have shown high potential in modelling complex data distributions while stochastic calculus has provided a unified point of view on these techniques allowing for flexible inference schemes. In this paper we introduce Grad-TTS, a novel text-to-speech model with score-based decoder producing mel-spectrograms by gradually transforming noise predicted by encoder and aligned with text input by means of Monotonic Alignment Search. The framework of stochastic differential equations helps us to generalize conventional diffusion probabilistic models to the case of reconstructing data from noise with different parameters and allows to make this reconstruction flexible by explicitly controlling trade-off between sound quality and inference speed. Subjective human evaluation shows that Grad-TTS is competitive with state-of-the-art text-to-speech approaches in terms of Mean Opinion Score.\n\n## References\n\n* HiFi-GAN model is used as vocoder, official github repository: [link](https://github.com/jik876/hifi-gan).\n* Monotonic Alignment Search algorithm is used for unsupervised duration modelling, official github repository: [link](https://github.com/jaywalnut310/glow-tts).\n* Phonemization utilizes CMUdict, official github repository: [link](https://github.com/cmusphinx/cmudict).\n\n\n# BigVGAN 官方信息\n\n### BigVGAN: A Universal Neural Vocoder with Large-Scale Training\n#### Sang-gil Lee, Wei Ping, Boris Ginsburg, Bryan Catanzaro, Sungroh Yoon\n\n\u003ccenter\u003e\u003cimg src=\"https://user-images.githubusercontent.com/15963413/218609148-881e39df-33af-4af9-ab95-1427c4ebf062.png\" width=\"800\"\u003e\u003c/center\u003e\n\nproject link: https://github.com/NVIDIA/BigVGAN\n\n### Infer Test\n\ndowdload pretrain model [bigvgan_base_24khz_100band](https://drive.google.com/drive/folders/19WyD7wN3BeIwBtr9ei1bBcdNEuiq_Avr)\n\n```shell\npython bigvgan/inference.py \\\n--input_wavs_dir bigvgan_debug \\\n--output_dir bigvgan_out\n```\n\n## Train with baker\n\n\u003e python bigvgan/train.py --config bigvgan_pretrain/config.json\n\n## References\n* [HiFi-GAN](https://github.com/jik876/hifi-gan) (for generator and multi-period discriminator)\n\n* [Snake](https://github.com/EdwardDixon/snake) (for periodic activation)\n\n* [Alias-free-torch](https://github.com/junjun3518/alias-free-torch) (for anti-aliasing)\n\n* [Julius](https://github.com/adefossez/julius) (for low-pass filter)\n\n* [UnivNet](https://github.com/mindslab-ai/univnet) (for multi-resolution discriminator)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaxmax2016%2Fgrad-tts-chinese","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmaxmax2016%2Fgrad-tts-chinese","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaxmax2016%2Fgrad-tts-chinese/lists"}