{"id":13585877,"url":"https://github.com/r9y9/deepvoice3_pytorch","last_synced_at":"2025-05-14T19:08:38.013Z","repository":{"id":41175043,"uuid":"108992863","full_name":"r9y9/deepvoice3_pytorch","owner":"r9y9","description":"PyTorch implementation of convolutional neural networks-based text-to-speech synthesis models","archived":false,"fork":false,"pushed_at":"2023-12-19T04:21:47.000Z","size":7107,"stargazers_count":1976,"open_issues_count":45,"forks_count":487,"subscribers_count":92,"default_branch":"master","last_synced_at":"2025-04-13T13:59:44.878Z","etag":null,"topics":["end-to-end","machine-learning","multi-speaker","python","pytorch","speech-processing","speech-synthesis","tts"],"latest_commit_sha":null,"homepage":"https://r9y9.github.io/deepvoice3_pytorch/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/r9y9.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-10-31T12:31:44.000Z","updated_at":"2025-04-10T16:01:21.000Z","dependencies_parsed_at":"2024-06-19T03:02:39.392Z","dependency_job_id":"b41d6a6e-b7c5-487e-b9ec-0569c7e0047f","html_url":"https://github.com/r9y9/deepvoice3_pytorch","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/r9y9%2Fdeepvoice3_pytorch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/r9y9%2Fdeepvoice3_pytorch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/r9y9%2Fdeepvoice3_pytorch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/r9y9%2Fdeepvoice3_pytorch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/r9y9","download_url":"https://codeload.github.com/r9y9/deepvoice3_pytorch/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254209859,"owners_count":22032897,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["end-to-end","machine-learning","multi-speaker","python","pytorch","speech-processing","speech-synthesis","tts"],"created_at":"2024-08-01T15:05:11.862Z","updated_at":"2025-05-14T19:08:36.088Z","avatar_url":"https://github.com/r9y9.png","language":"Python","funding_links":[],"categories":["Python","Speech synthesizer or Text-to-Speech(TTS)","语音合成","Paper implementations｜论文实现","Paper implementations","Deepfake Voices"],"sub_categories":["网络服务_其他","Other libraries｜其他库:","Other libraries:","Codes Mainly on Generation"],"readme":"![alt text](assets/banner.jpg)\n\n# Deepvoice3_pytorch\n\n[![PyPI](https://img.shields.io/pypi/v/deepvoice3_pytorch.svg)](https://pypi.python.org/pypi/deepvoice3_pytorch)\n[![Build Status](https://travis-ci.org/r9y9/deepvoice3_pytorch.svg?branch=master)](https://travis-ci.org/r9y9/deepvoice3_pytorch)\n[![Build status](https://ci.appveyor.com/api/projects/status/8eurjakfaofbr24k?svg=true)](https://ci.appveyor.com/project/r9y9/deepvoice3-pytorch)\n[![DOI](https://zenodo.org/badge/108992863.svg)](https://zenodo.org/badge/latestdoi/108992863)\n\nPyTorch implementation of convolutional networks-based text-to-speech synthesis models:\n\n1. [arXiv:1710.07654](https://arxiv.org/abs/1710.07654): Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learning.\n2. [arXiv:1710.08969](https://arxiv.org/abs/1710.08969): Efficiently Trainable Text-to-Speech System Based on Deep Convolutional Networks with Guided Attention.\n\nAudio samples are available at https://r9y9.github.io/deepvoice3_pytorch/.\n\n## Folks\n\n- https://github.com/hash2430/dv3_world: DeepVoice3 with WORLD vocoder support. [#166](https://github.com/r9y9/deepvoice3_pytorch/issues/166)\n\n## Online TTS demo\n\nNotebooks supposed to be executed on https://colab.research.google.com are available:\n\n- [DeepVoice3: Multi-speaker text-to-speech demo](https://colab.research.google.com/github/r9y9/Colaboratory/blob/master/DeepVoice3_multi_speaker_TTS_en_demo.ipynb)\n- [DeepVoice3: Single-speaker text-to-speech demo](https://colab.research.google.com/github/r9y9/Colaboratory/blob/master/DeepVoice3_single_speaker_TTS_en_demo.ipynb)\n\n## Highlights\n\n- Convolutional sequence-to-sequence model with attention for text-to-speech synthesis\n- Multi-speaker and single speaker versions of DeepVoice3\n- Audio samples and pre-trained models\n- Preprocessor for [LJSpeech (en)](https://keithito.com/LJ-Speech-Dataset/), [JSUT (jp)](https://sites.google.com/site/shinnosuketakamichi/publication/jsut) and [VCTK](http://homepages.inf.ed.ac.uk/jyamagis/page3/page58/page58.html) datasets, as well as [carpedm20/multi-speaker-tacotron-tensorflow](https://github.com/carpedm20/multi-Speaker-tacotron-tensorflow) compatible custom dataset (in JSON format)\n- Language-dependent frontend text processor for English and Japanese\n\n### Samples\n\n- [Ja Step000380000 Predicted](https://soundcloud.com/user-623907374/ja-step000380000-predicted)\n- [Ja Step000370000 Predicted](https://soundcloud.com/user-623907374/ja-step000370000-predicted)\n- [Ko_single Step000410000 Predicted](https://soundcloud.com/user-623907374/ko-step000410000-predicted)\n- [Ko_single Step000400000 Predicted](https://soundcloud.com/user-623907374/ko-step000400000-predicted)\n- [Ko_multi Step001680000 Predicted](https://soundcloud.com/user-623907374/step001680000-predicted)\n- [Ko_multi Step001700000 Predicted](https://soundcloud.com/user-623907374/step001700000-predicted)\n\n## Pretrained models\n\n**NOTE**: pretrained models are not compatible to master. To be updated soon.\n\n | URL | Model      | Data     | Hyper paramters                                  | Git commit | Steps  |\n |-----|------------|----------|--------------------------------------------------|----------------------|--------|\n | [link](https://www.dropbox.com/s/5ucl9remrwy5oeg/20180505_deepvoice3_checkpoint_step000640000.pth?dl=0) | DeepVoice3 | LJSpeech | [link](https://www.dropbox.com/s/0ck82unm0bo0rxd/20180505_deepvoice3_ljspeech.json?dl=0) | [abf0a21](https://github.com/r9y9/deepvoice3_pytorch/tree/abf0a21f83aeb451b918f867bc23378f1e2e608b)| 640k |\n |  [link](https://www.dropbox.com/s/1y8bt6bnggbzzlp/20171129_nyanko_checkpoint_step000585000.pth?dl=0)   | Nyanko     | LJSpeech | `builder=nyanko,preset=nyanko_ljspeech`     | [ba59dc7](https://github.com/r9y9/deepvoice3_pytorch/tree/ba59dc75374ca3189281f6028201c15066830116) | 585k |\n  |  [link](https://www.dropbox.com/s/uzmtzgcedyu531k/20171222_deepvoice3_vctk108_checkpoint_step000300000.pth?dl=0)   | Multi-speaker DeepVoice3     | VCTK | `builder=deepvoice3_multispeaker,preset=deepvoice3_vctk`     | [0421749](https://github.com/r9y9/deepvoice3_pytorch/tree/0421749af908905d181f089f06956fddd0982d47) | 300k + 300k |\n\nTo use pre-trained models, it's highly recommended that you are on the **specific git commit** noted above. i.e.,\n\n```\ngit checkout ${commit_hash}\n```\n\nThen follow the \"Synthesize from a checkpoint\" section in the README of the specific git commit. Please notice that the latest development version of the repository may not work.\n\nYou could try for example:\n\n```\n# pretrained model (20180505_deepvoice3_checkpoint_step000640000.pth)\n# hparams (20180505_deepvoice3_ljspeech.json)\ngit checkout 4357976\npython synthesis.py --preset=20180505_deepvoice3_ljspeech.json \\\n  20180505_deepvoice3_checkpoint_step000640000.pth \\\n  sentences.txt \\\n  output_dir\n```\n\n## Notes on hyper parameters\n\n- Default hyper parameters, used during preprocessing/training/synthesis stages, are turned for English TTS using LJSpeech dataset. You will have to change some of parameters if you want to try other datasets. See `hparams.py` for details.\n- `builder` specifies which model you want to use. `deepvoice3`, `deepvoice3_multispeaker` [1] and `nyanko` [2] are surpprted.\n- Hyper parameters described in DeepVoice3 paper for single speaker didn't work for LJSpeech dataset, so I changed a few things. Add dilated convolution, more channels, more layers and add guided attention loss, etc. See code for details. The changes are also applied for multi-speaker model.\n- Multiple attention layers are hard to learn. Empirically, one or two (first and last) attention layers seems enough.\n- With guided attention (see https://arxiv.org/abs/1710.08969), alignments get monotonic more quickly and reliably if we use multiple attention layers. With guided attention, I can confirm five attention layers get monotonic, though I cannot get speech quality improvements.\n- Binary divergence (described in https://arxiv.org/abs/1710.08969) seems stabilizes training particularly for deep (\u003e 10 layers) networks.\n- Adam with step lr decay works. However, for deeper networks, I find Adam + noam's lr scheduler is more stable.\n\n## Requirements\n\n- Python \u003e= 3.5\n- CUDA \u003e= 8.0\n- PyTorch \u003e= v1.0.0\n- [nnmnkwii](https://github.com/r9y9/nnmnkwii) \u003e= v0.0.11\n- [MeCab](http://taku910.github.io/mecab/) (Japanese only)\n\n## Installation\n\nPlease install packages listed above first, and then\n\n```\ngit clone https://github.com/r9y9/deepvoice3_pytorch \u0026\u0026 cd deepvoice3_pytorch\npip install -e \".[bin]\"\n```\n\n## Getting started\n\n### Preset parameters\n\nThere are many hyper parameters to be turned depends on what model and data you are working on. For typical datasets and models, parameters that known to work good (**preset**) are provided in the repository. See `presets` directory for details. Notice that\n\n1. `preprocess.py`\n2. `train.py`\n3. `synthesis.py`\n\naccepts `--preset=\u003cjson\u003e` optional parameter, which specifies where to load preset parameters. If you are going to use preset parameters, then you must use same `--preset=\u003cjson\u003e` throughout preprocessing, training and evaluation. e.g.,\n\n```\npython preprocess.py --preset=presets/deepvoice3_ljspeech.json ljspeech ~/data/LJSpeech-1.0\npython train.py --preset=presets/deepvoice3_ljspeech.json --data-root=./data/ljspeech\n```\n\ninstead of\n\n```\npython preprocess.py ljspeech ~/data/LJSpeech-1.0\n# warning! this may use different hyper parameters used at preprocessing stage\npython train.py --preset=presets/deepvoice3_ljspeech.json --data-root=./data/ljspeech\n```\n\n### 0. Download dataset\n\n- LJSpeech (en): https://keithito.com/LJ-Speech-Dataset/\n- VCTK (en): http://homepages.inf.ed.ac.uk/jyamagis/page3/page58/page58.html\n- JSUT (jp): https://sites.google.com/site/shinnosuketakamichi/publication/jsut\n- NIKL (ko) (**Need korean cellphone number to access it**): http://www.korean.go.kr/front/board/boardStandardView.do?board_id=4\u0026mn_id=17\u0026b_seq=464\n\n### 1. Preprocessing\n\nUsage:\n\n```\npython preprocess.py ${dataset_name} ${dataset_path} ${out_dir} --preset=\u003cjson\u003e\n```\n\nSupported `${dataset_name}`s are:\n\n- `ljspeech` (en, single speaker)\n- `vctk` (en, multi-speaker)\n- `jsut` (jp, single speaker)\n- `nikl_m` (ko, multi-speaker)\n- `nikl_s` (ko, single speaker)\n\nAssuming you use preset parameters known to work good for LJSpeech dataset / DeepVoice3 and have data in `~/data/LJSpeech-1.0`, then you can preprocess data by:\n\n```\npython preprocess.py --preset=presets/deepvoice3_ljspeech.json ljspeech ~/data/LJSpeech-1.0/ ./data/ljspeech\n```\n\nWhen this is done, you will see extracted features (mel-spectrograms and linear spectrograms) in `./data/ljspeech`.\n\n#### 1-1. Building custom dataset. (using json_meta)\nBuilding your own dataset, with metadata in JSON format (compatible with [carpedm20/multi-speaker-tacotron-tensorflow](https://github.com/carpedm20/multi-Speaker-tacotron-tensorflow)) is currently supported.\nUsage:\n\n```\npython preprocess.py json_meta ${list-of-JSON-metadata-paths} ${out_dir} --preset=\u003cjson\u003e\n```\nYou may need to modify pre-existing preset JSON file, especially `n_speakers`. For english multispeaker, start with `presets/deepvoice3_vctk.json`.\n\nAssuming you have dataset A (Speaker A) and dataset B (Speaker B), each described in the JSON metadata file `./datasets/datasetA/alignment.json` and `./datasets/datasetB/alignment.json`, then you can preprocess  data by:\n\n```\npython preprocess.py json_meta \"./datasets/datasetA/alignment.json,./datasets/datasetB/alignment.json\" \"./datasets/processed_A+B\" --preset=(path to preset json file)\n```\n\n#### 1-2. Preprocessing custom english datasets with long silence. (Based on [vctk_preprocess](vctk_preprocess/))\n\nSome dataset, especially automatically generated dataset may include long silence and undesirable leading/trailing noises, undermining the char-level seq2seq model.\n(e.g. VCTK, although this is covered in vctk_preprocess)\n\nTo deal with the problem, `gentle_web_align.py` will\n- **Prepare phoneme alignments for all utterances**\n- Cut silences during preprocessing\n\n`gentle_web_align.py` uses [Gentle](https://github.com/lowerquality/gentle), a kaldi based speech-text alignment tool. This accesses web-served Gentle application, aligns given sound segments with transcripts and converts the result to HTK-style label files, to be processed in `preprocess.py`. Gentle can be run in Linux/Mac/Windows(via Docker).\n\nPreliminary results show that while HTK/festival/merlin-based method in `vctk_preprocess/prepare_vctk_labels.py` works better on VCTK, Gentle is more stable with audio clips with ambient noise. (e.g. movie excerpts)\n\nUsage:\n(Assuming Gentle is running at `localhost:8567` (Default when not specified))\n1. When sound file and transcript files are saved in separate folders. (e.g. sound files are at `datasetA/wavs` and transcripts are at `datasetA/txts`)\n```\npython gentle_web_align.py -w \"datasetA/wavs/*.wav\" -t \"datasetA/txts/*.txt\" --server_addr=localhost --port=8567\n```\n\n2. When sound file and transcript files are saved in nested structure. (e.g. `datasetB/speakerN/blahblah.wav` and `datasetB/speakerN/blahblah.txt`)\n```\npython gentle_web_align.py --nested-directories=\"datasetB\" --server_addr=localhost --port=8567\n```\n**Once you have phoneme alignment for each utterance, you can extract features by running `preprocess.py`**\n\n### 2. Training\n\nUsage:\n\n```\npython train.py --data-root=${data-root} --preset=\u003cjson\u003e --hparams=\"parameters you may want to override\"\n```\n\nSuppose you build a DeepVoice3-style model using LJSpeech dataset, then you can train your model by:\n\n```\npython train.py --preset=presets/deepvoice3_ljspeech.json --data-root=./data/ljspeech/\n```\n\nModel checkpoints (.pth) and alignments (.png) are saved in `./checkpoints` directory per 10000 steps by default.\n\n#### NIKL\n\nPleae check [this](https://github.com/homink/deepvoice3_pytorch/blob/master/nikl_preprocess/README.md) in advance and follow the commands below.\n\n```\npython preprocess.py nikl_s ${your_nikl_root_path} data/nikl_s --preset=presets/deepvoice3_nikls.json\n\npython train.py --data-root=./data/nikl_s --checkpoint-dir checkpoint_nikl_s --preset=presets/deepvoice3_nikls.json\n```\n\n### 4. Monitor with Tensorboard\n\nLogs are dumped in `./log` directory by default. You can monitor logs by tensorboard:\n\n```\ntensorboard --logdir=log\n```\n\n### 5. Synthesize from a checkpoint\n\nGiven a list of text, `synthesis.py` synthesize audio signals from trained model. Usage is:\n\n```\npython synthesis.py ${checkpoint_path} ${text_list.txt} ${output_dir} --preset=\u003cjson\u003e\n```\n\nExample test_list.txt:\n\n```\nGenerative adversarial network or variational auto-encoder.\nOnce upon a time there was a dear little girl who was loved by every one who looked at her, but most of all by her grandmother, and there was nothing that she would not have given to the child.\nA text-to-speech synthesis system typically consists of multiple stages, such as a text analysis frontend, an acoustic model and an audio synthesis module.\n```\n\n## Advanced usage\n\n### Multi-speaker model\n\nVCTK and NIKL are supported dataset for building a multi-speaker model.\n\n#### VCTK\nSince some audio samples in VCTK have long silences that affect performance, it's recommended to do phoneme alignment and remove silences according to [vctk_preprocess](vctk_preprocess/).\n\nOnce you have phoneme alignment for each utterance, you can extract features by:\n\n```\npython preprocess.py vctk ${your_vctk_root_path} ./data/vctk\n```\n\nNow that you have data prepared, then you can train a multi-speaker version of DeepVoice3 by:\n\n```\npython train.py --data-root=./data/vctk --checkpoint-dir=checkpoints_vctk \\\n   --preset=presets/deepvoice3_vctk.json \\\n   --log-event-path=log/deepvoice3_multispeaker_vctk_preset\n```\n\nIf you want to reuse learned embedding from other dataset, then you can do this instead by:\n\n```\npython train.py --data-root=./data/vctk --checkpoint-dir=checkpoints_vctk \\\n   --preset=presets/deepvoice3_vctk.json \\\n   --log-event-path=log/deepvoice3_multispeaker_vctk_preset \\\n   --load-embedding=20171213_deepvoice3_checkpoint_step000210000.pth\n```\n\nThis may improve training speed a bit.\n\n#### NIKL\n\nYou will be able to obtain cleaned-up audio samples in ../nikl_preprocoess. Details are found in [here](https://github.com/homink/speech.ko).\n\n\nOnce NIKL corpus is ready to use from the preprocessing, you can extract features by:\n\n```\npython preprocess.py nikl_m ${your_nikl_root_path} data/nikl_m\n```\n\nNow that you have data prepared, then you can train a multi-speaker version of DeepVoice3 by:\n\n```\npython train.py --data-root=./data/nikl_m  --checkpoint-dir checkpoint_nikl_m \\\n   --preset=presets/deepvoice3_niklm.json\n```\n\n### Speaker adaptation\n\nIf you have very limited data, then you can consider to try fine-turn pre-trained model. For example, using pre-trained model on LJSpeech, you can adapt it to data from VCTK speaker `p225` (30 mins) by the following command:\n\n```\npython train.py --data-root=./data/vctk --checkpoint-dir=checkpoints_vctk_adaptation \\\n    --preset=presets/deepvoice3_ljspeech.json \\\n    --log-event-path=log/deepvoice3_vctk_adaptation \\\n    --restore-parts=\"20171213_deepvoice3_checkpoint_step000210000.pth\"\n    --speaker-id=0\n```\n\nFrom my experience, it can get reasonable speech quality very quickly rather than training the model from scratch.\n\nThere are two important options used above:\n\n- `--restore-parts=\u003cN\u003e`: It specifies where to load model parameters. The differences from the option `--checkpoint=\u003cN\u003e` are 1) `--restore-parts=\u003cN\u003e` ignores all invalid parameters, while `--checkpoint=\u003cN\u003e` doesn't. 2) `--restore-parts=\u003cN\u003e` tell trainer to start from 0-step, while `--checkpoint=\u003cN\u003e` tell trainer to continue from last step. `--checkpoint=\u003cN\u003e` should be ok if you are using exactly same model and continue to train, but it would be useful if you want to customize your model architecture and take advantages of pre-trained model.\n- `--speaker-id=\u003cN\u003e`: It specifies what speaker of data is used for training. This should only be specified if you are using multi-speaker dataset. As for VCTK, speaker id is automatically assigned incrementally (0, 1, ..., 107) according to the `speaker_info.txt` in the dataset.\n\nIf you are training multi-speaker model, speaker adaptation will only work **when `n_speakers` is identical**.\n\n## Trouble shooting\n\n### [#5](https://github.com/r9y9/deepvoice3_pytorch/issues/5) RuntimeError: main thread is not in main loop\n\n\nThis may happen depending on backends you have for matplotlib. Try changing backend for matplotlib and see if it works as follows:\n\n```\nMPLBACKEND=Qt5Agg python train.py ${args...}\n```\n\nIn [#78](https://github.com/r9y9/deepvoice3_pytorch/pull/78#issuecomment-385327057), engiecat reported that changing the backend of matplotlib from Tkinter(TkAgg) to PyQt5(Qt5Agg) fixed the problem.\n\n## Sponsers\n\n- https://github.com/echelon\n\n## Acknowledgements\n\nPart of code was adapted from the following projects:\n\n- https://github.com/keithito/tacotron\n- https://github.com/facebookresearch/fairseq-py\n\nBanner and logo created by [@jraulhernandezi](https://github.com/jraulhernandezi) ([#76](https://github.com/r9y9/deepvoice3_pytorch/issues/76))\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fr9y9%2Fdeepvoice3_pytorch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fr9y9%2Fdeepvoice3_pytorch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fr9y9%2Fdeepvoice3_pytorch/lists"}