{"id":13564382,"url":"https://github.com/spring-media/ForwardTacotron","last_synced_at":"2025-04-03T21:30:51.501Z","repository":{"id":37391317,"uuid":"244631442","full_name":"as-ideas/ForwardTacotron","owner":"as-ideas","description":"⏩ Generating speech in a single forward pass without any attention!","archived":false,"fork":true,"pushed_at":"2024-05-06T10:52:48.000Z","size":213123,"stargazers_count":579,"open_issues_count":34,"forks_count":111,"subscribers_count":31,"default_branch":"master","last_synced_at":"2024-05-06T12:00:46.681Z","etag":null,"topics":["axelspringerai","deep-learning","forwardtacotron","python","pytorch","tacotron","text-to-speech","tts"],"latest_commit_sha":null,"homepage":"https://as-ideas.github.io/ForwardTacotron/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":"fatchord/WaveRNN","license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/as-ideas.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2020-03-03T12:33:51.000Z","updated_at":"2024-05-06T12:00:46.682Z","dependencies_parsed_at":"2023-10-03T13:55:11.187Z","dependency_job_id":null,"html_url":"https://github.com/as-ideas/ForwardTacotron","commit_stats":null,"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/as-ideas%2FForwardTacotron","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/as-ideas%2FForwardTacotron/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/as-ideas%2FForwardTacotron/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/as-ideas%2FForwardTacotron/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/as-ideas","download_url":"https://codeload.github.com/as-ideas/ForwardTacotron/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223040561,"owners_count":17078041,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["axelspringerai","deep-learning","forwardtacotron","python","pytorch","tacotron","text-to-speech","tts"],"created_at":"2024-08-01T13:01:30.431Z","updated_at":"2025-04-03T21:30:46.281Z","avatar_url":"https://github.com/as-ideas.png","language":"Python","funding_links":[],"categories":["Python","Audio and music generation/processing"],"sub_categories":["Project:"],"readme":"# ⏩ ForwardTacotron\n\nInspired by Microsoft's [FastSpeech](https://www.microsoft.com/en-us/research/blog/fastspeech-new-text-to-speech-model-improves-on-speed-accuracy-and-controllability/)\nwe modified Tacotron (Fork from fatchord's [WaveRNN](https://github.com/fatchord/WaveRNN)) to generate speech in a single forward pass using a duration predictor to align text and generated mel spectrograms. Hence, we call the model ForwardTacotron (see Figure 1).\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/model.png\" width=\"700\" /\u003e\n\u003c/p\u003e\n\u003cp align=\"center\"\u003e\n  \u003cb\u003eFigure 1:\u003c/b\u003e Model Architecture.\n\u003c/p\u003e\n\nThe model has following advantages:\n- **Robustness:** No repeats and failed attention modes for challenging sentences.\n- **Speed:** The generation of a mel spectogram takes about 0.04s on a GeForce RTX 2080.\n- **Controllability:** It is possible to control the speed of the generated utterance.\n- **Efficiency:** In contrast to FastSpeech and Tacotron, the model of ForwardTacotron\ndoes not use any attention. Hence, the required memory grows linearly with text size, which makes it possible to synthesize large articles at once.\n\n\n## UPDATE Improved attention mechanism (30.08.2023)\n- Faster tacotron attention buildup by adding alignment conditioning based on [one alignment to rule them all](https://arxiv.org/abs/2108.10447) \n- Improved attention translates to improved synth quality.\n\n## 🔈 Samples\n\n[Can be found here.](https://as-ideas.github.io/ForwardTacotron/)\n\nThe samples are generated with a model trained on LJSpeech and vocoded with WaveRNN, [MelGAN](https://github.com/seungwonpark/melgan), or [HiFiGAN](https://github.com/jik876/hifi-gan). \nYou can try out the latest pretrained model with the following notebook:  \n\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/as-ideas/ForwardTacotron/blob/master/notebooks/synthesize.ipynb)\n\n## ⚙️ Installation\n\nMake sure you have:\n\n* Python \u003e= 3.6\n\nInstall espeak as phonemizer backend (for macOS use brew):\n```\nsudo apt-get install espeak\n```\n\nThen install the rest with pip:\n```\npip install -r requirements.txt\n```\n\n## 🚀 Training your own Model (Singlespeaker)\n\nChange the params in the config.yaml according to your needs and follow the steps below:\n\n(1) Download and preprocess the [LJSpeech](https://keithito.com/LJ-Speech-Dataset/) dataset:\n ```\npython preprocess.py --path /path/to/ljspeech\n```\n(2) Train Tacotron with:\n```\npython train_tacotron.py\n```\nOnce the training is finished, the model will automatically extract the alignment features from the dataset. In case you stopped the training early, you \ncan use the latest checkpoint to manually run the process with:\n```\npython train_tacotron.py --force_align\n```\n(3) Train ForwardTacotron with:\n```\npython train_forward.py\n```\n(4) Generate Sentences with Griffin-Lim vocoder:\n```\npython gen_forward.py --alpha 1 --input_text 'this is whatever you want it to be' griffinlim\n```\nIf you want to use the [MelGAN](https://github.com/seungwonpark/melgan) vocoder, you can produce .mel files with:\n```\npython gen_forward.py --input_text 'this is whatever you want it to be' melgan\n```\nIf you want to use the [HiFiGAN](https://github.com/jik876/hifi-gan) vocoder, you can produce .npy files with:\n```\npython gen_forward.py --input_text 'this is whatever you want it to be' hifigan\n```\nTo vocode the resulting .mel or .npy files use the inference.py script from the MelGAN or HiFiGAN repo and point to the model output folder.\n\nFor training the model on your own dataset just bring it to the LJSpeech-like format:\n```\n|- dataset_folder/\n|   |- metadata.csv\n|   |- wav/\n|       |- file1.wav\n|       |- ...\n```\n\nFor languages other than English, change the language and cleaners params in the hparams.py, e.g. for French:\n```\nlanguage = 'fr'\ntts_cleaner_name = 'no_cleaners'\n```\n\n____\nYou can monitor the training processes for Tacotron and ForwardTacotron with \n```\ntensorboard --logdir checkpoints\n```\nHere is what the ForwardTacotron tensorboard looks like:\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/tensorboard.png\" width=\"700\" /\u003e\n\u003c/p\u003e\n\u003cp align=\"center\"\u003e\n  \u003cb\u003eFigure 2:\u003c/b\u003e Tensorboard example for training a ForwardTacotron model.\n\u003c/p\u003e\n\n\n## Multispeaker Training\nPrepare the data in ljspeech format:\n```\n|- dataset_folder/\n|   |- metadata.csv\n|   |- wav/\n|       |- file1.wav\n|       |- ...\n```\nThe metadata.csv is expected to have the speaker id in the second column:\n```\nid_001|speaker_1|this is the first text.\nid_002|speaker_1|this is the second text.\nid_003|speaker_2|this is the third text.\n...\n```\nWe also support the VCTK and a pandas format \n(can be set in the config multispeaker.yaml under preprocesing.metafile_format)\n\nFollow the same steps as for singlespaker, but provide the multispeaker config:\n ```\npython preprocess.py --config configs/multispeaker.yaml --path /path/to/ljspeech\npython train_tacotron.py --config configs/multispeaker.yaml\npython train_forward.py --config configs/multispeaker.yaml\n```\n\n## Pretrained Models\n\n| Model | Dataset | Commit Tag |\n|---|---|------------|\n|[forward_tacotron](https://public-asai-dl-models.s3.eu-central-1.amazonaws.com/ForwardTacotron/forward_step90k.pt)| ljspeech | v3.1     |\n|[fastpitch](https://public-asai-dl-models.s3.eu-central-1.amazonaws.com/ForwardTacotron/thorsten_fastpitch_50k.pt)| [thorstenmueller (german)](https://github.com/thorstenMueller/deep-learning-german-tts) | v3.1     |\n\nOur pre-trained LJSpeech model is compatible with the pre-trained vocoders:\n- [MelGAN](https://github.com/seungwonpark/melgan)\n- [HiFiGAN](https://github.com/jik876/hifi-gan)\n\n\nAfter downloading the models you can synthesize text using the pretrained models with\n```\npython gen_forward.py --input_text 'Hi there!' --checkpoint forward_step90k.pt wavernn --voc_checkpoint wave_step_575k.pt\n\n```\n\n## Export Model with TorchScript\n\nHere is a dummy example of exporting the model in TorchScript:\n```\nimport torch\nfrom models.forward_tacotron import ForwardTacotron\n\ntts_model = ForwardTacotron.from_checkpoint('checkpoints/ljspeech_tts.forward/latest_model.pt')\ntts_model.eval()\nmodel_script = torch.jit.script(tts_model)\nx = torch.ones((1, 5)).long()\ny = model_script.generate_jit(x)\n```\nFor the necessary preprocessing steps (text to tokens) please refer to:\n```\ngen_forward.py\n```\n\n## References\n\n* [FastSpeech: Fast, Robust and Controllable Text to Speech](https://arxiv.org/abs/1905.09263)\n* [FastPitch: Parallel Text-to-speech with Pitch Prediction](https://arxiv.org/abs/2006.06873)\n* [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://arxiv.org/abs/2010.05646)\n* [MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis](https://arxiv.org/abs/1910.06711)\n* [Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis](https://arxiv.org/abs/1806.04558)\n\n## Acknowlegements\n\n* [https://github.com/keithito/tacotron](https://github.com/keithito/tacotron)\n* [https://github.com/fatchord/WaveRNN](https://github.com/fatchord/WaveRNN)\n* [https://github.com/seungwonpark/melgan](https://github.com/seungwonpark/melgan)\n* [https://github.com/jik876/hifi-gan](https://github.com/jik876/hifi-gan)\n* [https://github.com/xcmyz/LightSpeech](https://github.com/xcmyz/LightSpeech)\n* [https://github.com/resemble-ai/Resemblyzer](https://github.com/resemble-ai/Resemblyzer)\n* [https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/SpeechSynthesis/FastPitch](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/SpeechSynthesis/FastPitch)\n* [https://github.com/resemble-ai/Resemblyzer](https://github.com/resemble-ai/Resemblyzer)\n\n## Maintainers\n\n* Christian Schäfer, github: [cschaefer26](https://github.com/cschaefer26)\n\n## Copyright\n\nSee [LICENSE](LICENSE) for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspring-media%2FForwardTacotron","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fspring-media%2FForwardTacotron","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspring-media%2FForwardTacotron/lists"}