{"id":50480920,"url":"https://github.com/keonlee9420/comprehensive-e2e-tts","last_synced_at":"2026-06-01T17:30:33.274Z","repository":{"id":38358115,"uuid":"475779002","full_name":"keonlee9420/Comprehensive-E2E-TTS","owner":"keonlee9420","description":"A Non-Autoregressive End-to-End Text-to-Speech (text-to-wav), supporting a family of SOTA unsupervised duration modelings. This project grows with the research community, aiming to achieve the ultimate E2E-TTS","archived":false,"fork":false,"pushed_at":"2022-06-06T12:30:10.000Z","size":3620,"stargazers_count":146,"open_issues_count":4,"forks_count":19,"subscribers_count":10,"default_branch":"main","last_synced_at":"2025-09-05T03:43:47.694Z","etag":null,"topics":["deep-learning","end-to-end","fastspeech2","hifi-gan","jets","multi-speaker","neural-tts","non-ar","non-autoregressive","pytorch","single-speaker","sota","speech-synthesis","text-to-speech","text-to-wav","tts","ultimate-tts","unsupervised"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/keonlee9420.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-03-30T08:02:49.000Z","updated_at":"2025-02-08T09:03:05.000Z","dependencies_parsed_at":"2022-09-06T16:11:54.761Z","dependency_job_id":null,"html_url":"https://github.com/keonlee9420/Comprehensive-E2E-TTS","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/keonlee9420/Comprehensive-E2E-TTS","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/keonlee9420%2FComprehensive-E2E-TTS","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/keonlee9420%2FComprehensive-E2E-TTS/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/keonlee9420%2FComprehensive-E2E-TTS/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/keonlee9420%2FComprehensive-E2E-TTS/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/keonlee9420","download_url":"https://codeload.github.com/keonlee9420/Comprehensive-E2E-TTS/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/keonlee9420%2FComprehensive-E2E-TTS/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33786894,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-01T02:00:06.963Z","response_time":115,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","end-to-end","fastspeech2","hifi-gan","jets","multi-speaker","neural-tts","non-ar","non-autoregressive","pytorch","single-speaker","sota","speech-synthesis","text-to-speech","text-to-wav","tts","ultimate-tts","unsupervised"],"created_at":"2026-06-01T17:30:32.434Z","updated_at":"2026-06-01T17:30:33.260Z","avatar_url":"https://github.com/keonlee9420.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Comprehensive-E2E-TTS - PyTorch Implementation\n\n**A Non-Autoregressive End-to-End** Text-to-Speech (generating waveform given text), supporting a family of SOTA unsupervised duration modelings. This project grows with the research community, **aiming to achieve the ultimate E2E-TTS**. Any suggestions toward the best End-to-End TTS are welcome :)\n\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"./img/model.png\" width=\"60%\"\u003e\n\u003c/p\u003e\n\n### Architecture Design\n- [ ] [WavThruVec: Latent speech representation as intermediate features for neural speech synthesis](https://arxiv.org/abs/2203.16930) (Siuzdak et al., 2022)\n- [x] [JETS: Jointly Training FastSpeech2 and HiFi-GAN for End to End Text to Speech](https://arxiv.org/abs/2203.16852) (Lim et al., 2022)\n\n### Linguistic Encoder\n- [x] [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://arxiv.org/abs/2006.04558) (Ren et al., 2020)\n\n### Audio Upsampler\n- [ ] [HiFi++: a Unified Framework for Neural Vocoding, Bandwidth Extension and Speech Enhancement](https://arxiv.org/abs/2203.13086) (Andreev et al., 2022)\n- [x] [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://arxiv.org/abs/2010.05646) (Kong et al., 2020)\n\n### Duration Modeling\n- [ ] [Differentiable Duration Modeling for End-to-End Text-to-Speech](https://arxiv.org/abs/2203.11049) (Nguyen et al., 2022)\n- [x] [One TTS Alignment To Rule Them All](https://arxiv.org/abs/2108.10447) (Badlani et al., 2021)\n\n\u003c!-- ## Audio Samples\nAudio samples are available at [/demo]() --\u003e\n\n# Quickstart\n\n***DATASET*** refers to the names of datasets such as `LJSpeech` and `VCTK` in the following documents.\n\n## Dependencies\nYou can install the Python dependencies with\n```\npip3 install -r requirements.txt\n```\nAlso, `Dockerfile` is provided for `Docker` users.\n\n## Inference\n\nYou have to download the [pretrained models]() (will be shared soon) and put them in `output/ckpt/DATASET/`.\n\nFor a **single-speaker TTS**, run\n```\npython3 synthesize.py --text \"YOUR_DESIRED_TEXT\" --restore_step RESTORE_STEP --mode single --dataset DATASET\n```\n\nFor a **multi-speaker TTS**, run\n```\npython3 synthesize.py --text \"YOUR_DESIRED_TEXT\" --speaker_id SPEAKER_ID --restore_step RESTORE_STEP --mode single --dataset DATASET\n```\n\nThe dictionary of learned speakers can be found at `preprocessed_data/DATASET/speakers.json`, and the generated utterances will be put in `output/result/`.\n\n\n## Batch Inference\nBatch inference is also supported, try\n\n```\npython3 synthesize.py --source preprocessed_data/DATASET/val.txt --restore_step RESTORE_STEP --mode batch --dataset DATASET\n```\nto synthesize all utterances in `preprocessed_data/DATASET/val.txt`.\n\n## Controllability\nThe pitch/volume/speaking rate of the synthesized utterances can be controlled by specifying the desired pitch/energy/duration ratios.\nFor example, one can increase the speaking rate by 20 % and decrease the volume by 20 % by\n\n```\npython3 synthesize.py --text \"YOUR_DESIRED_TEXT\" --restore_step RESTORE_STEP --mode single --dataset DATASET --duration_control 0.8 --energy_control 0.8\n```\nAdd ***--speaker_id SPEAKER_ID*** for a multi-speaker TTS.\n\n# Training\n\n## Datasets\n\nThe supported datasets are\n\n- [LJSpeech](https://keithito.com/LJ-Speech-Dataset/): a **single-speaker** English dataset consists of 13100 short audio clips of a female speaker reading passages from 7 non-fiction books, approximately 24 hours in total.\n- [VCTK](https://datashare.ed.ac.uk/handle/10283/3443): The CSTR VCTK Corpus includes speech data uttered by 110 English speakers (**multi-speaker TTS**) with various accents. Each speaker reads out about 400 sentences, which were selected from a newspaper, the rainbow passage and an elicitation paragraph used for the speech accent archive.\n\nAny of both **single-speaker TTS** dataset (e.g., [Blizzard Challenge 2013](https://www.synsig.org/index.php/Blizzard_Challenge_2013)) and **multi-speaker TTS** dataset (e.g., [LibriTTS](https://openslr.org/60/)) can be added following LJSpeech and VCTK, respectively. Moreover, **your own language and dataset** can be adapted following [here](https://github.com/keonlee9420/Expressive-FastSpeech2).\n\n## Preprocessing\n\n- For a **multi-speaker TTS** with external speaker embedder, download [ResCNN Softmax+Triplet pretrained model](https://drive.google.com/file/d/1F9NvdrarWZNktdX9KlRYWWHDwRkip_aP) of [philipperemy's DeepSpeaker](https://github.com/philipperemy/deep-speaker) for the speaker embedding and locate it in `./deepspeaker/pretrained_models/`.\n- Run the preprocessing script by\n  ```\n  python3 preprocess.py --dataset DATASET\n  ```\n\n## Training\n\nTrain your model with\n```\npython3 train.py --dataset DATASET\n```\nUseful options:\n\u003c!-- - To use [Automatic Mixed Precision](https://pytorch.org/tutorials/recipes/recipes/amp_recipe.html), append `--use_amp` argument to the above command. --\u003e\n- The trainer assumes single-node multi-GPU training. To use specific GPUs, specify `CUDA_VISIBLE_DEVICES=\u003cGPU_IDs\u003e` at the beginning of the above command.\n\n# TensorBoard\n\nUse\n```\ntensorboard --logdir output/log\n```\n\nto serve TensorBoard on your localhost.\n\u003c!-- The loss curves, synthesized mel-spectrograms, and audios are shown.\n\n![](./img/tensorboard_loss.png)\n![](./img/tensorboard_spec.png)\n![](./img/tensorboard_audio.png) --\u003e\n\n# Notes\n\n- Two options for embedding for the **multi-speaker TTS** setting: training speaker embedder from scratch or using a pre-trained [philipperemy's DeepSpeaker](https://github.com/philipperemy/deep-speaker) model (as [STYLER](https://github.com/keonlee9420/STYLER) did). You can toggle it by setting the config (between `'none'` and `'DeepSpeaker'`).\n- DeepSpeaker on VCTK dataset shows clear identification among speakers. The following figure shows the T-SNE plot of extracted speaker embedding.\n\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"./preprocessed_data/VCTK/spker_embed_tsne.png\" width=\"40%\"\u003e\n\u003c/p\u003e\n\n# Citation\n\nPlease cite this repository by the \"[Cite this repository](https://github.blog/2021-08-19-enhanced-support-citations-github/)\" of **About** section (top right of the main page).\n\n# References\n- [keonlee9420's Comprehensive-Transformer-TTS](https://github.com/keonlee9420/Comprehensive-Transformer-TTS)\n- [keonlee9420's DiffGAN-TTS](https://github.com/keonlee9420/DiffGAN-TTS)\n- [jik876' hifi-gan](https://github.com/jik876/hifi-gan)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkeonlee9420%2Fcomprehensive-e2e-tts","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkeonlee9420%2Fcomprehensive-e2e-tts","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkeonlee9420%2Fcomprehensive-e2e-tts/lists"}