{"id":13958565,"url":"https://github.com/p0p4k/pflowtts_pytorch","last_synced_at":"2025-07-21T00:31:16.408Z","repository":{"id":206716913,"uuid":"717485333","full_name":"p0p4k/pflowtts_pytorch","owner":"p0p4k","description":"Unofficial implementation of NVIDIA P-Flow TTS paper","archived":false,"fork":false,"pushed_at":"2024-12-24T00:53:24.000Z","size":11013,"stargazers_count":219,"open_issues_count":23,"forks_count":33,"subscribers_count":14,"default_branch":"master","last_synced_at":"2024-12-24T01:35:32.736Z","etag":null,"topics":["deep-learning","flow-matching","pflow","pytorch","text-to-speech","tts"],"latest_commit_sha":null,"homepage":"https://neurips.cc/virtual/2023/poster/69899","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/p0p4k.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-11-11T16:08:17.000Z","updated_at":"2024-12-24T00:53:28.000Z","dependencies_parsed_at":"2023-11-11T20:27:52.676Z","dependency_job_id":"b09e5e7a-4f2b-4637-bb4e-c1614fafee7b","html_url":"https://github.com/p0p4k/pflowtts_pytorch","commit_stats":null,"previous_names":["p0p4k/pflowtts_pytorch"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/p0p4k/pflowtts_pytorch","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/p0p4k%2Fpflowtts_pytorch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/p0p4k%2Fpflowtts_pytorch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/p0p4k%2Fpflowtts_pytorch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/p0p4k%2Fpflowtts_pytorch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/p0p4k","download_url":"https://codeload.github.com/p0p4k/pflowtts_pytorch/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/p0p4k%2Fpflowtts_pytorch/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266221259,"owners_count":23894965,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","flow-matching","pflow","pytorch","text-to-speech","tts"],"created_at":"2024-08-08T13:01:44.172Z","updated_at":"2025-07-21T00:31:16.400Z","avatar_url":"https://github.com/p0p4k.png","language":"Python","funding_links":[],"categories":["语音合成"],"sub_categories":["网络服务_其他"],"readme":"## P-Flow: A Fast and Data-Efficient Zero-Shot TTS through Speech Prompting\n### Authors : Sungwon Kim, Kevin J Shih, Rohan Badlani, Joao Felipe Santos, Evelina Bhakturina, Mikyas Desta, Rafael Valle, Sungroh Yoon, Bryan Catanzaro\n#### Affiliations: NVIDIA\n\n## Status : Added newer samples with better prosody and pronunciation. Check out `samples` folder. LJSpeech pretrained ckpt - [GDrive Link](https://drive.google.com/drive/folders/1x-A2Ezmmiz01YqittO_GLYhngJXazaF0?usp=sharing) Multispeaker VCTK pretrained ckpt (1100 epoch on VCTK) - [Huggingface](https://huggingface.co/datasets/isjwdu/DFADD/blob/main/pflowtts_checkpoint_epoch%3D1099.ckpt)\n\nUnofficial implementation of the paper [P-Flow: A Fast and Data-Efficient Zero-Shot TTS through Speech Prompting](https://openreview.net/pdf?id=zNA7u7wtIN) by NVIDIA.\n\n![P-Flow](architecture.jpg)\n\nWhile recent large-scale neural codec language models have shown significant improvement in zero-shot TTS by training on thousands of hours of data, they suffer from drawbacks such as a lack of robustness, slow sampling speed similar to previous autoregressive TTS methods, and reliance on pre-trained neural codec representations. Our work proposes P-Flow, a fast and data-efficient zero-shot TTS model that uses speech prompts for speaker adaptation. P-Flow comprises a speechprompted text encoder for speaker adaptation and a flow matching generative decoder for high-quality and fast speech synthesis. Our speech-prompted text encoder uses speech prompts and text input to generate speaker-conditional text representation. The flow matching generative decoder uses the speaker-conditional output to synthesize high-quality personalized speech significantly faster than in real-time. Unlike the neural codec language models, we specifically train P-Flow on LibriTTS dataset using a continuous mel-representation. Through our training method using continuous speech prompts, P-Flow matches the speaker similarity performance of the large-scale zero-shot TTS models with two orders of magnitude less training data and has more than 20× faster sampling speed. Our results show that P-Flow has better pronunciation and is preferred in human likeness and speaker similarity to its recent state-of-the-art counterparts, thus defining P-Flow as an attractive and desirable alternative.\n\n## Credits\n- Of course the kind author of the paper for taking some time to explain me some details of the paper that I didn't understand at first. \n- We will build this repo based on the [VITS2 repo](https://github.com/p0p4k/vits2_pytorch), [MATCHA-TTS repo](https://github.com/shivammehta25/Matcha-TTS/) and [VoiceFlow-TTS repo](https://github.com/cantabile-kwok/VoiceFlow-TTS)\n- Folks at LMNT-com. Try out their ultrafast, realistic TTS models at [lmnt-com](https://app.lmnt.com/). If you like what we're building here, [come join us at LMNT](https://explore.lmnt.com).\n\n# Dry run\n``` sh\ncd pflowtts_pytorch/notebooks\n```\n``` python\nimport sys\nsys.path.append('..')\n\nfrom pflow.models.pflow_tts import pflowTTS\nimport torch\nfrom dataclasses import dataclass\n\n@dataclass\nclass DurationPredictorParams:\n    filter_channels_dp: int\n    kernel_size: int\n    p_dropout: float\n\n@dataclass\nclass EncoderParams:\n    n_feats: int\n    n_channels: int\n    filter_channels: int\n    filter_channels_dp: int\n    n_heads: int\n    n_layers: int\n    kernel_size: int\n    p_dropout: float\n    spk_emb_dim: int\n    n_spks: int\n    prenet: bool\n\n@dataclass\nclass CFMParams:\n    name: str\n    solver: str\n    sigma_min: float\n\n# Example usage\nduration_predictor_params = DurationPredictorParams(\n    filter_channels_dp=256,\n    kernel_size=3,\n    p_dropout=0.1\n)\n\nencoder_params = EncoderParams(\n    n_feats=80,\n    n_channels=192,\n    filter_channels=768,\n    filter_channels_dp=256,\n    n_heads=2,\n    n_layers=6,\n    kernel_size=3,\n    p_dropout=0.1,\n    spk_emb_dim=64,\n    n_spks=1,\n    prenet=True\n)\n\ncfm_params = CFMParams(\n    name='CFM',\n    solver='euler',\n    sigma_min=1e-4\n)\n\n@dataclass\nclass EncoderOverallParams:\n    encoder_type: str\n    encoder_params: EncoderParams\n    duration_predictor_params: DurationPredictorParams\n\nencoder_overall_params = EncoderOverallParams(\n    encoder_type='RoPE Encoder',\n    encoder_params=encoder_params,\n    duration_predictor_params=duration_predictor_params\n)\n\n@dataclass\nclass DecoderParams:\n    channels: tuple\n    dropout: float\n    attention_head_dim: int\n    n_blocks: int\n    num_mid_blocks: int\n    num_heads: int\n    act_fn: str\n\ndecoder_params = DecoderParams(\n    channels=(256, 256),\n    dropout=0.05,\n    attention_head_dim=64,\n    n_blocks=1,\n    num_mid_blocks=2,\n    num_heads=2,\n    act_fn='snakebeta',\n)\n    \nmodel = pflowTTS(\n    n_vocab=100,\n    n_feats=80,\n    encoder=encoder_overall_params,\n    decoder=decoder_params.__dict__,\n    cfm=cfm_params,\n    data_statistics=None,\n)\n\nx = torch.randint(0, 100, (4, 20))\nx_lengths = torch.randint(10, 20, (4,))\ny = torch.randn(4, 80, 500)\ny_lengths = torch.randint(300, 500, (4,))\n\ndur_loss, prior_loss, diff_loss, attn = model(x, x_lengths, y, y_lengths)\n# backpropagate the loss \n\n# now synthesises\nx = torch.randint(0, 100, (1, 20))\nx_lengths = torch.randint(10, 20, (1,))\ny_slice = torch.randn(1, 80, 264)\n\nmodel.synthesise(x, x_lengths, y_slice, n_timesteps=10)\n```\n# Quick run in Google Colab\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1tyWDhqfP8ff5O4YcSv9S7TgSpiEyzIvz?usp=sharing)\n\n# Instructions to run\n0. Create an environment (suggested but optional)\n``` sh\nconda create -n pflowtts python=3.10 -y\nconda activate pflowtts\n```\n\nStay in the root directory (of course clone the repo first!)\n``` sh\ncd pflowtts_pytorch\npip install -r requirements.txt\n```\n\n1. Build Monotonic Alignment Search \n```sh\n# Cython-version Monotonoic Alignment Search\npython setup.py build_ext --inplace\n```\n\nLet's assume we are training with LJ Speech\n\n2. Download the dataset from [here](https://keithito.com/LJ-Speech-Dataset/), extract it to `data/LJSpeech-1.1`, and prepare the file lists to point to the extracted data like for [item 5 in the setup of the NVIDIA Tacotron 2 repo](https://github.com/NVIDIA/tacotron2#setup).\n\n3a. Go to `configs/data/ljspeech.yaml` and change\n```yaml\ntrain_filelist_path: data/filelists/ljs_audio_text_train_filelist.txt\nvalid_filelist_path: data/filelists/ljs_audio_text_val_filelist.txt\n```\n3b. Helper commands for the lazy \n```sh\n!mkdir -p /home/ubuntu/LJSpeech/LJSpeech-1.1/filelists\n!wget -O /home/ubuntu/LJSpeech/LJSpeech-1.1/filelists/ljs_audio_text_test_filelist.txt https://raw.githubusercontent.com/NVIDIA/tacotron2/master/filelists/ljs_audio_text_test_filelist.txt\n!wget -O /home/ubuntu/LJSpeech/LJSpeech-1.1/filelists/ljs_audio_text_train_filelist.txt https://raw.githubusercontent.com/NVIDIA/tacotron2/master/filelists/ljs_audio_text_train_filelist.txt\n!wget -O /home/ubuntu/LJSpeech/LJSpeech-1.1/filelists/ljs_audio_text_val_filelist.txt https://raw.githubusercontent.com/NVIDIA/tacotron2/master/filelists/ljs_audio_text_val_filelist.txt\n\n!sed -i -- 's,DUMMY,/home/ubuntu/LJSpeech/LJSpeech-1.1/wavs,g' /home/ubuntu/LJSpeech/LJSpeech-1.1/filelists/*.txt\n\n!sed -i -- 's,train_filelist_path: data/filelists/ljs_audio_text_train_filelist.txt,train_filelist_path: /home/ubuntu/LJSpeech/LJSpeech-1.1/filelists/ljs_audio_text_train_filelist.txt,g' /home/ubuntu/LJSpeech/pflowtts_pytorch/configs/data/ljspeech.yaml\n!sed -i -- 's,valid_filelist_path: data/filelists/ljs_audio_text_val_filelist.txt,valid_filelist_path: /home/ubuntu/LJSpeech/LJSpeech-1.1/filelists/ljs_audio_text_val_filelist.txt,g' /home/ubuntu/LJSpeech/pflowtts_pytorch/configs/data/ljspeech.yaml\n```\n4. Generate normalisation statistics with the yaml file of dataset configuration\n\n```bash\ncd pflowtts_pytorch/pflow/utils\npython generate_data_statistics.py -i ljspeech.yaml\n# Output:\n#{'mel_mean': -5.53662231756592, 'mel_std': 2.1161014277038574}\n```\n\nUpdate these values in `configs/data/ljspeech.yaml` under `data_statistics` key.\n\n```bash\ndata_statistics:  # Computed for ljspeech dataset\n  mel_mean: -5.536622\n  mel_std: 2.116101\n```\nto the paths of your train and validation filelists.\n\n5. Run the training script\n\n```bash\npython pflow/train.py experiment=ljspeech\n```\n\n- for multi-gpu training, run\n\n```bash\npython pflow/train.py experiment=ljspeech trainer.devices=[0,1]\n```\n\n## Architecture details\n- [x] Speech prompted text encoder with Prenet and RoPE Transformer\n- [x] Duration predictor with MAS\n- [x] Flow matching generative decoder with CFM (paper uses wavenet decoder; we use modified wavenet and optional U-NET decoder is included to experiment with)\n- [x] Speech prompt input currently slices the input spectrogram and concatenates it with the text embedding. Can support external speech prompt input (during training as well)\n- [x] pflow prompt masking loss for training\n- [x] HiFiGan for vocoder\n- [x] Guidance for sampling\n\n## TODOs, features and update notes\n- [x] (11/12/2023) Currently it is an experimental repo with many features substituted with quick architecture implementations I found online. I will add the original architectures soon.\n- [x] (11/12/2023) Check out `notebooks` for a quick dry run and architecture testing of the model.\n- [x] ~(11/12/2023) Training fails on my dataset at the moment, will debug and fix it soon. But the training the code runs error-free.~\n- [x] (11/13/2023) \n    - fixed big mistake in monotonic align build\n    - lot of combinations possible within model\n    - architecture big picture is same like paper, but internals are different\n    - if model doesnt converge, will eventually settle to paper's exact architecture\n- [x] (11/13/2023) Tensorboard screenshot\n    ![Alt text](init_tensorboard_11_13_23.png)\n- [x] (11/13/2023)\n    - added installation instructions\n- [x] (11/13/2023)\n    - looks like the model is learning and is on the right track. \n    ![Alt text](val_out_tboard.png)\n- [x] (11/14/2023)\n    ![Alt text](20k_tboard.png)\n- [x] (11/14/2023)\n    - added Google Colab notebook for quick run\n- [x] (11/16/2023)\n    - added sample audio\n    - some architecture changes\n    - we know the model learns, now we need to try multispeaker and check for prosody.\n- [x] (11/17/2023)\n    - added 3 new branches -\u003e\n        - dev/stochastic -\u003e some changes to posterior sampling and text encoder (prior) to make it stochastic\n        - dev/encodec -\u003e predicts encodec continuous latent instead of mel spectrogram; if works, use encodec for decoding instead of hifi-gan\n        - exp/end2end -\u003e end to end training of pflow with hi-fi gan to generate audio directly from text and speech prompt input; if it works, vits-tts will be obsolete.\n- [x] (11/17/2023)\n    - 24 epochs encodec sample (although robotic, it is a proof of concept) [encodec_poc.wav](encodec_poc.wav)\n- [x] (11/20/2023)\n    - Model is more or less ready.\n    - Added 3 choices for estimators, need to make them into hyperparameters and add them to the config file.\n    - Thanks to [@zidsi](https://github.com/zidsi) for pointing out typos in the code.\n- [x] (11/23/2023)\n    - Added newer samples with better prosody and pronunciation. Check out `samples` folder. (LJSpeech trained for 300k steps) _Paper recommends 800k steps._\n- [x] (12/02/2023)\n    - Added guidance for euler solver as recommended by the paper. Improves the quality of the audio drastically. Thanks to @robbit on discord for pointing this out.\n- [x] (12/03/2023)\n    - Added descript-codec branch for the curious. (not tested yet)\n- [x] (01/17/2024)\n    - Added minimum wav sample size (in seconds) parameter in the configs (data/[dataset].yaml), `min_sample_size` (default is 4s; so that prompt is at least 3s and prediction is at least 1s)\n    - Added `prompt_size` in the configs (models/pflow.yaml) to control the size of the prompt. It is number of mel frames to be used as prompt from the wav sample at traning. (default is ~3s; 3*22050//256 = 258; rounded to 264 in configs) \n    - Added `dur_p_use_log` in the configs (models/pflow.yaml) to control whether to use log of duration prediction or not for loss calculation. (default is False now) My hypothesis is that log durations mse loss doesn't work well for longer pauses etc (due to nature of log function). So, we just `e` power the log durations before calculating the loss. Alternative way can be using relu instead of log.\n    - Added `transfer_ckpt_path` in the configs (train.yaml) to control the path of the ckpt to be used for transfer learning. (default is None) If None, then the model is trained from scratch. If not None, then the model is loaded from the ckpt path and trained from step 0. In case, `ckpt_path` is also not None, then the model is loaded from `ckpt_path` and trained from the step it was saved at. `transfer_ckpt_path` can handle layer size mismatches between the ckpt and the model.\n- [x] (01/28/2024)\n    - Added ONNX export support based on Matcha-TTS repo. (not tested yet, will test soon) {draft}\n- [x] (01/30/2024)\n    - ONNX tested and works well. Use the arguments in `export.py` and `inference.py` to export and test the model. (the arguemnts are self-explanatory)\n- [x] (03/18/2024)\n    - added pos embeddings in speech text encoder.\n- [x] Anyone is welcome to contribute to this repo. Please feel free to open an issue or a PR.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fp0p4k%2Fpflowtts_pytorch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fp0p4k%2Fpflowtts_pytorch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fp0p4k%2Fpflowtts_pytorch/lists"}