https://github.com/SWivid/F5-TTS

Official code for "F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching"
https://github.com/SWivid/F5-TTS

Last synced: over 1 year ago
JSON representation

Official code for "F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching"

Host: GitHub
URL: https://github.com/SWivid/F5-TTS
Owner: SWivid
License: mit
Created: 2024-10-08T13:36:55.000Z (over 1 year ago)
Default Branch: main
Last Pushed: 2025-02-18T10:42:12.000Z (over 1 year ago)
Last Synced: 2025-02-18T11:26:04.448Z (over 1 year ago)
Language: Python
Homepage: https://arxiv.org/abs/2410.06885
Size: 1.76 MB
Stars: 9,680
Watchers: 95
Forks: 1,303
Open Issues: 42
Metadata Files:
- Readme: README.md
- License: LICENSE

Awesome Lists containing this project

awesome-tts-stt - F5-TTS
awesome-llm-tools - F5-TTS
awesome-ai - F5-TTS - An open-source project for high-quality text-to-speech. Explore a fast, high-performance voice generator. Possibility of cloning a voice with great precision (排行榜 [2025-03-18])
3d-resources - F5 TTS - source zero-shot voice cloning TTS.<br><small class="see-also">See also: <a href="#ai-audio--music-generation-software">Software Reference → AI Audio & Music Generation Software</a></small> | Open Source | Zero-Shot · Voice Clone | Zero-shot voice cloning | (AI & Machine Learning for CG / AI Audio & Music)
awesome-opensource-ai - F5-TTS - Flow matching-based TTS with fluent and faithful speech synthesis. Zero-shot voice cloning with high naturalness and prosody accuracy. MIT licensed. ![GitHub stars](https://img.shields.io/github/stars/SWivid/F5-TTS?style=social) (2. Open Foundation Models)
voiceai - F5-TTS - transformer TTS with high-quality zero-shot voice cloning. **🟡 Intermediate** (4. Text-to-speech (TTS) / Open source)

README

          # F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching

[![python](https://img.shields.io/badge/Python-3.10-brightgreen)](https://github.com/SWivid/F5-TTS)

[![arXiv](https://img.shields.io/badge/arXiv-2410.06885-b31b1b.svg?logo=arXiv)](https://arxiv.org/abs/2410.06885)

[![demo](https://img.shields.io/badge/GitHub-Demo%20page-orange.svg)](https://swivid.github.io/F5-TTS/)

[![hfspace](https://img.shields.io/badge/🤗-Space%20demo-yellow)](https://huggingface.co/spaces/mrfakename/E2-F5-TTS)

[![msspace](https://img.shields.io/badge/🤖-Space%20demo-blue)](https://modelscope.cn/studios/modelscope/E2-F5-TTS)

[![lab](https://img.shields.io/badge/X--LANCE-Lab-grey?labelColor=lightgrey)](https://x-lance.sjtu.edu.cn/)

[![lab](https://img.shields.io/badge/Peng%20Cheng-Lab-grey?labelColor=lightgrey)](https://www.pcl.ac.cn)

**F5-TTS**: Diffusion Transformer with ConvNeXt V2, faster trained and inference.

**E2 TTS**: Flat-UNet Transformer, closest reproduction from [paper](https://arxiv.org/abs/2406.18009).

**Sway Sampling**: Inference-time flow step sampling strategy, greatly improves performance

### Thanks to all the contributors !

## News

- **2024/10/08**: F5-TTS & E2 TTS base models on [🤗 Hugging Face](https://huggingface.co/SWivid/F5-TTS), [🤖 Model Scope](https://www.modelscope.cn/models/SWivid/F5-TTS_Emilia-ZH-EN), [🟣 Wisemodel](https://wisemodel.cn/models/SJTU_X-LANCE/F5-TTS_Emilia-ZH-EN).

## Installation

### Create a separate environment if needed

```bash

# Create a python 3.10 conda env (you could also use virtualenv)

conda create -n f5-tts python=3.10

conda activate f5-tts

```

### Install PyTorch with matched device

NVIDIA GPU

> ```bash

> # Install pytorch with your CUDA version, e.g.

> pip install torch==2.3.0+cu118 torchaudio==2.3.0+cu118 --extra-index-url https://download.pytorch.org/whl/cu118

> ```

AMD GPU

> ```bash

> # Install pytorch with your ROCm version (Linux only), e.g.

> pip install torch==2.5.1+rocm6.2 torchaudio==2.5.1+rocm6.2 --extra-index-url https://download.pytorch.org/whl/rocm6.2

> ```

Intel GPU

> ```bash

> # Install pytorch with your XPU version, e.g.

> # Intel® Deep Learning Essentials or Intel® oneAPI Base Toolkit must be installed

> pip install torch torchaudio --index-url https://download.pytorch.org/whl/test/xpu

> 

> # Intel GPU support is also available through IPEX (Intel® Extension for PyTorch)

> # IPEX does not require the Intel® Deep Learning Essentials or Intel® oneAPI Base Toolkit

> # See: https://pytorch-extension.intel.com/installation?request=platform

> ```

Apple Silicon

> ```bash

> # Install the stable pytorch, e.g.

> pip install torch torchaudio

> ```

### Then you can choose one from below:

> ### 1. As a pip package (if just for inference)

> 

> ```bash

> pip install git+https://github.com/SWivid/F5-TTS.git

> ```

> 

> ### 2. Local editable (if also do training, finetuning)

> 

> ```bash

> git clone https://github.com/SWivid/F5-TTS.git

> cd F5-TTS

> # git submodule update --init --recursive  # (optional, if need > bigvgan)

> pip install -e .

> ```

### Docker usage also available

```bash

# Build from Dockerfile

docker build -t f5tts:v1 .

# Or pull from GitHub Container Registry

docker pull ghcr.io/swivid/f5-tts:main

```

## Inference

### 1. Gradio App

Currently supported features:

- Basic TTS with Chunk Inference

- Multi-Style / Multi-Speaker Generation

- Voice Chat powered by Qwen2.5-3B-Instruct

- [Custom inference with more language support](src/f5_tts/infer/SHARED.md)

```bash

# Launch a Gradio app (web interface)

f5-tts_infer-gradio

# Specify the port/host

f5-tts_infer-gradio --port 7860 --host 0.0.0.0

# Launch a share link

f5-tts_infer-gradio --share

```

NVIDIA device docker compose file example

```yaml

services:

  f5-tts:

    image: ghcr.io/swivid/f5-tts:main

    ports:

      - "7860:7860"

    environment:

      GRADIO_SERVER_PORT: 7860

    entrypoint: ["f5-tts_infer-gradio", "--port", "7860", "--host", "0.0.0.0"]

    deploy:

      resources:

        reservations:

          devices:

            - driver: nvidia

              count: 1

              capabilities: [gpu]

volumes:

  f5-tts:

    driver: local

```

### 2. CLI Inference

```bash

# Run with flags

# Leave --ref_text "" will have ASR model transcribe (extra GPU memory usage)

f5-tts_infer-cli \

--model "F5-TTS" \

--ref_audio "ref_audio.wav" \

--ref_text "The content, subtitle or transcription of reference audio." \

--gen_text "Some text you want TTS model generate for you."

# Run with default setting. src/f5_tts/infer/examples/basic/basic.toml

f5-tts_infer-cli

# Or with your own .toml file

f5-tts_infer-cli -c custom.toml

# Multi voice. See src/f5_tts/infer/README.md

f5-tts_infer-cli -c src/f5_tts/infer/examples/multi/story.toml

```

### 3. More instructions

- In order to have better generation results, take a moment to read [detailed guidance](src/f5_tts/infer).

- The [Issues](https://github.com/SWivid/F5-TTS/issues?q=is%3Aissue) are very useful, please try to find the solution by properly searching the keywords of problem encountered. If no answer found, then feel free to open an issue.

## Training

### 1. Gradio App

Read [training & finetuning guidance](src/f5_tts/train) for more instructions.

```bash

# Quick start with Gradio web interface

f5-tts_finetune-gradio

```

## [Evaluation](src/f5_tts/eval)

## Development

Use pre-commit to ensure code quality (will run linters and formatters automatically)

```bash

pip install pre-commit

pre-commit install

```

When making a pull request, before each commit, run: 

```bash

pre-commit run --all-files

```

Note: Some model components have linting exceptions for E722 to accommodate tensor notation

## Acknowledgements

- [E2-TTS](https://arxiv.org/abs/2406.18009) brilliant work, simple and effective

- [Emilia](https://arxiv.org/abs/2407.05361), [WenetSpeech4TTS](https://arxiv.org/abs/2406.05763), [LibriTTS](https://arxiv.org/abs/1904.02882), [LJSpeech](https://keithito.com/LJ-Speech-Dataset/) valuable datasets

- [lucidrains](https://github.com/lucidrains) initial CFM structure with also [bfs18](https://github.com/bfs18) for discussion

- [SD3](https://arxiv.org/abs/2403.03206) & [Hugging Face diffusers](https://github.com/huggingface/diffusers) DiT and MMDiT code structure

- [torchdiffeq](https://github.com/rtqichen/torchdiffeq) as ODE solver, [Vocos](https://huggingface.co/charactr/vocos-mel-24khz) and [BigVGAN](https://github.com/NVIDIA/BigVGAN) as vocoder

- [FunASR](https://github.com/modelscope/FunASR), [faster-whisper](https://github.com/SYSTRAN/faster-whisper), [UniSpeech](https://github.com/microsoft/UniSpeech), [SpeechMOS](https://github.com/tarepan/SpeechMOS) for evaluation tools

- [ctc-forced-aligner](https://github.com/MahmoudAshraf97/ctc-forced-aligner) for speech edit test

- [mrfakename](https://x.com/realmrfakename) huggingface space demo ~

- [f5-tts-mlx](https://github.com/lucasnewman/f5-tts-mlx/tree/main) Implementation with MLX framework by [Lucas Newman](https://github.com/lucasnewman)

- [F5-TTS-ONNX](https://github.com/DakeQQ/F5-TTS-ONNX) ONNX Runtime version by [DakeQQ](https://github.com/DakeQQ)

## Citation

If our work and codebase is useful for you, please cite as:

```

@article{chen-etal-2024-f5tts,

      title={F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching}, 

      author={Yushen Chen and Zhikang Niu and Ziyang Ma and Keqi Deng and Chunhui Wang and Jian Zhao and Kai Yu and Xie Chen},

      journal={arXiv preprint arXiv:2410.06885},

      year={2024},

}

```

## License

Our code is released under MIT License. The pre-trained models are licensed under the CC-BY-NC license due to the training data Emilia, which is an in-the-wild dataset. Sorry for any inconvenience this may cause.

ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Awesome

https://github.com/SWivid/F5-TTS

Awesome Lists containing this project

README