{"id":50131932,"url":"https://github.com/Aratako/Irodori-TTS","last_synced_at":"2026-07-12T03:00:42.260Z","repository":{"id":346376560,"uuid":"1166706525","full_name":"Aratako/Irodori-TTS","owner":"Aratako","description":"A Flow Matching-based Text-to-Speech Model with Emoji-driven Style Control","archived":false,"fork":false,"pushed_at":"2026-05-31T06:41:49.000Z","size":465,"stargazers_count":794,"open_issues_count":10,"forks_count":98,"subscribers_count":14,"default_branch":"main","last_synced_at":"2026-05-31T08:14:03.952Z","etag":null,"topics":["diffusion-models","flow-matching","python","speech-synthesis","text-to-speech","tts","voice-cloning"],"latest_commit_sha":null,"homepage":"https://huggingface.co/collections/Aratako/irodori-tts","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Aratako.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-02-25T14:16:22.000Z","updated_at":"2026-05-31T06:41:53.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/Aratako/Irodori-TTS","commit_stats":null,"previous_names":["aratako/irodori-tts"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/Aratako/Irodori-TTS","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aratako%2FIrodori-TTS","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aratako%2FIrodori-TTS/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aratako%2FIrodori-TTS/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aratako%2FIrodori-TTS/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Aratako","download_url":"https://codeload.github.com/Aratako/Irodori-TTS/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aratako%2FIrodori-TTS/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35380480,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-12T02:00:06.386Z","response_time":87,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["diffusion-models","flow-matching","python","speech-synthesis","text-to-speech","tts","voice-cloning"],"created_at":"2026-05-23T21:00:29.573Z","updated_at":"2026-07-12T03:00:42.253Z","avatar_url":"https://github.com/Aratako.png","language":"Python","funding_links":[],"categories":["Text-to-Speech (TTS) Models","Python"],"sub_categories":["Irodori-TTS-500M-v2"],"readme":"# Irodori-TTS\n\n[![Model](https://img.shields.io/badge/Model-HuggingFace-yellow)](https://huggingface.co/Aratako/Irodori-TTS-500M-v3)\n[![VoiceDesign](https://img.shields.io/badge/VoiceDesign-HuggingFace-orange)](https://huggingface.co/Aratako/Irodori-TTS-600M-v3-VoiceDesign)\n[![Demo](https://img.shields.io/badge/Demo-HuggingFace%20Space-blue)](https://huggingface.co/spaces/Aratako/Irodori-TTS-500M-v3-Demo)\n[![VoiceDesign Demo](https://img.shields.io/badge/VoiceDesign%20Demo-HuggingFace%20Space-red)](https://huggingface.co/spaces/Aratako/Irodori-TTS-600M-v3-VoiceDesign-Demo)\n[![License: MIT](https://img.shields.io/badge/Code%20License-MIT-green.svg)](LICENSE)\n\nTraining and inference code for **Irodori-TTS**, a Flow Matching-based Text-to-Speech model. The architecture and training design largely follow [Echo-TTS](https://jordandarefsky.com/blog/2025/echo/), using [DACVAE](https://github.com/facebookresearch/dacvae) continuous latents as the generation target.\n\nFor an OpenAI-compatible inference API server, see [Irodori-TTS-Server](https://github.com/Aratako/Irodori-TTS-Server).\n\n\u003e [!IMPORTANT]\n\u003e `main` tracks the **v3** codebase and is intended for use with the **Irodori-TTS-500M-v3** base model release.\n\u003e It also supports the **Irodori-TTS-600M-v3-VoiceDesign** 3-branch VoiceDesign release.\n\u003e The current code remains backward-compatible with **Irodori-TTS-500M-v2** checkpoints, including **Irodori-TTS-500M-v2-VoiceDesign**.\n\u003e If you need the previous v2 codebase state, use the `v2` tag. If you need the previous v1 code, use the `v1` tag.\n\u003e v1 checkpoints / preprocessing are not compatible with v2/v3.\n\u003e The previous public v1 model is available at [Aratako/Irodori-TTS-500M](https://huggingface.co/Aratako/Irodori-TTS-500M).\n\nFor model weights and audio samples, please refer to the [base model card](https://huggingface.co/Aratako/Irodori-TTS-500M-v3) and the [VoiceDesign model card](https://huggingface.co/Aratako/Irodori-TTS-600M-v3-VoiceDesign).\n\n## Features\n\n- **Flow Matching TTS**: Rectified Flow Diffusion Transformer (RF-DiT) over continuous DACVAE latents\n- **Voice Cloning**: Zero-shot voice cloning from reference audio\n- **Multi-modal Voice Design**: v3 VoiceDesign can combine text, reference speech, and caption text for voice identity plus style/emotion control\n- **Emoji-based Style Control**: Emoji annotations in input text can influence delivery and non-verbal vocal expressions in supported checkpoints\n- **Automatic Duration Prediction**: v3 base and v3 VoiceDesign checkpoints estimate output length without manual `--seconds`\n- **Automatic Watermarking**: Generated audio is watermarked with [SilentCipher](https://github.com/sony/silentcipher) when available\n- **Multi-GPU Training**: Distributed training via `uv run --no-sync torchrun` with gradient accumulation, mixed precision (bf16), and W\u0026B logging\n- **PEFT LoRA Fine-Tuning**: Parameter-efficient adaptation with PEFT/LoRA for released checkpoints\n- **Speaker Inversion**: Learn reusable speaker embedding tokens for a target voice while freezing the base model\n- **Flexible Inference**: CLI, Gradio Web UI, and HuggingFace Hub checkpoint support\n\n## Architecture\n\nThe current codebase supports two closely related checkpoint families:\n\n1. **Base model (`Aratako/Irodori-TTS-500M-v3`)**:\n   Text encoder + reference latent encoder + diffusion transformer + duration predictor. The reference latent encoder consumes patched DACVAE latents from reference audio for speaker/style conditioning. v2 base checkpoints remain supported for inference.\n2. **VoiceDesign model (`Aratako/Irodori-TTS-600M-v3-VoiceDesign`)**:\n   Text encoder + reference latent encoder + caption encoder + diffusion transformer + duration predictor. The v3 VoiceDesign path supports 3-branch conditioning from text, reference speech, and caption text. v2 VoiceDesign remains supported as a backward-compatible caption-only checkpoint family.\n\nShared building blocks:\n\n1. **Text Encoder**: Token embeddings initialized from a pretrained LLM, followed by self-attention + SwiGLU transformer layers with RoPE\n2. **Reference Latent Encoder**: Encodes patched reference audio latents for speaker identity conditioning\n3. **Caption Encoder**: Encodes style-control text for emotion, tone, speaking style, and acoustic context\n4. **Diffusion Transformer**: Joint-attention DiT blocks with Low-Rank AdaLN (timestep-conditioned adaptive layer normalization), half-RoPE, and SwiGLU MLPs\n5. **Duration Predictor**: v3 checkpoints include an integrated predictor for automatic output length estimation\n\nAudio is represented as continuous latent sequences via the codec configured by the checkpoint. The released v2/v3 checkpoints use the 32-dim [Semantic-DACVAE-Japanese-32dim](https://huggingface.co/Aratako/Semantic-DACVAE-Japanese-32dim) codec for 48kHz waveform reconstruction.\n\n## Installation\n\n```bash\ngit clone https://github.com/Aratako/Irodori-TTS.git\ncd Irodori-TTS\nuv sync --extra cu128  # NVIDIA CUDA 12.8 (Linux/Windows)\n```\n\nIf you want to explicitly select a PyTorch backend, use one of the backend\nextras below:\n\n```bash\n# NVIDIA CUDA 12.8 on Linux/Windows\nuv sync --extra cu128\n\n# AMD ROCm on Linux/WSL\nuv sync --extra rocm\n\n# Intel XPU on Linux/Windows\nuv sync --extra xpu\n\n# CPU-only, or macOS CPU/MPS via PyPI\nuv sync --extra cpu\n```\n\nThe PyTorch backend extras are mutually exclusive. The `cu128` extra uses the\nPyTorch CUDA 12.8 index, the `rocm` extra uses the PyTorch ROCm index on\nLinux, and the `xpu` extra uses the PyTorch XPU index on Linux/Windows.\nThe `cpu` extra uses the CPU PyTorch index on Linux/Windows and falls\nback to the standard PyPI PyTorch wheels on macOS.\n\nAfter syncing with a backend extra, use `uv run --no-sync ...` for the commands\nbelow to avoid re-syncing the environment without the selected PyTorch backend\nextra.\n\nThe `rocm` extra includes `pytorch-triton-rocm` because `triton-rocm` alone does\nnot provide `triton.language` for the `transformers` to `torch._dynamo` import\npath. This was validated with AMD GPU inference.\n\n## Quick Start\n\n### Simple Inference\n\n```bash\nuv run --no-sync python infer.py \\\n  --hf-checkpoint Aratako/Irodori-TTS-500M-v3 \\\n  --text \"こんにちは、私はAIです。これは音声合成のテストです。\" \\\n  --ref-wav path/to/reference.wav \\\n  --output-wav outputs/sample.wav\n```\n\n### Inference without Reference Audio\n\n```bash\nuv run --no-sync python infer.py \\\n  --hf-checkpoint Aratako/Irodori-TTS-500M-v3 \\\n  --text \"こんにちは、私はAIです。これは音声合成のテストです。\" \\\n  --no-ref \\\n  --output-wav outputs/sample.wav\n```\n\n### VoiceDesign Inference\n\nPure VoiceDesign from text + caption:\n\n```bash\nuv run --no-sync python infer.py \\\n  --hf-checkpoint Aratako/Irodori-TTS-600M-v3-VoiceDesign \\\n  --text \"こんにちは、私はAIです。これは音声合成のテストです。\" \\\n  --caption \"落ち着いた女性の声で、近い距離感でやわらかく自然に読み上げてください。\" \\\n  --no-ref \\\n  --output-wav outputs/sample_voice_design.wav\n```\n\nStyle-controlled voice cloning with text + reference speech + caption:\n\n```bash\nuv run --no-sync python infer.py \\\n  --hf-checkpoint Aratako/Irodori-TTS-600M-v3-VoiceDesign \\\n  --text \"どうしてもっと早く教えてくれなかったの？私、ずっと待ってたのに。\" \\\n  --ref-wav path/to/reference.wav \\\n  --caption \"深く傷つき、今にも泣き出しそうな様子。声が震えており、悲痛なトーンで弱々しく話す。\" \\\n  --output-wav outputs/sample_voice_design_clone.wav\n```\n\n### Speaker Inversion Inference\n\nUse a learned Speaker Inversion embedding instead of reference audio:\n\n```bash\nuv run --no-sync python infer.py \\\n  --checkpoint path/to/Irodori-TTS-500M-v3.safetensors \\\n  --ref-embed path/to/my.speaker.safetensors \\\n  --text \"こんにちは、私はAIです。これは音声合成のテストです。\" \\\n  --output-wav outputs/sample_speaker_inversion.wav\n```\n\n### Gradio Web UI\n\n```bash\nuv run --no-sync python gradio_app.py --server-name 0.0.0.0 --server-port 7860\n```\n\nThen access the UI at `http://localhost:7860`.\nThe hosted v3 demo is available at [Aratako/Irodori-TTS-500M-v3-Demo](https://huggingface.co/spaces/Aratako/Irodori-TTS-500M-v3-Demo).\nThe reference input area supports either reference audio/latent input or a Speaker Inversion embedding via tabs.\n\nFor VoiceDesign checkpoints, use the dedicated UI:\n\n```bash\nuv run --no-sync python gradio_app_voicedesign.py --server-name 0.0.0.0 --server-port 7861\n```\n\nThe hosted VoiceDesign demo is available at [Aratako/Irodori-TTS-600M-v3-VoiceDesign-Demo](https://huggingface.co/spaces/Aratako/Irodori-TTS-600M-v3-VoiceDesign-Demo).\n\n`gradio_app.py` is for `Aratako/Irodori-TTS-500M-v3`. `gradio_app_voicedesign.py` is for `Aratako/Irodori-TTS-600M-v3-VoiceDesign` and remains compatible with v2 VoiceDesign checkpoints.\n\n## Inference\n\n### CLI\n\n```bash\nuv run --no-sync python infer.py \\\n  --hf-checkpoint Aratako/Irodori-TTS-500M-v3 \\\n  --text \"こんにちは、私はAIです。これは音声合成のテストです。\" \\\n  --ref-wav path/to/reference.wav \\\n  --output-wav outputs/sample.wav\n```\n\nLocal checkpoints (`.pt` or `.safetensors`) are also supported:\n\n```bash\nuv run --no-sync python infer.py \\\n  --checkpoint outputs/checkpoint_final.safetensors \\\n  --text \"こんにちは、私はAIです。これは音声合成のテストです。\" \\\n  --ref-wav path/to/reference.wav \\\n  --output-wav outputs/sample.wav\n```\n\nVoiceDesign checkpoints support caption conditioning. The v3 VoiceDesign model can run with\ncaption only by passing `--no-ref`, or with both reference speech and caption by passing\n`--ref-wav`, `--ref-latent`, or `--ref-embed`.\n\n```bash\nuv run --no-sync python infer.py \\\n  --hf-checkpoint Aratako/Irodori-TTS-600M-v3-VoiceDesign \\\n  --text \"こんにちは、私はAIです。これは音声合成のテストです。\" \\\n  --caption \"落ち着いた、近い距離感の女性話者\" \\\n  --no-ref \\\n  --output-wav outputs/sample_voice_design.wav\n```\n\n```bash\nuv run --no-sync python infer.py \\\n  --hf-checkpoint Aratako/Irodori-TTS-600M-v3-VoiceDesign \\\n  --text \"あははっ🤭、それ本当に言ってるの？…😮‍💨まぁ、君らしいけどね。\" \\\n  --caption \"余裕のある大人の男性。親しい相手に対して、くだけた雰囲気で呆れながらも楽しそうに話している。\" \\\n  --ref-wav path/to/reference.wav \\\n  --output-wav outputs/sample_voice_design_ref_caption.wav\n```\n\nThe older `Aratako/Irodori-TTS-500M-v2-VoiceDesign` checkpoint is still supported, but it is caption-only and intentionally ignores speaker/reference conditioning.\n\nLoRA adapter directories can be loaded dynamically at inference time without\nexporting a merged checkpoint:\n\n```bash\nuv run --no-sync python infer.py \\\n  --checkpoint path/to/base_model.safetensors \\\n  --lora-adapter outputs/irodori_tts_lora/checkpoint_final \\\n  --text \"こんにちは、私はAIです。これはLoRA推論のテストです。\" \\\n  --ref-wav path/to/reference.wav \\\n  --output-wav outputs/sample_lora.wav\n```\n\nSpeaker Inversion embedding checkpoints can be used with the same base model that\nwas used for inversion training. Pass the embedding with `--ref-embed`;\nit is mutually exclusive with `--ref-wav`, `--ref-latent`, and `--no-ref`.\n\n```bash\nuv run --no-sync python infer.py \\\n  --checkpoint path/to/Irodori-TTS-500M-v3.safetensors \\\n  --ref-embed outputs/speaker_inversion/name/checkpoint_final.speaker.safetensors \\\n  --text \"こんにちは、私はAIです。これはSpeaker Inversion推論のテストです。\" \\\n  --output-wav outputs/sample_speaker_inversion.wav\n```\n\n### Output Duration\n\nThe v3 base and v3 VoiceDesign models integrate duration prediction into inference.\nWhen `--seconds` is omitted, the runtime estimates the output length from the input\ntext and enabled conditions, then generates audio for that estimated duration. Use\n`--duration-scale` to multiply the predicted length (`\u003e1` longer, `\u003c1` shorter). For\nexact control, pass `--seconds` manually.\n\nOlder v2 checkpoints were trained with fixed-length 30-second targets. They remain\nsupported by the v3 codebase and still accept manual `--seconds`, but forcing a\nnon-default duration can reduce audio quality; prefer the v3 base model for automatic\nor scaled duration control.\n\n### Sway Sampling\n\nFor faster experimental inference, Sway Sampling can be combined with fewer Euler\nsteps:\n\n```bash\nuv run --no-sync python infer.py \\\n  --hf-checkpoint Aratako/Irodori-TTS-500M-v3 \\\n  --text \"こんにちは、私はAIです。これは音声合成のテストです。\" \\\n  --ref-wav path/to/reference.wav \\\n  --num-steps 6 \\\n  --t-schedule-mode sway \\\n  --sway-coeff -1.0 \\\n  --output-wav outputs/sample_sway.wav\n```\n\n### Additional Inference Notes\n\nFor tuning guidance and detailed explanations of inference options, see the\n[Parameter Guide](docs/parameters.md).\n\nGenerated audio is passed through [SilentCipher](https://github.com/sony/silentcipher) watermarking automatically when the dependency and model files are available.\n\n## Training\n\n### 1. Prepare Manifest (Precompute DACVAE Latents)\n\nEncodes audio from a Hugging Face dataset into DACVAE latents and produces a JSONL manifest for training.\n\n```bash\nuv run --no-sync python prepare_manifest.py \\\n  --dataset myorg/my_dataset \\\n  --split train \\\n  --audio-column audio \\\n  --text-column text \\\n  --output-manifest data/train_manifest.jsonl \\\n  --latent-dir data/latents \\\n  --device cuda\n```\n\nTo include `speaker_id` in the manifest (for speaker-conditioned training):\n\n```bash\nuv run --no-sync python prepare_manifest.py \\\n  --dataset myorg/my_dataset \\\n  --split train \\\n  --audio-column audio \\\n  --text-column text \\\n  --speaker-column speaker \\\n  --output-manifest data/train_manifest.jsonl \\\n  --latent-dir data/latents \\\n  --device cuda\n```\n\nTo include `caption` in the manifest (for caption-conditioned voice design training):\n\n```bash\nuv run --no-sync python prepare_manifest.py \\\n  --dataset myorg/my_dataset \\\n  --split train \\\n  --audio-column audio \\\n  --text-column text \\\n  --caption-column caption \\\n  --speaker-column speaker \\\n  --output-manifest data/train_manifest.jsonl \\\n  --latent-dir data/latents \\\n  --device cuda\n```\n\nSpeaker/reference labels depend on the training mode:\n\n- For v2 VoiceDesign training, `speaker_id` is optional because the model learns from\n  `text + caption`.\n- For v3 VoiceDesign training, keep `speaker_id` available so the model can learn from\n  `text + speaker/reference + caption`.\n- For Speaker Inversion training, `speaker_id` is not required because the run learns one\n  shared speaker embedding from the target speaker samples.\n\nThe manifest `caption` value may also be a list of strings; training randomly selects one\nnon-empty caption each time that row is loaded.\n\nThis produces a JSONL manifest with entries like:\n\n```json\n{\"text\": \"こんにちは\", \"caption\": \"落ち着いた、近い距離感の女性話者\", \"latent_path\": \"data/latents/00001.pt\", \"speaker_id\": \"myorg/my_dataset:speaker_001\", \"num_frames\": 750}\n```\n\n### 2. Training\n\nSingle-GPU training:\n\n```bash\nuv run --no-sync python train.py \\\n  --config configs/train_500m_v3_phase1_body.yaml \\\n  --manifest data/train_manifest.jsonl \\\n  --output-dir outputs/irodori_tts\n```\n\nv3 release training uses two phases. After training the body, initialize the integrated\nduration predictor from the phase-1 checkpoint:\n\n```bash\nuv run --no-sync python train.py \\\n  --config configs/train_500m_v3_phase2_duration.yaml \\\n  --manifest data/train_manifest.jsonl \\\n  --output-dir outputs/irodori_tts_duration \\\n  --init-checkpoint outputs/irodori_tts/checkpoint_final.pt\n```\n\nv2 VoiceDesign training uses a dedicated config:\n\n```bash\nuv run --no-sync python train.py \\\n  --config configs/train_500m_v2_voice_design.yaml \\\n  --manifest data/train_manifest.jsonl \\\n  --output-dir outputs/irodori_tts_voice_design\n```\n\n`configs/train_500m_v2_voice_design.yaml` sets `use_caption_condition: true` and disables the\nspeaker/reference branch. Caption-free configs continue to use speaker conditioning when\n`speaker_id` / reference inputs are available.\n\nv3 VoiceDesign training uses two phases. Phase 1 initializes the RF/DiT body from the\nv3 base checkpoint while adding the caption branch and skipping the base duration\npredictor:\n\n```bash\nuv run --no-sync python train.py \\\n  --config configs/train_500m_v3_voice_design_phase1_body.yaml \\\n  --manifest data/train_manifest.jsonl \\\n  --output-dir outputs/irodori_tts_voice_design_phase1 \\\n  --init-checkpoint path/to/Irodori-TTS-500M-v3.safetensors\n```\n\nPhase 2 adds and trains a newly initialized duration predictor with text + speaker +\ncaption conditioning:\n\n```bash\nuv run --no-sync python train.py \\\n  --config configs/train_500m_v3_voice_design_phase2_duration.yaml \\\n  --manifest data/train_manifest.jsonl \\\n  --output-dir outputs/irodori_tts_voice_design_phase2 \\\n  --init-checkpoint outputs/irodori_tts_voice_design_phase1/checkpoint_final.pt\n```\n\nThe VoiceDesign config also enables `caption_warmup: true` for optional caption-branch warmup.\n`warmup_steps` controls the LR scheduler, while `caption_warmup_steps` controls how long\nnon-caption gradients are discarded before normal joint training resumes.\n\n### v3 Duration Predictor Training\n\nv3 training uses two phases: `configs/train_500m_v3_phase1_body.yaml` trains the\nvariable-length DiT body, then `configs/train_500m_v3_phase2_duration.yaml` freezes the\nbody and trains the duration predictor.\n\nThe duration predictor regresses `log1p(num_frames)` with Huber loss. The current v3 phase2\nconfig uses the token-sum duration predictor selected from ablations; see the parameter\nguide for the architecture details.\n\nMulti-GPU DDP training:\n\n```bash\nuv run --no-sync torchrun --nproc_per_node 4 train.py \\\n  --config configs/train_500m_v3_phase1_body.yaml \\\n  --manifest data/train_manifest.jsonl \\\n  --output-dir outputs/irodori_tts \\\n  --device cuda\n```\n\nTraining supports YAML config files with `model` and `train` sections. CLI arguments take precedence over YAML values. See `uv run --no-sync python train.py --help` for all available options.\nFor a more detailed explanation of model and training config fields, see [Parameter Guide](docs/parameters.md).\n\n#### Fine-Tuning from Released Weights\n\nStart a new training run from released inference weights (`.safetensors`). This initializes only the model weights; optimizer / scheduler state starts fresh. For the v3 base release, the LoRA config keeps the duration predictor as part of the saved adapter by default.\n\n```bash\nuv run --no-sync python train.py \\\n  --config configs/train_500m_v3_lora.yaml \\\n  --manifest data/train_manifest.jsonl \\\n  --output-dir outputs/irodori_tts_lora \\\n  --init-checkpoint path/to/Irodori-TTS-500M-v3.safetensors\n```\n\nv3 VoiceDesign LoRA fine-tuning:\n\n```bash\nuv run --no-sync python train.py \\\n  --config configs/train_500m_v3_voice_design_lora.yaml \\\n  --manifest data/train_manifest.jsonl \\\n  --output-dir outputs/irodori_tts_voice_design_lora \\\n  --init-checkpoint path/to/Irodori-TTS-600M-v3-VoiceDesign.safetensors\n```\n\nFor the older v2 VoiceDesign checkpoint, use `configs/train_500m_v2_voice_design_lora.yaml`\nand initialize from `Irodori-TTS-500M-v2-VoiceDesign.safetensors`.\n\nLoRA target presets, adapter saving behavior, and resume details are covered in the\n[Parameter Guide](docs/parameters.md).\n\n#### Speaker Inversion\n\nSpeaker Inversion trains only a small set of speaker embedding tokens while keeping the\nbase Irodori-TTS model frozen. It is useful when you want a reusable speaker identity\ncheckpoint instead of providing reference audio at every inference call.\n\nPrepare a manifest from the target speaker's audio, then initialize from the released v3\nbase checkpoint:\n\n```bash\nuv run --no-sync python train.py \\\n  --config configs/train_500m_v3_speaker_inversion.yaml \\\n  --manifest data/target_speaker_manifest.jsonl \\\n  --init-checkpoint path/to/Irodori-TTS-500M-v3.safetensors \\\n  --output-dir outputs/speaker_inversion/name\n```\n\nThe saved checkpoints are embedding-only `.speaker.safetensors` files, for example\n`outputs/speaker_inversion/name/checkpoint_final.speaker.safetensors`. Use that file\nwith the base model during inference:\n\n```bash\nuv run --no-sync python infer.py \\\n  --checkpoint path/to/Irodori-TTS-500M-v3.safetensors \\\n  --ref-embed outputs/speaker_inversion/name/checkpoint_final.speaker.safetensors \\\n  --text \"こんにちは、これは学習した話者埋め込みを使った推論です。\" \\\n  --output-wav outputs/sample_speaker_inversion.wav\n```\n\nTo continue from a saved embedding, set `speaker_inversion_init_embedding` in the\nconfig or pass `--speaker-inversion-init-embedding path/to/checkpoint.speaker.safetensors`.\nFull trainer `--resume` is intentionally not used for Speaker Inversion checkpoints.\nEnable `gradient_checkpointing: true` or pass `--gradient-checkpointing` if GPU memory is tight.\n\n#### Resuming Interrupted Training\n\nResume an existing training run from a training checkpoint. Full-model runs use `.pt`; LoRA runs use checkpoint directories. Both restore optimizer, scheduler, and step state.\n\n```bash\nuv run --no-sync python train.py \\\n  --config configs/train_500m_v3_phase1_body.yaml \\\n  --manifest data/train_manifest.jsonl \\\n  --output-dir outputs/irodori_tts \\\n  --resume outputs/irodori_tts/checkpoint_0010000.pt\n```\n\nLoRA resume example:\n\n```bash\nuv run --no-sync python train.py \\\n  --config configs/train_500m_v3_lora.yaml \\\n  --manifest data/train_manifest.jsonl \\\n  --output-dir outputs/irodori_tts_lora \\\n  --resume outputs/irodori_tts_lora/checkpoint_0010000\n```\n\nIf you move a LoRA checkpoint to another environment and the original base-checkpoint path is no longer valid, pass `--init-checkpoint path/to/base_model.safetensors` together with `--resume` to override the saved base-model path.\n\n### 3. Checkpoint Conversion\n\nConvert a training checkpoint to inference-only safetensors format:\n\n```bash\nuv run --no-sync python convert_checkpoint_to_safetensors.py outputs/checkpoint_final.pt\n```\n\nLoRA adapter checkpoints can also be converted directly:\n\n```bash\nuv run --no-sync python convert_checkpoint_to_safetensors.py outputs/irodori_tts_lora/checkpoint_final\n```\n\nLoRA adapter checkpoints are merged into the base model automatically during conversion, so the exported `.safetensors` file is directly usable for inference. If you do not want to merge the adapter, pass the adapter directory directly to `infer.py --lora-adapter` or the matching Gradio field.\n\n## Project Structure\n\n```text\nIrodori-TTS/\n├── train.py                    # Training entry point (DDP support)\n├── infer.py                    # CLI inference\n├── gradio_app.py               # Gradio web UI\n├── gradio_app_voicedesign.py   # Gradio web UI for VoiceDesign checkpoints\n├── prepare_manifest.py         # Dataset -\u003e DACVAE latent preprocessing\n├── convert_checkpoint_to_safetensors.py  # Checkpoint converter\n│\n├── docs/\n│   └── parameters.md         # Detailed parameter guide\n│\n├── irodori_tts/                # Core library\n│   ├── model.py                # TextToLatentRFDiT architecture\n│   ├── rf.py                   # Rectified Flow utilities \u0026 Euler CFG sampling\n│   ├── codec.py                # DACVAE codec wrapper\n│   ├── dataset.py              # Dataset and collator\n│   ├── tokenizer.py            # Pretrained LLM tokenizer wrapper\n│   ├── config.py               # Model / Train / Sampling config dataclasses\n│   ├── inference_runtime.py    # Cached, thread-safe inference runtime\n│   ├── lora.py                 # PEFT LoRA integration helpers\n│   ├── speaker_inversion.py    # Speaker Inversion embedding save/load helpers\n│   ├── text_normalization.py   # Japanese text normalization\n│   ├── optim.py                # Muon + AdamW optimizer\n│   └── progress.py             # Training progress tracker\n│\n└── configs/\n    ├── train_500m_v3_phase1_body.yaml        # 500M v3 body training config\n    ├── train_500m_v3_phase2_duration.yaml    # 500M v3 duration-predictor training config\n    ├── train_500m_v3_voice_design_phase1_body.yaml     # 600M v3 VoiceDesign body config\n    ├── train_500m_v3_voice_design_phase2_duration.yaml # 600M v3 VoiceDesign duration config\n    ├── train_500m_v3_voice_design_lora.yaml            # 600M v3 VoiceDesign RF+duration LoRA config\n    ├── train_500m_v3_lora.yaml               # 500M v3 LoRA fine-tuning config\n    ├── train_500m_v3_speaker_inversion.yaml  # 500M v3 Speaker Inversion config\n    ├── train_500m_v2.yaml                    # 500M v2 backward-compatible model config\n    ├── train_500m_v2_lora.yaml               # 500M v2 LoRA fine-tuning config\n    ├── train_500m_v2_voice_design.yaml       # 500M v2 VoiceDesign full fine-tuning config\n    ├── train_500m_v2_voice_design_lora.yaml  # 500M v2 VoiceDesign LoRA fine-tuning config\n    ├── train_500m.yaml                       # 500M v1 model config\n    └── train_2.5b.yaml                       # 2.5B parameter model config\n```\n\n## License\n\n- **Code**: [MIT License](LICENSE)\n- **Model Weights**: Please refer to the [base model card](https://huggingface.co/Aratako/Irodori-TTS-500M-v3) and the [VoiceDesign model card](https://huggingface.co/Aratako/Irodori-TTS-600M-v3-VoiceDesign) for licensing details\n\n## Acknowledgments\n\nThis project builds upon the following works:\n\n- [Echo-TTS](https://jordandarefsky.com/blog/2025/echo/) — Architecture and training design reference\n- [DACVAE](https://github.com/facebookresearch/dacvae) — Audio VAE\n- [SilentCipher](https://github.com/sony/silentcipher) — Audio watermarking\n\n## Citation\n\n```bibtex\n@misc{irodori-tts,\n  author = {Chihiro Arata},\n  title = {Irodori-TTS: A Flow Matching-based Text-to-Speech Model with Emoji-driven Style Control},\n  year = {2026},\n  publisher = {GitHub},\n  journal = {GitHub repository},\n  howpublished = {\\url{https://github.com/Aratako/Irodori-TTS}}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FAratako%2FIrodori-TTS","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FAratako%2FIrodori-TTS","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FAratako%2FIrodori-TTS/lists"}