{"id":18386563,"url":"https://github.com/playvoice/grad-svc","last_synced_at":"2025-09-13T07:21:24.409Z","repository":{"id":190540099,"uuid":"681045552","full_name":"PlayVoice/Grad-SVC","owner":"PlayVoice","description":"Diffusion Singing Voice Conversion based on Grad-TTS from HuaWei","archived":false,"fork":false,"pushed_at":"2023-09-28T07:15:24.000Z","size":2334,"stargazers_count":76,"open_issues_count":4,"forks_count":9,"subscribers_count":6,"default_branch":"20230920-V2-96","last_synced_at":"2023-09-28T09:59:00.545Z","etag":null,"topics":["diff-svc","diffusion","flow-matching","grad-tts","svc","vits","vits-svc","vits2","voice-change"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/PlayVoice.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-08-21T06:33:40.000Z","updated_at":"2023-09-30T03:35:05.412Z","dependencies_parsed_at":"2023-08-25T05:54:54.048Z","dependency_job_id":"3b652419-c2d0-4435-af0d-4cd995332024","html_url":"https://github.com/PlayVoice/Grad-SVC","commit_stats":null,"previous_names":["playvoice/grad-svc"],"tags_count":7,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PlayVoice%2FGrad-SVC","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PlayVoice%2FGrad-SVC/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PlayVoice%2FGrad-SVC/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PlayVoice%2FGrad-SVC/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/PlayVoice","download_url":"https://codeload.github.com/PlayVoice/Grad-SVC/tar.gz/refs/heads/20230920-V2-96","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223268062,"owners_count":17116741,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["diff-svc","diffusion","flow-matching","grad-tts","svc","vits","vits-svc","vits2","voice-change"],"created_at":"2024-11-06T01:22:30.919Z","updated_at":"2024-11-06T01:22:31.587Z","avatar_url":"https://github.com/PlayVoice.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\u003ch1\u003e Grad-SVC based on Grad-TTS from HUAWEI Noah's Ark Lab \u003c/h1\u003e\n\n[![Hugging Face Spaces](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue)](https://huggingface.co/spaces/maxmax20160403/grad-svc)\n\u003cimg alt=\"GitHub Repo stars\" src=\"https://img.shields.io/github/stars/PlayVoice/Grad-SVC\"\u003e\n\u003cimg alt=\"GitHub forks\" src=\"https://img.shields.io/github/forks/PlayVoice/Grad-SVC\"\u003e\n\u003cimg alt=\"GitHub issues\" src=\"https://img.shields.io/github/issues/PlayVoice/Grad-SVC\"\u003e\n\u003cimg alt=\"GitHub\" src=\"https://img.shields.io/github/license/PlayVoice/Grad-SVC\"\u003e\n\nThis project is named as [Grad-SVC](), or [GVC]() for short. Its core technology is diffusion, but so different from other diffusion based SVC models. Codes are adapted from `Grad-TTS` and `whisper-vits-svc`. So the features from `whisper-vits-svc` are used in this project. By the way, [Diff-VC](https://github.com/huawei-noah/Speech-Backbones/tree/main/DiffVC) is a follow-up of [Grad-TTS](), [Diffusion-Based Any-to-Any Voice Conversion](https://arxiv.org/abs/2109.13821)\n\n[Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech](https://arxiv.org/abs/2105.06337)\n\n![grad_tts](./assets/grad_tts.jpg)\n\n![grad_svc](./assets/grad_svc.jpg)\n\nThe framework of grad-svc-v1\n\n![grad_svc_v2](./assets/grad_svc_v2.jpg)\n\nThe framework of grad-svc-v2 \u0026 v3, encoder:768-\u003e512, diffusion:64-\u003e96\n\nhttps://github.com/PlayVoice/Grad-SVC/assets/16432329/f9b66af7-b5b5-4efb-b73d-adb0dc84a0ae\n\n\u003c/div\u003e\n\n## Features\n1. Such beautiful codes from Grad-TTS\n\n    `easy to read`\n\n2. Multi-speaker based on speaker encoder\n\n3. No speaker leaky based on `Perturbation` \u0026 `Instance Normlize` \u0026 `GRL`\n\n\t[One-shot Voice Conversion by Separating Speaker and Content Representations with Instance Normalization](https://arxiv.org/abs/1904.05742)\n\n4. No electronic sound\n\n5. Integrated [DPM Solver-k](https://github.com/LuChengTHU/dpm-solver) for less steps\n\n6. Integrated [Fast Maximum Likelihood Sampling Scheme](https://github.com/huawei-noah/Speech-Backbones/tree/main/DiffVC), for less steps\n\n7. [Conditional Flow Matching](https://voicebox.metademolab.com/) (V3), first used in SVC\n\n8. [Rectified Flow Matching](https://github.com/cantabile-kwok/VoiceFlow-TTS) (TODO)\n\n## Setup Environment\n1. Install project dependencies\n\n    ```shell\n    pip install -r requirements.txt\n    ```\n\n2. Download the Timbre Encoder: [Speaker-Encoder by @mueller91](https://drive.google.com/drive/folders/15oeBYf6Qn1edONkVLXe82MzdIi3O_9m3), put `best_model.pth.tar`  into `speaker_pretrain/`.\n\n3. Download [hubert_soft model](https://github.com/bshall/hubert/releases/tag/v0.1)，put `hubert-soft-0d54a1f4.pt` into `hubert_pretrain/`.\n\n4. Download pretrained [nsf_bigvgan_pretrain_32K.pth](https://github.com/PlayVoice/NSF-BigVGAN/releases/augment), and put it into `bigvgan_pretrain/`.\n   \n\t**Performance Bottleneck: Generator and Discriminator are 116Mb, but Generator is only 22Mb**\n\n\t**系统性能瓶颈：生成器和判别器一共116M，而生成器只有22M**\n\n6. Download pretrain model [gvc.pretrain.pth](https://github.com/PlayVoice/Grad-SVC/releases/tag/20230920), and put it into `grad_pretrain/`.\n    ```\n    python gvc_inference.py --model ./grad_pretrain/gvc.pretrain.pth --spk ./assets/singers/singer0001.npy --wave test.wav\n    ```\n    \n    For this pretrain model, `temperature` is set `temperature=1.015` in `gvc_inference.py` to get good result.\n   \n## Dataset preparation\nPut the dataset into the `data_raw` directory following the structure below.\n```\ndata_raw\n├───speaker0\n│   ├───000001.wav\n│   ├───...\n│   └───000xxx.wav\n└───speaker1\n    ├───000001.wav\n    ├───...\n    └───000xxx.wav\n```\n\n## Data preprocessing\nAfter preprocessing you will get an output with following structure.\n```\ndata_gvc/\n└── waves-16k\n│    └── speaker0\n│    │      ├── 000001.wav\n│    │      └── 000xxx.wav\n│    └── speaker1\n│           ├── 000001.wav\n│           └── 000xxx.wav\n└── waves-32k\n│    └── speaker0\n│    │      ├── 000001.wav\n│    │      └── 000xxx.wav\n│    └── speaker1\n│           ├── 000001.wav\n│           └── 000xxx.wav\n└── mel\n│    └── speaker0\n│    │      ├── 000001.mel.pt\n│    │      └── 000xxx.mel.pt\n│    └── speaker1\n│           ├── 000001.mel.pt\n│           └── 000xxx.mel.pt\n└── pitch\n│    └── speaker0\n│    │      ├── 000001.pit.npy\n│    │      └── 000xxx.pit.npy\n│    └── speaker1\n│           ├── 000001.pit.npy\n│           └── 000xxx.pit.npy\n└── hubert\n│    └── speaker0\n│    │      ├── 000001.vec.npy\n│    │      └── 000xxx.vec.npy\n│    └── speaker1\n│           ├── 000001.vec.npy\n│           └── 000xxx.vec.npy\n└── speaker\n│    └── speaker0\n│    │      ├── 000001.spk.npy\n│    │      └── 000xxx.spk.npy\n│    └── speaker1\n│           ├── 000001.spk.npy\n│           └── 000xxx.spk.npy\n└── singer\n    ├── speaker0.spk.npy\n    └── speaker1.spk.npy\n```\n\n1.  Re-sampling\n    - Generate audio with a sampling rate of 16000Hz in `./data_gvc/waves-16k` \n    ```\n    python prepare/preprocess_a.py -w ./data_raw -o ./data_gvc/waves-16k -s 16000\n    ```\n    - Generate audio with a sampling rate of 32000Hz in `./data_gvc/waves-32k`\n    ```\n    python prepare/preprocess_a.py -w ./data_raw -o ./data_gvc/waves-32k -s 32000\n    ```\n2. Use 16K audio to extract pitch\n    ```\n    python prepare/preprocess_f0.py -w data_gvc/waves-16k/ -p data_gvc/pitch\n    ```\n3. use 32k audio to extract mel\n    ```\n    python prepare/preprocess_spec.py -w data_gvc/waves-32k/ -s data_gvc/mel\n    ``` \n4. Use 16K audio to extract hubert\n    ```\n    python prepare/preprocess_hubert.py -w data_gvc/waves-16k/ -v data_gvc/hubert\n    ```\n5. Use 16k audio to extract timbre code\n    ```\n    python prepare/preprocess_speaker.py data_gvc/waves-16k/ data_gvc/speaker\n    ```\n6. Extract the average value of the timbre code for inference\n    ```\n    python prepare/preprocess_speaker_ave.py data_gvc/speaker/ data_gvc/singer\n    ``` \n8. Use 32k audio to generate training index\n    ```\n    python prepare/preprocess_train.py\n    ```\n9. Training file debugging\n    ```\n    python prepare/preprocess_zzz.py\n    ```\n\n## Train\n1. Start training\n   ```\n   python gvc_trainer.py\n   ``` \n2. Resume training\n   ```\n   python gvc_trainer.py -p logs/grad_svc/grad_svc_***.pth\n   ```\n3. Log visualization\n   ```\n   tensorboard --logdir logs/\n   ```\n\n## Train Loss\n\n![loss_96_v2](./assets/loss_96_v2.jpg)\n\n![grad_svc_mel](./assets/grad_svc_mel.jpg)\n\n\n## Inference\n\n1. Export inference model\n   ```\n   python gvc_export.py --checkpoint_path logs/grad_svc/grad_svc_***.pth\n   ```\n\n2. Inference\n    ```\n    python gvc_inference.py --model gvc.pth --spk ./data_gvc/singer/your_singer.spk.npy --wave test.wav --rature 1.015 --shift 0\n    ```\n    temperature=1.015, needs to be adjusted to get good results; Recommended range is (1.001, 1.035).\n\n2. Inference step by step\n    - Extract hubert content vector\n        ```\n        python hubert/inference.py -w test.wav -v test.vec.npy\n        ```\n    - Extract pitch to the csv text format\n        ```\n        python pitch/inference.py -w test.wav -p test.csv\n        ```\n    - Convert hubert \u0026 pitch to wave\n        ```\n        python gvc_inference.py --model gvc.pth --spk ./data_gvc/singer/your_singer.spk.npy --wave test.wav --vec test.vec.npy --pit test.csv --shift 0\n        ```\n\n## Data\n\n| Name | URL |\n| :--- | :--- |\n|PopCS          |https://github.com/MoonInTheRiver/DiffSinger/blob/master/resources/apply_form.md|\n|opencpop       |https://wenet.org.cn/opencpop/download/|\n|Multi-Singer   |https://github.com/Multi-Singer/Multi-Singer.github.io|\n|M4Singer       |https://github.com/M4Singer/M4Singer/blob/master/apply_form.md|\n|VCTK           |https://datashare.ed.ac.uk/handle/10283/2651|\n\n## Code sources and references\n\nhttps://github.com/huawei-noah/Speech-Backbones/blob/main/Grad-TTS\n\nhttps://github.com/huawei-noah/Speech-Backbones/tree/main/DiffVC\n\nhttps://github.com/facebookresearch/speech-resynthesis\n\nhttps://github.com/cantabile-kwok/VoiceFlow-TTS\n\nhttps://github.com/shivammehta25/Matcha-TTS\n\nhttps://github.com/shivammehta25/Diff-TTSG\n\nhttps://github.com/majidAdibian77/ResGrad\n\nhttps://github.com/LuChengTHU/dpm-solver\n\nhttps://github.com/gmltmd789/UnitSpeech\n\nhttps://github.com/zhenye234/CoMoSpeech\n\nhttps://github.com/seahore/PPG-GradVC\n\nhttps://github.com/thuhcsi/LightGrad\n\nhttps://github.com/lmnt-com/wavegrad\n\nhttps://github.com/naver-ai/facetts\n\nhttps://github.com/jaywalnut310/vits\n\nhttps://github.com/NVIDIA/BigVGAN\n\nhttps://github.com/bshall/soft-vc\n\nhttps://github.com/mozilla/TTS\n\nhttps://github.com/ubisoft/ubisoft-laforge-daft-exprt\n\n##\n\nhttps://github.com/yl4579/StyleTTS-VC\n\nhttps://github.com/MingjieChen/DYGANVC\n\nhttps://github.com/sony/ai-research-code/tree/master/nvcnet\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fplayvoice%2Fgrad-svc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fplayvoice%2Fgrad-svc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fplayvoice%2Fgrad-svc/lists"}