{"id":13958566,"url":"https://github.com/TencentGameMate/chinese_speech_pretrain","last_synced_at":"2025-07-21T00:31:15.171Z","repository":{"id":37369285,"uuid":"496115889","full_name":"TencentGameMate/chinese_speech_pretrain","owner":"TencentGameMate","description":"chinese speech pretrained models","archived":false,"fork":false,"pushed_at":"2024-08-23T03:14:03.000Z","size":4214,"stargazers_count":1037,"open_issues_count":32,"forks_count":87,"subscribers_count":10,"default_branch":"master","last_synced_at":"2024-11-28T02:34:49.164Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/TencentGameMate.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-05-25T06:51:59.000Z","updated_at":"2024-11-27T01:41:13.000Z","dependencies_parsed_at":"2024-08-23T04:27:15.055Z","dependency_job_id":"983f3f6b-e35e-42e5-a569-74af50919769","html_url":"https://github.com/TencentGameMate/chinese_speech_pretrain","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/TencentGameMate/chinese_speech_pretrain","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TencentGameMate%2Fchinese_speech_pretrain","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TencentGameMate%2Fchinese_speech_pretrain/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TencentGameMate%2Fchinese_speech_pretrain/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TencentGameMate%2Fchinese_speech_pretrain/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/TencentGameMate","download_url":"https://codeload.github.com/TencentGameMate/chinese_speech_pretrain/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TencentGameMate%2Fchinese_speech_pretrain/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266221259,"owners_count":23894965,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-08T13:01:44.274Z","updated_at":"2025-07-21T00:31:11.988Z","avatar_url":"https://github.com/TencentGameMate.png","language":"Shell","funding_links":[],"categories":["语音识别与合成_其他"],"sub_categories":["网络服务_其他"],"readme":"# chinese_speech_pretrain\n\n### 简介\n我们使用 WenetSpeech [1] train_l 集的 1 万小时中文数据作为无监督预训练数据。数据主要来源于 YouTube 和 Podcast，覆盖了各种类型录制场景、背景噪声、说话方式等，其领域主要包括有声书、解说、纪录片、电视剧、访谈、新闻、朗读、演讲、综艺和其他等10大场景。我们基于 Fairseq 工具包 [2] 分别训练了 wav2vec 2.0 [3] 和 HuBERT [4] 模型，遵循 [3,4] 中模型配置，每个预训练模型模型包括 BASE 和 LARGE 两种大小。对于 BASE 模型，我们使用 8 张 A100 显卡，梯度累计为 8，模拟 64 张显卡进行训练。对于 LARGE 模型，我们使用 16 张 A100 显卡，梯度累计为 8，模拟 128 张显卡进行训练。\n\n### 模型下载\n为了方便下载，在huggingface模型库里有fairseq模型，如[chinese-wav2vec2-base](https://huggingface.co/TencentGameMate/chinese-wav2vec2-base) 里的chinese-wav2vec2-base-fairseq-ckpt.pt\n\n(We also provide fairseq checkpoint in huggingface model link, e.g chinese-wav2vec2-base-fairseq-ckpt.pt in [chinese-wav2vec2-base](https://huggingface.co/TencentGameMate/chinese-wav2vec2-base) )\n\n| 模型                   | 预训练数据          | fairseq模型下载(百度盘)                                                                    | huggingface \u0026 fairseq模型下载 |\n| ---------------------- | ------------------- | ---------------------------------------------------------------------------------- | ------------------- |\n| chinese-wav2vec2-base  | WenetSpeech train L | [chinese-wav2vec2-base](https://pan.baidu.com/s/1TwlSNDmihs_mjjPpNLhzoA) 提取码: d2hq |  [chinese-wav2vec2-base](https://huggingface.co/TencentGameMate/chinese-wav2vec2-base)  |\n| chinese-wav2vec2-large | WenetSpeech train L | [chinese-wav2vec2-large](https://pan.baidu.com/s/1WbAv3PUqRWmHwwp6GsmLnw) 提取码: 7p8r | [chinese-wav2vec2-large](https://huggingface.co/TencentGameMate/chinese-wav2vec2-large)  |\n| chinese-hubert-base    | WenetSpeech train L | [chinese-hubert-base](https://pan.baidu.com/s/1F3i1u27szmLtBnbMufEv0w) 提取码: xjiy | [chinese-hubert-base](https://huggingface.co/TencentGameMate/chinese-hubert-base)  |\n| chinese-hubert-large   | WenetSpeech train L | [chinese-hubert-large](https://pan.baidu.com/s/1ReagTulgkESGpGJhB5DWRQ) 提取码: hhn7 | [chinese-hubert-large](https://huggingface.co/TencentGameMate/chinese-hubert-large)  |\n\n## 下游任务：中文语音识别\n\n为了验证预训练模型在下游 ASR 任务的效果，我们遵循 ESPnet [5,6,7] 工具包中的 Conformer [8] 模型实验配置，即将预训练模型作为特征提取器，对于输入语音提取预训练模型各隐层表征进行加权求和，得到的语音表征将替换传统 FBank 特征作为 Conformer ASR 模型的输入。\n\n### Aishell 数据集 实验结果\n我们使用 Aishell 178 小时训练集作为有监督数据进行训练，分别对比了使用 FBank 特征、wav2vec 2.0 BASE/LARGE 模型特征和 HuBERT BASE/LARGE 模型特征的字错误率 (Character Error Rate, CER) 结果。同时，我们额外对比了使用 WenetSpeech train_l 集 1 万小时中文数据进行训练时，其在 Aishell 测试集上的效果。训练数据使用了变速（0.9、1.0、1.1 倍）和 SpecAugment 数据增广技术，解码方式为 beam search，使用了基于 Transformer 的语言模型进行 rescoring。具体实验结果见下表：\n\n| 输入特征      | 训练数据 | Dev | Test |\n| ----------------- | -------- | --- | ---- |\n| FBank [6]         | 178h     | 4.4 | 4.7  |\n| FBank [1]         | 1wh      | /   | 3.9  |\n| Wav2vec 2.0 BASE  | 178h     | 4.2 | 4.7  |\n| Wav2vec 2.0 LARGE | 178h     | 3.8 | 4.1  |\n| HuBERT Base       | 178h     | 4.1 | 4.3  |\n| HuBERT LARGE      | 178h     | 3.1 | 3.3  |\n\n### WenetSpeech 实验结果\n\n我们使用 WenetSpeech train_s 100h 数据集作为有监督数据进行训练，分别对比了使用 FBank 特征、wav2vec 2.0 模型特征和 HuBERT 模型特征的字错误率 (Character Error Rate, CER) 结果。同时，额外对比了使用 train_m 集 1000h 和 train_l 集 1wh 中文数据 FBank 特征训练的模型结果。训练数据没有使用变速或 SpecAugment 数据增广技术，解码方式为 beam search，没有使用语言模型 rescoring。具体实验结果见下表：\n\n| 输入特征          | 训练数据 | Dev 集 | Test_Net 集 | Test_Meeting 集 |\n| ----------------- | -------- | ------ | ----------- | --------------- |\n| FBank             | 100h     | 17.4   | 22.6        | 32.7            |\n| FBank             | 1000h    | 11.6   | 14.6        | 22.4            |\n| FBank             | 1wh      | 9.7    | 8.9         | 15.9            |\n| wav2vec 2.0 BASE  | 100h     | 13.1   | 16.1        | 25.5            |\n| wav2vec 2.0 LARGE | 100h     | 11.7   | 13.8        | 25.5            |\n| HuBERT BASE       | 100h     | 12.6   | 14.7        | 21.3            |\n| HuBERT LARGE      | 100h     | 10.0   | 10.2        | 14.5            |\n\n\n### 模型使用\n\n```python\n# This model does not have a tokenizer as it was pretrained on audio alone. \n# In order to use this model speech recognition, a tokenizer should be created and the model should be fine-tuned on labeled text data.\n\n# python package\n# transformers==4.16.2\n\n# fairseq 使用\nimport torch\nimport torch.nn.functional as F\nimport soundfile as sf\nfrom fairseq import checkpoint_utils\n\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n\nmodel_path=\"\"\nwav_path=\"\"\n\ndef postprocess(feats, normalize=False):\n    if feats.dim() == 2:\n        feats = feats.mean(-1)\n\n    assert feats.dim() == 1, feats.dim()\n\n    if normalize:\n        with torch.no_grad():\n            feats = F.layer_norm(feats, feats.shape)\n    return feats\n\nprint(\"loading model(s) from {}\".format(model_path))\nmodels, saved_cfg, task = checkpoint_utils.load_model_ensemble_and_task(\n    [model_path],\n    suffix=\"\",\n)\nprint(\"loaded model(s) from {}\".format(model_path))\nprint(f\"normalize: {saved_cfg.task.normalize}\")\n\n\nmodel = models[0]\nmodel = model.to(device)\nmodel = model.half()\nmodel.eval()\n\nwav, sr = sf.read(wav_path)\nfeat = torch.from_numpy(wav).float()\nfeat = postprocess(feat, normalize=saved_cfg.task.normalize)\nfeats = feat.view(1, -1)\npadding_mask = (\n    torch.BoolTensor(feats.shape).fill_(False)\n)\ninputs = {\n    \"source\": feats.half().to(device),\n    \"padding_mask\": padding_mask.to(device),\n}\n\nwith torch.no_grad():\n    logits = model.extract_features(**inputs)\n\n\n# huggingface 使用\n\nimport torch\nimport torch.nn.functional as F\nimport soundfile as sf\nfrom fairseq import checkpoint_utils\n\nfrom transformers import (\n    Wav2Vec2FeatureExtractor,\n    Wav2Vec2ForPreTraining,\n    Wav2Vec2Model,\n)\nfrom transformers.models.wav2vec2.modeling_wav2vec2 import _compute_mask_indices\n\nmodel_path=\"\"\nwav_path=\"\"\nmask_prob=0.0\nmask_length=10\n\nfeature_extractor = Wav2Vec2FeatureExtractor.from_pretrained(model_path)\nmodel = Wav2Vec2Model.from_pretrained(model_path)\n\n# for pretrain: Wav2Vec2ForPreTraining\n# model = Wav2Vec2ForPreTraining.from_pretrained(model_path)\n\nmodel = model.to(device)\nmodel = model.half()\nmodel.eval()\n\nwav, sr = sf.read(wav_path)\ninput_values = feature_extractor(wav, return_tensors=\"pt\").input_values\ninput_values = input_values.half()\ninput_values = input_values.to(device)\n\n# for Wav2Vec2ForPreTraining\n# batch_size, raw_sequence_length = input_values.shape\n# sequence_length = model._get_feat_extract_output_lengths(raw_sequence_length)\n# mask_time_indices = _compute_mask_indices((batch_size, sequence_length), mask_prob=0.0, mask_length=2)\n# mask_time_indices = torch.tensor(mask_time_indices, device=input_values.device, dtype=torch.long)\n\nwith torch.no_grad():\n    outputs = model(input_values)\n    last_hidden_state = outputs.last_hidden_state\n\n    # for Wav2Vec2ForPreTraining\n    # outputs = model(input_values, mask_time_indices=mask_time_indices, output_hidden_states=True)\n    # last_hidden_state = outputs.hidden_states[-1]\n\n```\n\n欢迎大家使用我们提供的中文语音预训练模型开展研究工作，一起探索语音预训练模型在中文和相关众多场景下的应用。\n\n\u003cp align=\"center\"\u003e\n    \u003cbr\u003e\n    \u003cimg src=\"./figs/tencent.png\" width=\"300\"/\u003e\u003cimg src=\"./figs/zhijiyouxi.png\" width=\"300\"/\u003e\n    \u003cbr\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n    \u003cbr\u003e\n    \u003cimg src=\"./figs/npu@aslp.jpeg\" width=\"500\"/\u003e\u003cimg src=\"./figs/npu_wechat.jpeg\" width=\"200\"/\u003e\n    \u003cbr\u003e\n\u003c/p\u003e\n\n\n### 使用了我们模型的项目\n以下项目使用了我们的模型\n\n| 项目 | 项目地址 |\n| --- | --- |\n|GPT-SoVITS | [GPT-SoVITS](https://github.com/RVC-Boss/GPT-SoVITS) |\n\n### 引用本项目\n```\n@misc{TencentGameMate,\ntitle={chinese_speech_pretrain},\nauthor = {Pengcheng Guo and Shixing Liu},\nyear = {2022},\nurl = {https://github.com/TencentGameMate/chinese_speech_pretrain},\n}\n```\n\n## 参考文献\n\n[1] Binbin Zhang, Hang Lv, Pengcheng Guo, Qijie Shao, Chao Yang, Lei Xie, Xin Xu, Hui Bu, Xiaoyu Chen, Chenhen Zeng, Di Wu, and Zhendong Peng, \"WenetSpeech: A 10000+ hours multi-domain Mandarin corpus for speech recognition,\" in Proc. ICASSP, 2021\n\n[2] Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli, \"fairseq: A fast, extensible toolkit for sequence modeling,\" in Proc. NAACL, 2019.\n\n[3] Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli, \"wav2vec 2.0: A framework for self-supervised learning of speech representations,\" in Proc. NeurIPS, 2020.\n\n[4] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed, \"HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,\" IEEE/ACM Transactions of Audio, Speech, and Language Processing, vol. 29, pp. 3451-3460, 2021\n\n[5] Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson Enrique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, Adithya Renduchintala, and Tsubasa Ochiai, \"ESPnet: End-to-end speech processing toolkit,\" in Proc. Interspeech, 2018, pp. 2207–2211\n\n[6] Pengcheng Guo, Florian Boyer, Xuankai Chang, Tomoki Hayashi, Yosuke Higuchi, Hirofumi Inaguma, Naoyuki Kamo, Chenda Li, Daniel Garcia-Romero, Jiatong Shi, Jing Shi, Shinji Watanabe, Kun Wei, Wangyou Zhang and Yuekai Zhang, \"Recent development on ESPnet tookit boosted by Conformer,\" in Proc. ICASSP, 2021\n\n[7] Xuankai Chang, Takashi Maekaku, Pengcheng Guo, Jing Shi, Yen-Ju Lu, Aswin Shanmugam Subramanian, Tianzi Wang, Shu-wen Yang, Yu Tsao, Hung-yi Lee, and Shinji Watanabe, \"An exploratino of self-supervised pretrained representations for end-to-end speech recognition,\" in Proc. ASRU, 2021\n\n[8] Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pan, \"Conformer: Convolution-augmented Transformer for speech recognition,\" in Proc. Interspeech, 2020, pp.5036–5040\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FTencentGameMate%2Fchinese_speech_pretrain","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FTencentGameMate%2Fchinese_speech_pretrain","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FTencentGameMate%2Fchinese_speech_pretrain/lists"}