{"id":13935497,"url":"https://github.com/jefflai108/pytorch-kaldi-neural-speaker-embeddings","last_synced_at":"2025-07-19T20:32:54.819Z","repository":{"id":63732851,"uuid":"220392667","full_name":"jefflai108/pytorch-kaldi-neural-speaker-embeddings","owner":"jefflai108","description":"A light weight neural speaker embeddings extraction based on Kaldi and PyTorch.","archived":false,"fork":false,"pushed_at":"2020-01-27T22:29:25.000Z","size":9806,"stargazers_count":135,"open_issues_count":5,"forks_count":34,"subscribers_count":8,"default_branch":"master","last_synced_at":"2024-08-08T23:21:02.747Z","etag":null,"topics":["kaldi","learnable-dictionary-encoding","pytorch","speaker-identification","speaker-recognition","speaker-verification","speech-processing"],"latest_commit_sha":null,"homepage":"","language":"Perl","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jefflai108.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-11-08T05:23:07.000Z","updated_at":"2024-07-29T06:47:08.000Z","dependencies_parsed_at":"2022-11-24T21:11:38.293Z","dependency_job_id":null,"html_url":"https://github.com/jefflai108/pytorch-kaldi-neural-speaker-embeddings","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jefflai108%2Fpytorch-kaldi-neural-speaker-embeddings","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jefflai108%2Fpytorch-kaldi-neural-speaker-embeddings/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jefflai108%2Fpytorch-kaldi-neural-speaker-embeddings/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jefflai108%2Fpytorch-kaldi-neural-speaker-embeddings/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jefflai108","download_url":"https://codeload.github.com/jefflai108/pytorch-kaldi-neural-speaker-embeddings/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":226677027,"owners_count":17665999,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kaldi","learnable-dictionary-encoding","pytorch","speaker-identification","speaker-recognition","speaker-verification","speech-processing"],"created_at":"2024-08-07T23:01:48.945Z","updated_at":"2024-11-27T03:30:42.091Z","avatar_url":"https://github.com/jefflai108.png","language":"Perl","funding_links":[],"categories":["Perl"],"sub_categories":[],"readme":"# pytorch-kaldi-neural-speaker-embeddings\n**A light weight neural speaker embeddings extraction based on Kaldi and PyTorch.** \\\nThe repository serves as a starting point for users to reproduce and experiment several recent advances in speaker recognition literature. \nKaldi is used for pre-processing and post-processing and PyTorch is used for training the neural speaker embeddings.\n**I want to note that this repo is not meant for keeping track of state-of-the-art on speaker recognition, and most likely the models will be considered outdated in a few months (or sooner :().** \n\nThis repository contains a PyTorch+Kaldi pipeline to reproduce the core results for: \n* [Exploring the Encoding Layer and Loss Function in End-to-End Speaker and Language Recognition System](https://arxiv.org/pdf/1804.05160.pdf)\n* [A Novel Learnable Dictionary Encoding Layer for End-to-End Language Identification](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=\u0026arnumber=8462025)\n\nWith some modifications, you can easily adapt the pipeline for:\n* [Angular softmax for short-duration text-independent speaker verification](https://pdfs.semanticscholar.org/93f9/24fa3d9920edc10469b04c90960c03dce856.pdf)\n* [Utterance-level aggregation for speaker recognition in the wild](https://arxiv.org/pdf/1902.10107.pdf)\n* [Margin Matters: Towards More Discriminative Deep Neural Network Embeddings for Speaker Recognition](https://arxiv.org/pdf/1906.07317.pdf)\n* [Spatial Pyramid Encoding with Convex Length Normalization for Text-Independent Speaker Verification](https://arxiv.org/pdf/1906.08333.pdf)\n\nIf one wants to go further, take a look at our recent work on [multi-speaker text-to-speech](https://arxiv.org/pdf/1910.10838.pdf), where the same speaker embeddings are employed to model speaker characterisitcs in a text-to-speech system. \n\nLastly, kindly cite our paper(s) if you find this repository useful. Cite both if you are kind enough!\n```\n@article{villalba2019state,\n  title={State-of-the-art speaker recognition with neural network embeddings in nist sre18 and speakers in the wild evaluations},\n  author={Villalba, Jes{\\'u}s and Chen, Nanxin and Snyder, David and Garcia-Romero, Daniel and McCree, Alan and Sell, Gregory and Borgstrom, Jonas and Garc{\\'\\i}a-Perera, Leibny Paola and Richardson, Fred and Dehak, R{\\'e}da and others},\n  journal={Computer Speech \\\u0026 Language},\n  pages={101026},\n  year={2019},\n  publisher={Elsevier}\n}\n```\n```\n@article{cooper2019zero,\n  title={Zero-Shot Multi-Speaker Text-To-Speech with State-of-the-art Neural Speaker Embeddings},\n  author={Cooper, Erica and Lai, Cheng-I and Yasuda, Yusuke and Fang, Fuming and Wang, Xin and Chen, Nanxin and Yamagishi, Junichi},\n  journal={arXiv preprint arXiv:1910.10838},\n  year={2019}\n}\n```\n\nOne should also check out the very nicely written [TensorFlow version](https://github.com/mycrazycracy/tf-kaldi-speaker) by [Yi Lu](http://yiliu.org.cn).\n\n# Overview\nNeural speaker embeddings: Encoder --\u003e Pooling --\u003e Classification \\\nLDE pooling method illustration:\n\u003cp align=\"center\"\u003e\n \u003cimg src=\"imgs/LDE-6.png\" width=\"50%\"\u003e\n\u003c/p\u003e\n\n# Requirements\n`pip install -r requirements.txt`\nPlease also [download and properly setup Kaldi](https://kaldi-asr.org/doc/install.html). If you are stuck in this phase, this repository is liekly not for you. \n\n# Getting Started \nThe bash file `pipeline.sh` contains the 12-stage speaker recognition pipeline, including feature extraction, the neural model training and decoding/evaluation. A more detailed description of each step is described in `pipeline.sh`. To get started, simply run: \n`./pipeline.sh`\n\n# Datasets \nThe models are trained on [VoxCeleb I+II](http://www.robots.ox.ac.uk/~vgg/data/voxceleb/), which is free for downloads (the trial lists are also there). One can easily adapt `pipeline.sh` for different datasets.\n\n# Pre-Trained Models \nDue to Youtube's privacy policy, unfortunately I am not allowed to upload pre-trained models for VoxCeleb I+II. \n\n# Benchmarking Speaker Verification EERs\n\n|     Embedding name       |  dimension | normalization|   pooling type |  train objective   |    EER      |  DCF\u003csup\u003emin\u003c/sup\u003e\u003csub\u003e0.01\u003c/sub\u003e |\n| :-----------------------:| :--------: | :----------: | :------------: | :----------------: | :---------: |  :---------: |\n|    i-vectors             |      400   |   no         |     mean       |      EM            |    5.329    |  0.493       |\n|    x-vectors             |      512   |   no         |     mean, std  |      Softmax       |    3.298    |  0.343       |\n|    x-vectors\u003csup\u003eN\u003c/sup\u003e |      512   |   yes        |     mean, std  |      Softmax       |    3.213    |  0.342       |\n|    LDE-1                 |      512   |   no         |     mean       |      Softmax       |    3.415    |  0.366       |\n|    LDE-1\u003csup\u003eN\u003c/sup\u003e     |      512   |   yes        |     mean       |      Softmax       |    3.446    |  0.365       |\n|    LDE-2                 |      512   |   no         |     mean       |      ASoftmax (m=2)|    3.674    |  0.364       |\n|    LDE-2\u003csup\u003eN\u003c/sup\u003e     |      512   |   yes        |     mean       |      ASoftmax (m=2)|    3.664    |  0.386       |\n|    LDE-3                 |      512   |   no         |     mean       |      ASoftmax (m=3)|  **3.033**  |**0.314**     |\n|    LDE-3\u003csup\u003eN\u003c/sup\u003e     |      512   |   yes        |     mean       |      ASoftmax (m=3)|    3.171    |  0.327       |\n|    LDE-4                 |      512   |   no         |     mean       |      ASoftmax (m=4)|    3.112    |  0.315       |\n|    LDE-4\u003csup\u003eN\u003c/sup\u003e     |      512   |   yes        |     mean       |      ASoftmax (m=4)|    3.271    |  0.327       |\n|    LDE-5                 |      256   |   no         |     mean       |      ASoftmax (m=2)|    3.287    |  0.343       |\n|    LDE-5\u003csup\u003eN\u003c/sup\u003e     |      256   |   yes        |     mean       |      ASoftmax (m=2)|    3.367    |  0.351       |\n|    LDE-6                 |      200   |   no         |     mean       |      ASoftmax (m=2)|    3.266    |  0.396       |\n|    LDE-6\u003csup\u003eN\u003c/sup\u003e     |      200   |   yes        |     mean       |      ASoftmax (m=2)|    3.266    |  0.396       |\n|    LDE-7                 |      512   |   no         |     mean, std  |      ASoftmax (m=2)|  **3.091**  |**0.303**     |\n|    LDE-7\u003csup\u003eN\u003c/sup\u003e     |      512   |   yes        |     mean, std  |      ASoftmax (m=2)|    3.171    |  0.328       |\n\n# Using Speaker Embeddings for Tacotron2 Speaker Adaptation\n\u003cp align=\"center\"\u003e\n \u003cimg src=\"imgs/TTS-4.png\" width=\"60%\"\u003e\n\u003c/p\u003e\n\n# Speaker Embedding Space Visualization (cluster by speakers)\n\n**i-vectors (baseline)**\n\u003cp align=\"center\"\u003e\n \u003cimg src=\"imgs/sid/ivector-vctk-plot/ivector_43873_tsne_2d_plot.png\" width=\"100%\"\u003e\n\u003c/p\u003e\n\n**LDE** \n\u003cp align=\"center\"\u003e\n \u003cimg src=\"imgs/sid/lde-vctk-plot/resnet_mfcc_3-8_512_32_mean_lde_sqr_asoftmax_m3_43873_tsne_2d_plot.png\" width=\"100%\"\u003e\n\u003c/p\u003e\n\n# Benchmarking TTS MOS scores\n\n|     Embedding name       | Naturalness dev | Naturalness test | Similarity dev | Similarity test |\n| :-----------------------:| :-------------: | :--------------: | :------------: | :-------------: |\n|    vocoded               |      3.41       |       3.55       |      2.79      |       2.82      |\n|    x-vectors\u003csup\u003eN\u003c/sup\u003e |      3.19       |       3.19       |      1.86      |       2.37      | \n|    LDE-1                 |      3.16       |       3.21       |    **2.05**    |       2.34      |\n|    LDE-1\u003csup\u003eN\u003c/sup\u003e     |      3.13       |       3.46       |      1.97      |       2.45      |\n|    LDE-2                 |      3.28       |       3.35       |      2.00      |       2.37      |\n|    LDE-2\u003csup\u003eN\u003c/sup\u003e     |      3.19       |       3.33       |      2.00      |       2.35      |\n|    LDE-3                 |      3.24       |     **3.48**     |      1.88      |     **2.46**    |\n|    LDE-3\u003csup\u003eN\u003c/sup\u003e     |      3.16       |       3.33       |      2.00      |       2.37      |\n|    LDE-4                 |      3.10       |       3.29       |      2.00      |       2.31      |\n|    LDE-4\u003csup\u003eN\u003c/sup\u003e     |      3.20       |       3.29       |      1.98      |       2.39      |\n|    LDE-5                 |      3.26       |       3.40       |      1.99      |       2.45      |\n|    LDE-5\u003csup\u003eN\u003c/sup\u003e     |      3.07       |       3.37       |      2.02      |       2.41      |\n|    LDE-6                 |      3.25       |       3.33       |      1.95      |       2.43      |\n|    LDE-6\u003csup\u003eN\u003c/sup\u003e     |    **3.29**     |       3.23       |      1.94      |       2.39      |\n|    LDE-7                 |      3.03       |       3.18       |      1.86      |       2.28      |\n|    LDE-7\u003csup\u003eN\u003c/sup\u003e     |      3.02       |       3.24       |      2.02      |       2.42      |\n\n# Credits\nBase code written by [Nanxin Chen](https://github.com/bobchennan), Johns Hopkins University \\\nExperiments done by [Cheng-I Lai](http://people.csail.mit.edu/clai24/), MIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjefflai108%2Fpytorch-kaldi-neural-speaker-embeddings","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjefflai108%2Fpytorch-kaldi-neural-speaker-embeddings","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjefflai108%2Fpytorch-kaldi-neural-speaker-embeddings/lists"}