{"id":13668544,"url":"https://github.com/manojpamk/pytorch_xvectors","last_synced_at":"2025-04-26T22:31:35.634Z","repository":{"id":52274364,"uuid":"249904636","full_name":"manojpamk/pytorch_xvectors","owner":"manojpamk","description":"Deep speaker embeddings in PyTorch, including x-vectors. Code used in this work: https://arxiv.org/abs/2007.16196","archived":false,"fork":false,"pushed_at":"2020-11-11T15:58:11.000Z","size":365,"stargazers_count":302,"open_issues_count":5,"forks_count":65,"subscribers_count":8,"default_branch":"master","last_synced_at":"2024-08-02T08:06:57.740Z","etag":null,"topics":["speaker-diarization","speaker-embeddings","speaker-recognition","speaker-verification"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/manojpamk.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-03-25T06:34:48.000Z","updated_at":"2024-08-01T15:05:20.000Z","dependencies_parsed_at":"2022-09-11T12:42:38.828Z","dependency_job_id":null,"html_url":"https://github.com/manojpamk/pytorch_xvectors","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manojpamk%2Fpytorch_xvectors","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manojpamk%2Fpytorch_xvectors/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manojpamk%2Fpytorch_xvectors/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manojpamk%2Fpytorch_xvectors/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/manojpamk","download_url":"https://codeload.github.com/manojpamk/pytorch_xvectors/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224048753,"owners_count":17247080,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["speaker-diarization","speaker-embeddings","speaker-recognition","speaker-verification"],"created_at":"2024-08-02T08:00:40.530Z","updated_at":"2024-11-11T04:31:17.025Z","avatar_url":"https://github.com/manojpamk.png","language":"Python","funding_links":[],"categories":["Software","Code/Tools/Frameworks/Libraries","2. Method Table"],"sub_categories":["Speaker embedding","2.2. Metric"],"readme":"## \u003cdiv align=\"center\"\u003eDeep speaker embeddings in PyTorch\u003c/div\u003e\n\n * [Requirements:](#requirements)\n       * [Other Tools:](#other-tools)\n * [Installation:](#installation)\n * [Data preparation](#data-preparation)\n    * [Training data preparation](#training-data-preparation)\n    * [Dataset for data augmentation](#dataset-for-data-augmentation)\n * [Training](#training)\n * [Embedding extraction](#embedding-extraction)\n * [Pretrained model](#pretrained-model)\n    * [Downloading](#downloading)\n    * [Speaker Verification](#speaker-verification)\n    * [Speaker Diarization](#speaker-diarization)\n * [Results](#results)\n    * [1. Speaker Verification (%R)](#1-speaker-verification-eer)\n    * [2. Speaker Diarization (%R)](#2-speaker-diarization-der)\n\n\n\nThis repository contains code and models for training an x-vector speaker recognition model using Kaldi for feature preparation and PyTorch for DNN model training. MFCC feature configurations and TDNN model architecture follow the Voxceleb recipe in Kaldi (commit hash `9b4dc93c9`). Training procedures including optimizer and step count are similar to, but not exactly the same as Kaldi.\n\nAdditionally, code for training meta-learning embeddings are available in [train_proto.py](train_proto.py) and [train_relation.py](train_relation.py). An overview of these models is available at [https://arxiv.org/abs/2007.16196](https://arxiv.org/abs/2007.16196) and in the below figure:\n\n![Overview: Meta Learning Models](figs/meta_learning_arch.png)\n\n\n### Citation\n\nIf you found this toolkit useful in your research, consider citing the following:\n\n```\n@misc{kumar2020designing,\n    title={Designing Neural Speaker Embeddings with Meta Learning},\n    author={Manoj Kumar and Tae Jin-Park and Somer Bishop and Catherine Lord and Shrikanth Narayanan},\n    year={2020},    \n    eprint={2007.16196},\n    archivePrefix={arXiv}  \n}\n```\n\n### Requirements:\nPython Libraries\n```\npython==3.6.10\ntorch==1.4.0\nkaldiio==2.15.1\nkaldi-python-io==1.0.4\n```\n\n##### Other Tools:\n\n* Spectral Clustering using normalized maximum eigengap [GitHub](https://github.com/tango4j/Auto-Tuning-Spectral-Clustering)\n  * Used for speaker clustering during diarization\n* Diarization scoring tool [GitHub](https://github.com/nryant/dscore)\n  * Used for computing diarization error rate (DER)\n\n\n\n### Installation:\n\n* Install the python libraries listed in [Requirements](#requirements)\n* Install [Kaldi toolkit](https://github.com/kaldi-asr/kaldi/blob/master/INSTALL).\n  * This repository is tested with commit hash `9b4dc93c9` of the above [Kaldi repository](https://github.com/kaldi-asr/kaldi/blob/master/INSTALL).\n  * Kaldi is recommended to be installed in `$HOME/kaldi`.\n* Download this repository. NOTE: Destination need not be inside Kaldi installation.\n* Set the `voxcelebDir` variable inside [pytorch_run.sh](pytorch_run.sh)\n* (Optional) Install Other Tools listering in [Requirements](#requirements)\n\n### Data preparation\n\n#### Training data preparation\n\n* Training features are expected in Kaldi nnet3 egs format, and read using the `nnet3EgsDL` class defined in [train_utils.py](train_utils.py).\n* The voxceleb recipe is provided in [pytorch_run.sh](pytorch_run.sh) to prepare them.\n* Extracted embeddings are written in Kaldi vector format, similar to `xvector.ark`.\n\n#### Dataset for data augmentation\n\n[pytorch_run.sh](pytorch_run.sh) script augments the training data using the following two datasets.\n* Download [MUSAN](https://openslr.org/17/) and extract to ./musan.\n* Download [RIRS_NOISES](https://openslr.org/28/) and extract to ./RIRS_NOISES.\n\n\n### Training\n```\nCUDA_VISIBLE_DEVICES=0 python -m torch.distributed.launch --nproc_per_node=1 train_xent.py \u003cegsDir\u003e\n```\n```\nusage: train_xent.py [-h] [--local_rank LOCAL_RANK] [-modelType MODELTYPE]\n                     [-featDim FEATDIM] [-resumeTraining RESUMETRAINING]\n                     [-resumeModelDir RESUMEMODELDIR]\n                     [-numArchives NUMARCHIVES] [-numSpkrs NUMSPKRS]\n                     [-logStepSize LOGSTEPSIZE] [-batchSize BATCHSIZE]\n                     [-numEgsPerArk NUMEGSPERARK]\n                     [-preFetchRatio PREFETCHRATIO]\n                     [-optimMomentum OPTIMMOMENTUM] [-baseLR BASELR]\n                     [-maxLR MAXLR] [-numEpochs NUMEPOCHS]\n                     [-noiseEps NOISEEPS] [-pDropMax PDROPMAX]\n                     [-stepFrac STEPFRAC]\n                     egsDir\n\npositional arguments:\n  egsDir                Directory with training archives\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --local_rank LOCAL_RANK\n  -modelType MODELTYPE  Refer train_utils.py\n  -featDim FEATDIM      Frame-level feature dimension\n  -resumeTraining RESUMETRAINING\n                        (1) Resume training, or (0) Train from scratch\n  -resumeModelDir RESUMEMODELDIR\n                        Path containing training checkpoints\n  -numArchives NUMARCHIVES\n                        Number of egs.*.ark files\n  -numSpkrs NUMSPKRS    Number of output labels\n  -logStepSize LOGSTEPSIZE\n                        Iterations per log\n  -batchSize BATCHSIZE  Batch size\n  -numEgsPerArk NUMEGSPERARK\n                        Number of training examples per egs file\n  -preFetchRatio PREFETCHRATIO\n                        xbatchSize to fetch from dataloader\n  -optimMomentum OPTIMMOMENTUM\n                        Optimizer momentum\n  -baseLR BASELR        Initial LR\n  -maxLR MAXLR          Maximum LR\n  -numEpochs NUMEPOCHS  Number of training epochs\n  -noiseEps NOISEEPS    Noise strength before pooling\n  -pDropMax PDROPMAX    Maximum dropout probability\n  -stepFrac STEPFRAC    Training iteration when dropout = pDropMax\n\n```\n`egsDir` contains the nnet3 egs files.\n\n### Embedding extraction\n```\nusage: extract.py [-h] [-modelType MODELTYPE] [-numSpkrs NUMSPKRS]\n                  modelDirectory featDir embeddingDir\n\npositional arguments:\n  modelDirectory        Directory containing the model checkpoints\n  featDir               Directory containing features ready for extraction\n  embeddingDir          Output directory\n\noptional arguments:\n  -h, --help            show this help message and exit\n  -modelType MODELTYPE  Refer train_utils.py\n  -numSpkrs NUMSPKRS    Number of output labels for model\n```\nThe script [pytorch_run.sh](pytorch_run.sh) can be used to train embeddings on the voxceleb recipe on an end-to-end basis.\n\n### Pretrained model\n\n#### Downloading\nTwo ways to download the pre-trained model:\n1. Google Drive [link](https://drive.google.com/file/d/1gbAWDdWN_pkOim4rWVXUlfuYjfyJqUHZ/view?usp=sharing) *(or)*\n2. Command line ([reference](https://medium.com/@acpanjan/download-google-drive-files-using-wget-3c2c025a8b99))\n    ```\n    wget --load-cookies /tmp/cookies.txt \"https://docs.google.com/uc?export=download\u0026confirm=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download\u0026id=1gbAWDdWN_pkOim4rWVXUlfuYjfyJqUHZ' -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\\1\\n/p')\u0026id=1gbAWDdWN_pkOim4rWVXUlfuYjfyJqUHZ\" -O preTrainedModel.zip \u0026\u0026 rm -rf /tmp/cookies.txt\n    ```\n\n#### Speaker Verification\nTo reproduce voxceleb EER results with the pretrained model, follow the below steps.\nNOTE: The voxceleb features must be prepared using `prepare_feats_for_egs.sh` prior to evaluation.\n\n1) Extract `models/` and `xvectors/` from the pre-trained archive into the installation directory\n2) Set the following variables in [pytorch_run.sh](pytorch_run.sh):\n    ```\n    modelDir=models/xvec_preTrained\n    trainFeatDir=data/train_combined_no_sil\n    trainXvecDir=xvectors/xvec_preTrained/train\n    testFeatDir=data/voxceleb1_test_no_sil\n    testXvecDir=xvectors/xvec_preTrained/test\n    ```\n3) Extract embeddings and compute EER, minDCF. Set `stage=7` in [pytorch_run.sh](pytorch_run.sh) and execute:\n   ```\n   bash pytorch_run.sh\n   ```\n4) Alternatively, pretrained PLDA model is available inside `xvectors/train` directory. Set `stage=9` in [pytorch_run.sh](pytorch_run.sh) and execute:\n   ```\n   bash pytorch_run.sh\n   ```\n#### Speaker Diarization\n\n```\ncd egs/\n```\nPlace the audio files to diarize and their corresponding RTTM files in `demo_wav/` and `demo_rttm/` directories. Execute:\n```\nbash diarize.sh\n```\n\n### Results\n\n#### 1. Speaker Verification (%EER)\n\n|         | Kaldi           | pytorch_xvectors  |\n|:-------------|:-------------:|:-----:|\n| Vox1-test      | 3.13 | 2.82 |\n| VOICES-dev      | 10.30 | 8.59 |\n\n\n#### 2. Speaker Diarization (%DER)\n\nNOTE: Clustering using [https://github.com/tango4j/Auto-Tuning-Spectral-Clustering](https://github.com/tango4j/Auto-Tuning-Spectral-Clustering)\n\n|         | Kaldi           | pytorch_xvectors  |\n|:-------------|:-------------:|:-----:|\n| DIHARD2 dev (no collar, oracle #spk)      | 26.97 | 27.50 |\n| DIHARD2 dev (no collar, est #spk)      | 24.49 | 24.66 |\n| AMI dev+test (26 meetings, collar, oracle #spk) | 6.39 | 6.30 |\n| AMI dev+test (26 meetings, collar, est #spk) | 7.29 | 10.14 |\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmanojpamk%2Fpytorch_xvectors","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmanojpamk%2Fpytorch_xvectors","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmanojpamk%2Fpytorch_xvectors/lists"}