{"id":13785552,"url":"https://github.com/philipperemy/deep-speaker","last_synced_at":"2025-05-16T08:05:53.589Z","repository":{"id":43539928,"uuid":"91767564","full_name":"philipperemy/deep-speaker","owner":"philipperemy","description":"Deep Speaker: an End-to-End Neural Speaker Embedding System.","archived":false,"fork":false,"pushed_at":"2024-04-13T15:09:03.000Z","size":83477,"stargazers_count":923,"open_issues_count":9,"forks_count":241,"subscribers_count":47,"default_branch":"master","last_synced_at":"2025-04-08T20:17:18.117Z","etag":null,"topics":["deep-learning","deep-speaker","keras","tensorflow"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/philipperemy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["philipperemy"]}},"created_at":"2017-05-19T05:06:32.000Z","updated_at":"2025-04-07T07:28:44.000Z","dependencies_parsed_at":"2024-08-03T19:15:02.116Z","dependency_job_id":null,"html_url":"https://github.com/philipperemy/deep-speaker","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/philipperemy%2Fdeep-speaker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/philipperemy%2Fdeep-speaker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/philipperemy%2Fdeep-speaker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/philipperemy%2Fdeep-speaker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/philipperemy","download_url":"https://codeload.github.com/philipperemy/deep-speaker/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254493378,"owners_count":22080126,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","deep-speaker","keras","tensorflow"],"created_at":"2024-08-03T19:01:01.734Z","updated_at":"2025-05-16T08:05:48.580Z","avatar_url":"https://github.com/philipperemy.png","language":"Python","funding_links":["https://github.com/sponsors/philipperemy"],"categories":["Software","Table of Contents","Python"],"sub_categories":["Speaker embedding","Pretrained models/embeddings"],"readme":"## Deep Speaker: An End-to-End Neural Speaker Embedding System.\nUnofficial Tensorflow/Keras implementation of Deep Speaker | [Paper](https://arxiv.org/pdf/1705.02304.pdf) | [Pretrained Models](https://drive.google.com/open?id=18h2bmsAWrqoUMsh_FQHDDxp7ioGpcNBa).\n\nTested with Tensorflow 2.3, 2.4, 2.5 and 2.6.\n\n### Sample Results\n\nModels were trained on clean speech data. Keep in mind that the performance will be lower on noisy data. It is advised to remove silence and background noise before computing the embeddings (by using Sox for example). There is a discussion on the topic: [Silence / Background Noise similarity](https://github.com/philipperemy/deep-speaker/issues/62).\n\n *Model name* | *Testing dataset* | *Num speakers* | *F* | *TPR* | *ACC* | *EER* | Training Logs | Download model\n | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |\nResCNN Softmax trained          | [LibriSpeech](http://www.openslr.org/12/) all(*) | 2484 | 0.789 | 0.733 | 0.996 | 0.043 | [Click](https://docs.google.com/document/d/1ZZjBk5TgFgaY9GgOcHaieOpiyB9lB6oTRSdk6g8FPRs) | [Click](https://drive.google.com/open?id=1SJBmHpnaW1VcbFWP6JfvbT3wWP9PsqxS)\nResCNN Softmax+Triplet trained  | [LibriSpeech](http://www.openslr.org/12/) all(*) | 2484 | 0.843 | 0.825 | 0.997 | 0.025 | [Click](https://docs.google.com/document/d/1mL0Jb8IpA7DOzFci71RT1OYTq7Kkw2DjTkI4BRpEzKc) | [Click](https://drive.google.com/file/d/1F9NvdrarWZNktdX9KlRYWWHDwRkip_aP)\n\n(*) all includes: dev-clean, dev-other, test-clean, test-other, train-clean-100, train-clean-360, train-other-500.\n\nThe Softmax+Triplet checkpoint is also available on the [Chinese cloud - WeiYun](https://share.weiyun.com/V2suEUVh).\n\n### Overview\n\nDeep Speaker is a neural speaker embedding system that maps utterances to a hypersphere where speaker similarity is measured by cosine similarity. The embeddings generated by Deep Speaker can be used for many tasks, including speaker identification,\nverification, and clustering.\n\n## Getting started\n### Install dependencies\n#### Requirements\n- tensorflow\u003e=2.0\n- keras\u003e=2.3.1\n- python\u003e=3.6\n```bash\npip install -r requirements.txt\n```\n\nIf you see this error: `libsndfile not found`, run this: `sudo apt-get install libsndfile-dev`.\n\n### Training\n\nThe code for training is available in this repository. It takes a bit less than a week with a GTX1070 to train the models.\n\nSystem requirements for a complete training are:\n- At least 300GB of free disk space on a fast SSD (250GB just for all the uncompressed + processed data)\n- 32GB of memory and at least 32GB of swap (can create swap with SSD space).\n- A NVIDIA GPU such as the 1080Ti.\n\n```bash\npip uninstall -y tensorflow \u0026\u0026 pip install tensorflow-gpu\n./deep-speaker download_librispeech    # if the download is too slow, consider replacing [wget] by [axel -n 10 -a] in download_librispeech.sh.\n./deep-speaker build_mfcc              # will build MFCC for softmax pre-training and triplet training.\n./deep-speaker build_model_inputs      # will build inputs for softmax pre-training.\n./deep-speaker train_softmax           # takes ~3 days.\n./deep-speaker train_triplet           # takes ~3 days.\n```\n\nNOTE: If you want to use your own dataset, make sure you follow the directory structure of librispeech. Audio files have to be in `.flac`. format. If you have `.wav`, you can use `ffmpeg` to make the conversion. Both formats are flawless (FLAC is compressed WAV).\n\n### Test instruction using pretrained model\n- Download the trained models\n \n\n *Model name* | *Used datasets for training* | *Num speakers* | *Model Link* | \n | :--- | :--- | :--- | :--- |\nResCNN Softmax trained  | [LibriSpeech](http://www.openslr.org/12/) train-clean-360 | 921 | [Click](https://drive.google.com/open?id=1SJBmHpnaW1VcbFWP6JfvbT3wWP9PsqxS)\nResCNN Softmax+Triplet trained  | [LibriSpeech](http://www.openslr.org/12/) all | 2484 | [Click](https://drive.google.com/file/d/1F9NvdrarWZNktdX9KlRYWWHDwRkip_aP)\n\nNote: the pre-training was performed on a subset of all the speakers we have. This is to match the philosophy of the paper where they first trained the model with softmax and then trained it on the whole dataset (bigger than this repo!) with triplets.\n\n* Run with pretrained model\n\n```python\nimport random\n\nimport numpy as np\n\nfrom deep_speaker.audio import read_mfcc\nfrom deep_speaker.batcher import sample_from_mfcc\nfrom deep_speaker.constants import SAMPLE_RATE, NUM_FRAMES\nfrom deep_speaker.conv_models import DeepSpeakerModel\nfrom deep_speaker.test import batch_cosine_similarity\n\n# Reproducible results.\nnp.random.seed(123)\nrandom.seed(123)\n\n# Define the model here.\nmodel = DeepSpeakerModel()\n\n# Load the checkpoint. https://drive.google.com/file/d/1F9NvdrarWZNktdX9KlRYWWHDwRkip_aP.\n# Also available here: https://share.weiyun.com/V2suEUVh (Chinese users).\nmodel.m.load_weights('ResCNN_triplet_training_checkpoint_265.h5', by_name=True)\n\n# Sample some inputs for WAV/FLAC files for the same speaker.\n# To have reproducible results every time you call this function, set the seed every time before calling it.\n# np.random.seed(123)\n# random.seed(123)\nmfcc_001 = sample_from_mfcc(read_mfcc('samples/PhilippeRemy/PhilippeRemy_001.wav', SAMPLE_RATE), NUM_FRAMES)\nmfcc_002 = sample_from_mfcc(read_mfcc('samples/PhilippeRemy/PhilippeRemy_002.wav', SAMPLE_RATE), NUM_FRAMES)\n\n# Call the model to get the embeddings of shape (1, 512) for each file.\npredict_001 = model.m.predict(np.expand_dims(mfcc_001, axis=0))\npredict_002 = model.m.predict(np.expand_dims(mfcc_002, axis=0))\n\n# Do it again with a different speaker.\nmfcc_003 = sample_from_mfcc(read_mfcc('samples/1255-90413-0001.flac', SAMPLE_RATE), NUM_FRAMES)\npredict_003 = model.m.predict(np.expand_dims(mfcc_003, axis=0))\n\n# Compute the cosine similarity and check that it is higher for the same speaker.\nprint('SAME SPEAKER', batch_cosine_similarity(predict_001, predict_002)) # SAME SPEAKER [0.81564593]\nprint('DIFF SPEAKER', batch_cosine_similarity(predict_001, predict_003)) # DIFF SPEAKER [0.1419204]\n```\n\n* Commands to reproduce the test results after the training\n\n**NOTE**: For some reasons, the `test-model` does not work with tensorflow\u003e2.3. Make sure to run `pip install tensorflow==2.3` if you want to run those two commands below.\n\n```bash\n$ export CUDA_VISIBLE_DEVICES=0; python cli.py test-model --working_dir ~/.deep-speaker-wd/triplet-training/ --\ncheckpoint_file checkpoints-softmax/ResCNN_checkpoint_102.h5\nf-measure = 0.789, true positive rate = 0.733, accuracy = 0.996, equal error rate = 0.043\n```\n\n```bash\n$ export CUDA_VISIBLE_DEVICES=0; python cli.py test-model --working_dir ~/.deep-speaker-wd/triplet-training/ --checkpoint_file checkpoints-triplets/ResCNN_checkpoint_265.h5\nf-measure = 0.849, true positive rate = 0.798, accuracy = 0.997, equal error rate = 0.025\n```\n\nWhen the triplet loss select the hard examples, then the training loss does not really decrease. Because the hard samples are always hard meaning they are on average above alpha. The test set should however decreased.\n\n### Further work\n\n- LSTM model: https://github.com/philipperemy/deep-speaker/pull/53\n- Fusion score: https://github.com/philipperemy/deep-speaker/pull/76\n\n## Contributors\n\n\u003ca href=\"https://github.com/philipperemy/deep-speaker/graphs/contributors\"\u003e\n  \u003cimg src=\"https://contrib.rocks/image?repo=philipperemy/deep-speaker\" /\u003e\n\u003c/a\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fphilipperemy%2Fdeep-speaker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fphilipperemy%2Fdeep-speaker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fphilipperemy%2Fdeep-speaker/lists"}