{"id":20021745,"url":"https://github.com/bshall/acoustic-model","last_synced_at":"2025-08-26T23:38:24.393Z","repository":{"id":39604279,"uuid":"418888468","full_name":"bshall/acoustic-model","owner":"bshall","description":"Acoustic models for: A Comparison of Discrete and Soft Speech Units for Improved Voice Conversion","archived":false,"fork":false,"pushed_at":"2023-07-12T20:48:09.000Z","size":168,"stargazers_count":103,"open_issues_count":6,"forks_count":25,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-06-18T05:39:52.196Z","etag":null,"topics":["pytorch","representation-learning","speech","voice-conversion"],"latest_commit_sha":null,"homepage":"https://bshall.github.io/soft-vc/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bshall.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-10-19T11:09:48.000Z","updated_at":"2025-04-11T04:37:18.000Z","dependencies_parsed_at":"2024-11-13T08:38:10.862Z","dependency_job_id":"423418a3-7846-4c52-a8a0-e95191142e7f","html_url":"https://github.com/bshall/acoustic-model","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/bshall/acoustic-model","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bshall%2Facoustic-model","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bshall%2Facoustic-model/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bshall%2Facoustic-model/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bshall%2Facoustic-model/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bshall","download_url":"https://codeload.github.com/bshall/acoustic-model/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bshall%2Facoustic-model/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272267793,"owners_count":24903779,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-26T02:00:07.904Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["pytorch","representation-learning","speech","voice-conversion"],"created_at":"2024-11-13T08:38:05.804Z","updated_at":"2025-08-26T23:38:24.368Z","avatar_url":"https://github.com/bshall.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n    \u003ca target=\"_blank\" href=\"https://colab.research.google.com/github/bshall/soft-vc/blob/main/soft-vc-demo.ipynb\"\u003e\n        \u003cimg src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/\u003e\n    \u003c/a\u003e\n\u003c/p\u003e\n\n# Acoustic-Model\n\nTraining and inference scripts for the acoustic models in [A Comparison of Discrete and Soft Speech Units for Improved Voice Conversion](https://ieeexplore.ieee.org/abstract/document/9746484). For more details see [soft-vc](https://github.com/bshall/soft-vc). Audio samples can be found [here](https://bshall.github.io/soft-vc/). Colab demo can be found [here](https://colab.research.google.com/github/bshall/soft-vc/blob/main/soft-vc-demo.ipynb).\n\n\u003cdiv align=\"center\"\u003e\n    \u003cimg width=\"100%\" alt=\"Soft-VC\"\n      src=\"https://raw.githubusercontent.com/bshall/acoustic-model/main/acoustic-model.png\"\u003e\n\u003c/div\u003e\n\u003cdiv\u003e\n  \u003csup\u003e\n    \u003cstrong\u003eFig 1:\u003c/strong\u003e Architecture of the voice conversion system. a) The \u003cstrong\u003ediscrete\u003c/strong\u003e content encoder clusters audio features to produce a sequence of discrete speech units. b) The \u003cstrong\u003esoft\u003c/strong\u003e content encoder is trained to predict the discrete units. The acoustic model transforms the discrete/soft speech units into a target spectrogram. The vocoder converts the spectrogram into an audio waveform.\n  \u003c/sup\u003e\n\u003c/div\u003e\n\n## Example Usage\n\n### Programmatic Usage\n\n```python\nimport torch\nimport numpy as np\n\n# Load checkpoint (either hubert_soft or hubert_discrete)\nacoustic = torch.hub.load(\"bshall/acoustic-model:main\", \"hubert_soft\").cuda()\n\n# Load speech units\nunits = torch.from_numpy(np.load(\"path/to/units\"))\n\n# Generate mel-spectrogram\nmel = acoustic.generate(units)\n```\n\n### Script-Based Usage\n\n```\nusage: generate.py [-h] {soft,discrete} in-dir out-dir\n\nGenerate spectrograms from input speech units (discrete or soft).\n\npositional arguments:\n  {soft,discrete}  available models (HuBERT-Soft or HuBERT-Discrete)\n  in-dir           path to the dataset directory.\n  out-dir          path to the output directory.\n\noptional arguments:\n  -h, --help       show this help message and exit\n```\n\n## Training\n\n### Step 1: Dataset Preparation\n\nDownload and extract the [LJSpeech](https://keithito.com/LJ-Speech-Dataset/) dataset. The training script expects the following tree structure for the dataset directory:\n\n```\n└───wavs\n    ├───dev\n    │   ├───LJ001-0001.wav\n    │   ├───...\n    │   └───LJ050-0278.wav\n    └───train\n        ├───LJ002-0332.wav\n        ├───...\n        └───LJ047-0007.wav\n```\n\nThe `train` and `dev` directories should contain the training and validation splits respectively. The splits used for the paper can be found [here](https://github.com/bshall/acoustic-model/releases/tag/v0.1).\n\n### Step 2: Extract Spectrograms\n\nExtract mel-spectrograms using the `mel.py` script:\n\n```\nusage: mels.py [-h] in-dir out-dir\n\nExtract mel-spectrograms for an audio dataset.\n\npositional arguments:\n  in-dir      path to the dataset directory.\n  out-dir     path to the output directory.\n\noptional arguments:\n  -h, --help  show this help message and exit\n```\n\nfor example:\n\n```\npython mel.py path/to/LJSpeech-1.1/wavs path/to/LJSpeech-1.1/mels\n```\n\nAt this point the directory tree should look like:\n\n```\n├───mels\n│   ├───...\n└───wavs\n    ├───...\n```\n\n### Step 3: Extract Discrete or Soft Speech Units\n\nUse the HuBERT-Soft or HuBERT-Discrete content encoders to extract speech units. First clone the [content encoder repo](https://github.com/bshall/hubert) and then run `encode.py` (see the repo for details):\n\n```\nusage: encode.py [-h] [--extension EXTENSION] {soft,discrete} in-dir out-dir\n\nEncode an audio dataset.\n\npositional arguments:\n  {soft,discrete}       available models (HuBERT-Soft or HuBERT-Discrete)\n  in-dir                path to the dataset directory.\n  out-dir               path to the output directory.\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --extension EXTENSION\n                        extension of the audio files (defaults to .flac).\n```\n\nfor example:\n\n```\npython encode.py soft path/to/LJSpeech-1.1/wavs path/to/LJSpeech-1.1/soft --extension .wav\n```\n\nAt this point the directory tree should look like:\n\n```\n├───mels\n│   ├───...\n├───soft/discrete\n│   ├───...\n└───wavs\n    ├───...\n```\n\n### Step 4: Train the Acoustic-Model\n\n```\nusage: train.py [-h] [--resume RESUME] [--discrete] dataset-dir checkpoint-dir\n\nTrain the acoustic model.\n\npositional arguments:\n  dataset-dir      path to the data directory.\n  checkpoint-dir   path to the checkpoint directory.\n\noptional arguments:\n  -h, --help       show this help message and exit\n  --resume RESUME  path to the checkpoint to resume from.\n  --discrete       Use discrete units.\n```\n\n## Links\n\n- [Soft-VC repo](https://github.com/bshall/soft-vc)\n- [Soft-VC paper](https://ieeexplore.ieee.org/abstract/document/9746484)\n- [HuBERT content encoders](https://github.com/bshall/hubert)\n- [HiFiGAN vocoder](https://github.com/bshall/hifigan)\n\n## Citation\n\nIf you found this work helpful please consider citing our paper:\n\n```\n@inproceedings{\n    soft-vc-2022,\n    author={van Niekerk, Benjamin and Carbonneau, Marc-André and Zaïdi, Julian and Baas, Matthew and Seuté, Hugo and Kamper, Herman},\n    booktitle={ICASSP}, \n    title={A Comparison of Discrete and Soft Speech Units for Improved Voice Conversion}, \n    year={2022}\n}\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbshall%2Facoustic-model","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbshall%2Facoustic-model","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbshall%2Facoustic-model/lists"}