{"id":19662020,"url":"https://github.com/asappresearch/wav2seq","last_synced_at":"2025-04-28T21:31:48.305Z","repository":{"id":46296653,"uuid":"464955543","full_name":"asappresearch/wav2seq","owner":"asappresearch","description":"Official code for Wav2Seq","archived":false,"fork":false,"pushed_at":"2022-07-19T22:23:17.000Z","size":118,"stargazers_count":96,"open_issues_count":1,"forks_count":12,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-04-05T11:07:37.070Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/asappresearch.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-03-01T15:45:00.000Z","updated_at":"2024-11-25T07:52:19.000Z","dependencies_parsed_at":"2022-08-29T17:32:21.838Z","dependency_job_id":null,"html_url":"https://github.com/asappresearch/wav2seq","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/asappresearch%2Fwav2seq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/asappresearch%2Fwav2seq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/asappresearch%2Fwav2seq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/asappresearch%2Fwav2seq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/asappresearch","download_url":"https://codeload.github.com/asappresearch/wav2seq/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251391232,"owners_count":21582131,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-11T16:09:19.302Z","updated_at":"2025-04-28T21:31:48.003Z","avatar_url":"https://github.com/asappresearch.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Wav2Seq: Pre-training Speech-to-Text Encoder-Decoder Models Using Pseudo Languages\n\n[![made-with-python](https://img.shields.io/badge/Made%20with-Python-red.svg)](#python)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n\n**Authors:**: Felix Wu, Kwangyoun Kim, Shinji Watanabe, Kyu Han, Ryan McDonald, Kilian Q. Weinberger, Yoav Artzi\n\nPaper link: [https://arxiv.org/abs/2205.01086](https://arxiv.org/abs/2205.01086?context=cs)\n\n## Model Checkpoints\n### Pre-trained Models\nModel | Pre-training updates | Dataset | Link\n---|---|---|---\nWav2Seq (from HuBERT-base) | 400K + 25K | LibriSpeech 960h | [Download](https://public-dataset-model-store.awsdev.asapp.com/fwu/wav2seq/public/pt/wav2seq-hubert-base-25k.pt)\nWav2Seq (from HuBERT-base) | 400K + 100K | LibriSpeech 960h | [Download](https://public-dataset-model-store.awsdev.asapp.com/fwu/wav2seq/public/pt/wav2seq-hubert-base-100k.pt)\nWav2Seq (from HuBERT-large) | 400K + 25K | LibriLight 60Kh | [Download](https://public-dataset-model-store.awsdev.asapp.com/fwu/wav2seq/public/pt/wav2seq-hubert-large-25k.pt)\nWav2Seq (from XLS-R 0.3B) | 400K + 25K | XLS-R multi-lingual corpus 436Kh -\u003e LibriLight 60Kh | [Download](https://public-dataset-model-store.awsdev.asapp.com/fwu/wav2seq/public/pt/wav2seq-xlsr-300m-25k.pt)\n\n### Fine-tuned Models\nModel | Pre-training updates | Finetuning split | Link\n---|---|---|---\nWav2Seq (from HuBERT-base)  | 400K + 25K | LibriSpeech 10h | [Download](https://public-dataset-model-store.awsdev.asapp.com/fwu/wav2seq/public/ft/wav2seq-hubert-base-25k-ft-10h.pt)\nWav2Seq (from HuBERT-base)  | 400K + 100K | LibriSpeech 10h | [Download](https://public-dataset-model-store.awsdev.asapp.com/fwu/wav2seq/public/ft/wav2seq-hubert-base-100k-ft-10h.pt)\nWav2Seq (from HuBERT-base) | 400K + 25K | LibriSpeech 100h | [Download](https://public-dataset-model-store.awsdev.asapp.com/fwu/wav2seq/public/ft/wav2seq-hubert-base-25k-ft-100h.pt)\nWav2Seq (from HuBERT-base) | 400K + 100K | LibriSpeech 100h | [Download](https://public-dataset-model-store.awsdev.asapp.com/fwu/wav2seq/public/ft/wav2seq-hubert-base-100k-ft-100h.pt)\n\n### Pre-trained k-means Models for Psuedo Characters\nNumber of Clusters | Link\n---|---\n25  | [Download](https://public-dataset-model-store.awsdev.asapp.com/fwu/wav2seq/public/km/hubert_base-l9-k2s2-fp16-ls0.1-c25-km.pkl)\n100 | [Download](https://public-dataset-model-store.awsdev.asapp.com/fwu/wav2seq/public/km/hubert_base-l9-k2s2-fp16-ls0.1-c100-km.pkl)\n500 | [Download](https://public-dataset-model-store.awsdev.asapp.com/fwu/wav2seq/public/km/hubert_base-l9-k2s2-fp16-ls0.1-c500-km.pkl)\n\n\n### Pre-trained BPE model for Psuedo Subwords\nNumber of Clusters | Number of Subwords | Link\n---|---|---\n25 | 1000 | [Download](https://public-dataset-model-store.awsdev.asapp.com/fwu/wav2seq/public/tokenizer/c25-bpe-tokenizer-vocab1000.json) \n25 | 3000 | [Download](https://public-dataset-model-store.awsdev.asapp.com/fwu/wav2seq/public/tokenizer/c25-bpe-tokenizer-vocab3000.json) \n25 | 10000 | [Download](https://public-dataset-model-store.awsdev.asapp.com/fwu/wav2seq/public/tokenizer/c25-bpe-tokenizer-vocab10000.json) \n25 | 30000 | [Download](https://public-dataset-model-store.awsdev.asapp.com/fwu/wav2seq/public/tokenizer/c25-bpe-tokenizer-vocab30000.json) \n100 | 3000 | [Download](https://public-dataset-model-store.awsdev.asapp.com/fwu/wav2seq/public/tokenizer/c100-bpe-tokenizer-vocab3000.json) \n100 | 10000 | [Download](https://public-dataset-model-store.awsdev.asapp.com/fwu/wav2seq/public/tokenizer/c100-bpe-tokenizer-vocab10000.json) \n100 | 30000 | [Download](https://public-dataset-model-store.awsdev.asapp.com/fwu/wav2seq/public/tokenizer/c100-bpe-tokenizer-vocab30000.json) \n500 | 3000 | [Download](https://public-dataset-model-store.awsdev.asapp.com/fwu/wav2seq/public/tokenizer/c500-bpe-tokenizer-vocab3000.json) \n500 | 10000 | [Download](https://public-dataset-model-store.awsdev.asapp.com/fwu/wav2seq/public/tokenizer/c500-bpe-tokenizer-vocab10000.json) \n500 | 30000 | [Download](https://public-dataset-model-store.awsdev.asapp.com/fwu/wav2seq/public/tokenizer/c500-bpe-tokenizer-vocab30000.json) \n\n## Usage\n### Dependency\nThe code is tested with fairseq commit bba000d.\n```\ntorch==1.9.0+cu111\ntorchaudio==0.9.0\ntqdm==4.62.3\nhydra-core==1.0.7\nomegaconf==2.0.6\neinops==0.3.0\nfire==0.4.0\nfairseq==1.0.0a0+bba000d\n```\n\n### Installation\n```\ngit clone git@github.com:asappresearch/wav2seq.git\ncd wav2seq\npip install -e .\n```\n\n### Creatining Psuedo Subword Tokens\n1. Create wav2vec style manifest files\nPlease set `LIBRISPEECH_PATH` to your librispeech folder which contains three subfolders `train-clean-100`, `train-clean-360`, `train-other-500`.\n```sh\nmkdir -p manifest/librispeech/train-960\npython -m examples.wav2vec.wav2vec_manifest LIBRISPEECH_PATH  --dest manifest/librispeech/train-960 --ext flac --valid-percent 0.01 --path-must-contain train\n```\nWe also provide our manifest [here](https://public-dataset-model-store.awsdev.asapp.com/fwu/wav2seq/public/manifest/librispeech-pt.tar.gz) in case researcher want to use the same split. Please modify the first line of all the tsv files to ensure that the path of the data is set correctly.\n\n2. Train k-means model and get cluster indices.\nPlease make sure that you have download pre-trained hubert-base checkpoint at `HUBERT_PATH`.\nNotably, this step requires a GPU for feature extraction and 64GB main memory for k-means training.\nExtracting HuBERT features takes about 15 minutes, training k-means may take about an hour, dumping the cluster ids of the whole Librispeech 960h data takes more than two hours.\n```sh\nHUBERT_PATH=\"save/pretrained/hubert_base_ls960.pt\"\nmkdir -p save/pretrained\nif ! [ -f $HUBERT_PATH ]; then\n    wget https://dl.fbaipublicfiles.com/hubert/hubert_base_ls960.pt -O $HUBERT_PATH\nfi\nbash scripts/pl/extract-hubert-features.sh $HUBERT_PATH 9 2 2 500\n```\nwhere 9, 2, 2, 500 means that we use the 9-th layer of HuBERT, kernel size 2 and stride size 2 for average pooling, and 500 custers in k-means.\n\n3. Training BPE model and create pseudo subword tokens.\n```sh\nbash scripts/pl/create-hubert-pseudo-language.sh labels/hubert_base-l9-k2s2-fp16-ls0.1/c500 30000\n```\nWe also provide our extract pseudo language [here](https://public-dataset-model-store.awsdev.asapp.com/fwu/wav2seq/public/labels/hubert_base-l9-k2s2-fp16-ls0.1.tar.gz) for reproducibility.\n\n### Pre-training Wav2Seq\n```sh\nbash scripts/wav2seq-pt.sh wav2seq-hubert-base-ls960\n```\n\n### Fine-tuning Wav2Seq on LibriSpeech\n\nTo fine-tune a pretrained checkpoint on librispeech with 10h data. Please use this command.\n```sh\nbash scripts/wav2seq-ft-ls.sh $pretrained_ckpt ft-ls-10h\n```\nwhere `$pretrained_ckpt` is your pretrained checkpoint.\n\nWith 100h supervised data, please use this command.\n```sh\nbash scripts/wav2seq-ft-ls.sh $pretrained_ckpt ft-ls-100h\n```\nPlease make sure that your manifest files are stored in `manifest/librispeech`.\nWe provide our manifest [here](https://public-dataset-model-store.awsdev.asapp.com/fwu/wav2seq/public/manifest/librispeech-ft.tar.gz) for reproducibility. Please make sure that you change the first line of all tsv files so that the path of the data is set correctly.\nWe use a pretrained subword tokenizer [link](https://public-dataset-model-store.awsdev.asapp.com/fwu/wav2seq/public/tokenizer/ls_text_bpe_unigram1000.tar.gz) to convert LibriSpeech transcripts into subword tokens.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fasappresearch%2Fwav2seq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fasappresearch%2Fwav2seq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fasappresearch%2Fwav2seq/lists"}