{"id":18405466,"url":"https://github.com/jishengpeng/WavTokenizer","last_synced_at":"2025-04-07T08:32:07.836Z","repository":{"id":255404864,"uuid":"849379735","full_name":"jishengpeng/WavTokenizer","owner":"jishengpeng","description":"[ICLR 2025] SOTA discrete acoustic codec models with 40/75 tokens per second for audio language modeling ","archived":false,"fork":false,"pushed_at":"2025-03-02T03:53:58.000Z","size":395,"stargazers_count":1033,"open_issues_count":58,"forks_count":75,"subscribers_count":23,"default_branch":"main","last_synced_at":"2025-03-02T04:26:06.349Z","etag":null,"topics":["acoustic","audio-representation","codec","dac","encodec","gpt4o","music-representation-learning","semantic","soundstream","speech-language-model","speech-representation","text-to-speech"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jishengpeng.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-08-29T13:45:44.000Z","updated_at":"2025-03-02T03:54:01.000Z","dependencies_parsed_at":"2024-08-29T20:26:23.687Z","dependency_job_id":"b436e3c7-78fa-4bb5-86c8-68b4087cd851","html_url":"https://github.com/jishengpeng/WavTokenizer","commit_stats":null,"previous_names":["jishengpeng/wavtokenizer"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jishengpeng%2FWavTokenizer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jishengpeng%2FWavTokenizer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jishengpeng%2FWavTokenizer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jishengpeng%2FWavTokenizer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jishengpeng","download_url":"https://codeload.github.com/jishengpeng/WavTokenizer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247620195,"owners_count":20968151,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["acoustic","audio-representation","codec","dac","encodec","gpt4o","music-representation-learning","semantic","soundstream","speech-language-model","speech-representation","text-to-speech"],"created_at":"2024-11-06T03:01:35.121Z","updated_at":"2025-04-07T08:32:07.827Z","avatar_url":"https://github.com/jishengpeng.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# WavTokenizer\nSOTA Discrete Codec Models With Forty Tokens Per Second for Audio Language Modeling \n\n\n\n[![arXiv](https://img.shields.io/badge/arXiv-Paper-\u003cCOLOR\u003e.svg)](https://arxiv.org/abs/2408.16532)\n[![demo](https://img.shields.io/badge/WanTokenizer-Demo-red)](https://wavtokenizer.github.io/)\n[![model](https://img.shields.io/badge/%F0%9F%A4%97%20WavTokenizer-Models-blue)](https://huggingface.co/novateur/WavTokenizer)\n\n\n\n### 🎉🎉 with WavTokenizer, you can represent speech, music, and audio with only 40 tokens per second!\n### 🎉🎉 with WavTokenizer, You can get strong reconstruction results.\n### 🎉🎉 WavTokenizer owns rich semantic information and is build for audio language models such as GPT-4o.\n\n\u003c!--\n# Tips\nWe have noticed that several works (approximately exceed ten recent months) have incorrectly cited WavTokenizer. Below is the correct citation format. We sincerely appreciate the community's attention and interest.\n```\n@article{ji2024wavtokenizer,\n  title={Wavtokenizer: an efficient acoustic discrete codec tokenizer for audio language modeling},\n  author={Ji, Shengpeng and Jiang, Ziyue and Wang, Wen and Chen, Yifu and Fang, Minghui and Zuo, Jialong and Yang, Qian and Cheng, Xize and Wang, Zehan and Li, Ruiqi and others},\n  journal={arXiv preprint arXiv:2408.16532},\n  year={2024}\n}\n```\n--\u003e\n\n# 🔥 News\n- *2025.02.25*: We update WavTokenizer camera ready version for ICLR 2025 and update WavTokenizer-large-v2 checkpoint on [huggingface](https://huggingface.co/novateur/WavTokenizer-large-speech-75token). \n- *2024.10.22*: We update WavTokenizer on arxiv and release WavTokenizer-Large checkpoint.\n- *2024.09.09*: We release WavTokenizer-medium checkpoint on [huggingface](https://huggingface.co/collections/novateur/wavtokenizer-medium-large-66de94b6fd7d68a2933e4fc0).\n- *2024.08.31*: We release WavTokenizer on arxiv.\n\n![result](result.png)\n\n\n## Installation\n\nTo use WavTokenizer, install it using:\n\n```bash\nconda create -n wavtokenizer python=3.9\nconda activate wavtokenizer\npip install -r requirements.txt\n```\n\n## Infer\n\n### Part1: Reconstruct audio from raw wav\n\n```python\n\nfrom encoder.utils import convert_audio\nimport torchaudio\nimport torch\nfrom decoder.pretrained import WavTokenizer\n\n\ndevice=torch.device('cpu')\n\nconfig_path = \"./configs/xxx.yaml\"\nmodel_path = \"./xxx.ckpt\"\naudio_outpath = \"xxx\"\n\nwavtokenizer = WavTokenizer.from_pretrained0802(config_path, model_path)\nwavtokenizer = wavtokenizer.to(device)\n\n\nwav, sr = torchaudio.load(audio_path)\nwav = convert_audio(wav, sr, 24000, 1) \nbandwidth_id = torch.tensor([0])\nwav=wav.to(device)\nfeatures,discrete_code= wavtokenizer.encode_infer(wav, bandwidth_id=bandwidth_id)\naudio_out = wavtokenizer.decode(features, bandwidth_id=bandwidth_id) \ntorchaudio.save(audio_outpath, audio_out, sample_rate=24000, encoding='PCM_S', bits_per_sample=16)\n```\n\n\n### Part2: Generating discrete codecs\n```python\n\nfrom encoder.utils import convert_audio\nimport torchaudio\nimport torch\nfrom decoder.pretrained import WavTokenizer\n\ndevice=torch.device('cpu')\n\nconfig_path = \"./configs/xxx.yaml\"\nmodel_path = \"./xxx.ckpt\"\n\nwavtokenizer = WavTokenizer.from_pretrained0802(config_path, model_path)\nwavtokenizer = wavtokenizer.to(device)\n\nwav, sr = torchaudio.load(audio_path)\nwav = convert_audio(wav, sr, 24000, 1) \nbandwidth_id = torch.tensor([0])\nwav=wav.to(device)\n_,discrete_code= wavtokenizer.encode_infer(wav, bandwidth_id=bandwidth_id)\nprint(discrete_code)\n```\n\n\n\n### Part3: Audio reconstruction through codecs\n```python\n# audio_tokens [n_q,1,t]/[n_q,t]\nfeatures = wavtokenizer.codes_to_features(audio_tokens)\nbandwidth_id = torch.tensor([0])  \naudio_out = wavtokenizer.decode(features, bandwidth_id=bandwidth_id)\n```\n\n## Available models\n🤗 links to the Huggingface model hub.\n\n| Model name                                                          |                                                                                                            HuggingFace                                                                                                             |  Corpus  |  Token/s  | Domain | Open-Source |\n|:--------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|:--------:|:---------:|:----------:|:------:|\n| WavTokenizer-small-600-24k-4096             |             [🤗](https://huggingface.co/novateur/WavTokenizer/blob/main/WavTokenizer_small_600_24k_4096.ckpt)    | LibriTTS  | 40  |  Speech  | √ |\n| WavTokenizer-small-320-24k-4096             |             [🤗](https://huggingface.co/novateur/WavTokenizer/blob/main/WavTokenizer_small_320_24k_4096.ckpt)     | LibriTTS  | 75 |  Speech  | √|\n| WavTokenizer-medium-320-24k-4096                 |               [🤗](https://huggingface.co/collections/novateur/wavtokenizer-medium-large-66de94b6fd7d68a2933e4fc0)         | 10000 Hours | 75 |  Speech, Audio, Music  | √ |\n| WavTokenizer-large-600-24k-4096 | [🤗](https://huggingface.co/novateur/WavTokenizer-large-unify-40token) | 80000 Hours | 40 |   Speech, Audio, Music   | √|\n| WavTokenizer-large-320-24k-4096   | [🤗](https://huggingface.co/novateur/WavTokenizer-large-speech-75token) | 80000 Hours | 75 |   Speech, Audio, Music   | √ |\n\n      \n\n## Training\n\n### Step1: Prepare train dataset\n```python\n# Process the data into a form similar to ./data/demo.txt\n```\n\n### Step2: Modifying configuration files\n```python\n# ./configs/xxx.yaml\n# Modify the values of parameters such as batch_size, filelist_path, save_dir, device\n```\n\n### Step3: Start training process\nRefer to [Pytorch Lightning documentation](https://lightning.ai/docs/pytorch/stable/) for details about customizing the\ntraining pipeline.\n\n```bash\ncd ./WavTokenizer\npython train.py fit --config ./configs/xxx.yaml\n```\n\n\n## Citation\n\nIf this code contributes to your research, please cite our work, Language-Codec and WavTokenizer:\n\n```\n@article{ji2024wavtokenizer,\n  title={Wavtokenizer: an efficient acoustic discrete codec tokenizer for audio language modeling},\n  author={Ji, Shengpeng and Jiang, Ziyue and Wang, Wen and Chen, Yifu and Fang, Minghui and Zuo, Jialong and Yang, Qian and Cheng, Xize and Wang, Zehan and Li, Ruiqi and others},\n  journal={arXiv preprint arXiv:2408.16532},\n  year={2024}\n}\n\n@article{ji2024language,\n  title={Language-codec: Reducing the gaps between discrete codec representation and speech language models},\n  author={Ji, Shengpeng and Fang, Minghui and Jiang, Ziyue and Huang, Rongjie and Zuo, Jialung and Wang, Shulei and Zhao, Zhou},\n  journal={arXiv preprint arXiv:2402.12208},\n  year={2024}\n}\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjishengpeng%2FWavTokenizer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjishengpeng%2FWavTokenizer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjishengpeng%2FWavTokenizer/lists"}