{"id":15601022,"url":"https://github.com/lucidrains/spear-tts-pytorch","last_synced_at":"2025-04-05T01:04:25.727Z","repository":{"id":177018352,"uuid":"655790300","full_name":"lucidrains/spear-tts-pytorch","owner":"lucidrains","description":"Implementation of Spear-TTS - multi-speaker text-to-speech attention network, in Pytorch","archived":false,"fork":false,"pushed_at":"2023-10-30T17:24:47.000Z","size":203,"stargazers_count":268,"open_issues_count":5,"forks_count":19,"subscribers_count":27,"default_branch":"main","last_synced_at":"2025-03-29T00:09:19.941Z","etag":null,"topics":["artificial-intelligence","attention","deep-learning","text-to-speech","transformers"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lucidrains.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-06-19T15:48:42.000Z","updated_at":"2025-03-09T14:28:04.000Z","dependencies_parsed_at":"2024-01-16T15:44:54.777Z","dependency_job_id":"ef5552fb-046e-4091-a55f-167a79a43c98","html_url":"https://github.com/lucidrains/spear-tts-pytorch","commit_stats":{"total_commits":63,"total_committers":3,"mean_commits":21.0,"dds":"0.11111111111111116","last_synced_commit":"aa3b42a33a11d0c1455ddaee0b320ea3c4f385e1"},"previous_names":["lucidrains/spear-tts-pytorch"],"tags_count":54,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Fspear-tts-pytorch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Fspear-tts-pytorch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Fspear-tts-pytorch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Fspear-tts-pytorch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lucidrains","download_url":"https://codeload.github.com/lucidrains/spear-tts-pytorch/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247271519,"owners_count":20911587,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artificial-intelligence","attention","deep-learning","text-to-speech","transformers"],"created_at":"2024-10-03T02:12:23.937Z","updated_at":"2025-04-05T01:04:25.711Z","avatar_url":"https://github.com/lucidrains.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cimg src=\"./spear-tts.png\" width=\"450px\"\u003e\u003c/img\u003e\n\n## Spear-TTS - Pytorch\n\nImplementation of \u003ca href=\"https://arxiv.org/abs/2302.03540\"\u003eSpear-TTS\u003c/a\u003e - multi-speaker text-to-speech attention network, in Pytorch\n\nThe text-to-semantic module built here will be used for \u003ca href=\"https://github.com/lucidrains/soundstorm-pytorch\"\u003eSoundStorm\u003c/a\u003e for conditioning.\n\n## Appreciation\n\n- \u003ca href=\"https://stability.ai/\"\u003eStability\u003c/a\u003e for their generous sponsorships to work on and open source cutting edge artificial intelligence research\n\n- \u003ca href=\"https://github.com/lucasnewman\"\u003eLucas Newman\u003c/a\u003e for completing the \u003ca href=\"https://github.com/lucidrains/spear-tts-pytorch/pull/4\"\u003ebacktranslation\u003c/a\u003e portion, as well as beam search decoding!\n\n- \u003ca href=\"https://github.com/lucasnewman\"\u003eLucas Newman\u003c/a\u003e for completing the final text to semantic transformer training code!\n\n## Install\n\n```bash\n$ pip install spear-tts-pytorch\n```\n\n## Usage\n\n```python\nimport torch\n\nfrom audiolm_pytorch import HubertWithKmeans\n\nfrom spear_tts_pytorch import (\n    TextToSemantic,\n    SemanticToTextDatasetGenerator,\n    GeneratedAudioTextDataset,\n    MockDataset\n)\n\nwav2vec = HubertWithKmeans(\n    checkpoint_path = './hubert_base_ls960.pt',\n    kmeans_path = './hubert_base_ls960_L9_km500.bin'\n)\n\nmodel = TextToSemantic(\n    wav2vec = wav2vec,\n    dim = 512,\n    num_text_token_ids = 256,\n    heads = 8,\n    target_kv_heads = 2, # grouped query attention, for memory efficient decoding\n    source_depth = 1,\n    target_depth = 1\n)\n\nds = MockDataset(10)\n\ndataset_generator = SemanticToTextDatasetGenerator(\n    model = model,\n    dataset = ds,\n    folder = './output_folder'\n)\n\ndataset_generator(max_length = 2)\n\ngenerated_dataset = GeneratedAudioTextDataset(\n    folder = './output_folder'\n)\n\nassert len(generated_dataset) == 10\n```\n\n## Todo\n\n- [x] add eos logic + generate, and hook up end-to-end generation in soundstorm\n- [x] add first pretraining speech-to-speech with the reconstruction of 60% deleted tokens\n- [x] add dropouts for this project, as low-resource\n- [x] add total flexiblity of which layers of encoder / decoder to freeze during training\n- [x] add step for training on small speech -\u003e text corpus and generating pseudo-labelled dataset + finetuning (thanks to @lucasnewman)\n- [x] add final step of finetuning on text -\u003e speech + pseudolabelled dataset\n- [x] figure out the best way to store and manage the pseudo-labelled generated dataset\n- [x] batched beam search decoding\n- [x] allow for using rotary positions in decoder + flash attention, give Tri another citation\n- [x] integrate speculative decoding with some improvisation - done in same model using early exit strategy\n\n- [ ] add cached key / values for starter + single / grouped key values, make sure flash attention can support specialized causal mask before flash attention 2 is in pytorch core\n- [ ] polish the audio-text generation workflow\n- [ ] concatting the real audio-text dataset with the generated one -\u003e or being able to convert real audio-text dataset to generated\n\n## Citations\n\n```bibtex\n@misc{kharitonov2023speak,\n    title   = {Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervision}, \n    author  = {Eugene Kharitonov and Damien Vincent and Zalán Borsos and Raphaël Marinier and Sertan Girgin and Olivier Pietquin and Matt Sharifi and Marco Tagliasacchi and Neil Zeghidour},\n    year    = {2023},\n    eprint  = {2302.03540},\n    archivePrefix = {arXiv},\n    primaryClass = {cs.SD}\n}\n```\n\n```bibtex\n@inproceedings{dao2022flashattention,\n    title   = {Flash{A}ttention: Fast and Memory-Efficient Exact Attention with {IO}-Awareness},\n    author  = {Dao, Tri and Fu, Daniel Y. and Ermon, Stefano and Rudra, Atri and R{\\'e}, Christopher},\n    booktitle = {Advances in Neural Information Processing Systems},\n    year    = {2022}\n}\n```\n\n```bibtex\n@misc{shi2023enhance,\n    title   = {Enhance audio generation controllability through representation similarity regularization}, \n    author  = {Yangyang Shi and Gael Le Lan and Varun Nagaraja and Zhaoheng Ni and Xinhao Mei and Ernie Chang and Forrest Iandola and Yang Liu and Vikas Chandra},\n    year    = {2023},\n    eprint  = {2309.08773},\n    archivePrefix = {arXiv},\n    primaryClass = {cs.SD}\n}\n```\n\n```bibtex\n@article{Ainslie2023GQATG,\n    title   = {GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints},\n    author  = {Joshua Ainslie and James Lee-Thorp and Michiel de Jong and Yury Zemlyanskiy and Federico Lebr'on and Sumit K. Sanghai},\n    journal = {ArXiv},\n    year    = {2023},\n    volume  = {abs/2305.13245},\n    url     = {https://api.semanticscholar.org/CorpusID:258833177}\n}\n```\n\n```bibtex\n@inproceedings{Leviathan2022FastIF,\n    title   = {Fast Inference from Transformers via Speculative Decoding},\n    author  = {Yaniv Leviathan and Matan Kalman and Y. Matias},\n    booktitle = {International Conference on Machine Learning},\n    year    = {2022},\n    url     = {https://api.semanticscholar.org/CorpusID:254096365}\n}\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flucidrains%2Fspear-tts-pytorch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flucidrains%2Fspear-tts-pytorch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flucidrains%2Fspear-tts-pytorch/lists"}