{"id":13862268,"url":"https://github.com/lucidrains/audiolm-pytorch","last_synced_at":"2025-05-13T15:11:06.799Z","repository":{"id":58965902,"uuid":"534817329","full_name":"lucidrains/audiolm-pytorch","owner":"lucidrains","description":"Implementation of AudioLM, a SOTA Language Modeling Approach to Audio Generation out of Google Research, in Pytorch","archived":false,"fork":false,"pushed_at":"2025-01-12T15:40:45.000Z","size":585,"stargazers_count":2528,"open_issues_count":53,"forks_count":275,"subscribers_count":59,"default_branch":"main","last_synced_at":"2025-04-23T18:56:52.163Z","etag":null,"topics":["artificial-intelligence","attention-mechanisms","audio-synthesis","deep-learning","transformers"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lucidrains.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-09-09T21:55:02.000Z","updated_at":"2025-04-21T12:34:42.000Z","dependencies_parsed_at":"2022-09-11T01:50:29.665Z","dependency_job_id":"eb64a573-e52c-4588-8cb4-eb24b5325714","html_url":"https://github.com/lucidrains/audiolm-pytorch","commit_stats":{"total_commits":306,"total_committers":11,"mean_commits":"27.818181818181817","dds":0.09477124183006536,"last_synced_commit":"dd4784abb20fa3311fd4394f980a092aa233163b"},"previous_names":[],"tags_count":339,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Faudiolm-pytorch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Faudiolm-pytorch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Faudiolm-pytorch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Faudiolm-pytorch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lucidrains","download_url":"https://codeload.github.com/lucidrains/audiolm-pytorch/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253969249,"owners_count":21992263,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artificial-intelligence","attention-mechanisms","audio-synthesis","deep-learning","transformers"],"created_at":"2024-08-05T06:01:40.815Z","updated_at":"2025-05-13T15:11:01.783Z","avatar_url":"https://github.com/lucidrains.png","language":"Python","funding_links":[],"categories":["Python","2. Method Table"],"sub_categories":["2.3. Others"],"readme":"\u003cimg src=\"./audiolm.png\" width=\"600px\"\u003e\u003c/img\u003e\n\n## AudioLM - Pytorch\n\nImplementation of \u003ca href=\"https://google-research.github.io/seanet/audiolm/examples/\"\u003eAudioLM\u003c/a\u003e, a Language Modeling Approach to Audio Generation out of Google Research, in Pytorch\n\nIt also extends the work for conditioning with classifier free guidance with T5. This allows for one to do text-to-audio or TTS, not offered in the paper. Yes, this means \u003ca href=\"https://valle-demo.github.io/\"\u003eVALL-E\u003c/a\u003e can be trained from this repository. It is essentially the same.\n\nPlease join \u003ca href=\"https://discord.gg/xBPBXfcFHd\"\u003e\u003cimg alt=\"Join us on Discord\" src=\"https://img.shields.io/discord/823813159592001537?color=5865F2\u0026logo=discord\u0026logoColor=white\"\u003e\u003c/a\u003e if you are interested in replicating this work in the open\n\nThis repository now also contains a MIT licensed version of \u003ca href=\"https://arxiv.org/abs/2107.03312\"\u003eSoundStream\u003c/a\u003e. It is also compatible with \u003ca href=\"https://github.com/facebookresearch/encodec\"\u003eEnCodec\u003c/a\u003e, which is also [MIT-licensed](https://github.com/facebookresearch/encodec/commit/349b72939f57cb3bc7b60906c0ee8228c849485d) at the time of writing.\n\nUpdate: AudioLM was essentially used to 'solve' music generation in the new \u003ca href=\"https://github.com/lucidrains/musiclm-pytorch\"\u003eMusicLM\u003c/a\u003e\n\nIn the future, \u003ca href=\"https://www.youtube.com/watch?v=olNvmUCmY8o\"\u003ethis movie clip\u003c/a\u003e would no longer make any sense. You would just prompt an AI instead.\n\n## Appreciation\n\n- \u003ca href=\"https://stability.ai/\"\u003eStability.ai\u003c/a\u003e for the generous sponsorship to work and open source cutting edge artificial intelligence research\n\n- \u003ca href=\"https://huggingface.co/\"\u003e🤗 Huggingface\u003c/a\u003e for their amazing accelerate and transformers libraries\n\n- \u003ca href=\"https://ai.facebook.com/\"\u003eMetaAI\u003c/a\u003e for \u003ca href=\"https://github.com/facebookresearch/fairseq\"\u003eFairseq\u003c/a\u003e and the liberal license\n\n- \u003ca href=\"https://github.com/eonglints\"\u003e@eonglints\u003c/a\u003e and \u003ca href=\"https://github.com/turian\"\u003eJoseph\u003c/a\u003e for offering their professional advice and expertise as well as pull requests!\n\n- \u003ca href=\"https://github.com/djqualia\"\u003e@djqualia\u003c/a\u003e, \u003ca href=\"https://github.com/yigityu\"\u003e@yigityu\u003c/a\u003e, \u003ca href=\"https://github.com/inspirit\"\u003e@inspirit\u003c/a\u003e, and \u003ca href=\"https://github.com/BlackFox1197\"\u003e@BlackFox1197\u003c/a\u003e for helping with the debugging of soundstream\n\n- \u003ca href=\"https://github.com/zhvng\"\u003eAllen\u003c/a\u003e and \u003ca href=\"https://github.com/LWprogramming\"\u003eLWprogramming\u003c/a\u003e for reviewing the code and submitting bug fixes!\n\n- \u003ca href=\"https://github.com/ilya16\"\u003eIlya\u003c/a\u003e for finding an issue with multi-scale discriminator downsampling and for soundstream trainer improvements\n\n- \u003ca href=\"https://github.com/AndreyBocharnikov\"\u003eAndrey\u003c/a\u003e for identifying a missing loss in soundstream and guiding me through the proper mel spectrogram hyperparameters\n\n- \u003ca href=\"https://github.com/alexdemartos\"\u003eAlejandro\u003c/a\u003e and \u003ca href=\"https://github.com/ilya16\"\u003eIlya\u003c/a\u003e for sharing their results with training soundstream, and for working through a few issues with the local attention positional embeddings\n\n- \u003ca href=\"https://github.com/LWprogramming\"\u003eLWprogramming\u003c/a\u003e for adding Encodec compatibility!\n\n- \u003ca href=\"https://github.com/LWprogramming\"\u003eLWprogramming\u003c/a\u003e for finding an issue with handling of the EOS token when sampling from the `FineTransformer`!\n\n- \u003ca href=\"https://github.com/YoungloLee\"\u003e@YoungloLee\u003c/a\u003e for identifying a big bug in the 1d causal convolution for soundstream related to padding not accounting for strides!\n\n- \u003ca href=\"https://github.com/haydenshively\"\u003eHayden\u003c/a\u003e for pointing out some discrepancies in the multi-scale discriminator for Soundstream\n\n## Install\n\n```bash\n$ pip install audiolm-pytorch\n```\n\n## Usage\n\n### SoundStream \u0026 Encodec\n\nThere are two options for the neural codec. If you want to use the pretrained 24kHz Encodec, just create an Encodec object as follows:\n```python\nfrom audiolm_pytorch import EncodecWrapper\nencodec = EncodecWrapper()\n# Now you can use the encodec variable in the same way you'd use the soundstream variables below.\n```\n\nOtherwise, to stay more true to the original paper, you can use `SoundStream`. First, `SoundStream` needs to be trained on a large corpus of audio data\n\n```python\nfrom audiolm_pytorch import SoundStream, SoundStreamTrainer\n\nsoundstream = SoundStream(\n    codebook_size = 4096,\n    rq_num_quantizers = 8,\n    rq_groups = 2,                       # this paper proposes using multi-headed residual vector quantization - https://arxiv.org/abs/2305.02765\n    use_lookup_free_quantizer = True,    # whether to use residual lookup free quantization - there are now reports of successful usage of this unpublished technique\n    use_finite_scalar_quantizer = False, # whether to use residual finite scalar quantization\n    attn_window_size = 128,              # local attention receptive field at bottleneck\n    attn_depth = 2                       # 2 local attention transformer blocks - the soundstream folks were not experts with attention, so i took the liberty to add some. encodec went with lstms, but attention should be better\n)\n\ntrainer = SoundStreamTrainer(\n    soundstream,\n    folder = '/path/to/audio/files',\n    batch_size = 4,\n    grad_accum_every = 8,         # effective batch size of 32\n    data_max_length_seconds = 2,  # train on 2 second audio\n    num_train_steps = 1_000_000\n).cuda()\n\ntrainer.train()\n\n# after a lot of training, you can test the autoencoding as so\n\nsoundstream.eval() # your soundstream must be in eval mode, to avoid having the residual dropout of the residual VQ necessary for training\n\naudio = torch.randn(10080).cuda()\nrecons = soundstream(audio, return_recons_only = True) # (1, 10080) - 1 channel\n```\n\nYour trained `SoundStream` can then be used as a generic tokenizer for audio\n\n```python\n\naudio = torch.randn(1, 512 * 320)\n\ncodes = soundstream.tokenize(audio)\n\n# you can now train anything with the codebook ids\n\nrecon_audio_from_codes = soundstream.decode_from_codebook_indices(codes)\n\n# sanity check\n\nassert torch.allclose(\n    recon_audio_from_codes,\n    soundstream(audio, return_recons_only = True)\n)\n```\n\nYou can also use soundstreams that are specific to `AudioLM` and `MusicLM` by importing `AudioLMSoundStream` and `MusicLMSoundStream` respectively\n\n```python\nfrom audiolm_pytorch import AudioLMSoundStream, MusicLMSoundStream\n\nsoundstream = AudioLMSoundStream(...) # say you want the hyperparameters as in Audio LM paper\n\n# rest is the same as above\n```\n\nAs of version `0.17.0`, you can now invoke the class method on `SoundStream` to load from checkpoint files, without having to remember your configurations.\n\n```python\nfrom audiolm_pytorch import SoundStream\n\nsoundstream = SoundStream.init_and_load_from('./path/to/checkpoint.pt')\n```\n\nTo use \u003ca href=\"https://wandb.ai\"\u003eWeights \u0026 Biases\u003c/a\u003e tracking, first set `use_wandb_tracking = True` on the `SoundStreamTrainer`, then do the following\n\n```python\n\ntrainer = SoundStreamTrainer(\n    soundstream,\n    ...,\n    use_wandb_tracking = True\n)\n\n# wrap .train() with contextmanager, specifying project and run name\n\nwith trainer.wandb_tracker(project = 'soundstream', run = 'baseline'):\n    trainer.train()\n\n```\n\n### Hierarchical Transformers\n\nThen three separate transformers (`SemanticTransformer`, `CoarseTransformer`, `FineTransformer`) need to be trained\n\n\nex. `SemanticTransformer`\n\n```python\nimport torch\nfrom audiolm_pytorch import HubertWithKmeans, SemanticTransformer, SemanticTransformerTrainer\n\n# hubert checkpoints can be downloaded at\n# https://github.com/facebookresearch/fairseq/tree/main/examples/hubert\n\nwav2vec = HubertWithKmeans(\n    checkpoint_path = './hubert/hubert_base_ls960.pt',\n    kmeans_path = './hubert/hubert_base_ls960_L9_km500.bin'\n)\n\nsemantic_transformer = SemanticTransformer(\n    num_semantic_tokens = wav2vec.codebook_size,\n    dim = 1024,\n    depth = 6,\n    flash_attn = True\n).cuda()\n\n\ntrainer = SemanticTransformerTrainer(\n    transformer = semantic_transformer,\n    wav2vec = wav2vec,\n    folder ='/path/to/audio/files',\n    batch_size = 1,\n    data_max_length = 320 * 32,\n    num_train_steps = 1\n)\n\ntrainer.train()\n```\n\nex. `CoarseTransformer`\n\n```python\nimport torch\nfrom audiolm_pytorch import HubertWithKmeans, SoundStream, CoarseTransformer, CoarseTransformerTrainer\n\nwav2vec = HubertWithKmeans(\n    checkpoint_path = './hubert/hubert_base_ls960.pt',\n    kmeans_path = './hubert/hubert_base_ls960_L9_km500.bin'\n)\n\nsoundstream = SoundStream.init_and_load_from('/path/to/trained/soundstream.pt')\n\ncoarse_transformer = CoarseTransformer(\n    num_semantic_tokens = wav2vec.codebook_size,\n    codebook_size = 1024,\n    num_coarse_quantizers = 3,\n    dim = 512,\n    depth = 6,\n    flash_attn = True\n)\n\ntrainer = CoarseTransformerTrainer(\n    transformer = coarse_transformer,\n    codec = soundstream,\n    wav2vec = wav2vec,\n    folder = '/path/to/audio/files',\n    batch_size = 1,\n    data_max_length = 320 * 32,\n    num_train_steps = 1_000_000\n)\n\ntrainer.train()\n```\n\nex. `FineTransformer`\n\n```python\nimport torch\nfrom audiolm_pytorch import SoundStream, FineTransformer, FineTransformerTrainer\n\nsoundstream = SoundStream.init_and_load_from('/path/to/trained/soundstream.pt')\n\nfine_transformer = FineTransformer(\n    num_coarse_quantizers = 3,\n    num_fine_quantizers = 5,\n    codebook_size = 1024,\n    dim = 512,\n    depth = 6,\n    flash_attn = True\n)\n\ntrainer = FineTransformerTrainer(\n    transformer = fine_transformer,\n    codec = soundstream,\n    folder = '/path/to/audio/files',\n    batch_size = 1,\n    data_max_length = 320 * 32,\n    num_train_steps = 1_000_000\n)\n\ntrainer.train()\n```\n\nAll together now\n\n```python\nfrom audiolm_pytorch import AudioLM\n\naudiolm = AudioLM(\n    wav2vec = wav2vec,\n    codec = soundstream,\n    semantic_transformer = semantic_transformer,\n    coarse_transformer = coarse_transformer,\n    fine_transformer = fine_transformer\n)\n\ngenerated_wav = audiolm(batch_size = 1)\n\n# or with priming\n\ngenerated_wav_with_prime = audiolm(prime_wave = torch.randn(1, 320 * 8))\n\n# or with text condition, if given\n\ngenerated_wav_with_text_condition = audiolm(text = ['chirping of birds and the distant echos of bells'])\n\n```\n\n## Text Conditioned Audio Synthesis\n\nUpdate: Looks like this will work, given \u003ca href=\"https://valle-demo.github.io/\"\u003e'VALL-E'\u003c/a\u003e\n\nex. Semantic Transformer\n\n```python\nimport torch\nfrom audiolm_pytorch import HubertWithKmeans, SemanticTransformer, SemanticTransformerTrainer\n\nwav2vec = HubertWithKmeans(\n    checkpoint_path = './hubert/hubert_base_ls960.pt',\n    kmeans_path = './hubert/hubert_base_ls960_L9_km500.bin'\n)\n\nsemantic_transformer = SemanticTransformer(\n    num_semantic_tokens = 500,\n    dim = 1024,\n    depth = 6,\n    has_condition = True,               # this will have to be set to True\n    cond_as_self_attn_prefix = True     # whether to condition as prefix to self attention, instead of cross attention, as was done in 'VALL-E' paper\n).cuda()\n\n# mock text audio dataset (as an example)\n\n# you will have to extend your own from `Dataset`, and return an audio tensor as well as a string (the audio description) in any order (the framework will autodetect and route it into the transformer)\n\nfrom torch.utils.data import Dataset\n\nclass MockTextAudioDataset(Dataset):\n    def __init__(self, length = 100, audio_length = 320 * 32):\n        super().__init__()\n        self.audio_length = audio_length\n        self.len = length\n\n    def __len__(self):\n        return self.len\n\n    def __getitem__(self, idx):\n        mock_audio = torch.randn(self.audio_length)\n        mock_caption = 'audio caption'\n        return mock_caption, mock_audio\n\ndataset = MockTextAudioDataset()\n\n# instantiate semantic transformer trainer and train\n\ntrainer = SemanticTransformerTrainer(\n    transformer = semantic_transformer,\n    wav2vec = wav2vec,\n    dataset = dataset,\n    batch_size = 4,\n    grad_accum_every = 8,\n    data_max_length = 320 * 32,\n    num_train_steps = 1_000_000\n)\n\ntrainer.train()\n\n# after much training above\n\nsample = trainer.generate(text = ['sound of rain drops on the rooftops'], batch_size = 1, max_length = 2) # (1, \u003c 128) - may terminate early if it detects [eos]\n\n```\n\n## Multi-GPU\n\nBecause all the trainer classes uses \u003ca href=\"https://huggingface.co/docs/accelerate/accelerator\"\u003e🤗 Accelerator\u003c/a\u003e, you can easily do multi gpu training by using the `accelerate` command as so\n\nAt the project root\n\n```python\n$ accelerate config\n```\n\nThen, in the same directory\n\n```python\n$ accelerate launch train.py\n```\n\n## Todo\n\n- [x] complete CoarseTransformer\n- [x] use fairseq vq-wav2vec for embeddings\n- [x] add conditioning\n- [x] add classifier free guidance\n- [x] add unique consecutive for \n- [x] incorporate ability to use hubert intermediate features as semantic tokens, recommended by \u003ca href=\"https://github.com/lucidrains/audiolm-pytorch/discussions/13\"\u003eeonglints\u003c/a\u003e\n- [x] accommodate variable lengthed audio, bring in eos token\n- [x] make sure unique consecutive works with coarse transformer\n- [x] pretty printing all discriminator losses to log\n- [x] handle when generating semantic tokens, that last logits may not be necessarily the last in the sequence given unique consecutive processing\n- [x] complete sampling code for both Coarse and Fine Transformers, which will be tricky\n- [x] make sure full inference with or without prompting works on the `AudioLM` class\n- [x] complete full training code for soundstream, taking care of discriminator training\n- [x] add efficient gradient penalty for discriminators for soundstream\n- [x] wire up sample hz from sound dataset -\u003e transformers, and have proper resampling within during training - think about whether to allow for dataset to have sound files of varying or enforce same sample hz\n- [x] full transformer training code for all three transformers\n- [x] refactor so semantic transformer has a wrapper to that handles unique consecutives as well as wav to hubert or vq-wav2vec\n- [x] simply not self attend to eos token on the prompting side (semantic for coarse transformer, coarse for fine transformer)\n- [x] add structured dropout from forgetful causal masking, far better than traditional dropouts\n- [x] figure out how to suppress logging in fairseq\n- [x] assert that all three transformers passed into audiolm is compatible\n- [x] allow for specialized relative positional embeddings in fine transformer based on absolute matching positions of quantizers between coarse and fine\n- [x] allow for grouped residual vq in soundstream (use `GroupedResidualVQ` from vector-quantize-pytorch lib), from \u003ca href=\"https://arxiv.org/abs/2305.02765\"\u003ehifi-codec\u003c/a\u003e\n- [x] add flash attention with \u003ca href=\"https://arxiv.org/abs/2305.19466\"\u003eNoPE\u003c/a\u003e\n- [x] accept prime wave in `AudioLM` as a path to an audio file, and auto resample for semantic vs acoustic\n- [x] add key / value caching to all transformers, speeding up inference\n\n- [ ] design a hierarchical coarse and fine transformer\n- [ ] investigate \u003ca href=\"https://openreview.net/forum?id=H-VlwsYvVi\"\u003espec decoding\u003c/a\u003e, first test in x-transformers, then port over if applicable\n\n- [ ] redo the positional embeddings in the presence of groups in residual vq\n- [ ] test with speech synthesis for starters\n- [ ] cli tool, something like `audiolm generate \u003cwav.file | text\u003e` and save generated wav file to local directory\n- [ ] return a list of waves in the case of variable lengthed audio\n- [ ] just take care of the edge case in coarse transformer text conditioned training, where the raw wave is resampled at different frequencies. autodetermine how to route based on length\n\n## Citations\n\n```bibtex\n@inproceedings{Borsos2022AudioLMAL,\n  title  = {AudioLM: a Language Modeling Approach to Audio Generation},\n  author = {Zal{\\'a}n Borsos and Rapha{\\\"e}l Marinier and Damien Vincent and Eugene Kharitonov and Olivier Pietquin and Matthew Sharifi and Olivier Teboul and David Grangier and Marco Tagliasacchi and Neil Zeghidour},\n  year   = {2022}\n}\n```\n\n```bibtex\n@misc{https://doi.org/10.48550/arxiv.2107.03312,\n  title  = {SoundStream: An End-to-End Neural Audio Codec},\n  author = {Zeghidour, Neil and Luebs, Alejandro and Omran, Ahmed and Skoglund, Jan and Tagliasacchi, Marco},\n  publisher = {arXiv},\n  url    = {https://arxiv.org/abs/2107.03312},\n  year   = {2021}\n}\n```\n\n```bibtex\n@misc{shazeer2020glu,\n    title   = {GLU Variants Improve Transformer},\n    author  = {Noam Shazeer},\n    year    = {2020},\n    url     = {https://arxiv.org/abs/2002.05202}\n}\n```\n\n```bibtex\n@article{Shazeer2019FastTD,\n    title   = {Fast Transformer Decoding: One Write-Head is All You Need},\n    author  = {Noam M. Shazeer},\n    journal = {ArXiv},\n    year    = {2019},\n    volume  = {abs/1911.02150}\n}\n```\n\n```bibtex\n@article{Ho2022ClassifierFreeDG,\n    title   = {Classifier-Free Diffusion Guidance},\n    author  = {Jonathan Ho},\n    journal = {ArXiv},\n    year    = {2022},\n    volume  = {abs/2207.12598}\n}\n```\n\n```bibtex\n@misc{crowson2022,\n    author  = {Katherine Crowson},\n    url     = {https://twitter.com/rivershavewings}\n}\n```\n\n```bibtex\n@misc{ding2021cogview,\n    title   = {CogView: Mastering Text-to-Image Generation via Transformers},\n    author  = {Ming Ding and Zhuoyi Yang and Wenyi Hong and Wendi Zheng and Chang Zhou and Da Yin and Junyang Lin and Xu Zou and Zhou Shao and Hongxia Yang and Jie Tang},\n    year    = {2021},\n    eprint  = {2105.13290},\n    archivePrefix = {arXiv},\n    primaryClass = {cs.CV}\n}\n```\n\n```bibtex\n@article{Liu2022FCMFC,\n    title   = {FCM: Forgetful Causal Masking Makes Causal Language Models Better Zero-Shot Learners},\n    author  = {Hao Liu and Xinyang Geng and Lisa Lee and Igor Mordatch and Sergey Levine and Sharan Narang and P. Abbeel},\n    journal = {ArXiv},\n    year    = {2022},\n    volume  = {abs/2210.13432}\n}\n```\n\n```bibtex\n@inproceedings{anonymous2022normformer,\n    title   = {NormFormer: Improved Transformer Pretraining with Extra Normalization},\n    author  = {Anonymous},\n    booktitle = {Submitted to The Tenth International Conference on Learning Representations },\n    year    = {2022},\n    url     = {https://openreview.net/forum?id=GMYWzWztDx5},\n    note    = {under review}\n}\n```\n\n```bibtex\n@misc{liu2021swin,\n    title   = {Swin Transformer V2: Scaling Up Capacity and Resolution},\n    author  = {Ze Liu and Han Hu and Yutong Lin and Zhuliang Yao and Zhenda Xie and Yixuan Wei and Jia Ning and Yue Cao and Zheng Zhang and Li Dong and Furu Wei and Baining Guo},\n    year    = {2021},\n    eprint  = {2111.09883},\n    archivePrefix = {arXiv},\n    primaryClass = {cs.CV}\n}\n```\n\n```bibtex\n@article{Li2021LocalViTBL,\n    title   = {LocalViT: Bringing Locality to Vision Transformers},\n    author  = {Yawei Li and K. Zhang and Jie Cao and Radu Timofte and Luc Van Gool},\n    journal = {ArXiv},\n    year    = {2021},\n    volume  = {abs/2104.05707}\n}\n```\n\n```bibtex\n@article{Defossez2022HighFN,\n    title   = {High Fidelity Neural Audio Compression},\n    author  = {Alexandre D'efossez and Jade Copet and Gabriel Synnaeve and Yossi Adi},\n    journal = {ArXiv},\n    year    = {2022},\n    volume  = {abs/2210.13438}\n}\n```\n\n```bibtex\n@article{Hu2017SqueezeandExcitationN,\n    title   = {Squeeze-and-Excitation Networks},\n    author  = {Jie Hu and Li Shen and Gang Sun},\n    journal = {2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition},\n    year    = {2017},\n    pages   = {7132-7141}\n}\n```\n\n```bibtex\n@inproceedings{Yang2023HiFiCodecGV,\n    title   = {HiFi-Codec: Group-residual Vector quantization for High Fidelity Audio Codec},\n    author  = {Dongchao Yang and Songxiang Liu and Rongjie Huang and Jinchuan Tian and Chao Weng and Yuexian Zou},\n    year    = {2023}\n}\n```\n\n```bibtex\n@article{Kazemnejad2023TheIO,\n    title   = {The Impact of Positional Encoding on Length Generalization in Transformers},\n    author  = {Amirhossein Kazemnejad and Inkit Padhi and Karthikeyan Natesan Ramamurthy and Payel Das and Siva Reddy},\n    journal = {ArXiv},\n    year    = {2023},\n    volume  = {abs/2305.19466}\n}\n```\n\n```bibtex\n@inproceedings{dao2022flashattention,\n    title   = {Flash{A}ttention: Fast and Memory-Efficient Exact Attention with {IO}-Awareness},\n    author  = {Dao, Tri and Fu, Daniel Y. and Ermon, Stefano and Rudra, Atri and R{\\'e}, Christopher},\n    booktitle = {Advances in Neural Information Processing Systems},\n    year    = {2022}\n}\n```\n\n```bibtex\n@misc{yu2023language,\n    title   = {Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation},\n    author  = {Lijun Yu and José Lezama and Nitesh B. Gundavarapu and Luca Versari and Kihyuk Sohn and David Minnen and Yong Cheng and Agrim Gupta and Xiuye Gu and Alexander G. Hauptmann and Boqing Gong and Ming-Hsuan Yang and Irfan Essa and David A. Ross and Lu Jiang},\n    year    = {2023},\n    eprint  = {2310.05737},\n    archivePrefix = {arXiv},\n    primaryClass = {cs.CV}\n}\n```\n\n```bibtex\n@inproceedings{Katsch2023GateLoopFD,\n    title   = {GateLoop: Fully Data-Controlled Linear Recurrence for Sequence Modeling},\n    author  = {Tobias Katsch},\n    year    = {2023},\n    url     = {https://api.semanticscholar.org/CorpusID:265018962}\n}\n```\n\n```bibtex\n@article{Fifty2024Restructuring,\n    title   = {Restructuring Vector Quantization with the Rotation Trick},\n    author  = {Christopher Fifty, Ronald G. Junkins, Dennis Duan, Aniketh Iyengar, Jerry W. Liu, Ehsan Amid, Sebastian Thrun, Christopher Ré},\n    journal = {ArXiv},\n    year    = {2024},\n    volume  = {abs/2410.06424},\n    url     = {https://api.semanticscholar.org/CorpusID:273229218}\n}\n```\n\n```bibtex\n@inproceedings{Zhou2024ValueRL,\n    title   = {Value Residual Learning For Alleviating Attention Concentration In Transformers},\n    author  = {Zhanchao Zhou and Tianyi Wu and Zhiyun Jiang and Zhenzhong Lan},\n    year    = {2024},\n    url     = {https://api.semanticscholar.org/CorpusID:273532030}\n}\n```\n\n```bibtex\n@inproceedings{Huang2025TheGI,\n    title   = {The GAN is dead; long live the GAN! A Modern GAN Baseline},\n    author  = {Yiwen Huang and Aaron Gokaslan and Volodymyr Kuleshov and James Tompkin},\n    year    = {2025},\n    url     = {https://api.semanticscholar.org/CorpusID:275405495}\n}\n```\n\n```bibtex\n@article{Zhu2024HyperConnections,\n    title   = {Hyper-Connections},\n    author  = {Defa Zhu and Hongzhi Huang and Zihao Huang and Yutao Zeng and Yunyao Mao and Banggu Wu and Qiyang Min and Xun Zhou},\n    journal = {ArXiv},\n    year    = {2024},\n    volume  = {abs/2409.19606},\n    url     = {https://api.semanticscholar.org/CorpusID:272987528}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flucidrains%2Faudiolm-pytorch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flucidrains%2Faudiolm-pytorch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flucidrains%2Faudiolm-pytorch/lists"}