{"id":13861928,"url":"https://github.com/lucidrains/BS-RoFormer","last_synced_at":"2025-07-14T09:34:11.483Z","repository":{"id":193739832,"uuid":"689406476","full_name":"lucidrains/BS-RoFormer","owner":"lucidrains","description":"Implementation of Band Split Roformer, SOTA Attention network for music source separation out of ByteDance AI Labs","archived":false,"fork":false,"pushed_at":"2024-08-06T16:37:04.000Z","size":230,"stargazers_count":432,"open_issues_count":8,"forks_count":16,"subscribers_count":14,"default_branch":"main","last_synced_at":"2024-11-21T06:02:51.114Z","etag":null,"topics":["artificial-intelligence","attention-mechanisms","deep-learning","music-source-separation","transformers"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lucidrains.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-09-09T17:48:36.000Z","updated_at":"2024-11-20T18:45:53.000Z","dependencies_parsed_at":"2023-09-09T18:39:38.212Z","dependency_job_id":"c7d10fd0-7c99-4dca-b819-b0792302806e","html_url":"https://github.com/lucidrains/BS-RoFormer","commit_stats":{"total_commits":59,"total_committers":4,"mean_commits":14.75,"dds":0.0847457627118644,"last_synced_commit":"aca155d755d312ca47c3910948f0275018febf62"},"previous_names":["lucidrains/bs-roformer"],"tags_count":32,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2FBS-RoFormer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2FBS-RoFormer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2FBS-RoFormer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2FBS-RoFormer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lucidrains","download_url":"https://codeload.github.com/lucidrains/BS-RoFormer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225681147,"owners_count":17507215,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artificial-intelligence","attention-mechanisms","deep-learning","music-source-separation","transformers"],"created_at":"2024-08-05T06:01:33.113Z","updated_at":"2025-07-14T09:34:11.471Z","avatar_url":"https://github.com/lucidrains.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"\u003cimg src=\"./bs-roformer.png\" width=\"450px\"\u003e\u003c/img\u003e\n\n## BS-RoFormer\n\nImplementation of \u003ca href=\"https://arxiv.org/abs/2309.02612\"\u003eBand Split Roformer\u003c/a\u003e, SOTA Attention network for music source separation out of ByteDance AI Labs. They beat the previous first place by a large margin. The technique uses axial attention across frequency (hence multi-band) and time. They also have experiments to show that rotary positional encoding led to a huge improvement over learned absolute positions.\n\nIt also includes support for stereo training and outputting multiple stems.\n\nPlease join \u003ca href=\"https://discord.gg/xBPBXfcFHd\"\u003e\u003cimg alt=\"Join us on Discord\" src=\"https://img.shields.io/discord/823813159592001537?color=5865F2\u0026logo=discord\u0026logoColor=white\"\u003e\u003c/a\u003e if you are interested in replicating a SOTA music source separator out in the open\n\nUpdate: This paper has been replicated by \u003ca href=\"https://github.com/ZFTurbo\"\u003eRoman\u003c/a\u003e and weight open sourced \u003ca href=\"https://github.com/ZFTurbo/Music-Source-Separation-Training?tab=readme-ov-file#vocal-models\"\u003ehere\u003c/a\u003e\n\nUpdate 2: Used for \u003ca href=\"https://www.youtube.com/watch?v=rboAdham380\"\u003ethis Katy Perry remix!\u003c/a\u003e\n\nUpdate 3: \u003ca href=\"https://github.com/KimberleyJensen\"\u003eKimberley Jensen\u003c/a\u003e has open sourced a MelBand Roformer trained on vocals \u003ca href=\"https://github.com/KimberleyJensen/Mel-Band-Roformer-Vocal-Model\"\u003ehere\u003c/a\u003e!\n\n## Appreciation\n\n- \u003ca href=\"https://stability.ai/\"\u003eStabilityAI\u003c/a\u003e and \u003ca href=\"https://huggingface.co/\"\u003e🤗 Huggingface\u003c/a\u003e for the generous sponsorship, as well as my other sponsors, for affording me the independence to open source artificial intelligence.\n\n- \u003ca href=\"https://github.com/shenberg\"\u003eRoee\u003c/a\u003e and \u003ca href=\"https://github.com/faroit\"\u003eFabian-Robert\u003c/a\u003e for sharing their audio expertise and fixing audio hyperparameters\n\n- \u003ca href=\"https://github.com/chenht2010\"\u003e@chenht2010\u003c/a\u003e and \u003ca href=\"https://github.com/ZFTurbo\"\u003eRoman\u003c/a\u003e for working out the default band splitting hyperparameter!\n\n- \u003ca href=\"https://github.com/dorpxam\"\u003eMax Prod\u003c/a\u003e for reporting a big bug with Mel-Band Roformer with stereo training!\n\n- \u003ca href=\"https://github.com/ZFTurbo\"\u003eRoman\u003c/a\u003e for successfully training the model and open sourcing his training code and weights at \u003ca href=\"https://github.com/ZFTurbo/Music-Source-Separation-Training\"\u003ethis repository\u003c/a\u003e!\n\n- \u003ca href=\"https://github.com/crlandsc\"\u003eChristopher\u003c/a\u003e for fixing an issue with multiple stems in Mel-Band Roformer\n\n- \u003ca href=\"https://github.com/iver56\"\u003eIver Jordal\u003c/a\u003e for identifying that the default stft window function is not correct\n\n## Install\n\n```bash\n$ pip install BS-RoFormer\n```\n\n## Usage\n\n```python\nimport torch\nfrom bs_roformer import BSRoformer\n\nmodel = BSRoformer(\n    dim = 512,\n    depth = 12,\n    time_transformer_depth = 1,\n    freq_transformer_depth = 1\n)\n\nx = torch.randn(2, 352800)\ntarget = torch.randn(2, 352800)\n\nloss = model(x, target = target)\nloss.backward()\n\n# after much training\n\nout = model(x)\n```\n\nTo use the Mel-Band Roformer proposed in \u003ca href=\"https://arxiv.org/abs/2310.01809\"\u003ea recent follow up paper\u003c/a\u003e, simply import `MelBandRoformer` instead\n\n```python\nimport torch\nfrom bs_roformer import MelBandRoformer\n\nmodel = MelBandRoformer(\n    dim = 32,\n    depth = 1,\n    time_transformer_depth = 1,\n    freq_transformer_depth = 1\n)\n\nx = torch.randn(2, 352800)\ntarget = torch.randn(2, 352800)\n\nloss = model(x, target = target)\nloss.backward()\n\n# after much training\n\nout = model(x)\n```\n\n## Todo\n\n- [x] get the multiscale stft loss in there\n- [x] figure out what `n_fft` should be\n- [x] review band split + mask estimation modules\n\n## Citations\n\n```bibtex\n@inproceedings{Lu2023MusicSS,\n    title   = {Music Source Separation with Band-Split RoPE Transformer},\n    author  = {Wei-Tsung Lu and Ju-Chiang Wang and Qiuqiang Kong and Yun-Ning Hung},\n    year    = {2023},\n    url     = {https://api.semanticscholar.org/CorpusID:261556702}\n}\n```\n\n```bibtex\n@inproceedings{Wang2023MelBandRF,\n    title   = {Mel-Band RoFormer for Music Source Separation},\n    author  = {Ju-Chiang Wang and Wei-Tsung Lu and Minz Won},\n    year    = {2023},\n    url     = {https://api.semanticscholar.org/CorpusID:263608675}\n}\n```\n\n```bibtex\n@misc{ho2019axial,\n    title  = {Axial Attention in Multidimensional Transformers},\n    author = {Jonathan Ho and Nal Kalchbrenner and Dirk Weissenborn and Tim Salimans},\n    year   = {2019},\n    archivePrefix = {arXiv}\n}\n```\n\n```bibtex\n@misc{su2021roformer,\n    title   = {RoFormer: Enhanced Transformer with Rotary Position Embedding},\n    author  = {Jianlin Su and Yu Lu and Shengfeng Pan and Bo Wen and Yunfeng Liu},\n    year    = {2021},\n    eprint  = {2104.09864},\n    archivePrefix = {arXiv},\n    primaryClass = {cs.CL}\n}\n```\n\n```bibtex\n@inproceedings{dao2022flashattention,\n    title   = {Flash{A}ttention: Fast and Memory-Efficient Exact Attention with {IO}-Awareness},\n    author  = {Dao, Tri and Fu, Daniel Y. and Ermon, Stefano and Rudra, Atri and R{\\'e}, Christopher},\n    booktitle = {Advances in Neural Information Processing Systems},\n    year    = {2022}\n}\n```\n\n```bibtex\n@article{Bondarenko2023QuantizableTR,\n    title   = {Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing},\n    author  = {Yelysei Bondarenko and Markus Nagel and Tijmen Blankevoort},\n    journal = {ArXiv},\n    year    = {2023},\n    volume  = {abs/2306.12929},\n    url     = {https://api.semanticscholar.org/CorpusID:259224568}\n}\n```\n\n```bibtex\n@inproceedings{ElNouby2021XCiTCI,\n    title   = {XCiT: Cross-Covariance Image Transformers},\n    author  = {Alaaeldin El-Nouby and Hugo Touvron and Mathilde Caron and Piotr Bojanowski and Matthijs Douze and Armand Joulin and Ivan Laptev and Natalia Neverova and Gabriel Synnaeve and Jakob Verbeek and Herv{\\'e} J{\\'e}gou},\n    booktitle = {Neural Information Processing Systems},\n    year    = {2021},\n    url     = {https://api.semanticscholar.org/CorpusID:235458262}\n}\n```\n\n```bibtex\n@inproceedings{Zhou2024ValueRL,\n    title   = {Value Residual Learning For Alleviating Attention Concentration In Transformers},\n    author  = {Zhanchao Zhou and Tianyi Wu and Zhiyun Jiang and Zhenzhong Lan},\n    year    = {2024},\n    url     = {https://api.semanticscholar.org/CorpusID:273532030}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flucidrains%2FBS-RoFormer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flucidrains%2FBS-RoFormer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flucidrains%2FBS-RoFormer/lists"}