{"id":15601091,"url":"https://github.com/lucidrains/mirasol-pytorch","last_synced_at":"2025-10-30T16:15:06.179Z","repository":{"id":207957434,"uuid":"720500455","full_name":"lucidrains/mirasol-pytorch","owner":"lucidrains","description":"Implementation of 🌻 Mirasol, SOTA Multimodal Autoregressive model out of Google Deepmind, in Pytorch","archived":false,"fork":false,"pushed_at":"2023-12-22T13:45:49.000Z","size":1056,"stargazers_count":89,"open_issues_count":1,"forks_count":2,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-08-18T08:57:17.961Z","etag":null,"topics":["artificial-intelligence","attention-mechanism","deep-learning","multimodality","transformers"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lucidrains.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-11-18T17:16:16.000Z","updated_at":"2025-07-23T15:25:53.000Z","dependencies_parsed_at":"2023-11-18T18:25:51.276Z","dependency_job_id":"6d5e068e-aa84-4dc5-8f20-6c413fa9105e","html_url":"https://github.com/lucidrains/mirasol-pytorch","commit_stats":null,"previous_names":["lucidrains/mirasol-pytorch"],"tags_count":14,"template":false,"template_full_name":null,"purl":"pkg:github/lucidrains/mirasol-pytorch","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Fmirasol-pytorch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Fmirasol-pytorch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Fmirasol-pytorch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Fmirasol-pytorch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lucidrains","download_url":"https://codeload.github.com/lucidrains/mirasol-pytorch/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Fmirasol-pytorch/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271470303,"owners_count":24765370,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-21T02:00:08.990Z","response_time":74,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artificial-intelligence","attention-mechanism","deep-learning","multimodality","transformers"],"created_at":"2024-10-03T02:14:23.395Z","updated_at":"2025-10-30T16:15:01.128Z","avatar_url":"https://github.com/lucidrains.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cimg src=\"./mirasol.png\" width=\"450px\"\u003e\u003c/img\u003e\n\n## 🌻 Mirasol - Pytorch\n\nImplementation of \u003ca href=\"https://arxiv.org/abs/2311.05698\"\u003eMirasol\u003c/a\u003e, SOTA Multimodal Autoregressive model out of Google Deepmind, in Pytorch\n\nWill simply implement the Transformer Combiner and omit the other variants.\n\n## Appreciation\n\n- \u003ca href=\"https://stability.ai/\"\u003eStabilityAI\u003c/a\u003e, \u003ca href=\"https://a16z.com/supporting-the-open-source-ai-community/\"\u003eA16Z Open Source AI Grant Program\u003c/a\u003e, and \u003ca href=\"https://huggingface.co/\"\u003e🤗 Huggingface\u003c/a\u003e for the generous sponsorships, as well as my other sponsors, for affording me the independence to open source current artificial intelligence research\n\n## Install\n\n```bash\n$ pip install mirasol-pytorch\n```\n\n## Usage\n\n```python\nimport torch\nfrom mirasol_pytorch import Mirasol\n\nmodel = Mirasol(\n    dim = 512,\n    num_text_tokens = 256,\n    video_image_size = 128,\n    video_frames_per_timechunk = 2,\n    audio_freq_dim = 64,\n    audio_time_dim_per_timechunk = 32,\n    audio_patch_size = (32, 16),\n    video_patch_size = (64, 2),\n    audio_encoder = dict(\n        dim = 512,\n        depth = 2\n    ),\n    video_encoder = dict(\n        dim = 512,\n        depth = 2\n    )\n)\n\naudio = torch.randn(1, 64, 1024)\nvideo = torch.randn(1, 3, 12, 128, 128)\n\ntext = torch.randint(0, 256, (1, 1024))\n\nloss = model(\n    audio = audio,\n    video = video,\n    text = text\n)\n\nloss.backward()\n\n# after much training\n\nsampled_text = model.generate(\n    audio = audio,\n    video = video,\n    seq_len = 512\n)\n```\n\n## Todo\n\n- [x] text generation code\n- [x] auto-handle start token for decoder\n- [x] positional embeddings for video and audio encoder\n- [x] enable register tokens for both video and audio encoder, inline with new research\n- [x] add audio and video reconstruction losses\n- [x] add similarity regularization from TTS research\n\n## Citations\n\n```bibtex\n@article{Piergiovanni2023Mirasol3BAM,\n    title   = {Mirasol3B: A Multimodal Autoregressive model for time-aligned and contextual modalities},\n    author  = {A. J. Piergiovanni and Isaac Noble and Dahun Kim and Michael S. Ryoo and Victor Gomes and Anelia Angelova},\n    journal = {ArXiv},\n    year    = {2023},\n    volume  = {abs/2311.05698},\n    url     = {https://api.semanticscholar.org/CorpusID:265129010}\n}\n```\n\n```bibtex\n@inproceedings{Liu2022TowardsBF,\n    title   = {Towards Better Few-Shot and Finetuning Performance with Forgetful Causal Language Models},\n    author  = {Hao Liu and Xinyang Geng and Lisa Lee and Igor Mordatch and Sergey Levine and Sharan Narang and P. Abbeel},\n    year    = {2022},\n    url     = {https://api.semanticscholar.org/CorpusID:256416540}\n}\n```\n\n```bibtex\n@article{Darcet2023VisionTN,\n    title   = {Vision Transformers Need Registers},\n    author  = {Timoth'ee Darcet and Maxime Oquab and Julien Mairal and Piotr Bojanowski},\n    journal = {ArXiv},\n    year    = {2023},\n    volume  = {abs/2309.16588},\n    url     = {https://api.semanticscholar.org/CorpusID:263134283}\n}\n```\n\n```bibtex\n@article{Bondarenko2023QuantizableTR,\n    title   = {Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing},\n    author  = {Yelysei Bondarenko and Markus Nagel and Tijmen Blankevoort},\n    journal = {ArXiv},\n    year    = {2023},\n    volume  = {abs/2306.12929},\n    url     = {https://api.semanticscholar.org/CorpusID:259224568}\n}\n```\n\n```bibtex\n@misc{shi2023enhance,\n    title   = {Enhance audio generation controllability through representation similarity regularization}, \n    author  = {Yangyang Shi and Gael Le Lan and Varun Nagaraja and Zhaoheng Ni and Xinhao Mei and Ernie Chang and Forrest Iandola and Yang Liu and Vikas Chandra},\n    year    = {2023},\n    eprint  = {2309.08773},\n    archivePrefix = {arXiv},\n    primaryClass = {cs.SD}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flucidrains%2Fmirasol-pytorch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flucidrains%2Fmirasol-pytorch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flucidrains%2Fmirasol-pytorch/lists"}