{"id":13478083,"url":"https://github.com/kyegomez/MultiModalMamba","last_synced_at":"2025-03-27T07:30:44.486Z","repository":{"id":215364956,"uuid":"738771232","full_name":"kyegomez/MultiModalMamba","owner":"kyegomez","description":"A novel implementation of fusing ViT with Mamba into a fast, agile, and high performance Multi-Modal Model. Powered by Zeta, the simplest AI framework ever.","archived":false,"fork":false,"pushed_at":"2024-04-08T05:53:46.000Z","size":2308,"stargazers_count":386,"open_issues_count":1,"forks_count":20,"subscribers_count":7,"default_branch":"main","last_synced_at":"2024-04-14T03:24:13.871Z","etag":null,"topics":["ai","artificial-intelligence","attention-mechanism","machine-learning","mamba","ml","pytorch","ssm","torch","transformer-architecture","transformers","zeta"],"latest_commit_sha":null,"homepage":"https://discord.gg/GYbXvDGevY","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kyegomez.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null},"funding":{"github":["kyegomez"],"patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"lfx_crowdfunding":null,"custom":null}},"created_at":"2024-01-04T02:21:13.000Z","updated_at":"2024-04-22T06:37:57.768Z","dependencies_parsed_at":"2024-01-13T19:19:41.032Z","dependency_job_id":"69188693-d743-47ec-842a-44e5b9426d49","html_url":"https://github.com/kyegomez/MultiModalMamba","commit_stats":null,"previous_names":["kyegomez/multimodalmamba"],"tags_count":0,"template":false,"template_full_name":"kyegomez/Python-Package-Template","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kyegomez%2FMultiModalMamba","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kyegomez%2FMultiModalMamba/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kyegomez%2FMultiModalMamba/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kyegomez%2FMultiModalMamba/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kyegomez","download_url":"https://codeload.github.com/kyegomez/MultiModalMamba/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245802217,"owners_count":20674611,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","artificial-intelligence","attention-mechanism","machine-learning","mamba","ml","pytorch","ssm","torch","transformer-architecture","transformers","zeta"],"created_at":"2024-07-31T16:01:52.205Z","updated_at":"2025-03-27T07:30:43.899Z","avatar_url":"https://github.com/kyegomez.png","language":"Python","funding_links":["https://github.com/sponsors/kyegomez"],"categories":["Python"],"sub_categories":[],"readme":"[![Multi-Modality](agorabanner.png)](https://discord.gg/qUtxnK2NMf)\n\n# Multi Modal Mamba - [MMM]\nMulti Modal Mamba (MultiModalMamba) is an all-new AI model that integrates Vision Transformer (ViT) and Mamba, creating a high-performance multi-modal model. MultiModalMamba is built on Zeta, a minimalist yet powerful AI framework, designed to streamline and enhance machine learning model management. \n\nThe capacity to process and interpret multiple data types concurrently is essential, the world isn't 1dimensional. MultiModalMamba addresses this need by leveraging the capabilities of Vision Transformer and Mamba, enabling efficient handling of both text and image data. This makes MultiModalMamba a versatile solution for a broad spectrum of AI tasks. \n\n\n## Install\n`pip3 install mmm-zeta`\n\n\n## Usage\n\n### `MultiModalMambaBlock`\n\n\n```python\n# Import the necessary libraries\nimport torch \nfrom torch import nn\nfrom mm_mamba import MultiModalMambaBlock\n\n# Create some random input tensors\nx = torch.randn(1, 16, 64)  # Tensor with shape (batch_size, sequence_length, feature_dim)\ny = torch.randn(1, 3, 64, 64)  # Tensor with shape (batch_size, num_channels, image_height, image_width)\n\n# Create an instance of the MultiModalMambaBlock model\nmodel = MultiModalMambaBlock(\n    dim = 64,  # Dimension of the token embeddings\n    depth = 5,  # Number of Mamba layers\n    dropout = 0.1,  # Dropout probability\n    heads = 4,  # Number of attention heads\n    d_state = 16,  # Dimension of the state embeddings\n    image_size = 64,  # Size of the input image\n    patch_size = 16,  # Size of each image patch\n    encoder_dim = 64,  # Dimension of the encoder token embeddings\n    encoder_depth = 5,  # Number of encoder transformer layers\n    encoder_heads = 4,  # Number of encoder attention heads\n    fusion_method=\"mlp\",\n)\n\n# Pass the input tensors through the model\nout = model(x, y)\n\n# Print the shape of the output tensor\nprint(out.shape)\n\n```\n\n\n### `MultiModalMamba`, Ready to Train Model\n- Flexibility in Data Types: The MultiModalMamba model can handle both text and image data simultaneously. This allows it to be trained on a wider variety of datasets and tasks, including those that require understanding of both text and image data.\n\n- Customizable Architecture: The MultiModalMamba model has numerous parameters such as depth, dropout, heads, d_state, image_size, patch_size, encoder_dim, encoder_depth, encoder_heads, and fusion_method. These parameters can be tuned according to the specific requirements of the task at hand, allowing for a high degree of customization in the model architecture.\n\n- Option to Return Embeddings: The MultiModalMamba model has a return_embeddings option. When set to True, the model will return the embeddings instead of the final output. This can be useful for tasks that require access to the intermediate representations learned by the model, such as transfer learning or feature extraction tasks.\n\n```python\nimport torch  # Import the torch library\n\n# Import the MultiModalMamba model from the mm_mamba module\nfrom mm_mamba import MultiModalMamba\n\n# Generate a random tensor 'x' of size (1, 224) with random elements between 0 and 10000\nx = torch.randint(0, 10000, (1, 196))\n\n# Generate a random image tensor 'img' of size (1, 3, 224, 224)\nimg = torch.randn(1, 3, 224, 224)\n\n# Audio tensor 'aud' of size 2d\naud = torch.randn(1, 224)\n\n# Video tensor 'vid' of size 5d - (batch_size, channels, frames, height, width)\nvid = torch.randn(1, 3, 16, 224, 224)\n\n# Create a MultiModalMamba model object with the following parameters:\nmodel = MultiModalMamba(\n    vocab_size=10000,\n    dim=512,\n    depth=6,\n    dropout=0.1,\n    heads=8,\n    d_state=512,\n    image_size=224,\n    patch_size=16,\n    encoder_dim=512,\n    encoder_depth=6,\n    encoder_heads=8,\n    fusion_method=\"mlp\",\n    return_embeddings=False,\n    post_fuse_norm=True,\n)\n\n# Pass the tensor 'x' and 'img' through the model and store the output in 'out'\nout = model(x, img, aud, vid)\n\n# Print the shape of the output tensor 'out'\nprint(out.shape)\n\n```\n\n# Real-World Deployment\n\nAre you an enterprise looking to leverage the power of AI? Do you want to integrate state-of-the-art models into your workflow? Look no further!\n\nMulti Modal Mamba (MultiModalMamba) is a cutting-edge AI model that fuses Vision Transformer (ViT) with Mamba, providing a fast, agile, and high-performance solution for your multi-modal needs. \n\nBut that's not all! With Zeta, our simple yet powerful AI framework, you can easily customize and fine-tune MultiModalMamba to perfectly fit your unique quality standards. \n\nWhether you're dealing with text, images, or both, MultiModalMamba has got you covered. With its deep configuration and multiple fusion layers, you can handle complex AI tasks with ease and efficiency.\n\n### :star2: Why Choose Multi Modal Mamba?\n\n- **Versatile**: Handle both text and image data with a single model.\n- **Powerful**: Leverage the power of Vision Transformer and Mamba.\n- **Customizable**: Fine-tune the model to your specific needs with Zeta.\n- **Efficient**: Achieve high performance without compromising on speed.\n\nDon't let the complexities of AI slow you down. Choose Multi Modal Mamba and stay ahead of the curve!\n\n[Contact us here](https://calendly.com/swarm-corp/30min) today to learn how you can integrate Multi Modal Mamba into your workflow and supercharge your AI capabilities!\n\n---\n\n\n# License\nMIT\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkyegomez%2FMultiModalMamba","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkyegomez%2FMultiModalMamba","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkyegomez%2FMultiModalMamba/lists"}