{"id":19156254,"url":"https://github.com/kyegomez/mambatransformer","last_synced_at":"2025-05-16T12:04:06.851Z","repository":{"id":216897587,"uuid":"742696594","full_name":"kyegomez/MambaTransformer","owner":"kyegomez","description":"Integrating Mamba/SSMs with Transformer for Enhanced Long Context and High-Quality Sequence Modeling","archived":false,"fork":false,"pushed_at":"2025-01-27T13:15:34.000Z","size":2342,"stargazers_count":190,"open_issues_count":1,"forks_count":15,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-03T08:11:08.940Z","etag":null,"topics":["ai","artificial-intelligence","attention-is-all-you-need","attention-mechanisms","gpt4","language","machine-learning","multimodal","neural-network","neural-networks","pytorch","recurrent-neural-networks","rnns","ssm","tensorflow","zeta"],"latest_commit_sha":null,"homepage":"https://discord.gg/GYbXvDGevY","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kyegomez.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["kyegomez"],"patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"lfx_crowdfunding":null,"custom":null}},"created_at":"2024-01-13T04:58:52.000Z","updated_at":"2025-04-02T06:50:22.000Z","dependencies_parsed_at":"2024-01-13T14:55:19.279Z","dependency_job_id":"1c2f5328-ce3b-4e79-a572-2210c92cae47","html_url":"https://github.com/kyegomez/MambaTransformer","commit_stats":{"total_commits":19,"total_committers":4,"mean_commits":4.75,"dds":"0.26315789473684215","last_synced_commit":"6c3739adb876d18e8c7a85345d122ed2d2c5f72a"},"previous_names":["kyegomez/mambatransformer"],"tags_count":0,"template":false,"template_full_name":"kyegomez/Python-Package-Template","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kyegomez%2FMambaTransformer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kyegomez%2FMambaTransformer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kyegomez%2FMambaTransformer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kyegomez%2FMambaTransformer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kyegomez","download_url":"https://codeload.github.com/kyegomez/MambaTransformer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248525138,"owners_count":21118619,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","artificial-intelligence","attention-is-all-you-need","attention-mechanisms","gpt4","language","machine-learning","multimodal","neural-network","neural-networks","pytorch","recurrent-neural-networks","rnns","ssm","tensorflow","zeta"],"created_at":"2024-11-09T08:33:52.185Z","updated_at":"2025-04-12T06:14:50.265Z","avatar_url":"https://github.com/kyegomez.png","language":"Python","funding_links":["https://github.com/sponsors/kyegomez"],"categories":[],"sub_categories":[],"readme":"[![Multi-Modality](agorabanner.png)](https://discord.gg/qUtxnK2NMf)\n\n# Mamba Transformer\n\n![Mamba Transformer](/mm_transformer.png)\n\nIntegrating Mamba/SSMs with Transformer for Enhanced Long Context and High-Quality Sequence Modeling.\n\nThis is 100% novel architecture that I have designed to combine the strengths and weaknesses out of SSMs and Attention for an all-new advanced architecture with the purpose of surpassing our old limits. Faster processing speed, longer context lengths, lower perplexity over long sequences, enhanced and superior reasoning while remaining small and compact.\n\nThe architecture is essentially: `x -\u003e norm -\u003e mamba -\u003e norm -\u003e transformer -\u003e norm -\u003e ffn -\u003e norm -\u003e out`.\n\nI added in many normalizations as I believe by default training stability would be severly degraded due to 2 foreign architecture's integrating with one another.\n\n\n## Install\n`pip3 install mambatransformer`\n\n\n### Usage\n```python\nimport torch\nfrom mamba_transformer import MambaTransformer\n\n# Generate a random tensor of shape (1, 10) with values between 0 and 99\nx = torch.randint(0, 100, (1, 10))\n\n# Create an instance of the MambaTransformer model\nmodel = MambaTransformer(\n    num_tokens=100,  # Number of tokens in the input sequence\n    dim=512,  # Dimension of the model\n    heads=8,  # Number of attention heads\n    depth=4,  # Number of transformer layers\n    dim_head=64,  # Dimension of each attention head\n    d_state=512,  # Dimension of the state\n    dropout=0.1,  # Dropout rate\n    ff_mult=4,  # Multiplier for the feed-forward layer dimension\n    return_embeddings=False,  # Whether to return the embeddings,\n    transformer_depth=2,  # Number of transformer blocks\n    mamba_depth=10,  # Number of Mamba blocks,\n    use_linear_attn=True,  # Whether to use linear attention\n)\n\n# Pass the input tensor through the model and print the output shape\nout = model(x)\n\nprint(out.shape)\n\n\n# After many training\nmodel.eval()\n\n# Would you like to train this model? Zeta Corporation offers unmatchable GPU clusters at unbeatable prices, let's partner!\n\n# Tokenizer\nmodel.generate(text)\n\n\n```\n\n# License\nMIT\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkyegomez%2Fmambatransformer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkyegomez%2Fmambatransformer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkyegomez%2Fmambatransformer/lists"}