https://github.com/kyegomez/MambaByte

Implementation of MambaByte in "MambaByte: Token-free Selective State Space Model" in Pytorch and Zeta
https://github.com/kyegomez/MambaByte

ai artificial-intelligence gpt4v machine-learning mamba megabyte ml multi-modality tokenizer

Last synced: about 2 months ago
JSON representation

Implementation of MambaByte in "MambaByte: Token-free Selective State Space Model" in Pytorch and Zeta

• Host: GitHub
• URL: https://github.com/kyegomez/MambaByte
• Owner: kyegomez
• License: mit
• Created: 2024-01-26T01:23:25.000Z (8 months ago)
• Default Branch: main
• Last Pushed: 2024-06-17T06:19:06.000Z (3 months ago)
• Last Synced: 2024-07-11T12:02:56.455Z (2 months ago)
• Topics: ai, artificial-intelligence, gpt4v, machine-learning, mamba, megabyte, ml, multi-modality, tokenizer
• Language: Python
• Homepage: https://discord.gg/Czg5rpMZaC
• Size: 2.17 MB
• Stars: 91
• Watchers: 6
• Forks: 6
• Open Issues: 1
• Metadata Files:
  • Readme: README.md
  • Funding: .github/FUNDING.yml
  • License: LICENSE

Awesome Lists containing this project

• Awesome-Mamba - MambaByte: Token-free Selective State Space Model
• Awesome-state-space-models - Token-free language models

README

        
[![Multi-Modality](agorabanner.png)](https://discord.gg/qUtxnK2NMf)

# MambaByte

Implementation of MambaByte in "MambaByte: Token-free Selective State Space Model" in Pytorch and Zeta. Note this will be a higher performance implementation of Mamba with parallel scan 

## Installation

```bash

pip install mambabyte

```

# Usage

```python

import torch 

from mambabyte import MambaConfig, Mamba

x = torch.randn(2, 3, 4)

config = MambaConfig(

    dim = 4,

    depth = 3,

    dt_rank = 2,

    d_state = 2,

    expand_factor = 2,

    d_conv = 3,

    dt_min = 0.001,

    dt_max = 0.1,

    dt_init = "random",

    dt_scale = 1.0,

    bias = False,

    conv_bias = True,

    pscan = True

)

model = Mamba(config)

out = model(x)

print(out)

```

# License

MIT