https://github.com/kyegomez/vortexfusion
Transformers + Mambas + LSTMS All in One Model
https://github.com/kyegomez/vortexfusion
agora ai ai-research deep-learning lstms mambas ml multi-modality ssms transformers
Last synced: 1 day ago
JSON representation
Transformers + Mambas + LSTMS All in One Model
- Host: GitHub
- URL: https://github.com/kyegomez/vortexfusion
- Owner: kyegomez
- License: mit
- Created: 2024-07-31T04:10:45.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2025-04-19T12:56:17.000Z (6 months ago)
- Last Synced: 2025-04-19T20:17:00.341Z (6 months ago)
- Topics: agora, ai, ai-research, deep-learning, lstms, mambas, ml, multi-modality, ssms, transformers
- Language: Python
- Homepage: https://discord.com/servers/agora-999382051935506503
- Size: 2.16 MB
- Stars: 8
- Watchers: 1
- Forks: 1
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- Funding: .github/FUNDING.yml
- License: LICENSE
Awesome Lists containing this project
README
[](https://discord.gg/qUtxnK2NMf)
# Vortex Fusion
This is the first ever implementation of a joint Transformer + Mamba + LSTM architecture. The flow is the following: `mamba -> transformer -> lstm` in a loop. Perhaps with more iteration on model design, we can find a better architecture but this architecture is the future. This model is ready for training right now. If you're interested in training this model or working on entirely new model architectures, join the discord now!
## install
```bash
$ pip3 install -U vortex-fusion
```
## Usage
```python
import torch
from vortex_fusion import VortexFusion
# Generate random input tensor
x = torch.randint(0, 10000, (1, 10))
# Create an instance of the VortexFusion model with dimension 512
model = VortexFusion(dim=512)
# Pass the input tensor through the model to get the output
output = model(x)
# Print the shape of the output tensor
print(output.shape)
```
# License
MIT
# Todo
- [ ] Train this on the same dataset as Llama. Create one script that trains this model on a massive dataset to experiment with performance metrics.