Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/kyegomez/tinygptv
Simple Implementation of TinyGPTV in super simple Zeta lego blocks
https://github.com/kyegomez/tinygptv
artificial-intelligence attention attention-is-all-you-need deep-learning multi-modal multi-modality transformers
Last synced: 11 days ago
JSON representation
Simple Implementation of TinyGPTV in super simple Zeta lego blocks
- Host: GitHub
- URL: https://github.com/kyegomez/tinygptv
- Owner: kyegomez
- License: mit
- Created: 2023-12-29T06:52:36.000Z (11 months ago)
- Default Branch: main
- Last Pushed: 2024-10-07T16:53:05.000Z (about 1 month ago)
- Last Synced: 2024-10-29T01:09:50.415Z (23 days ago)
- Topics: artificial-intelligence, attention, attention-is-all-you-need, deep-learning, multi-modal, multi-modality, transformers
- Language: Python
- Homepage: https://discord.gg/GYbXvDGevY
- Size: 2.17 MB
- Stars: 15
- Watchers: 3
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- Funding: .github/FUNDING.yml
- License: LICENSE
Awesome Lists containing this project
README
[](https://discord.gg/qUtxnK2NMf)
# TinyGPTV
Simple Implementation of TinyGPTV in super simple Zeta lego blocks. Here all the modules from figure 2 are implemented in Zeta and Pytorch.
The flow is the following:
x -> skip connection -> layer norm -> lora -> mha + lora -> residual_rms_norm -> original_skip_connection -> mlp + rms norm
## Install
`pip3 install tiny-gptv`
## Usage
### TinyGPTVBlock, Figure3 (c):
- Layernorm
- MHA
- Lora
- QK Norm
- RMS Norm
- MLP
```python
import torch
from tiny_gptv.blocks import TinyGPTVBlock
# Random tensor, replace with your input data
x = torch.rand(2, 8, 512)
# TinyGPTVBlock
block = TinyGPTVBlock(512, 8, depth=10)
# Print the block
print(block)
# Forward pass
out = block(x)
# Print the output shape
print(out.shape)
```
### Figure3 (b) Lora Module for LLMS Block
- MHA,
- Lora,
- Normalization,
- MLP
- Skip connection
- Split then add
```python
import torch
from tiny_gptv import LoraMHA
x = torch.rand(2, 8, 512)
block = LoraMHA(512, 8)
out = block(x)
print(out.shape)
```
# Citation
```bibtex
@misc{yuan2023tinygptv,
title={TinyGPT-V: Efficient Multimodal Large Language Model via Small Backbones},
author={Zhengqing Yuan and Zhaoxu Li and Lichao Sun},
year={2023},
eprint={2312.16862},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
```
# License
MIT