Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/mathiasotnes/multi-stream-transformer
https://github.com/mathiasotnes/multi-stream-transformer
Last synced: 21 days ago
JSON representation
- Host: GitHub
- URL: https://github.com/mathiasotnes/multi-stream-transformer
- Owner: Mathiasotnes
- License: mit
- Created: 2024-11-12T21:27:52.000Z (about 2 months ago)
- Default Branch: main
- Last Pushed: 2024-11-22T22:07:58.000Z (about 2 months ago)
- Last Synced: 2024-11-22T22:32:30.863Z (about 2 months ago)
- Language: Python
- Size: 52.7 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Multi-Stream Transformer
---
### 1. Introduction
This project implements a novel transformer architecture that processes text through multiple parallel streams, each trained on different objectives. The core idea is to enhance the model's understanding and generation capabilities by combining different types of pattern recognition. The parallel streams will be fused into a common residual stream optimized for next-token prediction, and the goal is to enhance the model's underlying comprehension.
---
### 2. Setup
The required libraries can be downloaded throught the requirements file:
```
pip install -r requirements.txt
```
Run this afterwards for installing pytorch version compatible with CUDA 11.6:
```
pip install torch==1.13.1+cu116 torchvision==0.14.1+cu116 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/cu116
```
---
### 3. Repo activity
