https://github.com/mathiasotnes/multi-stream-transformer
This repository explores the usage of parallel residual connections within the transformer architecture for natural language processing tasks.
https://github.com/mathiasotnes/multi-stream-transformer
natural-language-processing python pytorch transformer
Last synced: 3 months ago
JSON representation
This repository explores the usage of parallel residual connections within the transformer architecture for natural language processing tasks.
- Host: GitHub
- URL: https://github.com/mathiasotnes/multi-stream-transformer
- Owner: Mathiasotnes
- License: mit
- Created: 2024-11-12T21:27:52.000Z (8 months ago)
- Default Branch: main
- Last Pushed: 2024-11-27T00:06:00.000Z (7 months ago)
- Last Synced: 2025-02-10T10:35:27.894Z (5 months ago)
- Topics: natural-language-processing, python, pytorch, transformer
- Language: Python
- Homepage:
- Size: 80.1 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Multi-Stream Transformer
---
### 1. Introduction
This project implements a novel transformer architecture that processes text through multiple parallel streams, each trained on different objectives. The core idea is to enhance the model's understanding and generation capabilities by combining different types of pattern recognition. The parallel streams will be fused into a common residual stream optimized for next-token prediction, and the goal is to enhance the model's underlying comprehension.
---
### 2. Setup
The required libraries can be downloaded throught the requirements file:
```
pip install -r requirements.txt
```
Run this afterwards for installing pytorch version compatible with CUDA 11.6:
```
pip install torch==1.13.1+cu116 torchvision==0.14.1+cu116 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/cu116
```
---
### 3. Repo activity
