https://github.com/ceciljoseph97/insigpt

Custom Generatively Pretrained Transformer with Multi Head Attention
https://github.com/ceciljoseph97/insigpt

gpt multihead-attention-networks python

Last synced: about 2 months ago
JSON representation

Custom Generatively Pretrained Transformer with Multi Head Attention

• Host: GitHub
• URL: https://github.com/ceciljoseph97/insigpt
• Owner: ceciljoseph97
• Created: 2023-03-28T07:45:52.000Z (almost 2 years ago)
• Default Branch: main
• Last Pushed: 2023-03-29T10:13:22.000Z (almost 2 years ago)
• Last Synced: 2024-04-16T07:22:32.515Z (9 months ago)
• Topics: gpt, multihead-attention-networks, python
• Language: Python
• Homepage:
• Size: 1.43 MB
• Stars: 0
• Watchers: 3
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# InsiGPT

InsiGPT is a take on the GPT architecture, trained using PyTorch. InsiGPT incorporates Multi-Head Attention mechanism, which enables the model to attend to different parts of the input sequence in parallel and capture more complex relationships between the tokens.

## Model Architecture

InsiGPT architecture is based on the Transformer model, which uses self-attention to capture the dependencies between tokens in the input sequence. The Multi-Head Attention mechanism is used in InsiGPT to allow the model to attend to multiple, different representations of the input sequence, which are projected into several smaller subspaces. These subspaces are then concatenated and linearly transformed to produce the final attention output.

The InsiGPT model consists of several layers of self-attention and feedforward neural networks, with residual connections and layer normalization applied at each layer. The final layer outputs a probability distribution over the vocabulary, which is used to generate the next token in the sequence during training and inference.

Uses tiny shakespeare dataset and custom model to generate text sequences.

To Execute: `Python modelInvoke.py -m_path 'pretrained model'`

Sample text Generated:

ROMEO:

What this word, not be't to vycound's friend ere's,

My slait this is eye?

LEONTES:

YORK:

Have, not I my fine.

ANGELO:

When you hast thou would not thing assay.

**Under Development**