https://github.com/techn0man1ac/simplegpt
A simple implementation of a GPT-like model using TensorFlow and Keras. It uses a Transformer architecture with multi-head attention to generate responses from input questions.
https://github.com/techn0man1ac/simplegpt
dataset keras neural-network tensorflow trainn
Last synced: 2 months ago
JSON representation
A simple implementation of a GPT-like model using TensorFlow and Keras. It uses a Transformer architecture with multi-head attention to generate responses from input questions.
- Host: GitHub
- URL: https://github.com/techn0man1ac/simplegpt
- Owner: techn0man1ac
- License: mit
- Created: 2024-11-18T17:33:11.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-11-24T20:31:37.000Z (over 1 year ago)
- Last Synced: 2025-01-23T03:29:49.584Z (over 1 year ago)
- Topics: dataset, keras, neural-network, tensorflow, trainn
- Language: Python
- Homepage:
- Size: 172 KB
- Stars: 0
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Simple GPT Model
A simple implementation of a GPT-like model using TensorFlow and Keras. It uses a Transformer architecture with multi-head attention to generate responses from input questions.
## Features
- Transformer-based architecture with multi-head attention
- Positional encoding added to input embeddings
- Customizable hyperparameters (e.g., embedding size, number of layers)
- Tokenizer-based input processing
- Option to train from scratch or load a pre-trained model
- Interactive question-answer generation
## Requirements
Install dependencies:
```bash
pip install tensorflow keras numpy
```
# Dataset
The model uses a custom dataset (simpleGPTDict) with question-answer pairs. Each pair is tokenized with and tokens.
# Model Overview
Input: Tokenized question sequences
Architecture: Embedding → Positional Encoding → Transformer Encoder → Output Layer
Training: Uses SparseCategoricalCrossentropy loss and Adam optimizer
# Running the Model
## Train Model
To train the model, run:
```bash
python SimpleGPT.py
```
Select option `1` to train from scratch. The trained model will be saved as `simple_gpt_model.keras`.
## Load Pre-trained Model
To load an existing model, select option `0` when prompted.
# Usage
Interact with the model to generate responses:
```bash
Question: What is your name?
Response: i am eliks
```
# License
This project is licensed under the MIT License - see the [LICENSE](https://github.com/techn0man1ac/SimpleGPT/blob/main/LICENSE) file for details.