https://github.com/prajwalamte/tiny-transformers
This repository contains two implementations of language models built to understand the "Attention is All You Need" paper and GPT architecture, following Andrej Karpathy's educational approach.
https://github.com/prajwalamte/tiny-transformers
ai gpt llm
Last synced: 3 months ago
JSON representation
This repository contains two implementations of language models built to understand the "Attention is All You Need" paper and GPT architecture, following Andrej Karpathy's educational approach.
- Host: GitHub
- URL: https://github.com/prajwalamte/tiny-transformers
- Owner: PrajwalAmte
- Created: 2025-06-08T18:36:14.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2026-04-11T09:21:15.000Z (3 months ago)
- Last Synced: 2026-04-11T11:22:49.438Z (3 months ago)
- Topics: ai, gpt, llm
- Language: Python
- Homepage:
- Size: 436 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Tiny Transformers
This repository contains two implementations of language models built to understand the "Attention is All You Need" paper and GPT architecture, following Andrej Karpathy's educational approach.
## Background
These implementations are based on:
- **"Attention is All You Need"** (Vaswani et al., 2017) - The foundational Transformer paper
- **Andrej Karpathy's GPT tutorials** - Excellent educational content on building GPT from scratch
- **Character-level language modeling** using the Tiny Shakespeare dataset
## Repository Structure
### 1. `Bigram.py` - Simple Bigram Model
A basic character-level language model that predicts the next character based only on the current character.
**Key Features:**
- Simple embedding lookup table
- No attention mechanism
- Baseline for comparison
- ~65 characters vocabulary size
### 2. `NanoGPT.py` - Mini GPT Implementation
A complete GPT-style transformer with self-attention mechanism.
**Key Features:**
- Multi-head self-attention
- Transformer blocks with residual connections
- Layer normalization
- Positional embeddings
- 10.7M parameters
## Core Concepts Explained
### Self-Attention Mechanism
The heart of the transformer architecture. Each token can "attend" to all previous tokens in the sequence, allowing the model to understand context and relationships.
**How it works:**
1. **Query, Key, Value**: Each input is transformed into three vectors
2. **Attention Scores**: Compute similarity between query and all keys
3. **Weighted Sum**: Use attention scores to weight the values
4. **Context**: Each position gets a context-aware representation
### Multi-Head Attention
Instead of one attention mechanism, we use multiple "heads" that can focus on different types of relationships simultaneously.
### Transformer Blocks
Each block contains:
- **Self-attention**: For understanding context
- **Feed-forward network**: For processing the attended information
- **Residual connections**: For stable training
- **Layer normalization**: For training stability
### Positional Embeddings
Since attention has no inherent sense of position, we add positional information to help the model understand sequence order.
## System Architecture
### Simple Bigram Model
```mermaid
flowchart LR
A[Input Token] --> B[Embedding Lookup]
B --> C[Logits]
C --> D[Next Token]
style A fill:#e1f5fe,color:#000000
style D fill:#c8e6c9,color:#000000
```
### GPT Model Architecture
```mermaid
flowchart TD
subgraph Main["Main Architecture"]
A[Input Tokens] --> B[Token Embeddings]
A --> C[Positional Embeddings]
B --> D["+"]
C --> D
D --> E[Transformer Block × 6]
E --> F[Layer Norm]
F --> G[Linear Head]
G --> H[Output Logits]
end
subgraph Detail["Transformer Block Detail"]
I[Input] --> J[Layer Norm]
J --> K[Multi-Head Attention]
K --> L[Residual Add]
I --> L
L --> M[Layer Norm]
M --> N[Feed Forward]
N --> O[Residual Add]
L --> O
end
style A fill:#e1f5fe,color:#000000
style H fill:#c8e6c9,color:#000000
style K fill:#fff3e0,color:#000000
```
### Multi-Head Attention
```mermaid
flowchart TD
A[Input] --> B[Linear Q]
A --> C[Linear K]
A --> D[Linear V]
B --> E["Scaled Dot-Product
Attention × h heads"]
C --> E
D --> E
E --> F[Concat]
F --> G[Linear Projection]
G --> H[Output]
style A fill:#e1f5fe,color:#000000
style H fill:#c8e6c9,color:#000000
style E fill:#fff3e0,color:#000000
```
## Model Specifications
| Specification | Bigram Model | GPT Model |
|---------------|-------------|-----------|
| **Parameters** | ~4K (vocab_size²) | 10.7M |
| **Context Length** | 1 character | 256 tokens |
| **Embedding Dimension** | N/A | 384 |
| **Attention Heads** | N/A | 6 |
| **Layers** | 1 | 6 |
| **Training Iterations** | 3,000 | 5,000 |
| **Learning Rate** | 1e-2 | 3e-4 |
| **Batch Size** | 32 | 64 |
| **Block Size** | 8 | 256 |
## Key Improvements from Bigram to GPT
1. **Context Understanding**: GPT sees up to 256 previous characters vs. just 1
2. **Attention Mechanism**: Can focus on relevant parts of the context
3. **Deeper Architecture**: 6 layers vs. 1 embedding lookup
4. **Positional Awareness**: Understands sequence order
5. **Better Text Quality**: More coherent and contextually appropriate generations
## Training Features
### Advanced Training Techniques
- **Gradient Clipping**: Prevents exploding gradients
- **Learning Rate Scheduling**: Optimized learning
- **Dropout**: Prevents overfitting
- **Weight Initialization**: Proper parameter initialization
- **Temperature Sampling**: Controlled randomness in generation
### Evaluation
- Separate train/validation splits
- Periodic loss evaluation
- Sample generation during training
- Multiple temperature settings for generation variety
## Learning Outcomes
By studying these implementations, you'll understand:
- **Transformer Architecture**: Core building blocks of modern LLMs
- **Attention Mechanisms**: How models focus on relevant information
- **Scaling Effects**: Difference between simple and complex models
- **Training Dynamics**: How language models learn from data
- **Text Generation**: Autoregressive language modeling
## Usage
```bash
# For the bigram model
python Bigram.py
# For the GPT model
python NanoGPT.py
```
**Note**: You'll need to download the Tiny Shakespeare dataset:
```bash
curl -o input.txt https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt
```
## Expected Results
**Bigram Model**: Basic character-level patterns with limited coherence due to single-character context.
**GPT Model**: More coherent Shakespeare-like text with proper sentence structure, character dialogue patterns, and contextual consistency.
## References
- Vaswani, A., et al. (2017). "Attention is All You Need"
- Andrej Karpathy's GPT tutorials