{"id":31642052,"url":"https://github.com/ambidextrous9/transformer-from-scratch","last_synced_at":"2025-10-07T03:29:02.092Z","repository":{"id":315864920,"uuid":"1056616068","full_name":"ambideXtrous9/Transformer-from-Scratch","owner":"ambideXtrous9","description":"Transformer from Scratch","archived":false,"fork":false,"pushed_at":"2025-09-21T07:48:18.000Z","size":45,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-21T09:25:35.201Z","etag":null,"topics":["attention","decoder","encoder","encoder-decoder-model","masked-language-models","transformer"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ambideXtrous9.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-14T13:17:09.000Z","updated_at":"2025-09-21T07:48:21.000Z","dependencies_parsed_at":"2025-09-21T09:25:37.121Z","dependency_job_id":"6fdf333e-132f-4116-8181-376f239bbd68","html_url":"https://github.com/ambideXtrous9/Transformer-from-Scratch","commit_stats":null,"previous_names":["ambidextrous9/transformer-from-scratch"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/ambideXtrous9/Transformer-from-Scratch","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ambideXtrous9%2FTransformer-from-Scratch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ambideXtrous9%2FTransformer-from-Scratch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ambideXtrous9%2FTransformer-from-Scratch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ambideXtrous9%2FTransformer-from-Scratch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ambideXtrous9","download_url":"https://codeload.github.com/ambideXtrous9/Transformer-from-Scratch/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ambideXtrous9%2FTransformer-from-Scratch/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278715506,"owners_count":26033295,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-07T02:00:06.786Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["attention","decoder","encoder","encoder-decoder-model","masked-language-models","transformer"],"created_at":"2025-10-07T03:28:57.625Z","updated_at":"2025-10-07T03:29:02.087Z","avatar_url":"https://github.com/ambideXtrous9.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🚀 Transformer from Scratch\n\n\u003cdiv align=\"center\"\u003e\n\n[![PyTorch](https://img.shields.io/badge/PyTorch-%23EE4C2C.svg?style=for-the-badge\u0026logo=PyTorch\u0026logoColor=white)](https://pytorch.org/)\n[![Lightning](https://img.shields.io/badge/Lightning-792EE5?style=for-the-badge\u0026logo=pytorchlightning\u0026logoColor=white)](https://pytorch-lightning.readthedocs.io/)\n[![Python](https://img.shields.io/badge/Python-3.11+-3776AB?style=for-the-badge\u0026logo=python\u0026logoColor=white)](https://www.python.org/)\n[![License](https://img.shields.io/badge/License-MIT-yellow.svg?style=for-the-badge)](https://opensource.org/licenses/MIT)\n\n**A complete, production-ready implementation of the Transformer architecture from \"Attention Is All You Need\"**\n\n*Built with PyTorch Lightning for scalable training and inference*\n\n\u003c/div\u003e\n\n---\n\n## ✨ What Makes This Special\n\n🎯 **Complete Implementation** - Every component from the original paper, meticulously crafted  \n⚡ **Lightning Fast** - PyTorch Lightning integration for distributed training  \n🧠 **Production Ready** - Proper error handling, logging, and checkpointing  \n🔧 **Modular Design** - Each component is independently testable and reusable  \n🧪 **Independent Testing** - Run each module separately for debugging and learning  \n📚 **Educational** - Clean, well-documented code perfect for learning  \n🎨 **Modern Stack** - Uses GPT-2 tokenizer and state-of-the-art practices  \n🚀 **Multiple Architectures** - CrossAttention, DecoderOnly, and MoE implementations  \n📊 **Comprehensive Metrics** - BLEU, ROUGE, METEOR, and BERTScore evaluation  \n🎛️ **Advanced Features** - Mixture of Experts with Top-K routing and sparse computation  \n\n---\n\n## 🏗️ Architecture Deep Dive\n\n### Core Components\n\n| Component | Description | Key Features |\n|-----------|-------------|--------------|\n| **🔤 TokenEmbedding** | Converts tokens to dense vectors | Scaling, padding handling, vocabulary mapping |\n| **📍 PositionalEmbedding** | Adds position information | Sinusoidal \u0026 learned encodings, flexible max positions |\n| **🎯 MultiHeadSelfAttention** | The heart of Transformers | Causal masking, cross-attention, scaled dot-product |\n| **🧠 PositionwiseFeedForward** | Non-linear transformations | GELU activation, configurable dimensions |\n| **➕ AddNorm** | Residual connections + normalization | Layer normalization, dropout, gradient flow |\n| **📥 Encoder** | Processes input sequences | Stacked layers, self-attention, context building |\n| **📤 Decoder** | Generates output sequences | Masked attention, cross-attention, autoregressive |\n| **🎛️ MoE Components** | Mixture of Experts implementation | Top-K routing, sparse computation, expert specialization |\n| **🔀 TopKRouter** | Expert selection mechanism | Dynamic routing, load balancing, efficient computation |\n\n### Model Architectures\n\n| Architecture | Description | Use Cases | Key Features |\n|--------------|-------------|-----------|--------------|\n| **🔄 CrossAttentionSeq2Seq** | Full encoder-decoder with cross-attention | Translation, summarization | Bidirectional encoding, cross-attention |\n| **📝 DecoderOnly** | GPT-style autoregressive model | Text generation, completion | Causal masking, next-token prediction |\n| **🎛️ DecoderOnlyMoE** | Decoder-only with Mixture of Experts | Large-scale text generation | Sparse activation, expert routing |\n\n### Data Flow\n\n```mermaid\ngraph TD\n    A[Input Text] --\u003e B[Tokenization]\n    B --\u003e C[Token Embedding]\n    C --\u003e D[Positional Encoding]\n    D --\u003e E[Encoder Stack]\n    E --\u003e F[Context Vectors]\n    F --\u003e G[Decoder Stack]\n    G --\u003e H[Output Logits]\n    H --\u003e I[Generated Text]\n```\n\n---\n\n## 🚀 Quick Start\n\n### 1. Installation\n\n```bash\n# Clone the repository\ngit clone https://github.com/yourusername/transformer-from-scratch.git\ncd transformer-from-scratch\n\n# Install dependencies\npip install torch pytorch-lightning transformers pandas numpy sacrebleu rouge_score bert_score nltk\n```\n\n### 2. Training\n\nChoose from multiple model architectures:\n\n#### CrossAttention Seq2Seq Model\n```bash\n# Train encoder-decoder with cross-attention\npython Trainer.py\n```\n\n#### Decoder-Only Model (GPT-style)\n```bash\n# Train decoder-only autoregressive model\npython DecoderOnlyTrainer.py\n```\n\n#### Decoder-Only with Mixture of Experts\n```bash\n# Train MoE model with expert routing\npython DecoderMoETrainer.py\n```\n\n**Training Features:**\n- 🎯 **Automatic checkpointing** - Best model saved automatically\n- 📊 **Real-time monitoring** - Loss tracking and validation metrics\n- 🔄 **GPU acceleration** - GPU support\n- 📈 **Progress tracking** - Detailed logging and progress bars\n- 🎛️ **MoE Support** - Sparse computation with expert routing\n- 📊 **Comprehensive Metrics** - BLEU, ROUGE, METEOR, BERTScore evaluation\n\n### 3. Inference\n\nChoose the appropriate inference script for your model:\n\n#### CrossAttention Seq2Seq Model\n```bash\n# Generate text completions with encoder-decoder\npython Inference.py\n```\n\n#### Decoder-Only Model\n```bash\n# Generate text with decoder-only model\npython DecoderOnlyInference.py\n```\n\n#### Decoder-Only with MoE\n```bash\n# Generate text with MoE model\npython DecoderMoEInference.py\n```\n\n**Inference Features:**\n- 🎲 **Greedy decoding** - Deterministic text generation\n- ⚡ **Fast inference** - Optimized for production use\n- 🎯 **Flexible input** - Handle variable length sequences\n- 🔧 **Easy integration** - Simple API for your applications\n- 🎛️ **MoE Support** - Efficient expert routing during inference\n- 📊 **Multiple Models** - Support for different architectures\n\n### 4. Independent Module Testing\n\nEach component can be run independently for testing and experimentation:\n\n```bash\n# Test individual components\npython Embedding.py              # Test token \u0026 positional embeddings\npython MultiHeadSelfAttention.py # Test attention mechanism\npython FFN.py                    # Test feed-forward network\npython AddNorm.py                # Test residual connections \u0026 normalization\npython Encoder.py                # Test encoder stack\npython Decoder.py                # Test decoder stack\npython Seq2SeqModel.py           # Test complete model\n```\n\n**Independent Testing Features:**\n- 🧪 **Component isolation** - Test each part separately\n- 🔍 **Debugging friendly** - Easy to identify issues in specific components\n- 📚 **Learning focused** - Understand each component's behavior individually\n- ⚡ **Quick validation** - Fast testing without full training pipeline\n\n---\n\n## 📊 Evaluation Metrics\n\nThe codebase includes comprehensive evaluation metrics for assessing model performance:\n\n### Automatic Metrics\n\n| Metric | Description | Range | Use Case |\n|--------|-------------|-------|----------|\n| **🎯 BLEU** | N-gram overlap with reference | 0-100 | Translation quality, text similarity |\n| **📝 ROUGE-1** | Unigram overlap | 0-1 | Content coverage, summarization |\n| **📝 ROUGE-2** | Bigram overlap | 0-1 | Phrase-level similarity |\n| **📝 ROUGE-L** | Longest common subsequence | 0-1 | Structural similarity |\n| **☄️ METEOR** | Semantic similarity with synonyms | 0-1 | Meaning preservation |\n| **🧠 BERTScore** | Contextual embedding similarity | 0-1 | Semantic understanding |\n\n### Implementation Features\n\n- **📊 Real-time Tracking** - Metrics computed during validation\n- **📈 Progress Monitoring** - All metrics logged to PyTorch Lightning\n- **🔄 Automatic Evaluation** - No manual intervention required\n- **⚡ Efficient Computation** - Optimized for large-scale evaluation\n- **📋 Comprehensive Coverage** - Multiple evaluation perspectives\n\n### Usage\n\nAll metrics are automatically computed during training validation steps and logged to the progress bar and tensorboard logs.\n\n---\n\n## 📊 Dataset \u0026 Task\n\n**Versatile Text Completion Dataset**\n- 📝 **2,000 examples** of diverse text completion pairs\n- 🎯 **Task**: Complete partial sentences with meaningful continuations\n- 📏 **Format**: `\"partial sentence...\" → \"completion text\"`\n- 🔄 **Train/Val Split**: 80/20 automatic split\n- 🌍 **Diverse Topics**: Covers multiple domains and contexts\n\n**Example:**\n```\nInput:  \"The rise of renewable energy is changing global markets and Experts predict this shift will redefine economies\"\nOutput: \"reducing dependence on fossil fuels and lowering emissions.\"\n```\n\n**Dataset Features:**\n- 📚 **Educational Content** - Science, technology, and general knowledge\n- 🔄 **Multiple Formats** - Various sentence structures and completion types\n- 🎯 **Quality Controlled** - Curated for meaningful learning objectives\n- 📊 **Balanced Distribution** - Even representation across different topics\n\n---\n\n## ⚙️ Configuration\n\n### Model Architecture\n\n| Parameter | Default | Description |\n|-----------|---------|-------------|\n| `d_model` | 256 | Model dimension (embedding size) |\n| `num_heads` | 4-8 | Number of attention heads |\n| `num_encoder_layers` | 2-6 | Encoder stack depth |\n| `num_decoder_layers` | 2-6 | Decoder stack depth |\n| `d_ff` | 128-1024 | Feed-forward dimension |\n| `dropout` | 0.1 | Dropout rate |\n| `max_positions` | 32-512 | Maximum sequence length |\n| `use_sinusoidal_pos` | True | Use sinusoidal positional encoding |\n\n### MoE Configuration (DecoderOnlyMoE)\n\n| Parameter | Default | Description |\n|-----------|---------|-------------|\n| `num_experts` | 4 | Number of expert networks |\n| `top_k` | 2 | Number of experts to activate per token |\n| `expert_capacity` | Auto | Maximum tokens per expert |\n\n### Training Configuration\n\n| Parameter | Value | Description |\n|-----------|-------|-------------|\n| `batch_size` | 4 | Training batch size |\n| `learning_rate` | 1e-3 | Adam optimizer learning rate |\n| `max_epochs` | 100 | Maximum training epochs |\n| `gradient_clip` | 1.0 | Gradient clipping threshold |\n| `checkpoint_monitor` | val_loss_epoch | Model selection metric |\n\n---\n\n## 📁 Project Structure\n\n```\ntransformer-from-scratch/\n├── 🧠 Core Components\n│   ├── Embedding.py              # Token \u0026 positional embeddings\n│   ├── MultiHeadSelfAttention.py # Multi-head attention mechanism\n│   ├── FFN.py                    # Position-wise feed-forward\n│   └── AddNorm.py                # Residual connections + normalization\n├── 🏗️ Architecture Models\n│   ├── Encoder.py                # Encoder stack implementation\n│   ├── Decoder.py                # Decoder stack implementation\n│   ├── CrossAttentionSeq2SeqModel.py  # Full encoder-decoder model\n│   ├── DecoderOnlySeq2SeqModel.py     # GPT-style decoder-only model\n│   └── DecoderMoE.py             # Decoder-only with Mixture of Experts\n├── 🚀 Training Scripts\n│   ├── Trainer.py                # CrossAttention training pipeline\n│   ├── DecoderOnlyTrainer.py     # Decoder-only training pipeline\n│   └── DecoderMoETrainer.py      # MoE training pipeline\n├── 🎯 Inference Scripts\n│   ├── Inference.py              # CrossAttention inference\n│   ├── DecoderOnlyInference.py   # Decoder-only inference\n│   └── DecoderMoEInference.py    # MoE inference\n├── 📊 Data\n│   ├── versatile_dataset_2000.csv     # Main training dataset\n│   └── synthetic_text_completion.csv  # Legacy dataset\n├── 📁 Checkpoints\n│   ├── Seq2SeqCheckpoints/       # CrossAttention model checkpoints\n│   ├── DecoderOnlyCheckpoints/   # Decoder-only model checkpoints\n│   └── DecoderMoECheckpoints/    # MoE model checkpoints\n└── 📈 Logs\n    └── lightning_logs/           # Training logs and metrics\n```\n\n---\n\n## 🎯 Use Cases\n\n### Perfect For:\n- 📚 **Learning** - Understanding Transformer architecture\n- 🔬 **Research** - Experimenting with attention mechanisms\n- 🚀 **Prototyping** - Quick seq2seq model development\n- 🧪 **Component Testing** - Debug and validate individual modules\n\n### Applications:\n\n#### CrossAttention Seq2Seq Model\n- 📄 **Summarization** - Generate concise summaries\n- 🔄 **Translation** - Sequence-to-sequence translation\n- 📝 **Question Answering** - Context-aware responses\n- 📊 **Data-to-Text** - Convert structured data to natural language\n\n#### Decoder-Only Models\n- 📝 **Text Completion** - Auto-complete sentences\n- 💬 **Chatbots** - Conversational AI systems\n- 🎨 **Creative Writing** - Story and content generation\n- 🔍 **Code Generation** - Programming assistance\n\n#### MoE Models\n- 🚀 **Large-Scale Generation** - Efficient text generation at scale\n- 🎯 **Specialized Tasks** - Expert routing for domain-specific content\n- ⚡ **Resource Optimization** - Sparse computation for better efficiency\n- 🧠 **Multi-Domain Learning** - Handle diverse topics with specialized experts\n\n---\n\n## 🎛️ Mixture of Experts (MoE) Implementation\n\n### Key Features\n\nThe MoE implementation includes several advanced features for efficient sparse computation:\n\n#### Expert Architecture\n- **🔧 ExpertMLP** - Individual expert networks with GELU activation\n- **🎯 TopKRouter** - Intelligent routing mechanism for expert selection\n- **⚡ Sparse Computation** - Only activate selected experts per token\n- **📊 Load Balancing** - Automatic expert capacity management\n\n#### Routing Strategy\n- **🎲 Softmax Gating** - Probabilistic expert selection\n- **🔝 Top-K Selection** - Activate only the most relevant experts\n- **📈 Dynamic Routing** - Adaptive expert selection based on input\n- **⚖️ Load Balancing** - Prevent expert overloading\n\n#### Performance Optimizations\n- **🚀 Sparse Activation** - Reduce computational overhead\n- **💾 Memory Efficient** - Only store active expert outputs\n- **🔄 Batch Processing** - Efficient parallel expert computation\n- **📊 Gradient Flow** - Proper backpropagation through routing\n\n### Usage Example\n\n```python\n# Initialize MoE model\nmodel = DecoderOnlyMoEModel(\n    vocab_size=vocab_size,\n    d_model=256,\n    num_experts=4,      # Number of expert networks\n    top_k=2,           # Activate top 2 experts per token\n    num_layers=6,\n    tokenizer=tokenizer\n)\n\n# Training automatically handles expert routing\ntrainer.fit(model, train_loader, val_loader)\n```\n\n---\n\n## 🤝 Contributing\n\nWe welcome contributions! Here's how you can help:\n\n1. 🍴 **Fork** the repository\n2. 🌟 **Create** a feature branch (`git checkout -b feature/AmazingFeature`)\n3. 💾 **Commit** your changes (`git commit -m 'Add AmazingFeature'`)\n4. 📤 **Push** to the branch (`git push origin feature/AmazingFeature`)\n5. 🔄 **Open** a Pull Request\n\n### Areas for Contribution:\n- 🚀 **Performance optimizations**\n- 🧪 **Additional attention mechanisms**\n- 📊 **More datasets and tasks**\n- 📚 **Documentation improvements**\n- 🐛 **Bug fixes and testing**\n\n---\n\n## 📚 References \u0026 Learning\n\n### Papers\n1. **Vaswani, A., et al.** (2017). \"Attention is all you need.\" *NeurIPS 2017*\n2. **Devlin, J., et al.** (2018). \"BERT: Pre-training of Deep Bidirectional Transformers.\" *NAACL 2019*\n\n### Resources\n- 📖 [The Illustrated Transformer](https://jalammar.github.io/illustrated-transformer/)\n- ⚡ [PyTorch Lightning Documentation](https://pytorch-lightning.readthedocs.io/)\n- 🎓 [Attention Mechanism Explained](https://distill.pub/2016/augmented-rnns/)\n- 🔥 [Transformer from Scratch](https://www.youtube.com/watch?v=ISNdQcPhsts)\n\n---\n\n\u003cdiv align=\"center\"\u003e\n\n**⭐ Star this repository if you found it helpful!**\n\nMade with ❤️ and lots of ☕\n\n[Report Bug](https://github.com/yourusername/transformer-from-scratch/issues) · [Request Feature](https://github.com/yourusername/transformer-from-scratch/issues) · [Documentation](https://github.com/yourusername/transformer-from-scratch/wiki)\n\n\u003c/div\u003e","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fambidextrous9%2Ftransformer-from-scratch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fambidextrous9%2Ftransformer-from-scratch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fambidextrous9%2Ftransformer-from-scratch/lists"}