{"id":49311415,"url":"https://github.com/prajwalamte/tiny-transformers","last_synced_at":"2026-04-26T13:02:55.832Z","repository":{"id":350639610,"uuid":"998501297","full_name":"PrajwalAmte/Tiny-Transformers","owner":"PrajwalAmte","description":"This repository contains two implementations of language models built to understand the \"Attention is All You Need\" paper and GPT architecture, following Andrej Karpathy's educational approach.","archived":false,"fork":false,"pushed_at":"2026-04-11T09:21:15.000Z","size":446,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-11T11:22:49.438Z","etag":null,"topics":["ai","gpt","llm"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/PrajwalAmte.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-06-08T18:36:14.000Z","updated_at":"2026-04-11T09:21:18.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/PrajwalAmte/Tiny-Transformers","commit_stats":null,"previous_names":["prajwalamte/tiny-transformers"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/PrajwalAmte/Tiny-Transformers","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PrajwalAmte%2FTiny-Transformers","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PrajwalAmte%2FTiny-Transformers/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PrajwalAmte%2FTiny-Transformers/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PrajwalAmte%2FTiny-Transformers/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/PrajwalAmte","download_url":"https://codeload.github.com/PrajwalAmte/Tiny-Transformers/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PrajwalAmte%2FTiny-Transformers/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32297907,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-26T09:34:17.070Z","status":"ssl_error","status_checked_at":"2026-04-26T09:34:00.993Z","response_time":129,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","gpt","llm"],"created_at":"2026-04-26T13:02:46.012Z","updated_at":"2026-04-26T13:02:55.809Z","avatar_url":"https://github.com/PrajwalAmte.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Tiny Transformers \n\nThis repository contains two implementations of language models built to understand the \"Attention is All You Need\" paper and GPT architecture, following Andrej Karpathy's educational approach.\n\n## Background\n\nThese implementations are based on:\n- **\"Attention is All You Need\"** (Vaswani et al., 2017) - The foundational Transformer paper\n- **Andrej Karpathy's GPT tutorials** - Excellent educational content on building GPT from scratch\n- **Character-level language modeling** using the Tiny Shakespeare dataset\n\n## Repository Structure\n\n### 1. `Bigram.py` - Simple Bigram Model\nA basic character-level language model that predicts the next character based only on the current character.\n\n**Key Features:**\n- Simple embedding lookup table\n- No attention mechanism\n- Baseline for comparison\n- ~65 characters vocabulary size\n\n### 2. `NanoGPT.py` - Mini GPT Implementation  \nA complete GPT-style transformer with self-attention mechanism.\n\n**Key Features:**\n- Multi-head self-attention\n- Transformer blocks with residual connections\n- Layer normalization\n- Positional embeddings\n- 10.7M parameters\n\n## Core Concepts Explained\n\n### Self-Attention Mechanism\nThe heart of the transformer architecture. Each token can \"attend\" to all previous tokens in the sequence, allowing the model to understand context and relationships.\n\n**How it works:**\n1. **Query, Key, Value**: Each input is transformed into three vectors\n2. **Attention Scores**: Compute similarity between query and all keys\n3. **Weighted Sum**: Use attention scores to weight the values\n4. **Context**: Each position gets a context-aware representation\n\n### Multi-Head Attention\nInstead of one attention mechanism, we use multiple \"heads\" that can focus on different types of relationships simultaneously.\n\n### Transformer Blocks\nEach block contains:\n- **Self-attention**: For understanding context\n- **Feed-forward network**: For processing the attended information\n- **Residual connections**: For stable training\n- **Layer normalization**: For training stability\n\n### Positional Embeddings\nSince attention has no inherent sense of position, we add positional information to help the model understand sequence order.\n\n## System Architecture\n\n### Simple Bigram Model\n\n```mermaid\nflowchart LR\n    A[Input Token] --\u003e B[Embedding Lookup]\n    B --\u003e C[Logits]\n    C --\u003e D[Next Token]\n    \n    style A fill:#e1f5fe,color:#000000\n    style D fill:#c8e6c9,color:#000000\n```\n\n### GPT Model Architecture\n\n```mermaid\nflowchart TD\n    subgraph Main[\"Main Architecture\"]\n        A[Input Tokens] --\u003e B[Token Embeddings]\n        A --\u003e C[Positional Embeddings]\n        B --\u003e D[\"+\"]\n        C --\u003e D\n        D --\u003e E[Transformer Block × 6]\n        E --\u003e F[Layer Norm]\n        F --\u003e G[Linear Head]\n        G --\u003e H[Output Logits]\n    end\n    \n    subgraph Detail[\"Transformer Block Detail\"]\n        I[Input] --\u003e J[Layer Norm]\n        J --\u003e K[Multi-Head Attention]\n        K --\u003e L[Residual Add]\n        I --\u003e L\n        L --\u003e M[Layer Norm]\n        M --\u003e N[Feed Forward]\n        N --\u003e O[Residual Add]\n        L --\u003e O\n    end\n    \n    style A fill:#e1f5fe,color:#000000\n    style H fill:#c8e6c9,color:#000000\n    style K fill:#fff3e0,color:#000000\n```\n\n### Multi-Head Attention\n\n```mermaid\nflowchart TD\n    A[Input] --\u003e B[Linear Q]\n    A --\u003e C[Linear K] \n    A --\u003e D[Linear V]\n    \n    B --\u003e E[\"Scaled Dot-Product\u003cbr/\u003eAttention × h heads\"]\n    C --\u003e E\n    D --\u003e E\n    \n    E --\u003e F[Concat]\n    F --\u003e G[Linear Projection]\n    G --\u003e H[Output]\n    \n    style A fill:#e1f5fe,color:#000000\n    style H fill:#c8e6c9,color:#000000\n    style E fill:#fff3e0,color:#000000\n```\n\n## Model Specifications\n\n| Specification | Bigram Model | GPT Model |\n|---------------|-------------|-----------|\n| **Parameters** | ~4K (vocab_size²) | 10.7M |\n| **Context Length** | 1 character | 256 tokens |\n| **Embedding Dimension** | N/A | 384 |\n| **Attention Heads** | N/A | 6 |\n| **Layers** | 1 | 6 |\n| **Training Iterations** | 3,000 | 5,000 |\n| **Learning Rate** | 1e-2 | 3e-4 |\n| **Batch Size** | 32 | 64 |\n| **Block Size** | 8 | 256 |\n\n## Key Improvements from Bigram to GPT\n\n1. **Context Understanding**: GPT sees up to 256 previous characters vs. just 1\n2. **Attention Mechanism**: Can focus on relevant parts of the context\n3. **Deeper Architecture**: 6 layers vs. 1 embedding lookup\n4. **Positional Awareness**: Understands sequence order\n5. **Better Text Quality**: More coherent and contextually appropriate generations\n\n## Training Features\n\n### Advanced Training Techniques\n- **Gradient Clipping**: Prevents exploding gradients\n- **Learning Rate Scheduling**: Optimized learning\n- **Dropout**: Prevents overfitting\n- **Weight Initialization**: Proper parameter initialization\n- **Temperature Sampling**: Controlled randomness in generation\n\n### Evaluation\n- Separate train/validation splits\n- Periodic loss evaluation\n- Sample generation during training\n- Multiple temperature settings for generation variety\n\n## Learning Outcomes\n\nBy studying these implementations, you'll understand:\n- **Transformer Architecture**: Core building blocks of modern LLMs\n- **Attention Mechanisms**: How models focus on relevant information\n- **Scaling Effects**: Difference between simple and complex models\n- **Training Dynamics**: How language models learn from data\n- **Text Generation**: Autoregressive language modeling\n\n## Usage\n\n```bash\n# For the bigram model\npython Bigram.py\n\n# For the GPT model  \npython NanoGPT.py\n```\n\n**Note**: You'll need to download the Tiny Shakespeare dataset:\n```bash\ncurl -o input.txt https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt\n```\n\n## Expected Results\n\n**Bigram Model**: Basic character-level patterns with limited coherence due to single-character context.\n\n**GPT Model**: More coherent Shakespeare-like text with proper sentence structure, character dialogue patterns, and contextual consistency.\n\n## References\n\n- Vaswani, A., et al. (2017). \"Attention is All You Need\"\n- Andrej Karpathy's GPT tutorials\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprajwalamte%2Ftiny-transformers","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fprajwalamte%2Ftiny-transformers","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprajwalamte%2Ftiny-transformers/lists"}