{"id":51002575,"url":"https://github.com/muhammad-fiaz/gpt-2-rust","last_synced_at":"2026-06-20T16:32:38.071Z","repository":{"id":364262075,"uuid":"1267129275","full_name":"muhammad-fiaz/gpt-2-rust","owner":"muhammad-fiaz","description":"A native Rust implementation of the GPT-2 transformer architecture built from scratch using the Burn deep-learning framework.","archived":false,"fork":false,"pushed_at":"2026-06-12T10:30:49.000Z","size":83,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-12T11:06:21.118Z","etag":null,"topics":["gpt","gpt-2-rust","gpt-rust","gpt-rust-implementation","gpt2","rust","rust-bot","rust-cli","rust-lang"],"latest_commit_sha":null,"homepage":"https://muhammad-fiaz.github.io/gpt-2-rust/","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/muhammad-fiaz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":{"github":["muhammad-fiaz"],"ko_fi":"muhammadfiaz"}},"created_at":"2026-06-12T08:46:38.000Z","updated_at":"2026-06-12T10:30:54.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/muhammad-fiaz/gpt-2-rust","commit_stats":null,"previous_names":["muhammad-fiaz/gpt-2-rust"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/muhammad-fiaz/gpt-2-rust","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/muhammad-fiaz%2Fgpt-2-rust","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/muhammad-fiaz%2Fgpt-2-rust/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/muhammad-fiaz%2Fgpt-2-rust/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/muhammad-fiaz%2Fgpt-2-rust/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/muhammad-fiaz","download_url":"https://codeload.github.com/muhammad-fiaz/gpt-2-rust/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/muhammad-fiaz%2Fgpt-2-rust/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34578089,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-20T02:00:06.407Z","response_time":98,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gpt","gpt-2-rust","gpt-rust","gpt-rust-implementation","gpt2","rust","rust-bot","rust-cli","rust-lang"],"created_at":"2026-06-20T16:32:37.053Z","updated_at":"2026-06-20T16:32:38.065Z","avatar_url":"https://github.com/muhammad-fiaz.png","language":"Rust","funding_links":["https://github.com/sponsors/muhammad-fiaz","https://ko-fi.com/muhammadfiaz"],"categories":[],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003eGPT-2 Rust 🦀🔥\u003c/h1\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cstrong\u003eA native Rust implementation of the GPT-2 transformer architecture built from scratch using the Burn deep-learning framework.\u003c/strong\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://github.com/muhammad-fiaz/gpt-2-rust/actions\"\u003e\u003cimg src=\"https://img.shields.io/github/actions/workflow/status/muhammad-fiaz/gpt-2-rust/rust.yml?branch=main\" alt=\"Build Status\" /\u003e\u003c/a\u003e\n  \u003ca href=\"https://muhammad-fiaz.github.io/gpt-2-rust/\"\u003e\u003cimg src=\"https://img.shields.io/badge/demo-GitHub%20Pages-blue\" alt=\"GitHub Pages\" /\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/muhammad-fiaz/gpt-2-rust/issues\"\u003e\u003cimg src=\"https://img.shields.io/github/issues/muhammad-fiaz/gpt-2-rust\" alt=\"GitHub Issues\" /\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/muhammad-fiaz/gpt-2-rust/pulls\"\u003e\u003cimg src=\"https://img.shields.io/github/issues-pr/muhammad-fiaz/gpt-2-rust\" alt=\"GitHub Pull Requests\" /\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/muhammad-fiaz/gpt-2-rust/graphs/commit-activity\"\u003e\u003cimg src=\"https://img.shields.io/github/last-commit/muhammad-fiaz/gpt-2-rust\" alt=\"GitHub Last Commit\" /\u003e\u003c/a\u003e\n  \u003ca href=\"LICENSE\"\u003e\u003cimg src=\"https://img.shields.io/github/license/muhammad-fiaz/gpt-2-rust\" alt=\"License: MIT\" /\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n---\n\n## Overview\n\nThis project is a native Rust implementation of the Transformer model from the seminal paper [Attention Is All You Need (arXiv)](https://arxiv.org/abs/1706.03762) / [(Hugging Face)](https://huggingface.co/papers/1706.03762) and OpenAI's GPT-2 architecture outlined in [Language Models are Unsupervised Multitask Learners](https://d4mucfpruptmv.cloudfront.net/better-language-models/language-models_are_unsupervised_multitask_learners.pdf). \n\nBuilt using the [Burn deep-learning framework](https://burn.dev), it includes:\n\n- ✅ **Custom Causal Self-Attention:** Custom Q/K/V combined projection with causal masking.\n- ✅ **Embeddings:** Learned token and absolute positional embeddings.\n- ✅ **Pre-norm Block Architecture:** Layer normalization applied before self-attention and MLP blocks.\n- ✅ **GELU-activated MLP:** Standard GPT-2 feedforward network.\n- ✅ **LM Head with Weight Tying:** Shares parameters with the token embedding table.\n- ✅ **Pure-Rust BPE Tokenizer:** Fast `tiktoken-rs` wrapper using GPT-2's vocabulary.\n- ✅ **Real-Time Word Streaming:** Outputs newly generated words instantly to the console.\n- ✅ **GPU Model Weight Offloading:** Automatically drops weights from GPU VRAM immediately after inference finishes.\n- ✅ **CUDA Execution:** GPU-accelerated by default using native Rust CUDA support (via CubeCL/cudarc) across Windows and Linux.\n- ✅ **Single CLI Entry Point:** Reuses dependencies and compiles quickly into a single main executable.\n\n---\n\n## Model Sizes \u0026 Parameters\n\n| Variant | Params | n_layer | n_head | n_embd |\n|---------|--------|---------|--------|--------|\n| `small` | 117 M  | 12      | 12     | 768    |\n| `medium`| 345 M  | 24      | 16     | 1024   |\n| `large` | 762 M  | 36      | 20     | 1280   |\n| `xl`    | 1.5 B  | 48      | 25     | 1600   |\n\n---\n\n## Command-Line Interface (CLI) Usages\n\nEverything compiles into a single, unified binary. Choose the operation mode using the flags:\n\n### 1. Download Model Parameters (`--download`)\nDownloads safetensors weights, model configurations, and vocabulary mapping for all variants from Hugging Face:\n```bash\n# Download small variant parameters into weights/small/\ncargo run --release -- --download --size small --weights-dir weights\n\n# Download all 4 variant parameters (small, medium, large, xl)\ncargo run --release -- --download --size all --weights-dir weights\n```\n* **Arguments:**\n  * `--size \u003csmall|medium|large|xl|all\u003e`: The GPT-2 variant to fetch.\n  * `--weights-dir \u003cpath\u003e`: Directory to save downloaded files.\n  * `--force`: Force download even if file already exists.\n\n### 2. Generate Text / Run Inference (`--generate`)\nGenerates text autoregressively with real-time word streaming:\n```bash\ncargo run --release -- --generate --model weights/small/model.safetensors --prompt \"The future of artificial intelligence is\" --size small --max-new-tokens 100 --temperature 0.8 --top-k 50\n```\n* **Arguments:**\n  * `--model \u003cpath\u003e`: Path to model weights file.\n  * `--size \u003csmall|medium|large|xl\u003e`: Size configuration to construct.\n  * `--prompt \"\u003ctext\u003e\"`: Conditioning prompt text.\n  * `--max-new-tokens \u003cN\u003e`: Maximum new tokens to produce.\n  * `--temperature \u003cF\u003e`: Softmax temperature scaling (0 = greedy).\n  * `--top-k \u003cN\u003e`: Top-K cutoff value (0 to disable).\n  * `--top-p \u003cF\u003e`: Top-P (nucleus) value (0.0 to disable).\n  * `--seed \u003cN\u003e`: Random seed for reproducibility.\n  * `--device \u003ccuda|wgpu|cpu\u003e`: Compute backend device (default is `cuda`).\n\n### 3. Evaluate Perplexity (`--evaluate`)\nEvaluates cross-entropy loss and perplexity on a test dataset:\n```bash\ncargo run --release -- --evaluate --model weights/small/model.safetensors --format safetensors --data data/input.txt --seq-len 128 --batch-size 4\n```\n* **Arguments:**\n  * `--model \u003cpath\u003e`: Path to weights file.\n  * `--format \u003csafetensors|compact\u003e`: Weights format.\n  * `--data \u003cpath\u003e`: Path to validation dataset file.\n  * `--seq-len \u003cN\u003e`: Sliding window sequence length.\n  * `--batch-size \u003cN\u003e`: Evaluation batch size.\n  * `--device \u003ccuda|wgpu|cpu\u003e`: Compute backend device (default is `cuda`).\n\n### 4. Pre-Train or Fine-Tune (`--train`)\nPerforms pre-training or fine-tuning from scratch:\n```bash\ncargo run --release -- --train --data data/input.txt --artifact-dir artifacts/ --size small --epochs 3 --batch-size 4 --seq-len 128 --lr 3e-4 --dropout 0.1\n```\n* **Arguments:**\n  * `--data \u003cpath\u003e`: Path to plain-text training file.\n  * `--artifact-dir \u003cpath\u003e`: Output directory for training checkpoints.\n  * `--epochs \u003cN\u003e`: Number of training epochs.\n  * `--lr \u003cF\u003e`: AdamW peak learning rate.\n  * `--dropout \u003cF\u003e`: Dropout probability.\n  * `--device \u003ccuda|wgpu|cpu\u003e`: Compute backend device (default is `cuda`).\n\n---\n\n## Tech Stack\n\n- **Deep Learning Framework:** [Burn](https://burn.dev) (v0.21)\n- **Compute Backends:** CUDA (default, native Rust GPU), WGPU (optional WebGPU), or NdArray (CPU fallback)\n- **Tokenizer:** `tiktoken-rs` (v0.5, BPE encoding)\n- **Serialization:** `safetensors` (v0.4) \u0026 `memmap2` (v0.9)\n\n---\n\n## Citations \u0026 References\n\n- **Burn Framework:**\n  ```bibtex\n  @misc{burn2024,\n    title = {Burn: A Flexible and Modern Deep Learning Framework in Rust},\n    howpublished = {\\url{https://burn.dev/}},\n    year = {2024}\n  }\n  ```\n- **Attention Is All You Need:**\n  - **arXiv:** [https://arxiv.org/abs/1706.03762](https://arxiv.org/abs/1706.03762)\n  - **Hugging Face Papers:** [https://huggingface.co/papers/1706.03762](https://huggingface.co/papers/1706.03762)\n  ```bibtex\n  @inproceedings{vaswani2017attention,\n    author    = {Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N and Kaiser, {\\L}ukasz and Polosukhin, Illia},\n    title     = {Attention is all you need},\n    booktitle = {Advances in neural information processing systems},\n    pages     = {5998--6008},\n    year      = {2017},\n    url       = {https://arxiv.org/abs/1706.03762 or https://huggingface.co/papers/1706.03762}\n  }\n  ```\n- **GPT-2 (Language Models are Unsupervised Multitask Learners):**\n  ```bibtex\n  @article{radford2019language,\n    title   = {Language models are unsupervised multitask learners},\n    author  = {Radford, Alec and Wu, Jeffrey and Child, Rewon and Luan, David and Amodei, Dario and Sutskever, Ilya},\n    journal = {OpenAI blog},\n    volume  = {1},\n    number  = {8},\n    pages   = {9},\n    year    = {2019},\n    url     = {https://d4mucfpruptmv.cloudfront.net/better-language-models/language-models_are_unsupervised_multitask_learners.pdf}\n  }\n  ```\n- **GPT-2 Rust (This repository):**\n  ```bibtex\n  @misc{gpt2rust2026,\n    author       = {Muhammad Fiaz},\n    title        = {GPT-2 Rust: A native Rust implementation of the GPT-2 transformer architecture built from scratch using the Burn deep-learning framework},\n    howpublished = {\\url{https://github.com/muhammad-fiaz/gpt-2-rust}},\n    year         = {2026},\n    note         = {Licensed under MIT License}\n  }\n  ```\n\n---\n\n## Contributing\n\nPlease see [CONTRIBUTING.md](CONTRIBUTING.md) for details on code style, linting, and pull requests.\n\n---\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmuhammad-fiaz%2Fgpt-2-rust","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmuhammad-fiaz%2Fgpt-2-rust","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmuhammad-fiaz%2Fgpt-2-rust/lists"}