{"id":19830015,"url":"https://github.com/2003harsh/transformer-based-decoder-only-language-model","last_synced_at":"2026-05-14T09:36:03.432Z","repository":{"id":244670362,"uuid":"815907061","full_name":"2003HARSH/Transformer-based-Decoder-only-Language-Model","owner":"2003HARSH","description":"This repository explores building a character-level transformer decoder in PyTorch, similar to GPT while focusing more on understanding individual components. My goal is to gain deep transformer knowledge and see if character-level learning improves handling of unseen words. The code allows for hyperparameter tuning and experiment customization. ","archived":false,"fork":false,"pushed_at":"2024-08-30T11:31:23.000Z","size":77951,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-11T09:47:47.911Z","etag":null,"topics":["from-scratch-in-python","gpt","language-model","pytorch","transformers"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/2003HARSH.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-06-16T13:46:35.000Z","updated_at":"2024-10-24T18:57:09.000Z","dependencies_parsed_at":"2024-11-12T11:35:21.049Z","dependency_job_id":null,"html_url":"https://github.com/2003HARSH/Transformer-based-Decoder-only-Language-Model","commit_stats":null,"previous_names":["2003harsh/tinygpt","2003harsh/transformer-based-decoder-only-language-model"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/2003HARSH%2FTransformer-based-Decoder-only-Language-Model","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/2003HARSH%2FTransformer-based-Decoder-only-Language-Model/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/2003HARSH%2FTransformer-based-Decoder-only-Language-Model/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/2003HARSH%2FTransformer-based-Decoder-only-Language-Model/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/2003HARSH","download_url":"https://codeload.github.com/2003HARSH/Transformer-based-Decoder-only-Language-Model/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241189107,"owners_count":19924752,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["from-scratch-in-python","gpt","language-model","pytorch","transformers"],"created_at":"2024-11-12T11:21:16.633Z","updated_at":"2026-05-14T09:36:03.402Z","avatar_url":"https://github.com/2003HARSH.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Transformer-based (Decoder-only) Language Model from Scratch\n\nThis repository contains the implementation of a language model built from scratch using PyTorch. The model leverages a transformer-based (decoder-only) architecture, similar to GPT.\n\n## Model Architecture\n\n- **Model Parameters**: 14.26 million\n- **Batch Size**: 256\n- **Block Size**: 64\n- **Max Iterations**: 10,000\n- **Learning Rate**: 3e-4\n- **Evaluation Iterations**: 100\n- **Embedding Size**: 384\n- **Number of Layers**: 12\n- **Number of Attention Heads**: 8\n- **Dropout Rate**: 0.2\n- **Context Window**: 64 characters\n- **Tokenizer**: Character-level\n- **Vocublary Size**: 145\n- **Training Data**: 2GB of open source textual data \n\n## Why Build a Language Model from Scratch?\n\nIn an era where we can easily import state-of-the-art language models (LLMs) like GPT, BERT, or T5, you might wonder why build one from scratch. Here’s why:\n\n1. **Deep Understanding**: Constructing a language model from scratch provides an in-depth understanding of the underlying mechanics of transformer architectures. This knowledge is crucial for debugging, fine-tuning, and optimizing models for specific tasks.\n2. **Customization**: Off-the-shelf models are often generalized to handle a broad range of tasks. Building a custom model allows for tailored adjustments, ensuring optimal performance for specialized applications.\n3. **Innovation**: By exploring the fundamental building blocks, we can experiment with novel architectures and techniques, potentially leading to innovative solutions and improvements beyond current state-of-the-art models.\n4. **Educational Value**: The process of designing and implementing a model from scratch is an invaluable learning experience. It sharpens problem-solving skills, deepens expertise in machine learning, and enhances coding proficiency.\n5. **Performance Optimization**: Understanding the intricacies of the model enables us to optimize its performance, memory usage, and speed, which is particularly beneficial for deploying models in resource-constrained environments.\n6. **Transparency**: Building from scratch ensures complete transparency and control over the model's architecture, data handling, and training process, which is essential for developing ethical and trustworthy AI solutions.\n\n## Getting Started\n\n### Prerequisites\n\n- Python 3.8+\n- PyTorch 1.8.1+\n\n### Installation\n\nClone the repository:\n\n```bash\ngit clone https://github.com/2003HARSH/Transformer-based-Decoder-only-Language-Model-from-Scratch.git\ncd Transformer-based-Decoder-only-Language-Model-from-Scratch\n```\n\nInstall the required packages:\n\n```bash\npip install -r requirements.txt\n```\n\n## Final Thoughts\n\nAlthough this model may not yet match the capabilities of the leading LLMs out there, the experience has been incredibly informative. It's particularly rewarding to see the character-level tokenizer accurately form meaningful words—a significant achievement in itself. Given enough data and computational power, this model has the potential to perform exceptionally well.\n\nFeel free to contribute to the project or use it as a learning tool for understanding transformer architectures and language model training from scratch.\n\n## Contributing\n\nWe welcome contributions to improve the model, add new features, or fix bugs. Please open an issue or submit a pull request.\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F2003harsh%2Ftransformer-based-decoder-only-language-model","html_url":"https://awesome.ecosyste.ms/projects/github.com%2F2003harsh%2Ftransformer-based-decoder-only-language-model","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F2003harsh%2Ftransformer-based-decoder-only-language-model/lists"}