{"id":26671392,"url":"https://github.com/prateeknigam9/transformers-from-scratch","last_synced_at":"2026-05-14T13:32:47.476Z","repository":{"id":258562576,"uuid":"865493229","full_name":"prateeknigam9/transformers-from-scratch","owner":"prateeknigam9","description":"This repository showcases an encoder-only transformer model built from scratch, focusing on intent classification.","archived":false,"fork":false,"pushed_at":"2024-10-25T15:38:49.000Z","size":74486,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-04T12:05:07.228Z","etag":null,"topics":["attention-mechanism","encoder-decoder-model","intent-classification","llm","pytorch","transformers"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/prateeknigam9.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-30T16:10:41.000Z","updated_at":"2024-10-25T15:38:53.000Z","dependencies_parsed_at":"2024-10-24T07:12:25.614Z","dependency_job_id":null,"html_url":"https://github.com/prateeknigam9/transformers-from-scratch","commit_stats":null,"previous_names":["prateeknigam9/transformers-from-scratch"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/prateeknigam9/transformers-from-scratch","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prateeknigam9%2Ftransformers-from-scratch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prateeknigam9%2Ftransformers-from-scratch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prateeknigam9%2Ftransformers-from-scratch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prateeknigam9%2Ftransformers-from-scratch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/prateeknigam9","download_url":"https://codeload.github.com/prateeknigam9/transformers-from-scratch/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prateeknigam9%2Ftransformers-from-scratch/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33026846,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-13T13:14:54.681Z","status":"online","status_checked_at":"2026-05-14T02:00:06.663Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["attention-mechanism","encoder-decoder-model","intent-classification","llm","pytorch","transformers"],"created_at":"2025-03-25T23:50:11.269Z","updated_at":"2026-05-14T13:32:47.418Z","avatar_url":"https://github.com/prateeknigam9.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Transformers from Scratch\n\nHi!\nThis repository contains an implementation of transformer models built from scratch, divided into three main components: \n1. **Encoder-Only Model** \n2. **Decoder-Only Model** \n3. **Combined Encoder-Decoder Model**\n\nTransformers have revolutionized natural language processing and are foundational to many state-of-the-art models. This project aims to provide a clear and modular implementation of transformers, inspired by the work of \n[Sebastian Raschka.](https://sebastianraschka.com/)\n\n## Motivation\n\nI started this project to better understand how transformers work. I wanted to learn the details of each step involved in building a transformer model. By coding everything from scratch, I could see how the different parts fit together and how they function.\n\nMy goal was to become more skilled at working with transformers so that I could apply this knowledge to real-world tasks and research. By understanding the inner workings of transformers, I can also explore ways to improve or customize them for specific needs. Overall, this project helped me gain a deeper appreciation for this important technology in natural language processing.\n\n## Features - \n\n**Modular Code**: Each component of the transformer is implemented in a modular format for easy understanding and extensibility. \n**Customizable Architecture**: The models can be easily modified to suit specific tasks or research interests.\n\n## Directory Structure\n\n/transformers-from-scratch\n│\n├── encoder_only/\n│   ├── model.py\n│   ├── train.py\n│   └── evaluate.py\n│\n├── decoder_only/\n│   ├── model.py\n│   ├── train.py\n│   └── evaluate.py\n│\n└── combined_model/\n    ├── model.py\n    ├── train.py\n    └── evaluate.py\n\n\n\nThankyou!\n[Prateek Nigam](https://prateeknigam9.github.io/PrateekNigam)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprateeknigam9%2Ftransformers-from-scratch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fprateeknigam9%2Ftransformers-from-scratch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprateeknigam9%2Ftransformers-from-scratch/lists"}