https://github.com/fkodom/transformer-from-scratch

Code implementation from my blog post: https://fkodom.substack.com/p/transformers-from-scratch-in-pytorch
https://github.com/fkodom/transformer-from-scratch

Last synced: about 2 months ago
JSON representation

Code implementation from my blog post: https://fkodom.substack.com/p/transformers-from-scratch-in-pytorch

• Host: GitHub
• URL: https://github.com/fkodom/transformer-from-scratch
• Owner: fkodom
• License: mit
• Created: 2021-12-23T16:34:57.000Z (over 2 years ago)
• Default Branch: main
• Last Pushed: 2023-08-07T15:25:30.000Z (about 1 year ago)
• Last Synced: 2024-06-01T20:41:33.613Z (4 months ago)
• Language: Python
• Homepage: https://fkodom.substack.com/p/transformers-from-scratch-in-pytorch
• Size: 10.7 KB
• Stars: 88
• Watchers: 3
• Forks: 19
• Open Issues: 1
• Metadata Files:
  • Readme: README.md
  • Funding: .github/FUNDING.yml
  • License: LICENSE

Awesome Lists containing this project

README

        
# transformer-from-scratch

Code for my blog post: [Transformers from Scratch in PyTorch](https://fkodom.substack.com/p/transformers-from-scratch-in-pytorch)

**Note:**  This Transformer code does **not** include masked attention.  That was intentional, because it led to a much cleaner implementation.  This repository is intended for educational purposes only.  I believe that everything here is correct, but make no guarantees if for some reason you decide to use it in your own project.

## Citations

```

@misc{vaswani2023attention,

      title={Attention Is All You Need}, 

      author={Ashish Vaswani and Noam Shazeer and Niki Parmar and Jakob Uszkoreit and Llion Jones and Aidan N. Gomez and Lukasz Kaiser and Illia Polosukhin},

      year={2023},

      eprint={1706.03762},

      archivePrefix={arXiv},

      primaryClass={cs.CL}

}

```