https://github.com/mukeshmithrakumar/llm-poc-2024
Popular Large Language Models from scratch - 2024
https://github.com/mukeshmithrakumar/llm-poc-2024
gpt llama llm llm-inference llm-training transformer
Last synced: 25 days ago
JSON representation
Popular Large Language Models from scratch - 2024
- Host: GitHub
- URL: https://github.com/mukeshmithrakumar/llm-poc-2024
- Owner: mukeshmithrakumar
- License: mit
- Created: 2024-04-03T19:40:34.000Z (about 2 years ago)
- Default Branch: main
- Last Pushed: 2024-06-14T21:24:00.000Z (almost 2 years ago)
- Last Synced: 2024-10-18T23:12:35.830Z (over 1 year ago)
- Topics: gpt, llama, llm, llm-inference, llm-training, transformer
- Language: Jupyter Notebook
- Homepage:
- Size: 9.7 MB
- Stars: 1
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
Large Language Models POC 2024
🚧 🚀 WIP... 🚧
About |
Models |
Helpful Notebooks |
Requirements |
License |
Author
## :dart: About
My goal is to work my way through certain LLM models starting from Transformers to help me understand how each model works and builds from the previous models. Once I have a set of models I am interested in, the next focus will be on fine-tuning and optimizing the models to run on the cheapest hardware possible.
## :sparkles: Models
✅ Transformer
✅ GPT
✅ LLaMA
◻️ LLM Inference Optimization
◻️ In-flight Batching
◻️ Speculative inference
◻️ Key-Value Caching
◻️ PagedAttention
◻️ Pipeline Parallelism
◻️ Tensor Parallelism
◻️ Sequence Parallelism
◻️ Flash Attention
◻️ Quantization
◻️ Sparsity
◻️ Distillation
◻️
◻️
## :fire: Helpful Notebooks
✅ Transformer Arithmetic
✅ [WIP] Transformer Scaling
◻️
## :test_tube: Requirements
Requirements for all the models are stored under a single `requirements.txt` file.
## :memo: License
This project is under license from MIT. For more details, see the [LICENSE](LICENSE) file.
---
Made with :heart: by Mukesh Mithrakumar