https://github.com/rasbt/pytorch-memory-optim

This code repository contains the code used for my "Optimizing Memory Usage for Training LLMs and Vision Transformers in PyTorch" blog post.
https://github.com/rasbt/pytorch-memory-optim

deep-learning llm memory-optimization pytorch vision vit

Last synced: about 2 months ago
JSON representation

This code repository contains the code used for my "Optimizing Memory Usage for Training LLMs and Vision Transformers in PyTorch" blog post.

• Host: GitHub
• URL: https://github.com/rasbt/pytorch-memory-optim
• Owner: rasbt
• License: apache-2.0
• Created: 2023-06-27T18:16:56.000Z (over 1 year ago)
• Default Branch: main
• Last Pushed: 2023-07-14T20:11:55.000Z (over 1 year ago)
• Last Synced: 2024-10-23T11:38:51.434Z (2 months ago)
• Topics: deep-learning, llm, memory-optimization, pytorch, vision, vit
• Language: Python
• Homepage: https://lightning.ai/pages/community/tutorial/pytorch-memory-vit-llm/
• Size: 1.14 MB
• Stars: 84
• Watchers: 4
• Forks: 11
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE.txt

Awesome Lists containing this project

README

        
# Optimizing PyTorch Memory Usage

This code repository contains the code used for my "Optimizing Memory Usage for Training LLMs and Vision Transformers in PyTorch" blog post. 

You can install the dependencies via

```bash

pip install -r requirements.txt

```

The scripts are all standalone scripts and can be run by executing 

```bash

python 01_pytorch-vit.py

```

and so forth. The only requirement is to have the [`local_utilities.py`](local_utilities.py) in the same folder as the script as it contains some data loading utilities that are reused across all the scripts.

I tracked the script outputs in the [`logs.md`](logs.md) file.

![overview](figures/overview.png)