https://github.com/aflah02/fsdp_experiments

https://github.com/aflah02/fsdp_experiments

Last synced: about 5 hours ago
JSON representation

• Host: GitHub
• URL: https://github.com/aflah02/fsdp_experiments
• Owner: aflah02
• Created: 2024-07-22T11:24:28.000Z (4 months ago)
• Default Branch: main
• Last Pushed: 2024-08-02T17:51:53.000Z (4 months ago)
• Last Synced: 2024-08-02T20:32:31.138Z (4 months ago)
• Language: HTML
• Size: 1.43 MB
• Stars: 1
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# FSDP Experiments

## Setup Env

I've used Conda for all my experiments and it has served me well. You may choose to use other environment managers such as Mamba, Poetry etc. and your Mileage may vary but any mature environment manager should do the job.

For Conda Users - 

- The python package details are present in `environment.yml`. The exact build details alongside python packages are present in `environment_with_build_info.yml`

- To use the env file run - `conda env create -f YAML_FILE_PATH`

## Running Instructions 

Since I am using Accelerate for DDP and FSDP, it is possible but hard to run it from the notebooks. So for the DDP and FSDP variants I have added `.py` files which can be run using `accelerate launch` commands.

- For FSDP - `accelerate launch --config_file fsdp_config.yaml train_model_fsdp.py`

- For DDP - `accelerate launch train_model_ddp.py`

All the file names clearly showcase the variant they present!