https://github.com/gunale0926/grams
Grams: Gradient Descent with Adaptive Momentum Scaling (ICLR 2025 Workshop)
https://github.com/gunale0926/grams
adam-optimizer deep-learning machine-learning optimization optimizer pytorch
Last synced: about 1 year ago
JSON representation
Grams: Gradient Descent with Adaptive Momentum Scaling (ICLR 2025 Workshop)
- Host: GitHub
- URL: https://github.com/gunale0926/grams
- Owner: Gunale0926
- License: apache-2.0
- Created: 2024-12-24T12:39:08.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2025-03-06T02:13:42.000Z (over 1 year ago)
- Last Synced: 2025-05-06T21:06:23.914Z (about 1 year ago)
- Topics: adam-optimizer, deep-learning, machine-learning, optimization, optimizer, pytorch
- Language: Python
- Homepage: https://arxiv.org/abs/2412.17107
- Size: 38.1 KB
- Stars: 11
- Watchers: 1
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Grams: Gradient Descent with Adaptive Momentum Scaling (ICLR 2025 SCOPE Workshop)
[](https://arxiv.org/abs/2412.17107) [](https://pypi.org/project/grams-pytorch/)
Authors: [Yang Cao](https://scholar.google.com/citations?user=pCrKkUQAAAAJ), [Xiaoyu Li](https://scholar.google.com/citations?hl=en&user=WgiSk4AAAAAJ), [Zhao Song](https://scholar.google.com/citations?user=yDZct7UAAAAJ)
This repository contains the official PyTorch implementation for Grams optimizer.
We introduce Gradient Descent with Adaptive Momentum Scaling (Grams), a novel optimization algorithm that decouples the direction and magnitude of parameter updates in deep learning. Unlike traditional optimizers that directly integrate momentum into updates, Grams separates the update direction, derived from current gradients, from momentum, which is used solely for adaptive magnitude scaling. This approach enables Grams to achieve improved loss descent compared to state-of-the-art cautious and momentum-based optimizers.
## Install
Use the following command to install our pytorch implementation for Grams:
```bash
pip install grams-pytorch
```
## How to use Grams
Switching from Adam/AdamW to Grams is simple and requires only two lines of code:
Before:
```python
import torch
optimizer = torch.optim.adam(model.parameters(), lr=1e-3, weight_decay=0.0)
```
Switching to Grams:
```python
from grams import Grams
optimizer = Grams(model.parameters(), lr=1e-3, weight_decay=0.0)
```
Just import Grams and swap the optimizer—everything else remains the same!
## Citation
Please cite our work!
```bibtex
@inproceedings{cao2025grams,
title={Grams: Gradient Descent with Adaptive Momentum Scaling},
author={Yang Cao and Xiaoyu Li and Zhao Song},
booktitle={ICLR 2025 First Workshop on Scalable Optimization for Efficient and Adaptive Foundation Models},
year={2025},
url={https://openreview.net/forum?id=GmKQnpQdsc}
}
```