https://github.com/adrianscott/adamwtest
Testing AdamW vs experimental optimizer w/ extra adaptive smoothing
https://github.com/adrianscott/adamwtest
Last synced: 11 months ago
JSON representation
Testing AdamW vs experimental optimizer w/ extra adaptive smoothing
- Host: GitHub
- URL: https://github.com/adrianscott/adamwtest
- Owner: AdrianScott
- Created: 2025-07-31T18:21:00.000Z (11 months ago)
- Default Branch: master
- Last Pushed: 2025-07-31T18:34:59.000Z (11 months ago)
- Last Synced: 2025-07-31T21:56:49.948Z (11 months ago)
- Language: Python
- Homepage:
- Size: 34.2 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# AdamW-McGinley Dynamic Optimizer
An implementation of an AdamW variant using McGinley Dynamic-inspired adaptive momentum smoothing. This optimizer dynamically adapts the beta smoothing coefficients based on gradient change rates, potentially offering better performance on non-stationary or noisy training regimes.
## Concept
The McGinley Dynamic indicator from technical analysis proposes adaptive smoothing based on the rate of change of the signal. This project applies that concept to the AdamW optimizer's beta coefficients, dynamically adjusting them based on gradient changes:
- Instead of fixed β₁ and β₂, we use dynamic β₁ₜ and β₂ₜ computed per step
- β₁ₜ adapts based on gradient changes: β₁ₜ = β₁ / (1 + |grad_change|²)
- β₂ₜ adapts based on squared gradient changes: β₂ₜ = β₂ / (1 + |grad_sq_change|²)
- Minimum thresholds for beta values ensure stability
## Features
- `CustomAdamW` optimizer with dynamic smoothing capability
- Optional global gradient norm scaling
- Beta statistics logging integration with Weights & Biases
- Comprehensive sweep infrastructure for comparative analysis
## Installation
To install the required dependencies, run:
```bash
pip install -r requirements.txt
```
Or set up a virtual environment:
```bash
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
pip install -r requirements.txt
```
## Usage
### Basic Usage
```python
from custom_optimizer import CustomAdamW
# Create optimizer with dynamic smoothing
optimizer = CustomAdamW(
model.parameters(),
lr=0.001,
weight_decay=0.01,
dynamic_smoothing=True, # Enable dynamic betas
min_beta1=0.5, # Minimum value for beta1
min_beta2=0.9, # Minimum value for beta2
log_betas=True # Track beta statistics
)
```
### Running Experiments
Before running experiments, make sure to set up your Weights & Biases API key:
```bash
export WANDB_API_KEY=your_api_key_here
```
To run experiments comparing CustomAdamW to standard AdamW:
```bash
python sweep_train.py
```
This launches a Weights & Biases sweep that compares both optimizers across different configurations.
## Results
### Sample Comparison Test
Full experiment results for this sweep on Weights & Biases are available; contact me for access.
## Project Structure
- `custom_optimizer.py`: Implementation of CustomAdamW optimizer
- `sweep_train.py`: Main training and sweep script for CIFAR-10 experiments
- `adamw-mcginley-dynamic-optimizer.md`: Detailed explanation of the algorithm
## License
Copyright (c) 2025 Adrian Scott.