https://github.com/yorkerlin/structuredngd-dl
Matrix-multiplication-only KFAC; Code for ICML 2023 paper on Simplifying Momentum-based Positive-definite Submanifold Optimization with Applications to Deep Learning
https://github.com/yorkerlin/structuredngd-dl
deep-learning fisher hessian-free kfac natural-gradients optimization-methods optimizer pytorch-implementation second-order-optimization
Last synced: 1 day ago
JSON representation
Matrix-multiplication-only KFAC; Code for ICML 2023 paper on Simplifying Momentum-based Positive-definite Submanifold Optimization with Applications to Deep Learning
- Host: GitHub
- URL: https://github.com/yorkerlin/structuredngd-dl
- Owner: yorkerlin
- Created: 2023-02-22T06:28:47.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2023-07-28T03:50:04.000Z (about 2 years ago)
- Last Synced: 2025-09-22T02:42:38.866Z (21 days ago)
- Topics: deep-learning, fisher, hessian-free, kfac, natural-gradients, optimization-methods, optimizer, pytorch-implementation, second-order-optimization
- Language: Python
- Homepage:
- Size: 40 KB
- Stars: 4
- Watchers: 1
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
PyTorch implementation of our [matrix-multiplication-only KFAC](http://github.com/yorkerlin/StructuredNGD-DL/blob/main/optimizers/local_cov.py) based on [Simplifying Momentum-based Positive-definite Submanifold Optimization with Applications to Deep Learning (ICML 2023)](https://arxiv.org/abs/2302.09738)
* Download the dataset from https://www.kaggle.com/datasets/ambityga/imagenet100
* The dataset should be saved in the [data](https://github.com/yorkerlin/StructuredNGD-DL/tree/main/data) folder (i.e., data/imagenet100/)
* You can reproduce the results on imagenet-100 by using the provided bash scripts
* Todo:
* Add code for optimization on SPD manifolds