https://github.com/kvignesh1420/single-index-ht
[TMLR] From Spikes to Heavy Tails: Unveiling the Spectral Evolution of Neural Networks
https://github.com/kvignesh1420/single-index-ht
adam-optimizer gradient-descent heavy-tailed-distributions neural-networks spectral-analysis
Last synced: 9 days ago
JSON representation
[TMLR] From Spikes to Heavy Tails: Unveiling the Spectral Evolution of Neural Networks
- Host: GitHub
- URL: https://github.com/kvignesh1420/single-index-ht
- Owner: kvignesh1420
- Created: 2023-12-14T02:34:01.000Z (almost 2 years ago)
- Default Branch: main
- Last Pushed: 2025-08-16T06:58:51.000Z (about 2 months ago)
- Last Synced: 2025-08-16T08:38:18.831Z (about 2 months ago)
- Topics: adam-optimizer, gradient-descent, heavy-tailed-distributions, neural-networks, spectral-analysis
- Language: Python
- Homepage: https://arxiv.org/abs/2406.04657
- Size: 103 KB
- Stars: 4
- Watchers: 2
- Forks: 0
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
## From Spikes to Heavy Tails: Unveiling the Spectral Evolution of Neural Networks
This repository explores various methods to generate heavy tails in the weight matrix spectrum of neural networks without the influence of gradient noise. We specifically train shallow neural networks using full-batch Gradient Descent (GD) or Adam optimizer with large learning rates over multiple steps.
## Setup
To get started, set up your virtual environment and install the required dependencies:
```bash
$ python3.9 -m venv .venv
$ source .venv/bin/activate
$ pip install -r requirements.txt
```
## Experiments
### Single Configuration Runs
Investigate the properties of weights, features, overlap matrices, and more for a single configuration:
```bash
(.venv) $ python main.py configs/main.yml
```
To run with a learning rate schedule:
```bash
(.venv) $ python main.py configs/main_lr_schedule.yml
```
### Varying Learning Rates for GD/Adam
Conduct experiments with multiple runs to plot losses, Kernel Target Alignment (KTA), and Power Law (PL) Alphas for different learning rates and optimizers:
```bash
(.venv) $ python bulk_lr.py configs/bulk_lr.yml
```
### Losses with Varying Parameters
Perform experiments with multiple runs to plot the losses for different parameter settings:
#### Varying Dataset Size: `n`
```bash
(.venv) $ python bulk_losses.py configs/bulk_losses_vary_n.yml
```
#### Varying Regularization Parameter for Regression: `reg_lambda`
```bash
(.venv) $ python bulk_losses.py configs/bulk_losses_vary_reg_lambda.yml
```
#### Varying Label Noise: `label_noise_std`
```bash
(.venv) $ python bulk_losses.py configs/bulk_losses_vary_label_noise_std.yml
```
#### Varying Decay Factor of `StepLR` Learning Rate Schedule: `gamma`
```bash
(.venv) $ python bulk_losses.py configs/bulk_losses_vary_step_lr_gamma.yml
```
### Output
The outputs of the experiments are stored in the `out/` directory, named according to a hash value based on the experiment context.
## Citation
```bibtex
@article{kothapalli2025from,
title={From Spikes to Heavy Tails: Unveiling the Spectral Evolution of Neural Networks},
author={Vignesh Kothapalli and Tianyu Pang and Shenyang Deng and Zongmin Liu and Yaoqing Yang},
journal={Transactions on Machine Learning Research},
issn={2835-8856},
year={2025},
url={https://openreview.net/forum?id=DJHB8eBUnt},
note={}
}
```