https://github.com/pmixer/zan.initializer
Sample to show how input dim(input_dim for FC layer, for CNN, it's prod(tensor.shape[1:])) reciprocal could be used for initialize NN weights rather than sqrt(input_dim_reciprocal) hinted by Xavier:dragon:
https://github.com/pmixer/zan.initializer
cnn initial-nn-weights tensor xavier
Last synced: 3 months ago
JSON representation
Sample to show how input dim(input_dim for FC layer, for CNN, it's prod(tensor.shape[1:])) reciprocal could be used for initialize NN weights rather than sqrt(input_dim_reciprocal) hinted by Xavier:dragon:
- Host: GitHub
- URL: https://github.com/pmixer/zan.initializer
- Owner: pmixer
- Created: 2020-08-12T04:16:44.000Z (almost 5 years ago)
- Default Branch: master
- Last Pushed: 2020-08-12T04:26:55.000Z (almost 5 years ago)
- Last Synced: 2025-01-07T18:31:13.929Z (5 months ago)
- Topics: cnn, initial-nn-weights, tensor, xavier
- Language: Jupyter Notebook
- Homepage:
- Size: 67.4 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# zan.initializer
Sample to show how input dim(input_dim for FC layer, for CNN, it's prod(tensor.shape[1:])) reciprocal could be used for initial NN weights:dragon:
Sometimes more effective than xavier initializer in my experiments if without BatchNorm layers inside the network.
```
@misc{zan2017initializer,
title={Input Dimension Reciprocal for Effective Weight Initialization},
author={Huang, Zan},
year={2017},
publisher={GitHub},
howpublished={\url{https://github.com/pmixer/zan.initializer}},
}
```