https://github.com/facebookresearch/projUNN

Fast training of unitary deep network layers from low-rank updates
https://github.com/facebookresearch/projUNN

Last synced: 14 days ago
JSON representation

Fast training of unitary deep network layers from low-rank updates

• Host: GitHub
• URL: https://github.com/facebookresearch/projUNN
• Owner: facebookresearch
• License: mit
• Archived: true
• Created: 2022-02-18T16:00:55.000Z (over 2 years ago)
• Default Branch: main
• Last Pushed: 2022-12-11T02:33:28.000Z (almost 2 years ago)
• Last Synced: 2024-08-01T16:52:47.324Z (3 months ago)
• Language: Python
• Homepage:
• Size: 113 KB
• Stars: 27
• Watchers: 49
• Forks: 6
• Open Issues: 2
• Metadata Files:
  • Readme: README.md
  • Contributing: CONTRIBUTING.md
  • License: LICENSE
  • Code of conduct: CODE_OF_CONDUCT.md

Awesome Lists containing this project

README

        
# ProjUNN: efficient learning of orthogonal or unitary weights by low-rank updates

## Convolutional orthogonality/unitarity constraints

To run the convolution case simply run

> python convolutional_experiment.py --unitary --dataset MNIST -lr 0.001 --projector projUNNT --optimizer SGD

with the desired settings. the projector option can be either `projUNND` or `projUNNT` for the two methods we proposed in the paper. The optimizer is either `SGD` or `RMSProp`. Note that the otpimizers (other than simple SGD) need to be rewritten to be sure that terms such as momentum etc. are computed on the gradients/projected gradients but that the update to the weights is projected (to ensure that the weights stay on the orthogonal/unitary manifolds). As of now, this code simply runs the Resnet9 model on `MNIST`, `CIFAR10`, and `CIFAR100`.

To run without the unitary constraint simply remove the `--unitary` flag.

## Requirements

This software only requires `pytorch` and all its dependencies.

## TERMS OF USE & PRIVACY POLICY

- Terms of Use - https://opensource.facebook.com/legal/terms

- Privacy Policy - https://opensource.facebook.com/legal/privacy

## COPYRIGHT STATEMENT

Copyright © 2022 Meta Platforms, Inc

## LICENSE

MIT