https://github.com/oelin/bitmixer
An implementation of BitMixer in PyTorch.
https://github.com/oelin/bitmixer
Last synced: 3 months ago
JSON representation
An implementation of BitMixer in PyTorch.
- Host: GitHub
- URL: https://github.com/oelin/bitmixer
- Owner: oelin
- Created: 2024-02-27T23:12:37.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-03-01T19:57:04.000Z (over 1 year ago)
- Last Synced: 2024-03-01T20:50:34.751Z (over 1 year ago)
- Language: Python
- Size: 7.81 KB
- Stars: 0
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# BitMixer
An implementation of BitMixer in PyTorch. BitMixer is an MLP-Mixer architecture that uses `tanh` for soft [binarization](https://arxiv.org/abs/1602.02830). During inference, the model's weights and activations can be quantized to 1-bit.
Also, check out this recent paper exploring binarized transformers: [https://arxiv.org/abs/2310.11453](https://arxiv.org/abs/2310.11453).