https://github.com/sayakpaul/barlow-twins-tf

TensorFlow implementation of Barlow Twins (https://arxiv.org/abs/2103.03230).
https://github.com/sayakpaul/barlow-twins-tf

computer-vision information-bottleneck-theory representation-learning self-supervised-learning tensorflow2

Last synced: 24 days ago
JSON representation

TensorFlow implementation of Barlow Twins (https://arxiv.org/abs/2103.03230).

• Host: GitHub
• URL: https://github.com/sayakpaul/barlow-twins-tf
• Owner: sayakpaul
• License: apache-2.0
• Created: 2021-05-10T11:41:00.000Z (over 3 years ago)
• Default Branch: main
• Last Pushed: 2021-06-11T10:21:44.000Z (over 3 years ago)
• Last Synced: 2024-10-03T12:37:50.390Z (about 1 month ago)
• Topics: computer-vision, information-bottleneck-theory, representation-learning, self-supervised-learning, tensorflow2
• Language: Jupyter Notebook
• Homepage:
• Size: 479 KB
• Stars: 41
• Watchers: 2
• Forks: 5
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# Barlow-Twins-TF








This repository implements **Barlow Twins** ([Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://arxiv.org/abs/2103.03230)) in TensorFlow and demonstrates it on the CIFAR10 dataset.

**Summary**:

With a ResNet20 as a trunk and a 3-layer MLP (each layer containing 2048 units) and 100 epochs of pre-training, [this training notebook](https://github.com/sayakpaul/Barlow-Twins-TF/blob/main/Barlow_Twins.ipynb) can give **62.61%** accuracy on the CIFAR10 test set. The pre-training total takes ~23 minutes on a single Tesla V100. There are minor differences from the [original implementation](https://github.com/facebookresearch/barlowtwins/). However, the original loss function and the other minor details like having a big enough projection dimension have been maintained.

For details on Barlow Twins, I suggest reading the original paper, it's really well-written. 

## Loss progress during pre-training



  



## Other notes

* Pre-trained model is available [here](https://github.com/sayakpaul/Barlow-Twins-TF/releases/download/v1.0.0/barlow_twins.tar.gz). 

* To follow the original implementation details as closely as possible, a WarmUpCosine learning rate schedule has been used during pre-training:

  

  


    

  

* During linear evaluation, Cosine Decay has been used. 


## Acknowledgements

Thanks to Stéphane Deny (one of the authors of the paper) for helping me catch a pesky bug.