https://github.com/giakoumoglou/synco
[arXiv 2024] PyTorch implementation of SynCo: https://arxiv.org/abs/2410.02401
https://github.com/giakoumoglou/synco
computer-vision contrastive-learning deep-learning pytorch representation-learning self-supervised-learning unsupervised-learning
Last synced: 4 months ago
JSON representation
[arXiv 2024] PyTorch implementation of SynCo: https://arxiv.org/abs/2410.02401
- Host: GitHub
- URL: https://github.com/giakoumoglou/synco
- Owner: giakoumoglou
- License: mit
- Created: 2024-06-29T21:45:08.000Z (almost 2 years ago)
- Default Branch: main
- Last Pushed: 2025-03-19T14:55:10.000Z (over 1 year ago)
- Last Synced: 2025-04-14T17:31:50.212Z (about 1 year ago)
- Topics: computer-vision, contrastive-learning, deep-learning, pytorch, representation-learning, self-supervised-learning, unsupervised-learning
- Language: Python
- Homepage: https://giakoumoglou.com/src/synco.pdf
- Size: 1 MB
- Stars: 2
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
## _SynCo_: Synthetic Hard Negatives for Contrastive Visual Representation Learning
This is a PyTorch implementation of the [SynCo paper](https://arxiv.org/abs/2410.02401):
```
@misc{giakoumoglou2024synco,
author = {Nikolaos Giakoumoglou and Tania Stathaki},
title = {SynCo: Synthetic Hard Negatives for Contrastive Visual Representation Learning},
journal = {arXiv preprint arXiv:2410.02401},
year = {2024},
}
```
### Preparation
Install PyTorch and ImageNet dataset following the [official PyTorch ImageNet training code](https://github.com/pytorch/examples/tree/master/imagenet).
This repo is based on [MoCo v2](https://github.com/facebookresearch/moco) and [Barlow Twins](https://github.com/facebookresearch/barlowtwins) code:
```
diff main_synco.py <(curl https://raw.githubusercontent.com/facebookresearch/moco/main_moco.py)
diff main_lincls.py <(curl https://raw.githubusercontent.com/facebookresearch/moco/main_lincls.py)
diff main_semisup.py <(curl https://raw.githubusercontent.com/facebookresearch/barlowtwins/evaluate.py)
```
### Unsupervised Training
This implementation only supports **multi-gpu**, **DistributedDataParallel** training, which is faster and simpler; single-gpu or DataParallel training is not supported.
To do unsupervised pre-training of a ResNet-50 model on ImageNet in an 8-gpu machine, run:
```
python main_synco.py \
-a resnet50 \
--lr 0.03 \
--batch-size 256 \
--dist-url 'tcp://localhost:10001' --multiprocessing-distributed --world-size 1 --rank 0 \
--mlp --moco-t 0.2 --aug-plus --cos \
--n-hard 1024 --n1 256 --n2 256 --n3 256 --n4 64 --n5 64 --n6 64 \
[your imagenet-folder with train and val folders]
```
This script uses all the default hyper-parameters as described in the [MoCo v2 paper](https://arxiv.org/abs/1911.05722).
### Linear Classification
With a pre-trained model, to train a supervised linear classifier on frozen features/weights in an 8-gpu machine, run:
```
python main_lincls.py \
-a resnet50 \
--lr 30.0 \
--batch-size 256 \
--pretrained [your checkpoint path]/checkpoint_0199.pth.tar \
--dist-url 'tcp://localhost:10001' --multiprocessing-distributed --world-size 1 --rank 0 \
[your imagenet-folder with train and val folders]
```
This script uses all the default hyper-parameters as described in the [MoCo v2 paper](https://arxiv.org/abs/1911.05722).
### Semi-supervised Learning
To fine-tune the model end-to-end, including training a linear classifier on features/weights using a pre-trained model on an 8-GPU machine with a subset of the ImageNet training set, run:
```
python main_semisup.py \
-a resnet50 \
--lr-backbone [YOUR_LR] --lr-classifier [YOUR_LR] \
--train-percent 1 --weights finetune \
--batch-size 256 \
--pretrained [your checkpoint path]/checkpoint_0199.pth.tar \
--dist-url 'tcp://localhost:10001' --multiprocessing-distributed --world-size 1 --rank 0 \
[your imagenet-folder with train and val folders]
```
### Transferring to Object Detection
See [./detection](detection).
### Models
Our pre-trained ResNet-50 models can be downloaded as follows:
epochs
top-1 acc.
model
### License
This project is under the CC-BY-NC 4.0 license. See [LICENSE](LICENSE) for details.