{"id":13709092,"url":"https://github.com/wgcban/mix-bt","last_synced_at":"2025-09-13T00:31:50.150Z","repository":{"id":210763129,"uuid":"724589834","full_name":"wgcban/mix-bt","owner":"wgcban","description":"Official PyTorch Implementation of Guarding Barlow Twins Against Overfitting with Mixed Samples","archived":false,"fork":false,"pushed_at":"2024-01-19T20:23:06.000Z","size":9095,"stargazers_count":13,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-08-03T23:14:49.927Z","etag":null,"topics":["barlow-twins","byol","cifar-10","cifar-100","contrastive-learning","imagenet","knn-classification","linear-probing","mixup","representation-learning","self-supervised-learning","simclr","stl-10","tiny-imagenet"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2312.02151","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wgcban.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-11-28T11:48:41.000Z","updated_at":"2024-05-23T22:45:10.000Z","dependencies_parsed_at":"2023-12-11T17:46:38.031Z","dependency_job_id":"953ebf0b-f7d6-44ed-84ac-d34578ce0a64","html_url":"https://github.com/wgcban/mix-bt","commit_stats":null,"previous_names":["wgcban/mix-bt"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wgcban%2Fmix-bt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wgcban%2Fmix-bt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wgcban%2Fmix-bt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wgcban%2Fmix-bt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wgcban","download_url":"https://codeload.github.com/wgcban/mix-bt/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":232802752,"owners_count":18578684,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["barlow-twins","byol","cifar-10","cifar-100","contrastive-learning","imagenet","knn-classification","linear-probing","mixup","representation-learning","self-supervised-learning","simclr","stl-10","tiny-imagenet"],"created_at":"2024-08-02T23:00:35.824Z","updated_at":"2025-01-06T23:51:23.828Z","avatar_url":"https://github.com/wgcban.png","language":"Python","funding_links":[],"categories":["Beyond Vision"],"sub_categories":["**Other**"],"readme":"# Mixed Barlow Twins for Self-Supervised Representation Learning\n\u003e[**Guarding Barlow Twins Against Overfitting with Mixed Samples**](https://arxiv.org/abs/2312.02151)\u003cbr\u003e\n\n\u003e[![arXiv](https://img.shields.io/badge/arXiv-2312.02151-b31b1b)](https://arxiv.org/abs/2312.02151)\n\u003e[![Hugging Face Model Card](https://img.shields.io/badge/Model%20Card-Hugging%20Face-%2334D058)](https://huggingface.co/wgcban/mix-bt)\n\n\n[Wele Gedara Chaminda Bandara](https://www.wgcban.com) (Johns Hopkins University), [Celso M. De Melo](https://celsodemelo.net) (U.S. Army Research Laboratory), and [Vishal M. Patel](https://engineering.jhu.edu/vpatel36/) (Johns Hopkins University) \u003cbr\u003e\n\n## 1 Overview of Mixed Barlow Twins\n\nTL;DR\n- Mixed Barlow Twins aims to improve sample interaction during Barlow Twins training via linearly interpolated samples. \n- We introduce an additional regularization term to the original Barlow Twins objective, assuming linear interpolation in the input space translates to linearly interpolated features in the feature space.\n- Pre-training with this regularization effectively mitigates feature overfitting and further enhances the downstream performance on `CIFAR-10`, `CIFAR-100`, `TinyImageNet`, `STL-10`, and `ImageNet` datasets.\n\n\u003cimg src=\"figs/mix-bt.svg\" width=\"1024\"\u003e\n\n$C^{MA} = (Z^M)^TZ^A$\n\n$C^{MB} = (Z^M)^TZ^B$\n\n$C^{MA}_{gt} = \\lambda (Z^A)^TZ^A + (1-\\lambda)\\mathtt{Shuffle}^*(Z^B)^TZ^A$\n\n$C^{MB}_{gt} = \\lambda (Z^A)^TZ^B + (1-\\lambda)\\mathtt{Shuffle}^*(Z^B)^TZ^B$\n\n## 2 Usage\n### 2.1 Requirements\n\nBefore using this repository, make sure you have the following prerequisites installed:\n\n- [Anaconda](https://www.anaconda.com/download/)\n- [PyTorch](https://pytorch.org)\n\nYou can install PyTorch with the following [command](https://pytorch.org/get-started/locally/) (in Linux OS):\n```bash\nconda install pytorch torchvision pytorch-cuda=11.8 -c pytorch -c nvidia\n```\n\n### 2.2 Installation\n\nTo get started, clone this repository:\n```bash\ngit clone https://github.com/wgcban/mix-bt.git\n```\n\nNext, create the [conda](https://docs.conda.io/projects/conda/en/stable/) environment named `ssl-aug` by executing the following command:\n```bash\nconda env create -f environment.yml\n```\n\nAll the train-val-test statistics will be automatically upload to [`wandb`](https://wandb.ai/home), and please refer [`wandb-quick-start`](https://wandb.ai/quickstart?utm_source=app-resource-center\u0026utm_medium=app\u0026utm_term=quickstart) documentation if you are not familiar with using `wandb`. \n\n### 2.3 Supported Pre-training Datasets\n\nThis repository supports the following pre-training datasets:\n- `CIFAR-10`: https://www.cs.toronto.edu/~kriz/cifar.html\n- `CIFAR-100`: https://www.cs.toronto.edu/~kriz/cifar.html\n- `Tiny-ImageNet`: https://github.com/rmccorm4/Tiny-Imagenet-200\n- `STL-10`: https://cs.stanford.edu/~acoates/stl10/\n- `ImageNet`: https://www.image-net.org\n\n`CIFAR-10`, `CIFAR-100`, and `STL-10` datasets are directly available in PyTorch. \n\nTo use `TinyImageNet`, please follow the preprocessing instructions provided in the [TinyImageNet-Script](https://gist.github.com/moskomule/2e6a9a463f50447beca4e64ab4699ac4). Download these datasets and place them in the `data` directory.\n\n### 2.4 Supported Transfer Learning Datasets\nYou can download and place transfer learning datasets under their respective paths, such as 'data/DTD'. The supported transfer learning datasets include:\n- `DTD`: https://www.robots.ox.ac.uk/~vgg/data/dtd/ \n- `MNIST`: http://yann.lecun.com/exdb/mnist/\n- `FashionMNIST`: https://github.com/zalandoresearch/fashion-mnist\n- `CUBirds`: http://www.vision.caltech.edu/visipedia/CUB-200-2011.html\n- `VGGFlower`: https://www.robots.ox.ac.uk/~vgg/data/flowers/102/\n- `Traffic Signs`: https://benchmark.ini.rub.de/gtsdb_dataset.html\n- `Aircraft`: https://www.robots.ox.ac.uk/~vgg/data/fgvc-aircraft/\n\n### 2.5 Supported SSL Methods\n\nThis repository supports the following Self-Supervised Learning (SSL) methods:\n\n- [`SimCLR`](https://arxiv.org/abs/2002.05709): contrastive learning for SSL \n- [`BYOL`](https://arxiv.org/abs/2006.07733): distilation for SSL\n- [`Witening MSE`](http://proceedings.mlr.press/v139/ermolov21a/ermolov21a.pdf): infomax for SSL\n- [`Barlow Twins`](https://arxiv.org/abs/2103.03230): infomax for SSL\n- **`Mixed Barlow Twins (ours)`**: infomax + mixed samples for SSL\n\n### 2.6 Pre-Training with Mixed Barlow Twins\nTo start pre-training and obtain k-NN evaluation results for Mixed Barlow Twins on `CIFAR-10`, `CIFAR-100`, `TinyImageNet`, and `STL-10` with `ResNet-18/50` backbones, please run:\n```bash\nsh scripts-pretrain-resnet18/[dataset].sh\n```\n```bash\nsh scripts-pretrain-resnet50/[dataset].sh\n```\n\nTo start the pre-training on `ImageNet` with `ResNet-50` backbone, please run:\n```bash\nsh scripts-pretrain-resnet18/imagenet.sh\n```\n\n### 2.7 Linear Evaluation of Pre-trained Models\nBefore running linear evaluation, *ensure that you specify the `model_path` argument correctly in the corresponding .sh file*. \n\nTo obtain linear evaluation results on `CIFAR-10`, `CIFAR-100`, `TinyImageNet`, `STL-10` with `ResNet-18/50` backbones, please run:\n```bash\nsh scripts-linear-resnet18/[dataset].sh\n```\n```bash\nsh scripts-linear-resnet50/[dataset].sh\n```\n\nTo obtain linear evaluation results on `ImageNet` with `ResNet-50` backbone, please run:\n```bash\nsh scripts-linear-resnet50/imagenet_sup.sh\n```\n\n\n### 2.8 Transfer Learning of Pre-trained Models\nTo perform transfer learning from pre-trained models on `CIFAR-10`, `CIFAR-100`, and `STL-10` to fine-grained classification datasets, execute the following command, making sure to specify the `model_path` argument correctly:\n```bash\nsh scripts-transfer-resnet18/[dataset]-to-x.sh\n```\n\n## 3 Pre-Trained Checkpoints\nDownload the pre-trained models from [GitHub (Releases v1.0.0)](https://github.com/wgcban/mix-bt/releases/tag/v1.0.0) and store them in `checkpoints/`. This repository provides pre-trained checkpoints for both [`ResNet-18`](https://arxiv.org/abs/1512.03385) and [`ResNet-50`](https://arxiv.org/abs/1512.03385) architectures.\n\n#### 3.1 ResNet-18 \\[`CIFAR-10`, `CIFAR-100`, `TinyImageNet`, and `STL-10`\\]\n| Dataset        |  $d$   | $\\lambda_{BT}$ | $\\lambda_{reg}$ | Download Link to Pretrained Model | KNN Acc. | Linear Acc. |\n| ----------     | ---  | ---------- | ---------- | ------------------------ | -------- | ----------- |\n| `CIFAR-10`       | 1024 | 0.0078125  | 4.0        | [4wdhbpcf_cifar10.pth](https://github.com/wgcban/mix-bt/releases/download/v1.0.0/4wdhbpcf_0.0078125_1024_256_cifar10_model.pth)     | 90.52    | 92.58        |\n| `CIFAR-100`     | 1024 | 0.0078125  | 4.0        | [76kk7scz_cifar100.pth](https://github.com/wgcban/mix-bt/releases/download/v1.0.0/76kk7scz_0.0078125_1024_256_cifar100_model.pth)     | 61.25     | 69.31        |\n| `TinyImageNet`   | 1024 | 0.0009765  | 4.0        | [02azq6fs_tiny_imagenet.pth](https://github.com/wgcban/mix-bt/releases/download/v1.0.0/02azq6fs_0.0009765_1024_256_tiny_imagenet_model.pth)     | 38.11    | 51.67        |\n| `STL-10`        | 1024 | 0.0078125  | 2.0        | [i7det4xq_stl10.pth](https://github.com/wgcban/mix-bt/releases/download/v1.0.0/i7det4xq_0.0078125_1024_256_stl10_model.pth)     | 88.94     | 91.02        |\n\n#### 3.2 ResNet-50 \\[`CIFAR-10`, `CIFAR-100`, `TinyImageNet`, and `STL-10`\\]\n| Dataset        |  $d$   | $\\lambda_{BT}$ | $\\lambda_{reg}$ | Download Link to Pretrained Model | KNN Acc. | Linear Acc. |\n| ----------     | ---  | ---------- | ---------- | ------------------------ | -------- | ----------- |\n| `CIFAR-10`       | 1024 | 0.0078125  | 4.0        | [v3gwgusq_cifar10.pth](https://github.com/wgcban/mix-bt/releases/download/v1.0.0/v3gwgusq_0.0078125_1024_256_cifar10_model.pth)     | 91.39     | 93.89        |\n| `CIFAR-100`      | 1024 | 0.0078125  | 4.0        | [z6ngefw7_cifar100.pth](https://github.com/wgcban/mix-bt/releases/download/v1.0.0/z6ngefw7_0.0078125_1024_256_cifar100_model.pth)     | 64.32     | 72.51        |\n| `TinyImageNet`   | 1024 | 0.0009765  | 4.0        | [kxlkigsv_tiny_imagenet.pth](https://github.com/wgcban/mix-bt/releases/download/v1.0.0/kxlkigsv_0.0009765_1024_256_tiny_imagenet_model.pth)     | 42.21     | 51.84        |\n| `STL-10`        | 1024 | 0.0078125  | 2.0        | [pbknx38b_stl10.pth](https://github.com/wgcban/mix-bt/releases/download/v1.0.0/pbknx38b_0.0078125_1024_256_stl10_model.pth)     | 87.79     | 91.70        |\n\n#### 3.3. ResNet-50 on `ImageNet` (300 epochs)\n\u003e Setting: epochs = 300, $d$ = 8192, $\\lambda_{BT}$ = 0.0051\n\n| $\\lambda_{reg}$ | Linear Acc. | Download Link to Pretrained Model | Train Log | Download Link to Linear-Probed Model | Val. Log |\n| ---------- | --------------------- | ------ | ----- | ------ | ----------- |\n| 0.0 (BT)   | 71.3     | [3on0l4wl_resnet50.pth](https://github.com/wgcban/mix-bt/releases/download/v1.0.0/3on0l4wl_0.0000_8192_1024_imagenet_resnet50.pth) | [train_log](https://github.com/wgcban/mix-bt/releases/download/v1.0.0/3on0l4wl_train.txt) | [checkpoint_3tb4tcvp.pth](https://github.com/wgcban/mix-bt/releases/download/v1.0.0/checkpoint_3tb4tcvp.pth) | [val_log](https://github.com/wgcban/mix-bt/releases/download/v1.0.0/3tb4tcvp_val.txt) |\n| 0.0025     | 70.9  | [l418b9zw_resnet50.pth](https://github.com/wgcban/mix-bt/releases/download/v1.0.0/l418b9zw_0.0025_8192_1024_imagenet_resnet50.pth) | [train_log](https://github.com/wgcban/mix-bt/releases/download/v1.0.0/l418b9zw_train.txt) | [checkpoint_09g7ytcz.pth](https://github.com/wgcban/mix-bt/releases/download/v1.0.0/checkpoint_09g7ytcz.pth)  | [val_log](https://github.com/wgcban/mix-bt/releases/download/v1.0.0/09g7ytcz_val.txt)  |\n| 0.1        | 71.6  | [13awtq23_resnet50.pth](https://github.com/wgcban/mix-bt/releases/download/v1.0.0/13awtq23_0.1000_8192_1024_imagenet_resnet50.pth) | [train_log](https://github.com/wgcban/mix-bt/releases/download/v1.0.0/13awtq23_train.txt) | [checkpoint_pgawzr4e.pth](https://github.com/wgcban/mix-bt/releases/download/v1.0.0/checkpoint_pgawzr4e.pth)  | [val_log](https://github.com/wgcban/mix-bt/releases/download/v1.0.0/pgawzr4e_val.txt)    |\n| 1.0        | **72.2** (best) | [3fb1op86_resnet50.pth](https://github.com/wgcban/mix-bt/releases/download/v1.0.0/3fb1op86_1.0000_8192_1024_imagenet_resnet50.pth) | [train_log](https://github.com/wgcban/mix-bt/releases/download/v1.0.0/3fb1op86_train.txt) |  [checkpoint_wvi0hle8.pth](https://github.com/wgcban/mix-bt/releases/download/v1.0.0/checkpoint_wvi0hle8.pth)  | [val_log](https://github.com/wgcban/mix-bt/releases/download/v1.0.0/wvi0hle8_val.txt)   |\n| 2.0        | 72.1 | [5n9yqio0_resnet50.pth](https://github.com/wgcban/mix-bt/releases/download/v1.0.0/5n9yqio0_1.0000_8192_1024_imagenet_resnet50.pth) | [train_log](https://github.com/wgcban/mix-bt/releases/download/v1.0.0/5n9yqio0_train.txt) | [checkpoint_p9aeo8ga.pth](https://github.com/wgcban/mix-bt/releases/download/v1.0.0/checkpoint_p9aeo8ga.pth)  | [val_log](https://github.com/wgcban/mix-bt/releases/download/v1.0.0/p9aeo8ga_val.txt)   |\n| 3.0        | 72.0 | [q03u2xjz_resnet50.pth](https://github.com/wgcban/mix-bt/releases/download/v1.0.0/q03u2xjz_3.0000_8192_1024_imagenet_resnet50.pth) | [train_log](https://github.com/wgcban/mix-bt/releases/download/v1.0.0/q03u2xjz_train.txt) | [checkpoint_00atvp6x.pth](https://github.com/wgcban/mix-bt/releases/download/v1.0.0/checkpoint_00atvp6x.pth)  | [val_log](https://github.com/wgcban/mix-bt/releases/download/v1.0.0/00atvp6x_val.txt)  |\n\n#### 3.4. ResNet-50 on `ImageNet` (1000 epochs)\n\u003e Setting: epochs = 1000, $d$ = 8192, $\\lambda_{BT}$ = 0.0051, $\\lambda_{reg}$=2.0\n\n| Linear Eval. Top1 | Linear Eval. Top5 | Download Link to Pretrained Model | Train Log | Download Link to Linear-Probed Model | Val. Log |\n| ----- | ----- | --------------------------------- | --------- | ------------------------------------ | -------- |\n| **74.06** (best) | 91.47 | [4wpu8wmd_resnet50.pth](https://github.com/wgcban/mix-bt/releases/download/v1.0.0/4wpu8wmd_resnet50.pth) | [train_log](https://github.com/wgcban/mix-bt/releases/download/v1.0.0/4wpu8wmd_stats.txt) | [vfd2nu64_checkpoint.pth](https://github.com/wgcban/mix-bt/releases/download/v1.0.0/vfd2nu64_checkpoint.pth) | [val_log](https://github.com/wgcban/mix-bt/releases/download/v1.0.0/vfd2nu64_stats.txt) |\n\n## 4 Training/Val Logs\n\n### 3.1 Pre-trianing for 300 epochs\nLogs are available on `wandb` and can access via following links:\n- imagenet pre-training: https://api.wandb.ai/links/cha-yas/5olb2sar\n- imagenet linear probing: https://api.wandb.ai/links/cha-yas/9tb0ksfp\n\nHere we provide some training and validation (linear probing) statistics for Barlow Twins *vs.* Mixed Barlow Twins with `ResNet-50` backbone on `ImageNet`:\n\n\u003cimg src=\"figs/in-loss-bt.png\" width=\"256\"/\u003e \u003cimg src=\"figs/in-loss-reg.png\" width=\"256\"/\u003e \u003cimg src=\"figs/in-linear.png\" width=\"256\"/\u003e \n\n### 3.1 Pre-trianing for 1000 epochs\nWe also provide trianing-val statistics for our pre-trained model for 1000 epochs.\n\u003cimg src=\"figs/in-loss-bt-1000e.png\" width=\"256\"/\u003e \u003cimg src=\"figs/in-loss-reg-1000e.png\" width=\"256\"/\u003e \u003cimg src=\"figs/in-linear-1000e.png\" width=\"256\"/\u003e \n\n:fire: Access pre-training statistcis on wandb: [`wandb-imagenet-pretrain`](https://wandb.ai/cha-yas/Barlow-Twins-MixUp-ImageNet?workspace=user-wgcban)\n\n## 5 Disclaimer\nA large portion of the code is from [Barlow Twins HSIC](https://github.com/yaohungt/Barlow-Twins-HSIC) (for experiments on small datasets: `CIFAR-10`, `CIFAR-100`, `TinyImageNet`, and `STL-10`) and official implementation of Barlow Twins [here](https://github.com/facebookresearch/barlowtwins) (for experiments on `ImageNet`), which is a great resource for academic development.\n\nAlso, note that the implementation of SOTA methods ([SimCLR](https://arxiv.org/abs/2002.05709), [BYOL](https://arxiv.org/abs/2006.07733), and [Witening-MSE](https://arxiv.org/abs/2007.06346)) in `ssl-sota` are copied from [Witening-MSE](https://github.com/htdt/self-supervised).\n\nWe would like to thank all of them for making their repositories publicly available for the research community. 🙏\n\n## 6 Reference\nIf you feel our work is useful, please consider citing our work. Thanks!\n```bibtex\n@misc{bandara2023guarding,\n      title={Guarding Barlow Twins Against Overfitting with Mixed Samples}, \n      author={Wele Gedara Chaminda Bandara and Celso M. De Melo and Vishal M. Patel},\n      year={2023},\n      eprint={2312.02151},\n      archivePrefix={arXiv},\n      primaryClass={cs.CV}\n}\n```\n\n## 7 License\nThis code is under MIT licence, you can find the complete file [here](https://github.com/wgcban/mix-bt/blob/main/LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwgcban%2Fmix-bt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwgcban%2Fmix-bt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwgcban%2Fmix-bt/lists"}