{"id":13476822,"url":"https://github.com/google-research/simclr","last_synced_at":"2025-09-27T10:31:26.214Z","repository":{"id":37284396,"uuid":"246357476","full_name":"google-research/simclr","owner":"google-research","description":"SimCLRv2 - Big Self-Supervised Models are Strong Semi-Supervised Learners","archived":true,"fork":false,"pushed_at":"2023-05-22T04:42:53.000Z","size":3957,"stargazers_count":4057,"open_issues_count":71,"forks_count":623,"subscribers_count":47,"default_branch":"master","last_synced_at":"2024-09-26T23:03:47.832Z","etag":null,"topics":["computer-vision","contrastive-learning","representation-learning","self-supervised-learning","simclr","simclrv2","unsupervised-learning"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2006.10029","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/google-research.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2020-03-10T16:53:22.000Z","updated_at":"2024-09-25T14:48:40.000Z","dependencies_parsed_at":"2023-01-21T10:31:55.389Z","dependency_job_id":null,"html_url":"https://github.com/google-research/simclr","commit_stats":{"total_commits":37,"total_committers":4,"mean_commits":9.25,"dds":"0.16216216216216217","last_synced_commit":"383d4143fd8cf7879ae10f1046a9baeb753ff438"},"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Fsimclr","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Fsimclr/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Fsimclr/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Fsimclr/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/google-research","download_url":"https://codeload.github.com/google-research/simclr/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":234426494,"owners_count":18830912,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","contrastive-learning","representation-learning","self-supervised-learning","simclr","simclrv2","unsupervised-learning"],"created_at":"2024-07-31T16:01:35.006Z","updated_at":"2025-09-27T10:31:20.200Z","avatar_url":"https://github.com/google-research.png","language":"Jupyter Notebook","readme":"# SimCLR - A Simple Framework for Contrastive Learning of Visual Representations\n\n\u003cspan style=\"color: red\"\u003e\u003cstrong\u003eNews! \u003c/strong\u003e\u003c/span\u003e We have released a TF2 implementation of SimCLR (along with converted checkpoints in TF2), they are in \u003ca href=\"tf2/\"\u003etf2/ folder\u003c/a\u003e.\n\n\u003cspan style=\"color: red\"\u003e\u003cstrong\u003eNews! \u003c/strong\u003e\u003c/span\u003e Colabs for \u003ca href=\"https://arxiv.org/abs/2011.02803\"\u003eIntriguing Properties of Contrastive Losses\u003c/a\u003e are added, see \u003ca href=\"colabs/intriguing_properties/\"\u003ehere\u003c/a\u003e.\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg width=\"50%\" alt=\"SimCLR Illustration\" src=\"https://1.bp.blogspot.com/--vH4PKpE9Yo/Xo4a2BYervI/AAAAAAAAFpM/vaFDwPXOyAokAC8Xh852DzOgEs22NhbXwCLcBGAsYHQ/s1600/image4.gif\"\u003e\n\u003c/div\u003e\n\u003cdiv align=\"center\"\u003e\n  An illustration of SimCLR (from \u003ca href=\"https://ai.googleblog.com/2020/04/advancing-self-supervised-and-semi.html\"\u003eour blog here\u003c/a\u003e).\n\u003c/div\u003e\n\n## Pre-trained models for SimCLRv2\n\u003ca href=\"colabs/finetuning.ipynb\" target=\"_parent\"\u003e\u003cimg src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/\u003e\u003c/a\u003e\n\nWe opensourced total 65 pretrained models here, corresponding to those in Table 1 of the \u003ca href=\"https://arxiv.org/abs/2006.10029\"\u003eSimCLRv2\u003c/a\u003e paper:\n\n|   Depth | Width   | SK    |   Param (M) |   F-T (1%) |   F-T(10%) |   F-T(100%) |   Linear eval |   Supervised |\n|--------:|--------:|------:|--------:|-------------:|--------------:|---------------:|-----------------:|--------------:|\n|      50 | 1X      | False |    24 |         57.9 |          68.4 |           76.3 |             71.7 |          76.6 |\n|      50 | 1X      | True  |    35 |         64.5 |          72.1 |           78.7 |             74.6 |          78.5 |\n|      50 | 2X      | False |    94   |         66.3 |          73.9 |           79.1 |             75.6 |          77.8 |\n|      50 | 2X      | True  |   140 |         70.6 |          77.0   |           81.3 |             77.7 |          79.3 |\n|     101 | 1X      | False |    43 |         62.1 |          71.4 |           78.2 |             73.6 |          78.0   |\n|     101 | 1X      | True  |    65 |         68.3 |          75.1 |           80.6 |             76.3 |          79.6 |\n|     101 | 2X      | False |   170   |         69.1 |          75.8 |           80.7 |             77.0   |          78.9 |\n|     101 | 2X      | True  |   257 |         73.2 |          78.8 |           82.4 |             79.0   |          80.1 |\n|     152 | 1X      | False |    58 |         64.0   |          73.0   |           79.3 |             74.5 |          78.3 |\n|     152 | 1X      | True  |    89 |         70.0   |          76.5 |           81.3 |             77.2 |          79.9 |\n|     152 | 2X      | False |   233 |         70.2 |          76.6 |           81.1 |             77.4 |          79.1 |\n|     152 | 2X      | True  |   354 |         74.2 |          79.4 |           82.9 |             79.4 |          80.4 |\n|     152 | 3X      | True  |   795 |         74.9 |          80.1 |           83.1 |             79.8 |          80.5 |\n\nThese checkpoints are stored in Google Cloud Storage:\n\n* Pretrained SimCLRv2 models (with linear eval head): [gs://simclr-checkpoints/simclrv2/pretrained](https://console.cloud.google.com/storage/browser/simclr-checkpoints/simclrv2/pretrained)\n* Fine-tuned SimCLRv2 models on 1% of labels: [gs://simclr-checkpoints/simclrv2/finetuned_1pct](https://console.cloud.google.com/storage/browser/simclr-checkpoints/simclrv2/finetuned_1pct)\n* Fine-tuned SimCLRv2 models on 10% of labels: [gs://simclr-checkpoints/simclrv2/finetuned_10pct](https://console.cloud.google.com/storage/browser/simclr-checkpoints/simclrv2/finetuned_10pct)\n* Fine-tuned SimCLRv2 models on 100% of labels: [gs://simclr-checkpoints/simclrv2/finetuned_100pct](https://console.cloud.google.com/storage/browser/simclr-checkpoints/simclrv2/finetuned_100pct)\n* Supervised models with the same architectures: [gs://simclr-checkpoints/simclrv2/supervised](https://console.cloud.google.com/storage/browser/simclr-checkpoints/simclrv2/supervised)\n* The distilled / self-trained models (after fine-tuning) are also provided:\n  * [gs://simclr-checkpoints/simclrv2/distill_1pct](https://console.cloud.google.com/storage/browser/simclr-checkpoints/simclrv2/distill_1pct)\n  * [gs://simclr-checkpoints/simclrv2/distill_10pct](https://console.cloud.google.com/storage/browser/simclr-checkpoints/simclrv2/distill_10pct)\n\nWe also provide examples on how to use the checkpoints in `colabs/` folder.\n\n## Pre-trained models for SimCLRv1\n\nThe pre-trained models (base network with linear classifier layer) can be found below. Note that for these SimCLRv1 checkpoints, the projection head is not available.\n\n|                             Model checkpoint and hub-module                             |     ImageNet Top-1     |\n|-----------------------------------------------------------------------------------------|------------------------|\n|[ResNet50 (1x)](https://storage.cloud.google.com/simclr-gcs/checkpoints/ResNet50_1x.zip) |          69.1          |\n|[ResNet50 (2x)](https://storage.cloud.google.com/simclr-gcs/checkpoints/ResNet50_2x.zip) |          74.2          |\n|[ResNet50 (4x)](https://storage.cloud.google.com/simclr-gcs/checkpoints/ResNet50_4x.zip) |          76.6          |\n\nAdditional SimCLRv1 checkpoints are available: [gs://simclr-checkpoints/simclrv1](https://console.cloud.google.com/storage/browser/simclr-checkpoints/simclrv1).\n\nA note on the signatures of the TensorFlow Hub module: `default` is the representation output of the base network; `logits_sup` is the supervised classification logits for ImageNet 1000 categories. Others (e.g. `initial_max_pool`, `block_group1`) are middle layers of ResNet; refer to resnet.py for the specifics. See this [tutorial](https://www.tensorflow.org/hub/tf1_hub_module) for additional information regarding use of TensorFlow Hub modules.\n\n## Enviroment setup\n\nOur models are trained with TPUs. It is recommended to run distributed training with TPUs when using our code for pretraining.\n\nOur code can also run on a *single* GPU. It does not support multi-GPUs, for reasons such as global BatchNorm and contrastive loss across cores.\n\nThe code is compatible with both TensorFlow v1 and v2. See requirements.txt for all prerequisites, and you can also install them using the following command.\n\n```\npip install -r requirements.txt\n```\n\n## Pretraining\n\nTo pretrain the model on CIFAR-10 with a *single* GPU, try the following command:\n\n```\npython run.py --train_mode=pretrain \\\n  --train_batch_size=512 --train_epochs=1000 \\\n  --learning_rate=1.0 --weight_decay=1e-4 --temperature=0.5 \\\n  --dataset=cifar10 --image_size=32 --eval_split=test --resnet_depth=18 \\\n  --use_blur=False --color_jitter_strength=0.5 \\\n  --model_dir=/tmp/simclr_test --use_tpu=False\n```\n\nTo pretrain the model on ImageNet with Cloud TPUs, first check out the [Google Cloud TPU tutorial](https://cloud.google.com/tpu/docs/tutorials/mnist) for basic information on how to use Google Cloud TPUs.\n\nOnce you have created virtual machine with Cloud TPUs, and pre-downloaded the ImageNet data for [tensorflow_datasets](https://www.tensorflow.org/datasets/catalog/imagenet2012), please set the following enviroment variables:\n\n```\nTPU_NAME=\u003ctpu-name\u003e\nSTORAGE_BUCKET=gs://\u003cstorage-bucket\u003e\nDATA_DIR=$STORAGE_BUCKET/\u003cpath-to-tensorflow-dataset\u003e\nMODEL_DIR=$STORAGE_BUCKET/\u003cpath-to-store-checkpoints\u003e\n```\n\nThe following command can be used to pretrain a ResNet-50 on ImageNet (which reflects the default hyperparameters in our paper):\n\n```\npython run.py --train_mode=pretrain \\\n  --train_batch_size=4096 --train_epochs=100 --temperature=0.1 \\\n  --learning_rate=0.075 --learning_rate_scaling=sqrt --weight_decay=1e-4 \\\n  --dataset=imagenet2012 --image_size=224 --eval_split=validation \\\n  --data_dir=$DATA_DIR --model_dir=$MODEL_DIR \\\n  --use_tpu=True --tpu_name=$TPU_NAME --train_summary_steps=0\n```\n\nA batch size of 4096 requires at least 32 TPUs. 100 epochs takes around 6 hours with 32 TPU v3s. Note that learning rate of 0.3 with `learning_rate_scaling=linear` is equivalent to that of 0.075 with `learning_rate_scaling=sqrt` when the batch size is 4096. However, using sqrt scaling allows it to train better when smaller batch size is used.\n\n## Finetuning the linear head (linear eval)\n\nTo fine-tune a linear head (with a single GPU), try the following command:\n\n```\npython run.py --mode=train_then_eval --train_mode=finetune \\\n  --fine_tune_after_block=4 --zero_init_logits_layer=True \\\n  --variable_schema='(?!global_step|(?:.*/|^)Momentum|head)' \\\n  --global_bn=False --optimizer=momentum --learning_rate=0.1 --weight_decay=0.0 \\\n  --train_epochs=100 --train_batch_size=512 --warmup_epochs=0 \\\n  --dataset=cifar10 --image_size=32 --eval_split=test --resnet_depth=18 \\\n  --checkpoint=/tmp/simclr_test --model_dir=/tmp/simclr_test_ft --use_tpu=False\n```\n\nYou can check the results using tensorboard, such as\n\n```\npython -m tensorboard.main --logdir=/tmp/simclr_test\n```\n\nAs a reference, the above runs on CIFAR-10 should give you around 91% accuracy, though it can be further optimized.\n\nFor fine-tuning a linear head on ImageNet using Cloud TPUs, first set the `CHKPT_DIR` to pretrained model dir and set a new `MODEL_DIR`, then use the following command:\n\n```\npython run.py --mode=train_then_eval --train_mode=finetune \\\n  --fine_tune_after_block=4 --zero_init_logits_layer=True \\\n  --variable_schema='(?!global_step|(?:.*/|^)Momentum|head)' \\\n  --global_bn=False --optimizer=momentum --learning_rate=0.1 --weight_decay=1e-6 \\\n  --train_epochs=90 --train_batch_size=4096 --warmup_epochs=0 \\\n  --dataset=imagenet2012 --image_size=224 --eval_split=validation \\\n  --data_dir=$DATA_DIR --model_dir=$MODEL_DIR --checkpoint=$CHKPT_DIR \\\n  --use_tpu=True --tpu_name=$TPU_NAME --train_summary_steps=0\n```\n\nAs a reference, the above runs on ImageNet should give you around 64.5% accuracy.\n\n## Semi-supervised learning and fine-tuning the whole network\n\nYou can access 1% and 10% ImageNet subsets used for semi-supervised learning via [tensorflow datasets](https://www.tensorflow.org/datasets/catalog/imagenet2012_subset): simply set `dataset=imagenet2012_subset/1pct` and `dataset=imagenet2012_subset/10pct` in the command line for fine-tuning on these subsets.\n\nYou can also find image IDs of these subsets in `imagenet_subsets/`.\n\nTo fine-tune the whole network on ImageNet (1% of labels), refer to the following command:\n\n```\npython run.py --mode=train_then_eval --train_mode=finetune \\\n  --fine_tune_after_block=-1 --zero_init_logits_layer=True \\\n  --variable_schema='(?!global_step|(?:.*/|^)Momentum|head_supervised)' \\\n  --global_bn=True --optimizer=lars --learning_rate=0.005 \\\n  --learning_rate_scaling=sqrt --weight_decay=0 \\\n  --train_epochs=60 --train_batch_size=1024 --warmup_epochs=0 \\\n  --dataset=imagenet2012_subset/1pct --image_size=224 --eval_split=validation \\\n  --data_dir=$DATA_DIR --model_dir=$MODEL_DIR --checkpoint=$CHKPT_DIR \\\n  --use_tpu=True --tpu_name=$TPU_NAME --train_summary_steps=0 \\\n  --num_proj_layers=3 --ft_proj_selector=1\n```\n\nSet the `checkpoint` to those that are only pre-trained but not fine-tuned. Given that SimCLRv1 checkpoints do not contain projection head, it is recommended to run with SimCLRv2 checkpoints (you can still run with SimCLRv1 checkpoints, but `variable_schema` needs to exclude `head`). The `num_proj_layers` and `ft_proj_selector` need to be adjusted accordingly following SimCLRv2 paper to obtain best performances.\n\n## Other resources\n\n### Model conversion to Pytorch format\n\nThis [repo](https://github.com/tonylins/simclr-converter) provides a solution for converting the pretrained SimCLRv1 Tensorflow checkpoints into Pytorch ones.\n\nThis [repo](https://github.com/Separius/SimCLRv2-Pytorch) provides a solution for converting the pretrained SimCLRv2 Tensorflow checkpoints into Pytorch ones.\n\n### Other *non-offical* / *unverified* implementations\n\n(Feel free to share your implementation by creating an issue)\n\nImplementations in PyTorch:\n* [leftthomas](https://github.com/leftthomas/SimCLR)\n* [AndrewAtanov](https://github.com/AndrewAtanov/simclr-pytorch)\n* [ae-foster](https://github.com/ae-foster/pytorch-simclr)\n* [Spijkervet](https://github.com/Spijkervet/SimCLR)\n* [williamFalcon](https://pytorch-lightning-bolts.readthedocs.io/en/latest/self_supervised_models.html#simclr)\n\nImplementations in Tensorflow 2 / Keras (official TF2 implementation was added in \u003ca href=\"tf2/\"\u003etf2/ folder\u003c/a\u003e):\n* [sayakpaul](https://github.com/sayakpaul/SimCLR-in-TensorFlow-2)\n* [mwdhont](https://github.com/mwdhont/SimCLRv1-keras-tensorflow)\n\n## Known issues\n\n* **Batch size**: original results of SimCLR were tuned under a large batch size (i.e. 4096), which leads to suboptimal results when training using a smaller batch size. However, with a good set of hyper-parameters (mainly learning rate, temperature, projection head depth), small batch sizes can yield results that are on par with large batch sizes (e.g., see Table 2 in [this paper](https://arxiv.org/pdf/2011.02803.pdf)).\n\n* **Pretrained models / Checkpoints**: SimCLRv1 and SimCLRv2 are pretrained with different weight decays, so the pretrained models from the two versions have very different weight norm scales (convolutional weights in SimCLRv1 ResNet-50 are on average 16.8X of that in SimCLRv2). For fine-tuning the pretrained models from both versions, it is fine if you use an LARS optimizer, but it requires very different hyperparameters (e.g. learning rate, weight decay) if you use the momentum optimizer. So for the latter case, you may want to either search for very different hparams according to which version used, or re-scale th weight (i.e. conv `kernel` parameters of `base_model` in the checkpoints) to make sure they're roughly in the same scale.\n\n## Cite\n\n[SimCLR paper](https://arxiv.org/abs/2002.05709):\n\n```\n@article{chen2020simple,\n  title={A Simple Framework for Contrastive Learning of Visual Representations},\n  author={Chen, Ting and Kornblith, Simon and Norouzi, Mohammad and Hinton, Geoffrey},\n  journal={arXiv preprint arXiv:2002.05709},\n  year={2020}\n}\n```\n\n[SimCLRv2 paper](https://arxiv.org/abs/2006.10029):\n\n```\n@article{chen2020big,\n  title={Big Self-Supervised Models are Strong Semi-Supervised Learners},\n  author={Chen, Ting and Kornblith, Simon and Swersky, Kevin and Norouzi, Mohammad and Hinton, Geoffrey},\n  journal={arXiv preprint arXiv:2006.10029},\n  year={2020}\n}\n```\n\n## Disclaimer\nThis is not an official Google product.\n","funding_links":[],"categories":["Jupyter Notebook","Computer Vision"],"sub_categories":["Image Representation Learning"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle-research%2Fsimclr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgoogle-research%2Fsimclr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle-research%2Fsimclr/lists"}