{"id":15639971,"url":"https://github.com/sayakpaul/simclr-in-tensorflow-2","last_synced_at":"2025-04-30T07:28:07.416Z","repository":{"id":106650169,"uuid":"257231867","full_name":"sayakpaul/SimCLR-in-TensorFlow-2","owner":"sayakpaul","description":"(Minimally) implements SimCLR (https://arxiv.org/abs/2002.05709) in TensorFlow 2.","archived":false,"fork":false,"pushed_at":"2021-04-06T07:07:31.000Z","size":96315,"stargazers_count":90,"open_issues_count":2,"forks_count":21,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-30T14:01:34.234Z","etag":null,"topics":["constrastive-learning","deep-learning","keras","self-supervised-learning","tensorflow","unsupervised-learning"],"latest_commit_sha":null,"homepage":"https://app.wandb.ai/sayakpaul/simclr/reports/Towards-self-supervised-image-understanding-with-SimCLR--VmlldzoxMDI5NDM","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sayakpaul.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-04-20T09:26:11.000Z","updated_at":"2024-12-31T13:30:35.000Z","dependencies_parsed_at":"2023-07-12T16:16:34.098Z","dependency_job_id":null,"html_url":"https://github.com/sayakpaul/SimCLR-in-TensorFlow-2","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sayakpaul%2FSimCLR-in-TensorFlow-2","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sayakpaul%2FSimCLR-in-TensorFlow-2/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sayakpaul%2FSimCLR-in-TensorFlow-2/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sayakpaul%2FSimCLR-in-TensorFlow-2/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sayakpaul","download_url":"https://codeload.github.com/sayakpaul/SimCLR-in-TensorFlow-2/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251659920,"owners_count":21623149,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["constrastive-learning","deep-learning","keras","self-supervised-learning","tensorflow","unsupervised-learning"],"created_at":"2024-10-03T11:29:13.953Z","updated_at":"2025-04-30T07:28:07.346Z","avatar_url":"https://github.com/sayakpaul.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# SimCLR-in-TensorFlow-2\n(Minimally) implements SimCLR ([A Simple Framework for Contrastive Learning of Visual Representations](https://arxiv.org/abs/2002.05709) by Chen et al.) in TensorFlow 2. Uses many delicious pieces of `tf.keras` and TensorFlow's core APIs. A report is available [here](https://app.wandb.ai/sayakpaul/simclr/reports/Towards-self-supervised-image-understanding-with-SimCLR--VmlldzoxMDI5NDM).\n\n## Acknowledgements\nI did not code everything from scratch. This particular research paper felt super amazing to read and often felt natural to understand, that's why I wanted to try it out myself and come up with a minimal implementation. I reused the works of the following for different purposes -\n- Data augmentation policies comes from here: https://github.com/google-research/simclr/blob/master/data_util.py.\n- Loss function comes from here: https://github.com/sthalles/SimCLR-tensorflow/blob/master/utils/losses.py.\n- TSNE visualization referred from here: https://github.com/thunderInfy/simclr/blob/master/resnet-simclr.py.\n\nFollowing are the articles I studied for understanding SimCLR other than the paper:\n- [Understanding SimCLR — A Simple Framework for Contrastive Learning of Visual Representations with Code](https://medium.com/analytics-vidhya/understanding-simclr-a-simple-framework-for-contrastive-learning-of-visual-representations-d544a9003f3c)\n- [Exploring SimCLR: A Simple Framework for Contrastive Learning of Visual Representations](https://sthalles.github.io/simple-self-supervised-learning/)\n- [Illustrated SimCLR](https://amitness.com/2020/03/illustrated-simclr/) (This one does an amazing job at explaining the loss function\" NT-XEnt Loss)\n\nThanks a ton to the **ML-GDE program** for providing the GCP Credits using which I could run the experiments, store the intermediate results on GCS buckets as necessary. _All the notebooks can be run on Colab though_.\n\n## Dataset\n- Subset of ImageNet: https://github.com/thunderInfy/imagenet-5-categories\n\n## Architecture\n```\nModel: \"model\"\n_________________________________________________________________\nLayer (type)                 Output Shape              Param #\n=================================================================\ninput_2 (InputLayer)         [(None, 224, 224, 3)]     0\n_________________________________________________________________\nresnet50 (Model)             (None, 7, 7, 2048)        23587712\n_________________________________________________________________\nglobal_average_pooling2d (Gl (None, 2048)              0\n_________________________________________________________________\ndense (Dense)                (None, 256)               524544\n_________________________________________________________________\nactivation (Activation)      (None, 256)               0\n_________________________________________________________________\ndense_1 (Dense)              (None, 128)               32896\n_________________________________________________________________\nactivation_1 (Activation)    (None, 128)               0\n_________________________________________________________________\ndense_2 (Dense)              (None, 50)                6450\n=================================================================\nTotal params: 24,151,602\nTrainable params: 24,098,482\nNon-trainable params: 53,120\n```\n\n## Contrastive learning progress\n![](https://i.ibb.co/9yM4RzQ/download.png)\n\n## Training with 10% training data using the learned representations (linear evaluation)\n![](https://i.ibb.co/GV44Xsk/download-1.png)\n\n```\nloss: 1.1009 - accuracy: 0.5840 - val_loss: 1.1486 - val_accuracy: 0.5280\n```\n\nThis is when I only took the base encoder network i.e. _without any non-linear projections_. I presented results with different projection heads as well (available [here](https://github.com/sayakpaul/SimCLR-in-TensorFlow-2/blob/master/Linear_Evaluation_Imagenet_Subset.ipynb)) but this one came to be the best.\n\n## Learned representations with TSNE\n![](https://i.ibb.co/ckz1mbv/download-2.png)\n\nThis is when I only took the base encoder network i.e. _without any non-linear projections_. I presented results with different projection heads as well (available [here](https://github.com/sayakpaul/SimCLR-in-TensorFlow-2/blob/master/Linear_Evaluation_Imagenet_Subset.ipynb)) but this one came to be the best.\n\n## Supervised training with the full training dataset\n\nHere's the architecture that was used:\n\n```\n\nModel: \"model\"\n_________________________________________________________________\nLayer (type)                 Output Shape              Param #\n=================================================================\ninput_4 (InputLayer)         [(None, 224, 224, 3)]     0\n_________________________________________________________________\nresnet50 (Model)             (None, 7, 7, 2048)        23587712\n_________________________________________________________________\nglobal_average_pooling2d_1 ( (None, 2048)              0\n_________________________________________________________________\ndense_1 (Dense)              (None, 256)               524544\n_________________________________________________________________\nactivation (Activation)      (None, 256)               0\n_________________________________________________________________\ndense_2 (Dense)              (None, 5)                 1285\n=================================================================\nTotal params: 24,113,541\nTrainable params: 24,060,421\nNon-trainable params: 53,120\n```\n\n![](https://i.ibb.co/WVwpZJk/download-3.png)\n\n```\nloss: 0.6623 - accuracy: 0.7528 - val_loss: 1.0171 - val_accuracy: 0.6440\n```\n\nWe see a 12% increase here. The accuracy with the SimCLR framework could further be increased with better pre-training in terms of the following aspect:\n- More unsupervised data. If we could gather a larger corpurs of images for the pre-training task (think of ImageNet) that would have definitely helped.\n- I only trained using the SimCLR framework for 200 epochs. Longer training could have definitely helped.\n- Architectural considerations and hyperparameter tuning:\n\t- Temperature (tau) (I used 0.1)\n\t- Mix and match between the different augmentation policies shown in the paper and the strength of the color distortion.\n\t- Different projection heads.\n\n_SimCLR benefits from larger data_. Ting Chen (the first author of the paper) suggested to go for an augmentation policy (when using custom datasets) that's not too easy nor too hard for the contrastive task i.e. the contrastive accuracy should be high (e.g. \u003e 80%).\n\n## Pre-trained weights\nAvailable here - `Pretrained_Weights`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsayakpaul%2Fsimclr-in-tensorflow-2","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsayakpaul%2Fsimclr-in-tensorflow-2","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsayakpaul%2Fsimclr-in-tensorflow-2/lists"}