{"id":21053600,"url":"https://github.com/andrewatanov/simclr-pytorch","last_synced_at":"2025-04-09T15:05:41.815Z","repository":{"id":43573192,"uuid":"320287122","full_name":"AndrewAtanov/simclr-pytorch","owner":"AndrewAtanov","description":"PyTorch implementation of SimCLR: supports multi-GPU training and closely reproduces results","archived":false,"fork":false,"pushed_at":"2024-04-29T10:29:25.000Z","size":1842,"stargazers_count":203,"open_issues_count":5,"forks_count":40,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-04-09T15:05:36.199Z","etag":null,"topics":["contrastive-learning","deep-learning","pytorch","pytorch-implementation","representation-learning","self-supervised-learning"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AndrewAtanov.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-12-10T13:52:17.000Z","updated_at":"2025-03-28T03:45:54.000Z","dependencies_parsed_at":"2025-01-16T00:12:14.423Z","dependency_job_id":"a519ccf3-d382-4cd5-a215-fa630bd3bc31","html_url":"https://github.com/AndrewAtanov/simclr-pytorch","commit_stats":{"total_commits":17,"total_committers":4,"mean_commits":4.25,"dds":0.3529411764705882,"last_synced_commit":"6493ecc3d0512b171d14de45b14dd3d8b726a5c4"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AndrewAtanov%2Fsimclr-pytorch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AndrewAtanov%2Fsimclr-pytorch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AndrewAtanov%2Fsimclr-pytorch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AndrewAtanov%2Fsimclr-pytorch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AndrewAtanov","download_url":"https://codeload.github.com/AndrewAtanov/simclr-pytorch/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248055284,"owners_count":21040157,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["contrastive-learning","deep-learning","pytorch","pytorch-implementation","representation-learning","self-supervised-learning"],"created_at":"2024-11-19T16:07:36.809Z","updated_at":"2025-04-09T15:05:41.790Z","avatar_url":"https://github.com/AndrewAtanov.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# SimCLR PyTorch\n\nThis is an unofficial repository reproducing results of the paper [A Simple Framework for Contrastive Learning of Visual Representations](https://arxiv.org/abs/2002.05709). The implementation supports multi-GPU distributed training on several nodes with PyTorch `DistributedDataParallel`.\n\n## How close are we to the original SimCLR?\n\nThe implementation closely reproduces the original ResNet50 results on ImageNet and CIFAR-10.\n\n\u003cp align=\"center\"\u003e\n\u003cimg height=\"480\" src=\"https://raw.githubusercontent.com/senya-ashukha/senya-ashukha.github.io/master/projects/imagenet_top1.png\"/\u003e\n\u003c/p\u003e\n\n| Dataset  | Batch Size | \\# Epochs | Training GPUs | Training time | Top\\-1 accuracy of Linear evaluation (100% labels)| Reference |\n|----------|------------|-----------|---------------|---------------|-----------------------------------|------------|\n| CIFAR-10 | 1024       | 1000      | 2v100         | 13h           | 93\\.44                             | 93.95      |\n| ImageNet | 512        | 100       | 4v100         | 85h           | 60\\.14                            | 60.62      |\n| ImageNet | 2048       | 200       | 16v100        | 55h           | 65\\.58                            | 65.83      |\n| ImageNet | 2048       | 600       | 16v100        | 170h          | 67\\.84                            | 68.71       |\n\n## Pre-trained weights\n\nTry out a pre-trained models [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/AndrewAtanov/simclr-pytorch/blob/master/colabs/model_apply.ipynb) \n\nYou can download pre-trained weights from [here](https://drive.google.com/file/d/13tjpWYTzV8qLB5yY5raBn5cwtIyFtt6-/view?usp=sharing).\n\nTo eval the preatrained CIFAR-10 linear model and encoder use the following command:\n```(bash)\npython train.py --problem eval --eval_only true --iters 1 --arch linear \\\n--ckpt pretrained_models/resnet50_cifar10_bs1024_epochs1000_linear.pth.tar \\\n--encoder_ckpt pretrained_models/resnet50_cifar10_bs1024_epochs1000.pth.tar\n```\n\nTo eval the preatrained ImageNet linear model and encoder use the following command:\n```(bash)\nexport IMAGENET_PATH=.../raw-data\npython train.py --problem eval --eval_only true --iters 1 --arch linear --data imagenet \\\n--ckpt pretrained_models/resnet50_imagenet_bs2k_epochs600_linear.pth.tar \\\n--encoder_ckpt pretrained_models/resnet50_imagenet_bs2k_epochs600.pth.tar\n```\n\n## Enviroment Setup\n\n\nCreate a python enviroment with the provided config file and [miniconda](https://docs.conda.io/en/latest/miniconda.html):\n\n```(bash)\nconda env create -f environment.yml\nconda activate simclr_pytorch\n\nexport IMAGENET_PATH=... # If you have enough RAM using /dev/shm usually accelerates data loading time\nexport EXMAN_PATH=... # A path to logs\n```\n\n## Training\nModel training consists of two steps: (1) self-supervised encoder pretraining and (2) classifier learning with the encoder representations. Both steps are done with the `train.py` script. To see the help for `sim-clr/eval` problem call the following command: `python source/train.py --help --problem sim-clr/eval`.\n\n### Self-supervised pretraining\n\n#### CIFAR-10\nThe config `cifar_train_epochs1000_bs1024.yaml` contains the parameters to reproduce results for CIFAR-10 dataset. It requires 2 V100 GPUs. The pretraining command is:\n\n```(bash)\npython train.py --config configs/cifar_train_epochs1000_bs1024.yaml\n```\n\n#### ImageNet\nThe configs `imagenet_params_epochs*_bs*.yaml` contain the parameters to reproduce results for ImageNet dataset. It requires at 4v100-16v100 GPUs depending on a batch size. The single-node (4 v100 GPUs) pretraining command is:\n\n```(bash)\npython train.py --config configs/imagenet_train_epochs100_bs512.yaml\n```\n\n#### Logs\nThe logs and the model will be stored at `./logs/exman-train.py/runs/\u003cexperiment-id\u003e/`. You can access all the experiments from python with `exman.Index('./logs/exman-train.py').info()`.\n\nSee how to work with logs [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/AndrewAtanov/simclr-pytorch/blob/master/colabs/read_logs.ipynb) \n\n### Linear Evaluation\nTo train a linear classifier on top of the pretrained encoder, run the following command:\n\n```(bash)\npython train.py --config configs/cifar_eval.yaml --encoder_ckpt \u003cpath-to-encoder\u003e\n```\n\nThe above model with batch size 1024 gives `93.5` linear eval test accuracy.\n \n### Pretraining with `DistributedDataParallel`\nTo train a model with larger batch size on several nodes you need to set `--dist ddp` flag and specify the following parameters: \n- `--dist_address`: the address and a port of the main node in the `\u003caddress\u003e:\u003cport\u003e` format\n- `--node_rank`: 0 for the main node and 1,... for the others.\n- `--world_size`: the number of nodes.\n\nFor example, to train with two nodes you need to run the following command on the main node:\n```(bash)\npython train.py --config configs/cifar_train_epochs1000_bs1024.yaml --dist ddp --dist_address \u003caddress\u003e:\u003cport\u003e --node_rank 0 --world_size 2\n```\nand on the second node:\n```(bash)\npython train.py --config configs/cifar_train_epochs1000_bs1024.yaml --dist ddp --dist_address \u003caddress\u003e:\u003cport\u003e --node_rank 1 --world_size 2\n```\n\nThe ImageNet the pretaining on 4 nodes all with 4 GPUs looks as follows:\n```\nnode1: python train.py --config configs/imagenet_train_epochs200_bs2k.yaml --dist ddp --world_size 4 --dist_address \u003caddress\u003e:\u003cport\u003e --node_rank 0\nnode2: python train.py --config configs/imagenet_train_epochs200_bs2k.yaml --dist ddp --world_size 4 --dist_address \u003caddress\u003e:\u003cport\u003e --node_rank 1\nnode3: python train.py --config configs/imagenet_train_epochs200_bs2k.yaml --dist ddp --world_size 4 --dist_address \u003caddress\u003e:\u003cport\u003e --node_rank 2\nnode4: python train.py --config configs/imagenet_train_epochs200_bs2k.yaml --dist ddp --world_size 4 --dist_address \u003caddress\u003e:\u003cport\u003e --node_rank 3\n```\n\n## Attribution\nParts of this code are based on the following repositories:v\n- [PyTorch](https://github.com/pytorch/pytorch), [PyTorch Examples](https://github.com/pytorch/examples/tree/ee964a2/imagenet), [PyTorch Lightning](https://github.com/PyTorchLightning/pytorch-lightning) for standard backbones, training loops, etc.\n- [SimCLR - A Simple Framework for Contrastive Learning of Visual Representations](https://github.com/google-research/simclr) for more details on the original implementation \n- [diffdist](https://github.com/ag14774/diffdist) for multi-gpu contrastive loss implementation, allows backpropagation through `all_gather` operation (see [models/losses.py#L58](https://github.com/AndrewAtanov/simclr-pytorch/blob/master/models/losses.py#L62)) \n- [Experiment Manager (exman)](https://github.com/ferrine/exman) a tool that distributes logs, checkpoints, and parameters-dicts via folders, and allows to load them in a pandas DataFrame, that is handly for processing in ipython notebooks.\n- [NVIDIA APEX](https://github.com/NVIDIA/apex) for LARS optimizer. We modeified LARC to make it consistent with SimCLR repo.\n\n## Acknowledgements\n- This work was supported in part through computational resources of HPC facilities at NRU HSE\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandrewatanov%2Fsimclr-pytorch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fandrewatanov%2Fsimclr-pytorch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandrewatanov%2Fsimclr-pytorch/lists"}