{"id":13688882,"url":"https://github.com/HobbitLong/SupContrast","last_synced_at":"2025-05-01T20:31:07.666Z","repository":{"id":37346969,"uuid":"262288044","full_name":"HobbitLong/SupContrast","owner":"HobbitLong","description":"PyTorch implementation of \"Supervised Contrastive Learning\"  (and SimCLR incidentally)","archived":false,"fork":false,"pushed_at":"2023-12-26T21:20:36.000Z","size":1472,"stargazers_count":3245,"open_issues_count":96,"forks_count":546,"subscribers_count":19,"default_branch":"master","last_synced_at":"2025-04-13T23:54:09.283Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/HobbitLong.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2020-05-08T09:58:30.000Z","updated_at":"2025-04-12T18:40:46.000Z","dependencies_parsed_at":"2022-07-08T04:42:33.875Z","dependency_job_id":"763b8f29-b616-46cf-a117-3ee02091984b","html_url":"https://github.com/HobbitLong/SupContrast","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HobbitLong%2FSupContrast","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HobbitLong%2FSupContrast/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HobbitLong%2FSupContrast/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HobbitLong%2FSupContrast/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/HobbitLong","download_url":"https://codeload.github.com/HobbitLong/SupContrast/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251940579,"owners_count":21668558,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-02T15:01:26.273Z","updated_at":"2025-05-01T20:31:07.661Z","avatar_url":"https://github.com/HobbitLong.png","language":"Python","funding_links":[],"categories":["Python","其他_机器视觉","Contrastive \u0026 Self-Supervised Learning"],"sub_categories":["网络服务_其他"],"readme":"# SupContrast: Supervised Contrastive Learning\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"figures/teaser.png\" width=\"700\"\u003e\n\u003c/p\u003e\n\nThis repo covers an reference implementation for the following papers in PyTorch, using CIFAR as an illustrative example:  \n(1) Supervised Contrastive Learning. [Paper](https://arxiv.org/abs/2004.11362)  \n(2) A Simple Framework for Contrastive Learning of Visual Representations. [Paper](https://arxiv.org/abs/2002.05709)  \n\n## Update\n\n${\\color{red}Note}$: if you found it not easy to parse the supcon loss implementation in this repo, we got you. Supcon loss essentially is just a cross-entropy loss (see eq 4 in the [StableRep](https://arxiv.org/pdf/2306.00984.pdf) paper). So we got a cleaner and simpler implementation [here](https://github.com/google-research/syn-rep-learn/blob/main/StableRep/models/losses.py#L49). Hope it helps.\n\nImageNet model (small batch size with the trick of the momentum encoder) is released [here](https://www.dropbox.com/s/l4a69ececk4spdt/supcon.pth?dl=0). It achieved \u003e 79% top-1 accuracy.\n\n## Loss Function\nThe loss function [`SupConLoss`](https://github.com/HobbitLong/SupContrast/blob/master/losses.py#L11) in `losses.py` takes `features` (L2 normalized) and `labels` as input, and return the loss. If `labels` is `None` or not passed to the it, it degenerates to SimCLR.\n\nUsage:\n```python\nfrom losses import SupConLoss\n\n# define loss with a temperature `temp`\ncriterion = SupConLoss(temperature=temp)\n\n# features: [bsz, n_views, f_dim]\n# `n_views` is the number of crops from each image\n# better be L2 normalized in f_dim dimension\nfeatures = ...\n# labels: [bsz]\nlabels = ...\n\n# SupContrast\nloss = criterion(features, labels)\n# or SimCLR\nloss = criterion(features)\n...\n```\n\n## Comparison\nResults on CIFAR-10:\n|          |Arch | Setting | Loss | Accuracy(%) |\n|----------|:----:|:---:|:---:|:---:|\n|  SupCrossEntropy | ResNet50 | Supervised   | Cross Entropy |  95.0  |\n|  SupContrast     | ResNet50 | Supervised   | Contrastive   |  96.0  | \n|  SimCLR          | ResNet50 | Unsupervised | Contrastive   |  93.6  |\n\nResults on CIFAR-100:\n|          |Arch | Setting | Loss | Accuracy(%) |\n|----------|:----:|:---:|:---:|:---:|\n|  SupCrossEntropy | ResNet50 | Supervised   | Cross Entropy |  75.3 |\n|  SupContrast     | ResNet50 | Supervised   | Contrastive   |  76.5 | \n|  SimCLR          | ResNet50 | Unsupervised | Contrastive   |  70.7 |\n\nResults on ImageNet (Stay tuned):\n|          |Arch | Setting | Loss | Accuracy(%) |\n|----------|:----:|:---:|:---:|:---:|\n|  SupCrossEntropy | ResNet50 | Supervised   | Cross Entropy |  -  |\n|  SupContrast     | ResNet50 | Supervised   | Contrastive   |  79.1 (MoCo trick)  | \n|  SimCLR          | ResNet50 | Unsupervised | Contrastive   |  -  |\n\n## Running\nYou might use `CUDA_VISIBLE_DEVICES` to set proper number of GPUs, and/or switch to CIFAR100 by `--dataset cifar100`.  \n**(1) Standard Cross-Entropy**\n```\npython main_ce.py --batch_size 1024 \\\n  --learning_rate 0.8 \\\n  --cosine --syncBN \\\n```\n**(2) Supervised Contrastive Learning**  \nPretraining stage:\n```\npython main_supcon.py --batch_size 1024 \\\n  --learning_rate 0.5 \\\n  --temp 0.1 \\\n  --cosine\n```\n\n\u003cs\u003eYou can also specify `--syncBN` but I found it not crucial for SupContrast (`syncBN` 95.9% v.s. `BN` 96.0%). \u003c/s\u003e\n\nWARN: Currently, `--syncBN` has no effect since the code is using `DataParallel` instead of `DistributedDataParaleel`\n\nLinear evaluation stage:\n```\npython main_linear.py --batch_size 512 \\\n  --learning_rate 5 \\\n  --ckpt /path/to/model.pth\n```\n**(3) SimCLR**  \nPretraining stage:\n```\npython main_supcon.py --batch_size 1024 \\\n  --learning_rate 0.5 \\\n  --temp 0.5 \\\n  --cosine --syncBN \\\n  --method SimCLR\n```\nThe `--method SimCLR` flag simply stops `labels` from being passed to `SupConLoss` criterion.\nLinear evaluation stage:\n```\npython main_linear.py --batch_size 512 \\\n  --learning_rate 1 \\\n  --ckpt /path/to/model.pth\n```\n\nOn custom dataset:\n```\npython main_supcon.py --batch_size 1024 \\\n  --learning_rate 0.5  \\ \n  --temp 0.1 --cosine \\\n  --dataset path \\\n  --data_folder ./path \\\n  --mean \"(0.4914, 0.4822, 0.4465)\" \\\n  --std \"(0.2675, 0.2565, 0.2761)\" \\\n  --method SimCLR\n```\n\nThe `--data_folder` must be of form ./path/label/xxx.png folowing https://pytorch.org/docs/stable/torchvision/datasets.html#torchvision.datasets.ImageFolder convension.\n\nand \n## t-SNE Visualization\n\n**(1) Standard Cross-Entropy**\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"figures/SupCE.jpg\" width=\"400\"\u003e\n\u003c/p\u003e\n\n**(2) Supervised Contrastive Learning**\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"figures/SupContrast.jpg\" width=\"800\"\u003e\n\u003c/p\u003e\n\n**(3) SimCLR**\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"figures/SimCLR.jpg\" width=\"800\"\u003e\n\u003c/p\u003e\n\n## Reference\n```\n@Article{khosla2020supervised,\n    title   = {Supervised Contrastive Learning},\n    author  = {Prannay Khosla and Piotr Teterwak and Chen Wang and Aaron Sarna and Yonglong Tian and Phillip Isola and Aaron Maschinot and Ce Liu and Dilip Krishnan},\n    journal = {arXiv preprint arXiv:2004.11362},\n    year    = {2020},\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FHobbitLong%2FSupContrast","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FHobbitLong%2FSupContrast","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FHobbitLong%2FSupContrast/lists"}