{"id":13738420,"url":"https://github.com/yukimasano/self-label","last_synced_at":"2025-05-08T16:33:50.607Z","repository":{"id":38458537,"uuid":"229603018","full_name":"yukimasano/self-label","owner":"yukimasano","description":"Self-labelling via simultaneous clustering and representation learning. (ICLR 2020)","archived":false,"fork":false,"pushed_at":"2024-07-25T10:49:30.000Z","size":94,"stargazers_count":528,"open_issues_count":4,"forks_count":53,"subscribers_count":13,"default_branch":"master","last_synced_at":"2024-08-04T03:12:30.293Z","etag":null,"topics":["clustering","iclr2020","representation-learning","resnet","resnet-v2","self-supervised-learning"],"latest_commit_sha":null,"homepage":"http://www.robots.ox.ac.uk/~vgg/research/self-label/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yukimasano.png","metadata":{"files":{"readme":"README.md","changelog":"change_state_dict.py","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2019-12-22T17:14:27.000Z","updated_at":"2024-07-25T07:58:31.000Z","dependencies_parsed_at":"2024-04-16T22:03:41.225Z","dependency_job_id":"24b1292e-ffcb-4b07-a000-63098578b18e","html_url":"https://github.com/yukimasano/self-label","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yukimasano%2Fself-label","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yukimasano%2Fself-label/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yukimasano%2Fself-label/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yukimasano%2Fself-label/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yukimasano","download_url":"https://codeload.github.com/yukimasano/self-label/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224746694,"owners_count":17363092,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clustering","iclr2020","representation-learning","resnet","resnet-v2","self-supervised-learning"],"created_at":"2024-08-03T03:02:21.986Z","updated_at":"2024-11-15T07:31:01.637Z","avatar_url":"https://github.com/yukimasano.png","language":"Python","funding_links":[],"categories":["Python","Computer Vision"],"sub_categories":["Image Representation Learning"],"readme":"# Self-labelling via simultaneous clustering and representation learning\n\n🆗🆗🎉 _NEW models (20th August 2020): Added standard SeLa pretrained torchvision ResNet models to make loading much easier + added baselines using better MoCov2 augmentation (~69% LP performance) + added evaluation with K=1000 for ImageNet \"unuspervised clustering\"_\n\n🆕✅🎉 _updated code: 23rd April 2020: bug fixes + CIFAR code + evaluation for resnet \u0026 alexnet._\n\nCheckout our [blogpost](http://www.robots.ox.ac.uk/~vgg/blog/self-labelling-via-simultaneous-clustering-and-representation-learning.html) for a quick non-technical overview and an interactive visualization of our clusters.\n\n## Self-Label\n\nThis code is the official implementation of the ICLR 2020 paper [Self-labelling via simultaneous clustering and representation learning](https://openreview.net/forum?id=Hyx-jyBFPr).\n\n### Abstract\nCombining clustering and representation learning is one of the most promising\napproaches for unsupervised learning of deep neural networks. However, doing\nso naively leads to ill posed learning problems with degenerate solutions. In this\npaper, we propose a novel and principled learning formulation that addresses\nthese issues. The method is obtained by maximizing the information between\nlabels and input data indices. We show that this criterion extends standard crossentropy minimization to an optimal transport problem, which we solve efficiently\nfor millions of input images and thousands of labels using a fast variant of the\nSinkhorn-Knopp algorithm. The resulting method is able to self-label visual data\nso as to train highly competitive image representations without manual labels. Our\nmethod achieves state of the art representation learning performance for AlexNet\nand ResNet-50 on SVHN, CIFAR-10, CIFAR-100 and ImageNet.\n\n### Results at a glance\n\n|                     | NMI(%) | aNMI(%) | ARI(%) | LP Acc (%) | \n|---------------------|--------|---------|--------|------------|\n| AlexNet 1k          | 50.5   | 12.2    | 2.7    | 42.1       |  \n| AlexNet 10k         | 66.4   | 4.7     | 4.7    | 43.8       |  \n| R50 10x3k           | 54.2   | 34.4    | 7.2    | 61.5       |  \n\n#### With better augmentations (all single crop)\n\n|                      | Label-Acc  | NMI(%) | aNMI(%) | ARI(%) | LP Acc (%) | model_weights |\n|----------------------|------|--------|---------|--------|------------|---------------|\n| Aug++ R18  1k (new)  | 26.9 | 62.7   | 36.4    | 12.5   | 53.3       | [here](http://www.robots.ox.ac.uk/~vgg/research/self-label/asset/new_models/resnet18-1k_pp.pth) |\n| Aug++ R50  1k (new)  | 30.5 | 65.7   | 42.0    | 16.2   | 63.5       | [here](http://www.robots.ox.ac.uk/~vgg/research/self-label/asset/new_models/resnet50-1k_pp.pth) |\n| Aug++ R50 10x3k (new)| 38.1 |75.7   | 52.8    | 27.6   | 68.8       | [here](http://www.robots.ox.ac.uk/~vgg/research/self-label/asset/new_models/resnet50-10x3k_pp.pth) |\n|(MoCo-v2 + k-means**, K=3k)      |  |71.4   | 39.6    | 15.8   | 71.1       |               |\n\n* \"Aug++\" refers to the better augmentations used in SimCLR, taken from the [MoCo-v2 repo](https://github.com/facebookresearch/moco/blob/master/main_moco.py#L225), but I still only trained for 280 epochs, with three lr-drops as in [CMC](https://github.com/HobbitLong/CMC/blob/master/train_CMC.py#L50).\n* There are still further improvements to be made with a MLP or training 800 epochs (I train 280), as done in SimCLR, [MoCov2](https://github.com/facebookresearch/moco) and [SwAV](https://github.com/facebookresearch/swav).\n* **MoCo-v2 uses 800 epochs, MLP and cos-lr-schedule. On MoCo-v2 I run k-means (K=3000) on the avg-pooled features (after the MLP-head it's pretty much the same performance) to obtain NMI, aNMI and ARI numbers.\n* Models above use standard torchvision ResNet backbones so loading is now super easy:\n```\nimport torch, torchvision\nmodel = torchvision.models.resnet50(pretrained=False, num_classes=3000)\nckpt = torch.load('resnet50-10x3k_pp.pth')\nmodel.load_state_dict(ckpt['state_dict'])\npseudolabels = ckpt['labels']\n```\n* note on improvement potential: by just using \"aug+\": I get LP-accuracy of 67.2% after 200 epochs. MoCo-v2 with \"aug+\" only has 63.4% after 200 epochs.\n\n## Clusters that were discovered by our method\n*Sorted*\n\n![Imagenet validation images with clusters sorted by imagenet purity](https://www.robots.ox.ac.uk/~vgg/research/self-label/asset/sorted-clusters.png)\n\n*Random*\n\n![Imagenet validation images with random clusters](https://www.robots.ox.ac.uk/~vgg/research/self-label/asset/random-clusters.png)\n\nThe edge-colors encode the true imagenet classes (which are not used for training).\nYou can view all clusters [here](http://www.robots.ox.ac.uk/~vgg/blog/self-labelling-via-simultaneous-clustering-and-representation-learning.html).\n\n## Requirements\n* Python \u003e3.6\n* PyTorch \u003e 1.0\n* CUDA\n* Numpy, SciPy\n* also, see requirements.txt\n* (optional:) TensorboardX\n\n## Running our code\nRun the self-supervised training of an AlexNet with the command\n```\n$./scripts/alexnet.sh\n```\nor train a ResNet-50 with\n```\n$./scripts/resnet.sh\n```\nNote: you need to specify your dataset directory (it expects a format just like ImageNet with \"train\" and \"val\" folders). You also need to give the code enough GPUs to allow for storage of activations on the GPU. Otherwise you need to use the CPU variant which is significantly slower.\n\nFull documentation of the unsupervised training code `main.py`:\n```\nusage: main.py [-h] [--epochs EPOCHS] [--batch-size BATCH_SIZE] [--lr LR]\n               [--lrdrop LRDROP] [--wd WD] [--dtype {f64,f32}] [--nopts NOPTS]\n               [--augs AUGS] [--paugs PAUGS] [--lamb LAMB] [--cpu]\n               [--arch ARCH] [--archspec {big,small}] [--ncl NCL] [--hc HC]\n               [--device DEVICE] [--modeldevice MODELDEVICE] [--exp EXP]\n               [--workers WORKERS] [--imagenet-path IMAGENET_PATH]\n               [--comment COMMENT] [--log-intv LOG_INTV] [--log-iter LOG_ITER]\n\nPyTorch Implementation of Self-Label\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --epochs EPOCHS       number of epochs\n  --batch-size BATCH_SIZE\n                        batch size (default: 256)\n  --lr LR               initial learning rate (default: 0.05)\n  --lrdrop LRDROP       multiply LR by 0.1 every (default: 150 epochs)\n  --wd WD               weight decay pow (default: (-5)\n  --dtype {f64,f32}     SK-algo dtype (default: f64)\n  --nopts NOPTS         number of pseudo-opts (default: 100)\n  --augs AUGS           augmentation level (default: 3)\n  --paugs PAUGS         for pseudoopt: augmentation level (default: 3)\n  --lamb LAMB           for pseudoopt: lambda (default:25)\n  --cpu                 use CPU variant (slow) (default: off)\n  --arch ARCH           alexnet or resnet (default: alexnet)\n  --archspec {big,small}\n                        alexnet variant (default:big)\n  --ncl NCL             number of clusters per head (default: 3000)\n  --hc HC               number of heads (default: 1)\n  --device DEVICE       GPU devices to use for storage and model\n  --modeldevice MODELDEVICE\n                        GPU numbers on which the CNN runs\n  --exp EXP             path to experiment directory\n  --workers WORKERS     number workers (default: 6)\n  --imagenet-path IMAGENET_PATH\n                        path to folder that contains `train` and `val`\n  --comment COMMENT     name for tensorboardX\n  --log-intv LOG_INTV   save stuff every x epochs (default: 1)\n  --log-iter LOG_ITER   log every x-th batch (default: 200)\n```\n\n## Evaluation\n### Linear Evaluation\nWe provide the linear evaluation methods in this repo.\nSimply download the models via `. ./scripts/download_models.sh` and then either run `scripts/eval-alexnet.sh` or `scripts/eval-resnet.sh`.\n\n### Pascal VOC\nWe follow the standard evaluation protocols for self-supervised visual representation learning.\n* for Classification: we follow the PyTorch implementation of [DeepCluster](https://github.com/facebookresearch/deepcluster) with frozen BatchNorm.\n* for Segmentation: we follow the implmentation from the [Colorization paper](https://github.com/richzhang/colorization) which uses the [FCN repo](https://github.com/shelhamer/fcn.berkeleyvision.org). Note: requires the Caffe framework\n* for Detection: we follow [Krähenbühl et al.'s implementation](https://www.philkr.net/2001/02/01/pub/)\nbased on the [Faster RCNN](https://github.com/rbgirshick/py-faster-rcnn). Note: requires the Caffe framework\n\n## Our extracted pseudolabels\nAs we show in the paper, the pseudolabels we generate from our training can be used to quickly train a neural network with regular cross-entropy.\nMoreover they seem to correctly group together similar images. Hence we provide the labels for everyone to use.\n### AlexNet\nYou can download the pseudolabels from our best (raw) AlexNet model with 10x3000 clusters [here](http://www.robots.ox.ac.uk/~vgg/research/self-label/asset/alexnet-labels.csv).\n### ResNet\nYou can download the pseudolabels from our best ResNet model with 10x3000 clusters [here](http://www.robots.ox.ac.uk/~vgg/research/self-label/asset/resnet-labels.csv).\n\n## Trained models\nYou can also download our trained models by running\n```\n$./scripts/download_models.sh\n```\nUse them like this:\n```\nimport torch\nimport models\nd = torch.load('self-label_models/resnet-10x3k.pth')\nm = models.resnet(num_classes = [3000]*10)\nm.load_state_dict(d)\n\nd = torch.load('self-label_models/alexnet-10x3k-wRot.pth')\nm = models.alexnet(num_classes = [3000]*10)\nm.load_state_dict(d)\n\n```\n\n## Reference\n\nIf you use this code etc., please cite the following paper:\n\nYuki M. Asano, Christian Rupprecht and Andrea Vedaldi.  \"Self-labelling via simultaneous clustering and representation learning.\" Proc. ICLR (2020)\n\n```\n@inproceedings{asano2020self,\n  title={Self-labelling via simultaneous clustering and representation learning},\n  author={Asano, Yuki M. and Rupprecht, Christian and Vedaldi, Andrea},\n  booktitle={International Conference on Learning Representations (ICLR)},\n  year={2020},\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyukimasano%2Fself-label","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyukimasano%2Fself-label","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyukimasano%2Fself-label/lists"}