{"id":16832383,"url":"https://github.com/yanndubs/invariant-self-supervised-learning","last_synced_at":"2025-03-22T04:30:40.652Z","repository":{"id":58029533,"uuid":"419591818","full_name":"YannDubs/Invariant-Self-Supervised-Learning","owner":"YannDubs","description":"Pytorch code for \"Improving Self-Supervised Learning by Characterizing Idealized Representations\"","archived":false,"fork":false,"pushed_at":"2022-11-27T21:38:04.000Z","size":364491,"stargazers_count":40,"open_issues_count":1,"forks_count":5,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-18T08:11:14.080Z","etag":null,"topics":["machine-learning","python","pytorch","representation-learning","self-supervised-learning"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/YannDubs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-10-21T05:22:55.000Z","updated_at":"2024-12-30T15:19:55.000Z","dependencies_parsed_at":"2022-09-04T16:11:40.960Z","dependency_job_id":null,"html_url":"https://github.com/YannDubs/Invariant-Self-Supervised-Learning","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/YannDubs%2FInvariant-Self-Supervised-Learning","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/YannDubs%2FInvariant-Self-Supervised-Learning/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/YannDubs%2FInvariant-Self-Supervised-Learning/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/YannDubs%2FInvariant-Self-Supervised-Learning/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/YannDubs","download_url":"https://codeload.github.com/YannDubs/Invariant-Self-Supervised-Learning/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244907420,"owners_count":20529850,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["machine-learning","python","pytorch","representation-learning","self-supervised-learning"],"created_at":"2024-10-13T11:48:46.734Z","updated_at":"2025-03-22T04:30:39.933Z","avatar_url":"https://github.com/YannDubs.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Lossy Compression for Lossless Prediction [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/YannDubs/lossyless/blob/main/LICENSE) [![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/release/python-380/)\n\n\nThis repostiory contains pretrained weights from and the original implementation of [Improving Self-Supervised Learning by Characterizing Idealized Representations](https://arxiv.org/abs/2209.06235),\nwhich derives a simple uniying framework for invariant self-supervised learning (ISSL).\nOur framework provides actionable insights into ISSL that lead to important empirical gains such as how to:\n- [**Simplify non-contrastive ISSL using our DISSL objective**](#dissl-tinyimagenet) (no momentum encoders / no stop-gradients / ... )\n- [**Choose the dimensionality of representations**](#dimensionality) \n- [**Choose the architecture of projection probes**](#projection-heads) \n- [**Choose the augmentations**](#augmentations)\n\nThe following provides the code load our ImageNet pretrained models, to reproduce our key results, and minimal notebook implementations of our DISSL [![DISSL](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/YannDubs/Invariant-Self-Supervised-Learning/blob/main/notebooks/minimal_dissl.ipynb) and CISSL [![Training](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/YannDubs/Invariant-Self-Supervised-Learning/blob/main/notebooks/minimal_cissl.ipynb).\n\n\n## DISSL\n\n\n\nhttps://user-images.githubusercontent.com/24327668/204160941-fd0a2d0d-fe4e-49b7-b621-6139bcfc52be.mov\n\n\n\n\n\nOur DISSL objective is a very simple non-contrastive objective that outperforms previous baselines. \n\nWe release our pretrained weights on torch hub. \nTo load any of our model use:\n```python\nimport torch\n\nmodel = torch.hub.load('YannDubs/Invariant-Self-Supervised-Learning:main', \n                       'dissl_resnet50_d8192_e800_m8')\n```\n\nHere are all available models with their respective linear probing performance on ImageNet.\nThey are all ResNet50 trained with a batch size of 2560 and 16fp on 8 A100.\n\n| Epochs | Dimensionality | Multi-crop    |  ImageNet top-1 acc. | ImageNet top-5 acc. |                TorchHub name |                                                                                                                                   Weights | \n|--------|----------------|---------------|---------------------:|--------------------:|-----------------------------:|------------------------------------------------------------------------------------------------------------------------------------------:|\n| 100    | 2048           | 2x224         |                 66.9 |                87.5 | dissl_resnet50_dNone_e100_m2 | [model](https://github.com/YannDubs/Invariant-Self-Supervised-Learning/releases/download/v1.0.0-alpha/dissl_resnet50_dNone_e100_m2.torch) | \n| 100    | 8192           | 2x224         |                 68.9 |                88.5 | dissl_resnet50_d8192_e100_m2 | [model](https://github.com/YannDubs/Invariant-Self-Supervised-Learning/releases/download/v1.0.0-alpha/dissl_resnet50_d8192_e100_m2.torch) |\n| 100    | 8192           | 2x160 + 4x96  |                 70.7 |                88.5 | dissl_resnet50_d8192_e100_m6 | [model](https://github.com/YannDubs/Invariant-Self-Supervised-Learning/releases/download/v1.0.0-alpha/dissl_resnet50_d8192_e100_m6.torch) |\n| 400    | 2048           | 2x224         |                 71.1 |                90.2 | dissl_resnet50_dNone_e400_m2 | [model](https://github.com/YannDubs/Invariant-Self-Supervised-Learning/releases/download/v1.0.0-alpha/dissl_resnet50_dNone_e400_m2.torch) | \n| 400    | 2048           | 2x160 + 4x96  |                 73.0 |                91.3 | dissl_resnet50_dNone_e400_m6 | [model](https://github.com/YannDubs/Invariant-Self-Supervised-Learning/releases/download/v1.0.0-alpha/dissl_resnet50_dNone_e400_m6.torch) |     \n| 400    | 8192           | 2x160 + 4x96  |                 74.0 |                91.9 | dissl_resnet50_d8192_e400_m6 | [model](https://github.com/YannDubs/Invariant-Self-Supervised-Learning/releases/download/v1.0.0-alpha/dissl_resnet50_d8192_e400_m6.torch) |\n| 800    | 8192           | 2x224 + 6x96  |                 73.9 |                91.9 | dissl_resnet50_d8192_e800_m8 | [model](https://github.com/YannDubs/Invariant-Self-Supervised-Learning/releases/download/v1.0.0-alpha/dissl_resnet50_d8192_e800_m8.torch) |\n\nFor an example of how to use the pretrained models see: [![Minimal training of DISSL](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/YannDubs/Invariant-Self-Supervised-Learning/blob/main/notebooks/torchhub.ipynb). \n\nWe also provide a minimal DISSL implementation: [![Minimal training of DISSL](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/YannDubs/Invariant-Self-Supervised-Learning/blob/main/notebooks/minimal_dissl.ipynb)\n\n\n## Reproducing main results \n\n\n\nTo reproduce our key TinyImageNet results you need to install ISSL (see below) and run the desired script in `bin/tinyimagenet/*.sh`.\nTo run the script without slurm use `bin/tinyimagenet/*.sh -s none`.\nIf you want to use slurm then you need to define the desired configs in `config/server` for an example see `nlprun` or `vector` which can be called using `bin/tinyimagenet/*.sh -s \u003cserver_name\u003e`.\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eInstallation\u003c/b\u003e\u003c/summary\u003e\n\n1. Clone repository\n2. Install [PyTorch](https://pytorch.org/) \u003e=  1.9\n3. `pip install -r requirements.txt`. \n\nIf you have issues try installing out exact dependencies using `conda env update --file environment.yaml`.\n\n\u003c/details\u003e\n\nFor our ImageNet models we used [VISSL](www.vissl.ai). The exact commands can be seen on this (still uncleaned/undocumented) [VISSL fork](https://github.com/YannDubs/vissl) and we aim to incorporate DISSL in the main VISSL soon.\n\n\n### DISSL TinyImageNet\n\nThe right column in Table 1 of our paper shows empirically that DISSL outperforms DINO on TinyImageNet.\nTo reproduce a similar table (single seed) run `bin/tinyimagenet/table1_distillation.sh -s none` (no servers/slurm).\nOnce the script is finished you can collect and print the results by running `bin/tinyimagenet/table1_distillation.sh -v \"\" -a is_force_gpu=False`.\nYou should get the following results printed:\n\n| Model    | TinyImageNet Linear probing acc. |\n|:---------|---------------------------------:|\n| DINO     |                            43.3% |\n| DISSL    |                            45.1% |\n| + dim.   |                            48.0% |\n| + epochs |                            49.0% |\n | + aug.   |                            50.7% |\n\nTraining curves: \u003ca href=\"https://wandb.ai/issl/issl_opensource/groups/table1_distillation\"\u003e\u003cimg src=\"https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg\" height=\"18\"  /\u003e\u003ca/\u003e\n\n\n### Dimensionality\n\nIn our paper we characterize exactly the minimal and sufficient dimensionality depending on the probing architecture.\nFor linear probes it's much larger than standard dimensionalities, which suggests that one would gain important gains by increasing dimensionality. \nFigure 7c of our paper shows empirically that this is indeed the case.\nTo reproduce a similar figure (single seed) run `bin/tinyimagenet/fig7c_dimensions.sh -s none`.\nOnce the script is finished you can collect and print the results by running `bin/tinyimagenet/fig7c_dimensions.sh -v \"\" -a is_force_gpu=False`.\nThe following figure will then be saved in `results/exp_fig7c_dimensions/fig7c.pdf`.\n\n\u003cp float=\"left\" align=\"middle\"\u003e\n\u003cimg src=\"assets/fig7c.png\" alt=\"Fig7c: Effect of dimensionality\" width=\"400\"/\u003e\n\u003c/p\u003e\n\n\nTraining curves: \u003ca href=\"https://wandb.ai/issl/issl_opensource/groups/fig7c_dimensions\"\u003e\u003cimg src=\"https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg\" height=\"18\"  /\u003e\u003ca/\u003e\n\n\n### Projection heads\n\n\n\n\nhttps://user-images.githubusercontent.com/24327668/204155086-04e07066-f988-47af-bc3c-3a3090f1c57e.mov\n\n\n\n\n\nIn our paper, we prove that one of the two projection heads needs to have the same architecture as the dowsntream probe.\nThis is to ensure that the SSL representations are pretrained the same way as they will be used in downstream tasks.\n\n\nThis is the difference between our CISSL and SimCLR. \nThe left column in Table 1 of our paper shows empirically that this improves performance.\nTo reproduce a similar table (single seed) run `bin/tinyimagenet/table1_contrastive.sh -s none` (no servers/slurm).\nOnce the script is finished you can collect and print the results by running `bin/tinyimagenet/table1_contrastive.sh -v \"\" -a is_force_gpu=False`.\nYou should get the following results printed:\n\n| Model    | TinyImageNet Linear probing acc. |\n|:---------|---------------------------------:|\n| SimCLR   |                            45.2% |\n| CISSL    |                            45.8% |\n| + dim.   |                            47.6% |\n| + epochs |                            48.7% |\n | + aug.   |                            51.2% |\n\nTraining curves: \u003ca href=\"https://wandb.ai/issl/issl_opensource/groups/table1_contrastive\"\u003e\u003cimg src=\"https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg\" height=\"18\"  /\u003e\u003ca/\u003e\n\n\nWe also provide a minimal CISSL implementation: [![Minimal training of CISSL](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/YannDubs/Invariant-Self-Supervised-Learning/blob/main/notebooks/minimal_cissl.ipynb)\n\n\n\n### Augmentations\n\nIn our paper we characterize exactly optimal sample efficiency as a function of how coarse the equivalence class induced by augmentations are.\nIn particular, our theory suggests that stronger label-preserving augmentations improve performance.\nFigure 7a of our paper shows empirically that this is indeed the case.\nTo reproduce a similar figure (single seed) run `bin/tinyimagenet/fig7a_augmentations.sh`.\nThe following figure will then be saved as `results/exp_fig7a_augmentations/fig7a.pdf`.\n\n\u003cp float=\"left\" align=\"middle\"\u003e\n\u003cimg src=\"assets/fig7a.png\" alt=\"Fig7a: Effect of augmentations\" width=\"600\"/\u003e\n\u003c/p\u003e\n\n\nTraining curves: \u003ca href=\"https://wandb.ai/issl/issl_opensource/groups/fig7a_augmentations\"\u003e\u003cimg src=\"https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg\" height=\"18\"  /\u003e\u003ca/\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyanndubs%2Finvariant-self-supervised-learning","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyanndubs%2Finvariant-self-supervised-learning","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyanndubs%2Finvariant-self-supervised-learning/lists"}