{"id":32994659,"url":"https://github.com/lif31up/ssl","last_synced_at":"2026-05-13T21:35:10.448Z","repository":{"id":323847749,"uuid":"1094936964","full_name":"lif31up/SSL","owner":"lif31up","description":"Self-supervised learning from scratch","archived":false,"fork":false,"pushed_at":"2025-11-12T11:42:45.000Z","size":9,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-11-12T13:21:08.825Z","etag":null,"topics":["attention-mechanism","bert","computer-vision","encoder","from-scratch","pretraining","pytorch","semi-supervised-learning","transfer-learning","transformer","vision-transformer"],"latest_commit_sha":null,"homepage":"https://colab.research.google.com/drive/19tnhhYqi4iBZ7nbZ9NJF6YQA3LlGW8JF?usp=sharing","language":"Python","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lif31up.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-11-12T11:25:26.000Z","updated_at":"2025-11-12T12:58:13.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/lif31up/SSL","commit_stats":null,"previous_names":["lif31up/ssl"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/lif31up/SSL","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lif31up%2FSSL","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lif31up%2FSSL/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lif31up%2FSSL/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lif31up%2FSSL/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lif31up","download_url":"https://codeload.github.com/lif31up/SSL/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lif31up%2FSSL/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":284202058,"owners_count":26964370,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-11-13T02:00:06.582Z","response_time":61,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["attention-mechanism","bert","computer-vision","encoder","from-scratch","pretraining","pytorch","semi-supervised-learning","transfer-learning","transformer","vision-transformer"],"created_at":"2025-11-13T11:00:51.796Z","updated_at":"2025-11-13T11:01:04.243Z","avatar_url":"https://github.com/lif31up.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Self-Supervised Learning for Baseline ViT\nThis implementation is inspired by:\n[Distilling the Knowledge in a Neural Network (2015)](https://arxiv.org/abs/1503.02531) by Geoffrey Hinton, Oriol Vinyals, Jeff Dean.\n[An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2021)](https://arxiv.org/abs/2010.11929) by Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby.\n\n\n\nThe Vision Transformer (ViT) attains excellent results when pretrained at sufficient scale and transferred to tasks with fewer datapoints. When pretrained on the public ImageNet-21k dataset or the in-house JFT-300M dataset, ViT approaches or beats state-of-the-art image recognition benchmarks.\n\n- **Task:** Image Recognition\n- **Dataset:** MNIST\n\n### Experiment on CoLab\n\u003ca href=\"https://colab.research.google.com/drive/19tnhhYqi4iBZ7nbZ9NJF6YQA3LlGW8JF?usp=sharing\"\u003e\n  \u003cimg alt=\"colab\" src=\"https://upload.wikimedia.org/wikipedia/commons/thumb/d/d0/Google_Colaboratory_SVG_Logo.svg/2560px-Google_Colaboratory_SVG_Logo.svg.png\" width=\"160\"\u003e\u003c/img\u003e\n\u003c/a\u003e\n\n|               | **Pretrained ViT with MLPs** |\n|---------------|------------------------------|\n| **ACC (1000)** | `90.9%`                      |\n### Requirements\nTo run the code on your own machine, `run pip install -r requirements.txt`.\n```text\ntqdm\u003e=4.67.1\n```\n\n### Configuration\n`confing.py` contains the configurations settings for the two model (adapter and ViT), including the number of heads, dimensions, learning rate, and other hyperparameters.\n\n```python\nBASE_SAVE_TO = \"bins/baseViT.bin\"\nBASE_LOAD_FROM = \"bins/baseViT.bin\"\nPRETRAINED_SAVE_TO = \"bins/pretrainedViT.bin\"\nPRETRAINED_FROM = \"bins/pretrainedViT.bin\"\n\nclass BaseConfig:\n  def __init__(self):\n    self.iters = 50\n    self.batch_size = 16\n    self.dataset_len, self.testset_len = 1000, 500\n    self.dummy = None\n\n    self.n_heads = 3\n    self.n_stacks = 6\n    self.n_hidden = 3\n    self.dim = 900\n    self.output_dim = 10\n    self.bias = True\n\n    self.dropout = 0.1\n    self.attention_dropout = 0.1\n    self.eps = 1e-3\n    self.betas = (0.9, 0.98)\n    self.epochs = 5\n    self.batch_size = 16\n    self.lr = 1e-4\n    self.clip_grad = False\n    self.mask_prob = 0.3\n    self.init_weights = init_weights\n\n    self.mask_val = -1e-9\n    self.mask_ratio = 768\n\nclass AdapterConfig(BaseConfig):\n  def __init__(self):\n    super().__init__()\n    self.output_dim = 10\n```\n\n### Pretraining\n`pretrain.py` is to pretrain the model on the MNIST dataset with SSL.\n\n```python\nif __name__ == \"__main__\":\n  config = BaseConfig()\n  device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n  mnist_transform = get_transform_MNIST(input_size=90)\n  traindata, _ = load_MNIST(path='./data', transform=mnist_transform, len=(10000, 1000))\n  trainset = Masker(dataset=traindata, config=config).consolidate()\n  config.dummy = trainset.__getitem__(0)[0]\n  trainloader = DataLoader(dataset=trainset, batch_size=config.batch_size)\n  model = ViTBase(config=config)\n  train(model=model, path=BASE_SAVE_TO, config=config, trainset=trainloader, device=device)\n```\n\n### Training\n`train.py` is to pretrain the model on the MNIST dataset with Transfer Learning.\n\n```python\nif __name__ == \"__main__\":\n  base_config = BaseConfig()\n  adapter_config = AdapterConfig()\n  device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n  mnist_10_transform = get_transform_MNIST(input_size=90)\n  traindata, testdata = load_MNIST(path='./data', transform=mnist_10_transform, len=(10000, 1000))\n  trainset = Embedder(dataset=traindata, config=base_config).consolidate()\n  base_config.dummy = trainset.__getitem__(0)[0]\n  trainloader = DataLoader(dataset=trainset, batch_size=base_config.batch_size)\n  testset = Embedder(dataset=testdata, config=base_config).consolidate()\n  testloader = DataLoader(dataset=testset, batch_size=base_config.batch_size)\n  data = torch.load(f=f\"{BASE_LOAD_FROM}\", weights_only=False, map_location=device)\n  base = ViTBase(base_config).load_state_dict(data['state'])\n  model = Adapter(config=adapter_config, base=base)\n  train(model=model, path=PRETRAINED_SAVE_TO, config=adapter_config, trainset=trainloader, device=device)\n  evaluate(model=model, dataset=testloader, device=device)\n```\n\n### Evaluation\n`evaluate.py` is used to evaluate the trained model on the MNIST-10 dataset. It loads the model and embedder, processes the dataset, and computes the accuracy of the model.\n\n```python\nif __name__ == \"__main__\":\n  device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n  base_config = BaseConfig()\n  adapter_config = AdapterConfig()\n  mnist_10_transform = get_transform_MNIST(input_size=90)\n  _, testdata = load_MNIST(path='./data', transform=mnist_10_transform, len=(1, 1000))\n  testset = Embedder(dataset=testdata, config=base_config).consolidate()\n  base_config.dummy = testset.__getitem__(0)[0]\n  testloader = DataLoader(dataset=testset, batch_size=base_config.batch_size)\n  base_data = torch.load(f=BASE_LOAD_FROM, map_location=torch.device('cpu'), weights_only=True)\n  base = ViTBase(base_config)\n  base.load_state_dict(base_data['sate'])\n  adapter_data = torch.load(f=PRETRAINED_FROM, map_location=torch.device('cpu'), weights_only=True)\n  adapter = Adapter(adapter_config, base=base)\n  evaluate(model=adapter, dataset=testloader, device=device)\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flif31up%2Fssl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flif31up%2Fssl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flif31up%2Fssl/lists"}