{"id":27128187,"url":"https://github.com/k-karna/moco_self_supervised","last_synced_at":"2025-04-07T18:47:49.750Z","repository":{"id":271892790,"uuid":"914888555","full_name":"k-karna/MoCo_Self_Supervised","owner":"k-karna","description":"Momentum Contrast (MoCo) Self Supervised Learning on STL-10 dataset. PyTorch implementation","archived":false,"fork":false,"pushed_at":"2025-03-12T11:08:43.000Z","size":9120,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-12T12:22:17.473Z","etag":null,"topics":["moco","self-supervised-learning","stl-10"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/k-karna.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-01-10T14:13:46.000Z","updated_at":"2025-03-12T11:08:46.000Z","dependencies_parsed_at":null,"dependency_job_id":"fd39d5ff-1c91-45ae-a17e-cff84e59b96f","html_url":"https://github.com/k-karna/MoCo_Self_Supervised","commit_stats":null,"previous_names":["k-karna/moco_self_supervised"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/k-karna%2FMoCo_Self_Supervised","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/k-karna%2FMoCo_Self_Supervised/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/k-karna%2FMoCo_Self_Supervised/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/k-karna%2FMoCo_Self_Supervised/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/k-karna","download_url":"https://codeload.github.com/k-karna/MoCo_Self_Supervised/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247713266,"owners_count":20983680,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["moco","self-supervised-learning","stl-10"],"created_at":"2025-04-07T18:47:48.783Z","updated_at":"2025-04-07T18:47:49.749Z","avatar_url":"https://github.com/k-karna.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"## MoCo Self Supervised Learning Method\n\nWe are using STL-10 dataset, following recommended protocol on [cs.stanford.edu/~acoates/stl10/](https://cs.stanford.edu/~acoates/stl10/)\n\u003e Perform unsupervised training on the unlabeled.\n\n\u003e Perform supervised training on the labeled data using 10 (pre-defined) folds of 100 examples from the training data. The indices of the examples to be used for each fold are provided.\n\n\u003e Report average accuracy on the full test set.\n\nAlso, following original paper [arxiv.org/pdf/1911.05722](https://arxiv.org/pdf/1911.05722) __Momentum Contrast for Unsupervised Visual Representation Learning__ modelled our MoCo from scratch with most of the hyper-parameters recommended there.\n\nNext, we are experiment with only 10 epochs (for not having GPU). Our results are given below:\n\n\n| Ex. No. | Base Encoder  | Queue Size | Mtm |Temp. | Trainable Params |Num Epochs | Learning Rate | Training Time (on unlabeled) | Accuracy on TrainingSet | Accuracy on TestSet|\n| ------ | ------ | ------ | ------ | ------ | ------ | ----- | ------| -------| ----| --- |\n| 1 | ResNet18  |  65,536 | 0.999 | 0.07 |11,242,176 | 10 | 0.001 | 8 Hours 13 Min| 45.76% | 29.21% |\n| 2 | ResNet18  |  65,536 | 0.999 | 0.07 |11,242,176 | 10 | 0.01 | 7 Hours 40 Min| 36.8% | 21.26% |\n| 3 | ResNet18  |  65,536 | 0.999 | 0.07 |11,242,176 | 10 | 0.03 | 8 Hours 10 Min| 33.84% | 20.19% |\n| 4 | ResNet18  |  65,536 | 0.999 | 0.09 |11,242,176 | 10 | 0.001 | 8 Hours 53 Min| 46.78% | 28.81% |\n| 5 | ResNet18  |  65,536 | 0.999 | 0.05 |11,242,176 | 10 | 0.001 | 8 Hours 48 Min| 45.56% | 24.86% |\n| 6 | ResNet50  |  65,536 | 0.999 | 0.05 |23,770,304 | 10 | 0.001 | 20 Hours 10 Min| 41.42% | 24.52% |\n\nCertainly, we need to train on ``unlabeled set`` for almost upto $100$ epoch as in __MoCo__ paper, and that will happen when we get access to GPU, and we will run with ResNet50 as base encoder. \n\nFor our evaluation, __we will from now on continue with ResNet18__ as ResNet50 took more than twice of ResNEt18 inference time. Next, experimenting by increasing ``num_epochs`` to 20 and 30.\nResults certainly improved. We definitely need to run the model for 100.\n\n| Ex. No. | Base Encoder | Queue Size | Mtm |Temp. | Trainable Params |Num Epochs | Learning Rate | Training Time (on unlabeled) | Accuracy on TrainingSet | Accuracy on TestSet|\n| ------ | ------ | ------ | ------ | ------ | ------ | ----- | ------| -------| ----| --- |\n| 7 | ResNet18  | 65,536 | 0.999 | 0.07 |11,242,176 | 20 | 0.001 | 16 Hours 8 Min| 46.24% | 30.63% |\n| 8 | ResNet18  | 65,536 | 0.999 | 0.07 |11,242,176 | 30 | 0.001 | 26 Hours 21 Min | 42.86% | 32.15% |\n\nNext, we are experimenting by decreasing __Momentum__ from $0.999$ to $0.95$ and reducing ``queue_size`` from 65,536 to 32768. We will run this for 15 epochs. Results are:\n\n| Ex. No. | Base Encoder  | Queue Size | Mtm |Temp. | Trainable Params |Num Epochs | Learning Rate | Training Time (on unlabeled) | Accuracy on TrainingSet | Accuracy on TestSet|\n| ------ | ------ | ------ | ------ | ------ | ------ | ----- | ------| -------| ----| --- |\n| 9 | ResNet18 |  32,768 | 0.95 | 0.07 |11,242,176 | 15 | 0.001 | 13 Hours 26 Min| 42.65% | 31.95% |\n\nDespite being used in the original MoCo paper, we were deliberately avoiding using transfomation of random resized scaling that first crops anywhere between 20-100% and resize them to image size again; but we will implement it now and see if results improve. Results, however, improved only a little (_as we ran it for 25 epochs with previous hyperparameter_)\n\n| Ex. No. | Base Encoder  | Queue Size | Mtm |Temp. | Trainable Params |Num Epochs | Learning Rate | Training Time (on unlabeled) | Accuracy on TrainingSet | Accuracy on TestSet|\n| ------ | ------ | ------ | ------ | ------ | ------ | ----- | ------| -------| ----| --- |\n| 10 | ResNet18  | 32,768 | 0.95 | 0.07 |11,242,176 | 25 | 0.001 | 19 Hours 15 Min| 55.9% | 33.89% |\n\nSo, we will try one more time without it again as we are using instead ``GaussianBlur``\n\n| Ex. No. | Base Encoder | Queue Size | Mtm |Temp. | Trainable Params |Num Epochs | Learning Rate | Training Time (on unlabeled) | Accuracy on TrainingSet | Accuracy on TestSet|\n| ------ | ------ | ------ | ------ | ------ | ------ | ----- | ------| -------| ----| --- |\n| 11 | ResNet18  | 32,768 | 0.95 | 0.07 |11,242,176 | 25 | 0.001 | 22 Hours 41 Min| 36.36% | 31.62% |\n\nResults instead perished.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fk-karna%2Fmoco_self_supervised","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fk-karna%2Fmoco_self_supervised","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fk-karna%2Fmoco_self_supervised/lists"}