{"id":15647740,"url":"https://github.com/sayakpaul/paws-tf","last_synced_at":"2025-07-07T03:05:26.587Z","repository":{"id":88793312,"uuid":"364478024","full_name":"sayakpaul/PAWS-TF","owner":"sayakpaul","description":"Minimal implementation of PAWS (https://arxiv.org/abs/2104.13963) in TensorFlow. ","archived":false,"fork":false,"pushed_at":"2021-05-25T15:36:59.000Z","size":33713,"stargazers_count":45,"open_issues_count":0,"forks_count":4,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-01-09T22:08:39.826Z","etag":null,"topics":["keras","multi-crop","representation-learning","semi-supervised-learning","soft-similarity","tensorflow","vision"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sayakpaul.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-05-05T06:10:18.000Z","updated_at":"2024-07-25T09:39:13.000Z","dependencies_parsed_at":null,"dependency_job_id":"d1b5722d-f3f9-4df8-82ea-f311ccf155f5","html_url":"https://github.com/sayakpaul/PAWS-TF","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sayakpaul%2FPAWS-TF","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sayakpaul%2FPAWS-TF/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sayakpaul%2FPAWS-TF/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sayakpaul%2FPAWS-TF/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sayakpaul","download_url":"https://codeload.github.com/sayakpaul/PAWS-TF/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":233686178,"owners_count":18714103,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["keras","multi-crop","representation-learning","semi-supervised-learning","soft-similarity","tensorflow","vision"],"created_at":"2024-10-03T12:20:48.960Z","updated_at":"2025-01-13T02:44:05.204Z","avatar_url":"https://github.com/sayakpaul.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PAWS-TF 🐾\nImplementation of [Semi-Supervised Learning of Visual Features by Non-Parametrically Predicting View Assignments with Support Samples](https://arxiv.org/abs/2104.13963)\n (PAWS) in TensorFlow (2.4.1). \n\nPAWS introduces a simple way to combine a very small fraction of labeled data with a comparatively larger corpus of unlabeled data _during_ pre-training. With its approach, it sets the state-of-the-art in semi-supervised learning (as of May 2021) beating methods like [SimCLRV2](https://arxiv.org/abs/2006.10029), [Meta Pseudo Labels](https://arxiv.org/abs/2003.10580) that too with **fewer parameters and a smaller pre-training schedule**. For details, I recommend checking out the original paper as well as [this blog post](https://ai.facebook.com/blog/dino-paws-computer-vision-with-self-supervised-transformers-and-10x-more-efficient-training/) by the authors. \n\nThis repository implements and includes all the major bits proposed in PAWS in TensorFlow. The only major difference is that the pre-training and subsequent fine-tuning weren't run for the original number of epochs (600 and 30 respectively) to save compute. I have reused the utility components for PAWS loss from the [original implementation](https://github.com/facebookresearch/suncet/).\n\n## Dataset ⌗\n\nThe current code works with CIFAR10 and uses 4000 labeled samples (**8%**) during pre-training (along with the unlabeled samples).\n\n## Features ✨\n\n- [x] Multi-crop augmentation strategy (originally introduced in [SwAV](https://arxiv.org/abs/2006.09882))\n- [x] Class stratified sampler (common in few-shot classification problems)\n- [x] WarmUpCosine learning rate schedule (which is typical for self-supervised and semi-supervised pre-training)\n- [x] LARS optimizer (comes from [TensorFlow Model Garden](https://github.com/tensorflow/models/blob/master/official/modeling/optimization/lars_optimizer.py))\n\nThe trunk portion (all, except the last classification layer) of a WideResNet-28-2 is used inside the encoder for CIFAR10. All the experimental configurations were followed from the Appendix C of the paper. \n\n## Setup and code structure 💻\n\nA GCP VM ([`n1-standard-8`](https://cloud.google.com/compute/docs/machine-types)) with a single V100 GPU was used for executing the code. \n\n* `paws_train.py` runs the pre-training as introduced in PAWS.\n* `fine_tune.py` runs the fine-tuning part as suggested in Appendix C. Note that this is only required for CIFAR10.\n* `nn_eval.py` runs the soft nearest neighbor classification on CIFAR10 test set.\n\nPre-training and fine-tuning total take **1.4 hours** to complete. All the logs are available in [`misc/logs.txt`](https://github.com/sayakpaul/PAWS-TF/blob/main/misc/logs.txt). Additionally, the indices that were used to sample the labeled examples from the CIFAR10 training set are available [here](https://github.com/sayakpaul/PAWS-TF/blob/main/misc/random_idx.npy).\n\n## Results 📊\n\n### Pre-training\n\nPAWS minimizes the cross-entropy loss (as well as maximizes mean-entropy) during pre-training. This is what the training plot indicates too:\n\n\u003cdiv align=\"center\"\u003e\n\u003cimg src=\"https://i.ibb.co/4MwCb9J/pretraining-ce-loss.png\" width=450/\u003e\n\u003c/div\u003e\n\nTo evaluate the effectivity of the pre-training, PAWS performs soft nearest neighbor classification to report the top-1 accuracy score on a given test set.\n\n### Top-1 Accuracy\n\nThis repository gets to **73.46%** top-1 accuracy on the CIFAR10 test set. Again, **note** that I only pre-trained for 50 epochs (as opposed to 600) and fine-tuned for 10 epochs (as opposed to 30). With the original schedule this score should be around 96.0%. \n\nIn the following PCA projection plot, we see that the embeddings of images (computed after fine-tuning) of PAWS are starting to be well separated:\n\n\u003cdiv align=\"center\"\u003e\n\u003cimg src=\"https://i.ibb.co/y0XB6pL/projections-viz.png\" width=450/\u003e\n\u003c/div\u003e\n\n## Notebooks 📘\n\nThere are two Colab Notebooks:\n\n* [`colabs/data_prep.ipynb`](https://github.com/sayakpaul/PAWS-TF/blob/main/colabs/data_prep.ipynb): It walks through the process of constructing a multi-crop dataset with CIFAR10.\n* [`colabs/visualization_paws_projections.ipynb`](https://github.com/sayakpaul/PAWS-TF/blob/main/colabs/visualization_paws_projections.ipynb): Visualizes the PCA projections of pre-computed embeddings. \n\n## Misc ⺟\n\n* Model weights are available [here](https://github.com/sayakpaul/PAWS-TF/releases/tag/v1.0.0) for reproducibility. \n* With mixed-precision training, the performance can further be improved. I am open to accepting contributions that would implement mixed-precision training in the current code.\n\n## Acknowledgements\n\n* Huge amount of thanks to [Mahmoud Assran](https://www.linkedin.com/in/midoassran/) (first author of PAWS) for patiently resolving my doubts.\n* [ML-GDE program](https://developers.google.com/programs/experts/) for providing GCP credit support. \n\n## Paper Citation\n\n```\n@misc{assran2021semisupervised,\n      title={Semi-Supervised Learning of Visual Features by Non-Parametrically Predicting View Assignments with Support Samples}, \n      author={Mahmoud Assran and Mathilde Caron and Ishan Misra and Piotr Bojanowski and Armand Joulin and Nicolas Ballas and Michael Rabbat},\n      year={2021},\n      eprint={2104.13963},\n      archivePrefix={arXiv},\n      primaryClass={cs.CV}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsayakpaul%2Fpaws-tf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsayakpaul%2Fpaws-tf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsayakpaul%2Fpaws-tf/lists"}