{"id":19383964,"url":"https://github.com/saadkh1/clip_dual_encoder","last_synced_at":"2026-05-07T16:40:50.662Z","repository":{"id":170395149,"uuid":"551999086","full_name":"saadkh1/clip_dual_encoder","owner":"saadkh1","description":"Visual and Vision-Language Representation Pre-Training with Contrastive Learning","archived":false,"fork":false,"pushed_at":"2023-02-19T11:33:27.000Z","size":12562,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-07T07:14:28.177Z","etag":null,"topics":["computer-vision","contrastive-language-image-pretraining","deep-learning","nlp","openai-clip","pytorch"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/saadkh1.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-10-15T15:28:17.000Z","updated_at":"2023-05-07T08:49:21.000Z","dependencies_parsed_at":null,"dependency_job_id":"a3058ae4-010f-44b6-9b7a-998585fb0eb3","html_url":"https://github.com/saadkh1/clip_dual_encoder","commit_stats":null,"previous_names":["saadkh1/clip_dual_encoder"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saadkh1%2Fclip_dual_encoder","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saadkh1%2Fclip_dual_encoder/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saadkh1%2Fclip_dual_encoder/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saadkh1%2Fclip_dual_encoder/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/saadkh1","download_url":"https://codeload.github.com/saadkh1/clip_dual_encoder/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240521037,"owners_count":19814694,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","contrastive-language-image-pretraining","deep-learning","nlp","openai-clip","pytorch"],"created_at":"2024-11-10T09:28:32.581Z","updated_at":"2026-05-07T16:40:45.609Z","avatar_url":"https://github.com/saadkh1.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Visual and Vision-Language Representation Pre-Training with Contrastive Learning\n\nIn this repository, I present my approach as a family of vision-language foundation systems. These systems, which are considered among the most advanced in the field of artificial intelligence, are used to solve a variety of important tasks, such as generation, retrieval, and classification tasks. The basis of the vision-language systems is a combination of pre-trained encoder models of computer vision and natural language processing.\n\n## Setup\n\nI used Python 3.7 and [PyTorch](https://pytorch.org/) 1.7.1 to train and test my models. It also requires the installation of timm and transformers packages with the following versions:\n\n```bash\n$ pip install timm==0.6.7\n$ pip install transformers\n```\n\n## Training the model:\n```\npython main.py\n```\n\nDuring this research, I trained our models on several image encoder models, such as deit3, efficientnet_b8, convnext, swinv2, and fbnetv3, as well as text encoder models, such as roberta, xlm-roberta, xlnet, albert, electra, and bert, on a small-scale dataset. I concluded that changing image and text encoder models would necessarily change the efficiency of the model. I was able to show that encoder models like ConvNeXt and RoBERTa produce better results than more popular encoder models like BERT and ViT. I also found that training our model on a small-scale data set produced accurate results.\n\nBelow in this table are the best results I got after testing more than 80 pairs of image and text encoders in 20 epochs.\n\n|  Image and Text Encoders  | Top-1 accuracy | Top-5 accuracy | \n|:-------------------:|:----------:|:----------:|\n|  Swin + BERT        |    13.99    |    31.65    |\n|  Swin + ALBERT      |    13.28    |    29.04    |\n| Swin + RoBERTa      |    14.83    |    32.64    |\n| ConvNeXt + BERT     |    16.63    |    35.31    |\n| ConvNeXt + ALBERT   |    13.46    |    29.46    |\n| ConvNeXt + RoBERTa  |    17.31    |    35.31    |\n\nThe model composed by the ConvNeXt and RoBERTa encoders has achieved satisfactory results on image-text retrieval tasks on some datasets, such as the ImageNetV2 dataset, the Unsplash dataset, and more other datasets, using different queries (textual queries, visual queries, and visual + textual queries) as shown in these notebooks [Search_In_Unsplash.ipynb](https://github.com/saadkh1/clip_dual_encoder/blob/main/Search_In_Unsplash.ipynb) and [Image_To_Text_Search.ipynb](https://github.com/saadkh1/clip_dual_encoder/blob/main/Image_To_Text_Search.ipynb).It also achieved these results on the video-text retrieval task for videos from YouTube [Test_video.ipynb](https://github.com/saadkh1/clip_dual_encoder/blob/main/Test_video.ipynb).This model has outperformed many other models in image classification tasks on some datasets, such as Food-101, CIFAR-10, CIFAR-100, Describable Textures (DTD), Oxford-IIIT Pets (Pets), MNIST, STL-10, the German Traffic Sign Recognition Benchmark (GTSRB), and Rendered SST2 (SST) [CR_classification.ipynb](https://github.com/saadkh1/clip_dual_encoder/blob/main/CR_classification.ipynb).Also, I tested this model on zero-shot image classification tasks as it showed its efficiency in some datasets, such as the (CIFAR-100 classes with some random images), the (puppy and bagel), and the (chihuahua and muffin) datasets [Zero-Shot Image Classification.ipynb](https://github.com/saadkh1/clip_dual_encoder/blob/main/Zero-Shot%20Image%20Classification.ipynb).\n\nI did this by applying the contrastive learning method to the image-text pair dataset, which was used to train my models. Then I used the Zero-Shot method to retrieve images or texts. My model achieved all of these results, considering the size of the Flickr30k dataset and the number of epochs used in training.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsaadkh1%2Fclip_dual_encoder","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsaadkh1%2Fclip_dual_encoder","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsaadkh1%2Fclip_dual_encoder/lists"}