{"id":15783648,"url":"https://github.com/deepmancer/clip-object-detection","last_synced_at":"2025-05-13T01:05:20.698Z","repository":{"id":189037427,"uuid":"679920081","full_name":"deepmancer/clip-object-detection","owner":"deepmancer","description":" Zero-shot object detection with CLIP, utilizing Faster R-CNN for region proposals.","archived":false,"fork":false,"pushed_at":"2024-12-09T14:43:58.000Z","size":28876,"stargazers_count":20,"open_issues_count":0,"forks_count":4,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-13T01:05:07.874Z","etag":null,"topics":["clip","deep-learning","faster-rcnn","object-detection","openai-clip","rcnn","region-proposal","zero-shot-object-detection"],"latest_commit_sha":null,"homepage":"https://deepmancer.github.io/clip-object-detection/","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/deepmancer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-08-17T23:37:02.000Z","updated_at":"2025-05-05T23:08:46.000Z","dependencies_parsed_at":"2023-08-18T00:58:53.371Z","dependency_job_id":"f9ba661c-ed13-49d8-a740-c74a3e40fb3b","html_url":"https://github.com/deepmancer/clip-object-detection","commit_stats":null,"previous_names":["alirezahr79/clip-object-detection","alirezaheidari-cs/clip-object-detection","deepmancer/clip-object-detection"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepmancer%2Fclip-object-detection","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepmancer%2Fclip-object-detection/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepmancer%2Fclip-object-detection/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepmancer%2Fclip-object-detection/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/deepmancer","download_url":"https://codeload.github.com/deepmancer/clip-object-detection/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253850874,"owners_count":21973672,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clip","deep-learning","faster-rcnn","object-detection","openai-clip","rcnn","region-proposal","zero-shot-object-detection"],"created_at":"2024-10-04T20:00:25.343Z","updated_at":"2025-05-13T01:05:20.682Z","avatar_url":"https://github.com/deepmancer.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🚀 CLIP Zero-Shot Object Detection\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://img.shields.io/badge/PyTorch-%23EE4C2C.svg?style=for-the-badge\u0026logo=PyTorch\u0026logoColor=white\" alt=\"PyTorch\"\u003e\n  \u003cimg src=\"https://img.shields.io/badge/OpenAI-412991.svg?style=for-the-badge\u0026logo=OpenAI\u0026logoColor=white\" alt=\"OpenAI\"\u003e\n  \u003cimg src=\"https://img.shields.io/badge/Python-3670A0?style=for-the-badge\u0026logo=Python\u0026logoColor=ffdd54\" alt=\"Python\"\u003e\n  \u003cimg src=\"https://img.shields.io/badge/Jupyter-F37626.svg?style=for-the-badge\u0026logo=Jupyter\u0026logoColor=white\" alt=\"Jupyter Notebook\"\u003e\n  \u003cimg src=\"https://img.shields.io/badge/license-MIT-blue.svg?style=for-the-badge\" alt=\"License\"\u003e\n\u003c/p\u003e\n\n\u003e **Detect objects in images without training!**\n\nWelcome to the **CLIP Zero-Shot Object Detection** project! This repository demonstrates how to perform zero-shot object detection by integrating OpenAI's **CLIP** (Contrastive Language-Image Pretraining) model with a **Faster R-CNN** for region proposal generation.\n\n---\n\n| **Source Code** | **Website** |\n|:-----------------|:------------|\n| \u003ca href=\"https://github.com/deepmancer/clip-object-detection\" target=\"_blank\"\u003egithub.com/deepmancer/clip-object-detection\u003c/a\u003e | \u003ca href=\"https://deepmancer.github.io/clip-object-detection/\" target=\"_blank\"\u003edeepmancer.github.io/clip-object-detection\u003c/a\u003e |\n\n\n---\n\n## 🎯 Quick Start\n\nSet up and run the pipeline in three simple steps:\n\n1. **Clone the Repository**:\n\n   ```bash\n   git clone https://github.com/deepmancer/clip-object-detection.git\n   cd clip-object-detection\n   ```\n\n2. **Install Dependencies**:\n  \n   ```bash\n   pip install -r requirements.txt\n   ```\n\n3. **Run the Notebook**:\n\n   ```bash\n   jupyter notebook clip_object_detection.ipynb\n   ```\n\n---\n\n## 🤔 What is CLIP?\n\n**CLIP** (Contrastive Language–Image Pretraining) is trained on 400 million image-text pairs. It embeds images and text into a shared space where the cosine similarity between embeddings reflects their semantic relationship.\n\n\u003cdiv align=\"center\"\u003e\n    \u003cfigure\u003e\n        \u003cimg \n            src=\"https://raw.githubusercontent.com/deepmancer/clip-object-detection/main/assets/CLIP.png\" \n            width=\"600\" \n            alt=\"CLIP Model Architecture\"\n        /\u003e\n        \u003cfigcaption\u003e\n            CLIP Model Architecture - \u003ca href=\"https://arxiv.org/abs/2103.00020\"\u003ePaper\u003c/a\u003e\n        \u003c/figcaption\u003e\n    \u003c/figure\u003e\n\u003c/div\u003e\n\n---\n\n## 🔍 Methodology\n\nOur approach combines CLIP and Faster R-CNN for zero-shot object detection:\n\n1. **📦 Region Proposal**: Use Faster R-CNN to identify potential object locations.\n2. **🎯 CLIP Embeddings**: Encode image regions and text descriptions into a shared embedding space.\n3. **🔍 Similarity Matching**: Compute cosine similarity between text and image embeddings to identify matches.\n4. **✨ Results**: Highlight detected objects with their confidence scores.\n\n---\n\n## 📊 Example Results\n\n### Input Image\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://raw.githubusercontent.com/deepmancer/clip-object-detection/main/assets/original_image.png\" width=\"450\" alt=\"Original Image\"\u003e\n\u003c/p\u003e\n\n### Region Proposals\n\nRegions proposed by Faster R-CNN's RPN:\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://raw.githubusercontent.com/deepmancer/clip-object-detection/main/assets/regions.png\" width=\"450\" alt=\"Candidate Regions\"\u003e\n\u003c/p\u003e\n\n### Detected Objects\n\nObjects detected by CLIP based on textual queries:\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://raw.githubusercontent.com/deepmancer/clip-object-detection/main/assets/clip_result.png\" width=\"450\" alt=\"Detected Objects\"\u003e\n\u003c/p\u003e\n\n---\n\n## 📦 Requirements\n\nEnsure the following are installed:\n\n- **PyTorch**: Deep learning framework.\n- **Torchvision**: Pre-trained Faster R-CNN.\n- **OpenAI CLIP**: [GitHub Repository](https://github.com/openai/CLIP.git).\n- Additional dependencies are listed in [requirements.txt](requirements.txt).\n\n---\n\n## 📝 License\n\nThis project is licensed under the [MIT License](LICENSE). Feel free to use, modify, and distribute the code.\n\n---\n\n## ⭐ Support the Project\n\nIf this project inspires or assists your work, please consider giving it a ⭐ on GitHub! Your support motivates us to continue improving and expanding this repository.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeepmancer%2Fclip-object-detection","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdeepmancer%2Fclip-object-detection","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeepmancer%2Fclip-object-detection/lists"}