{"id":24315088,"url":"https://github.com/ivan-zhou/image-search","last_synced_at":"2025-04-12T19:10:53.004Z","repository":{"id":165807837,"uuid":"600905095","full_name":"Ivan-Zhou/image-search","owner":"Ivan-Zhou","description":"Simple Image Search powered by Multimodal Foundation Models (OpenAI Clip and Microsoft GLIP)","archived":false,"fork":false,"pushed_at":"2023-03-22T18:36:11.000Z","size":17785,"stargazers_count":16,"open_issues_count":1,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-26T13:21:23.049Z","etag":null,"topics":["clip","deep-learning","foundation-models","glip","image-search-engine","openai","pytorch","streamlit"],"latest_commit_sha":null,"homepage":"https://www.ivanzhou.me/blog/2023/3/19/fast-and-simple-image-search-with-foundation-models","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Ivan-Zhou.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-02-12T23:35:16.000Z","updated_at":"2024-08-24T05:43:16.000Z","dependencies_parsed_at":null,"dependency_job_id":"92954257-5e22-44aa-8f6f-abf7fe859127","html_url":"https://github.com/Ivan-Zhou/image-search","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Ivan-Zhou%2Fimage-search","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Ivan-Zhou%2Fimage-search/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Ivan-Zhou%2Fimage-search/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Ivan-Zhou%2Fimage-search/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Ivan-Zhou","download_url":"https://codeload.github.com/Ivan-Zhou/image-search/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248618277,"owners_count":21134200,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clip","deep-learning","foundation-models","glip","image-search-engine","openai","pytorch","streamlit"],"created_at":"2025-01-17T10:24:54.935Z","updated_at":"2025-04-12T19:10:52.979Z","avatar_url":"https://github.com/Ivan-Zhou.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Fast and Easy Image Search\nDevelope an image search application that uses multimodal foundation models to search for highly accurate and relevant results instantly.\n\n## Installation\n### General Requirements\nCreate a Conda environment and install the required packages:\n```\nconda create -n image-search python=3.10\npip install -r requirements.txt\n```\n\nCheck out [PyTorch page](https://pytorch.org/get-started/locally/) for the installation guide for your system.\nFor example, I installed torch-1.13.1+cu116 torchaudio-0.13.1+cu116 torchvision-0.14.1+cu116 for my CUDA environment.\n```\npip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu116\n```\n\n### Install FAISS\nInstall faiss from conda-forge ([details](https://github.com/facebookresearch/faiss/blob/main/INSTALL.md#installing-from-conda-forge)) for 100x faster vector distance search.\n\n```\n# CPU version\nconda install -c conda-forge faiss-cpu\n\n# GPU version\nconda install -c conda-forge faiss-gpu\n```\n\n### Install GLIP\nWe found GLIP helpful at improving the precision of image search, but it is optional. You can launch this app without intalling GLIP.\nGLIP has strict requirements on the environment (e.g. PyTorch 1.9.1 and CUDA 11.1).\n\nPull the submodule:\n```\ngit submodule update --init --recursive\n```\n\nInstall dependencies and build the package:\n```\nsh .install_glip.sh\n```\n\n## Run a local app\nHere is the system design of the image search app:\n![System Design](resources/system_design.png)\n\n### Data\nYou can try searching with your own photos. That will be more fun! If you don’t have images to try on. I prepared a subset of ImageNet with 10,000 samples. You can download them [here](https://drive.google.com/file/d/1VfVpZ-Lrpx853WaoVtBygruOfLQGKODi/view?usp=share_link).\n\nWe manage the paths to the data directory and indexed file path in `app/constants.py`. You can change the paths to your own data directory and indexed file path.\n\n### Launch the app\nThe local app is a simple web app that allows you to select a dataset and type a query to search:\n\n![Local app](resources/app_screenshot.png)\n\nBefore you run the app, make sure you've prepared the dataset and set up the right path with `DATA_DIR` variable in `app/constants.py`.\n\nYou can run the local app by executing the following command:\n```\nsh execute.sh\n```\n\n### Optimized Implementation\nRun indexer first for faster image feature retrieval:\n```\npython app/indexer.py\n```\n\n## Benchmarks\n### Performance\nWe evaluated the performance of both systems using 20 image search queries. First, we generated candidate queries with ChatGPT, and then hand-picked the ones that were suitable.\nWe report the performance metrics of the CLIP and CLIP+GLIP systems in the table below.\n\n| Metric               | CLIP  | CLIP+GLIP |\n| -------------------- | ----- | --------- |\n| Precision            | 0.567 | 0.685     |\n| Mean Reciprocal Rank | 0.775 | 0.778     |\n| Average Precision    | 0.639 | 0.719     |\n\n### Latency\nWe conducted a benchmark study to measure the latency of image search using different implementations, hardware, and image sample sizes. The results are presented below. To ensure stable measurements, we prepared 10 search queries and ran each query three times. We then calculated the average latency required to search through the entire image set per query.\n\n| Model            | Search Time (sec) | Image Sample Size | Device             |\n| ---------------- | ----------------- | ----------------- | ------------------ |\n| CLIP (OpenAI)    | 15.751            | 1000              | MacBook Pro M1 Max |\n| CLIP (Optimized) | 0.035             | 1000              | MacBook Pro M1 Max |\n| CLIP (OpenAI)    | 4.287             | 1000              | A4000 GPU          |\n| CLIP (Optimized) | 0.005             | 1000              | A4000 GPU          |\n| CLIP (Optimized) | 0.0076            | 10000             | A4000 GPU          |\n| CLIP (OpenAI)    | 8.660             | 1000              | Nvidia P100        |\n| CLIP (Optimized) | 0.0073            | 1000              | Nvidia P100        |\n| CLIP (Optimized) | 0.0085            | 10000             | Nvidia P100        |\n| GLIP             | 0.418             | 1                 | Nvidia P100        |\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fivan-zhou%2Fimage-search","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fivan-zhou%2Fimage-search","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fivan-zhou%2Fimage-search/lists"}