{"id":28653894,"url":"https://github.com/tiger-ai-lab/uniir","last_synced_at":"2025-06-13T07:08:03.008Z","repository":{"id":209680686,"uuid":"724435692","full_name":"TIGER-AI-Lab/UniIR","owner":"TIGER-AI-Lab","description":"Official code for paper \"UniIR: Training and Benchmarking Universal Multimodal Information Retrievers\"","archived":false,"fork":false,"pushed_at":"2024-05-28T22:39:36.000Z","size":56161,"stargazers_count":69,"open_issues_count":2,"forks_count":11,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-05-29T10:25:15.860Z","etag":null,"topics":["language-model","retrieval"],"latest_commit_sha":null,"homepage":"https://tiger-ai-lab.github.io/UniIR/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/TIGER-AI-Lab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-11-28T04:16:10.000Z","updated_at":"2024-05-28T22:39:41.000Z","dependencies_parsed_at":"2023-11-28T17:11:18.329Z","dependency_job_id":"d226435b-2942-4b4a-b3e5-388dc2e93a4c","html_url":"https://github.com/TIGER-AI-Lab/UniIR","commit_stats":null,"previous_names":["tiger-ai-lab/uniir"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/TIGER-AI-Lab/UniIR","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TIGER-AI-Lab%2FUniIR","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TIGER-AI-Lab%2FUniIR/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TIGER-AI-Lab%2FUniIR/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TIGER-AI-Lab%2FUniIR/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/TIGER-AI-Lab","download_url":"https://codeload.github.com/TIGER-AI-Lab/UniIR/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TIGER-AI-Lab%2FUniIR/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259599331,"owners_count":22882357,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["language-model","retrieval"],"created_at":"2025-06-13T07:08:02.220Z","updated_at":"2025-06-13T07:08:02.990Z","avatar_url":"https://github.com/TIGER-AI-Lab.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# UniIR\n\n[**🌐 Homepage**](https://tiger-ai-lab.github.io/UniIR/) | [**🤗 Dataset(M-BEIR Benchmark)**](https://huggingface.co/datasets/TIGER-Lab/M-BEIR) | [**🤗 Checkpoints(UniIR models)**](https://huggingface.co/TIGER-Lab/UniIR) | [**📖 arXiv**](https://arxiv.org/pdf/2311.17136.pdf) | [**GitHub**](https://github.com/TIGER-AI-Lab/UniIR)\n\nThis repo contains the codebase for the ECCV-2024 paper \"[UniIR: Training and Benchmarking Universal Multimodal\nInformation Retrievers](https://arxiv.org/pdf/2311.17136.pdf)\"\n\n## 🔔News\n- **🔥[2024-04-13]**: We highlight another valuable and concurrent research on training instruction-following, multi-task multi-modal retrievers with Late-interaction:[PreFLMR: Scaling Up Fine-Grained Late-Interaction Multi-modal Retrievers](https://preflmr.github.io/) , which was done by the researchers of the University of Cambridge. They also introduced the M2KR benchmark which can be used to train and evaluate multi-modal universal information retrievers. We may combine the M2KR and M-BEIR benchmarks together to facilitate the advance of this field.\n- **🔥[2024-03-18]: Release the UniIR(CLIP_SF) large and UniIR(BLIP_FF) large checkpoints [**🤗 Checkpoints**](https://huggingface.co/TIGER-Lab/UniIR)**\n- **🔥[2023-12-21]: Our [🤗 M-BEIR Benchmark](https://huggingface.co/datasets/TIGER-Lab/M-BEIR) is now available for use.**\n\n\n## Introduction\nWe propose the **UniIR**(Universal multimodal Information Retrieval) **framework** to learn a single retriever to accomplish (possibly) any retrieval task. Unlike traditional IR systems, UniIR needs to follow the instructions to take a heterogeneous query to retrieve from a heterogeneous candidate pool with millions of candidates in diverse modalities.\n\n\u003cimg src=\"docs/images/uniir_teaser.jpg\" alt=\"UniIR Teaser\" style=\"width:80%;\"\u003e\n\n\n## Content\n\n1. [M-BEIR](#M-BEIR)\n2. [Training](#Training)\n3. [Evaluation](#Evaluation)\n2. [Model Zoo](#Model-Zoo)\n4. [Citations and Contact](#Citation-and-Contact)\n\n\n# M-BEIR\nTo train and evaluate universal multimodal retrieval models, we build a large-scale retrieval benchmark named **M-BEIR** (Multimodal BEnchmark for Instructed Retrieval).\n\n## M-BEIR Downloading\nWe provide the M-BEIR dataset in the [**🤗 Dataset**](https://huggingface.co/datasets/TIGER-Lab/M-BEIR).\n**Please follow the instructions provided on the HF page** to download the dataset and prepare the data for training and evaluation.\nYou need to set up GiT LFS and directly clone the repo:\n```\ngit clone https://huggingface.co/datasets/TIGER-Lab/M-BEIR\n```\n\n# UniIR Models\nWe provide the codebase for training and evaluating the UniIR CLIP-ScoreFusion, CLIP-FeatureFusion, BLIP-ScoreFusion, and BLIP-FeatureFusion models.\n\n## Environment\nPrepare the codebase of the UniIR project and Conda environment using the following commands:\n```bash\ngit clone https://github.com/TIGER-AI-Lab/UniIR\ncd UniIR\n\ncd src/models/\nconda env create -f uniir_env.yml\n```\n\n## Training\nTo train the UniIR models from pretrained CLIP and BLIP checkpoints, please follow the instructions below. \nThe scripts will automatically download the pretrained CLIP and BLIP checkpoints.\n\n### 1. Download the M-BEIR Benchmark\nPlease download the M-BEIR benchmark by following the instructions in the [**M-BEIR**](#M-BEIR) section.\n\n### 2. Scripts\n#### To train UniIR CLIP_SF Large with the default configuration:\n```bash\ncd src/models/uniir_clip/clip_scorefusion/configs_scripts/large/train/inbatch/\n```\nModify `inbatch.yaml` for hyperparameter tuning and `run_inbatch.sh` for your own environment and paths.\n\n#### Note:\n1. Modify the ```UNIIR_DIR``` in the `run_inbatch.sh` to the directory where you\nwant to store the checkpoints.\n2. Modify the ```MBEIR_DATA_DIR``` in the `run_inbatch.sh` to the directory where you store the M-BEIR benchmark.\n3. Modify the ```SRC_DIR``` in the `run_inbatch.sh` to the directory where you store the codebase of the UniIR project(This repo).\n4. By default, UniIR models are trained on M-BEIR with in-batch negatives, and the hard negatives provided by the original datasets \nare not used.\n5. We used wandb to log the training process. Please make sure a `.env` environment with `WANDB_API_KEY`, `WANDB_PROJECT`, and `WANDB_ENTITY`  is set. \n\nThen you can run the following command to train the UniIR CLIP_SF Large model.\n```bash\nbash run_inbatch.sh\n```\n\n#### To train UniIR BLIP_FF Large with the default configuration:\n```bash\ncd src/models/uniir_blip/blip_featurefusion/configs_scripts/large/train/inbatch/\n```\nModify `inbatch.yaml` for hyperparameter tuning and `run_inbatch.sh` for your own environment and paths.\n```bash\nbash run_inbatch.sh\n```\n\n#### Similarly, you can train the UniIR CLIP_FF and BLIP_SF models by modifying the corresponding scripts.\n\n## Evaluation\nWe provide the evaluation pipeline for the UniIR models on the M-BEIR benchmark.\n### 1. Environment\nPlease create an environment for the FAISS library:\n```bash\n# From the root directory of the project\ncd src/common/\nconda env create -f faiss_env.yml\n```\n\n### 2. Download the M-BEIR Benchmark\nPlease download the M-BEIR benchmark by following the instructions in the [**M-BEIR**](#M-BEIR) section.\n\n### 3. Download the UniIR Checkpoints\nYou can train the UniIR models from scratch or download the pre-trained UniIR checkpoints by following the instructions in the [**Model Zoo**](#Model-Zoo) section.\n\n### 4. Scripts\n#### To evaluate UniIR CLIP_SF Large with the default configuration:\n```bash\ncd src/models/uniir_clip/clip_scorefusion/configs_scripts/large/eval/inbatch/\n```\nModify `embed.yaml`, `index.yaml`, `retrieval.yaml` and `run_eval_pipeline_inbatch.sh` for your own environment, paths and evaluation settings.\n\n#### Note:\n1. If you download our pretrained UniIR model, please modify the ```UNIIR_DIR``` in the `run_eval_pipeline_inbatch.sh` to the directory where you\nwant to store large files including the checkpoints, embeddings, index and retrieval results.\nThen you can place the ```clip_sf_large.pth``` file in the following path:\n    ```bash\n    $UNIIR_DIR/checkpoint/CLIP_SF/Large/Instruct/InBatch/clip_sf_large.pth\n    ```\n    This the default path specified by ```model.ckpt_config``` in the `embed.yaml` file.\n2. Modify the ```MBEIR_DATA_DIR``` in the `run_eval_pipeline_inbatch.sh` to the directory where you store the M-BEIR benchmark.\n3. Modify the ```SRC_DIR``` in the `run_eval_pipeline_inbatch.sh` to the directory where you store the codebase of the UniIR project(This repo).\n\nThe default configuration will evaluate the UniIR CLIP_SF Large model on both the M-BEIR (5.6M heterogeneous candidate pool) and the M-BEIR_local (homogeneous candidate pool) benchmarks.\n```UNION``` in the yaml files refers to the M-BEIR (5.6M heterogeneous candidate pool).\nYou can follow the comments in the yaml files and modify the configurations to evaluate the model on the M-BEIR_local benchmark only.\n```bash\nbash run_eval_pipeline_inbatch.sh\n```\n```embed```, ```index```, ```logger``` and ```retrieval_results``` will be saved in the ```$UNIIR_DIR``` directory.\n\n#### To evaluate UniIR BLIP_FF Large with the default configuration:\n```bash\ncd src/models/unii_blip/blip_featurefusion/configs_scripts/large/eval/inbatch/\n```\nSimilarly, if you download our pretrained UniIR model, you can place the ```blip_ff_large.pth``` file in the following path:\n```bash\n$UNIIR_DIR/checkpoint/BLIP_FF/Large/Instruct/InBatch/blip_ff_large.pth\n```\n\nThe default configuration will evaluate the UniIR BLIP_FF Large model on both the M-BEIR and the M-BEIR_local benchmarks.\n```bash\nbash run_eval_pipeline_inbatch.sh\n```\n#### UniRAG evaluation\nUniRAG evaluation is very similar to the default evaluation with the following differences:\n- It stores jsonl files containing queries and their retrieved candidates under ```retrieval_results```. This is useful when retrieved results will be used in downstream applications like RAG.\n- When `retrieve_image_text_pairs` in `retrieval.yaml` is set to `True`, a complement candidate will be fetched for each candidate with `text` or `image` only modality. With this setting, the candidate and its complement will always have `image, text` modality. Complement candidates are fetched by using the original candidates as queries (e.g., _query\u003csub\u003etext\u003c/sub\u003e_ -\u003e _candidate\u003csub\u003eimage\u003c/sub\u003e_ -\u003e  _complement candidate\u003csub\u003etext\u003c/sub\u003e_).\n- To run evaluations in UniRAG mode follow the instructions provided above replacing `InBatch` and `inbatch` with `UniRAG` and `unirag`, respectively.\n\n#### You can train and evaluate the UniIR CLIP_FF and BLIP_SF models by modifying the corresponding scripts.\n\n## Model Zoo\nWe provide the UniIR model checkpoints in the [**🤗 Checkpoints**](https://huggingface.co/TIGER-Lab/UniIR).\nYou can directly use the checkpoints for retrieval tasks or fine-tune the models for your own retrieval tasks.\n\n### Available Checkpoints\n\n| Model Name     | Version | Model Size | Model Link                                                                                             |\n|----------------|---------|------------|--------------------------------------------------------------------------------------------------------|\n| UniIR(CLIP-SF) | Large   | 5.13 GB    | [Download Link](https://huggingface.co/TIGER-Lab/UniIR/blob/main/checkpoint/CLIP_SF/clip_sf_large.pth) |\n| UniIR(BLIP-FF) | Large   | 7.49 GB    | [Download Link](https://huggingface.co/TIGER-Lab/UniIR/blob/main/checkpoint/BLIP_FF/blip_ff_large.pth) |\n\nYou can download them by \n```\ngit clone https://huggingface.co/TIGER-Lab/UniIR\n```\n\n\n## Citation and Contact\n- Cong Wei: c58wei@uwaterloo.ca\n- Yang Chen: yangc@gatech.edu\n- Alan Ritter: alan.ritter@cc.gatech.edu\n- Wenhu Chen: wenhuchen@uwaterloo.ca\n\n\n**BibTeX:**\n```bibtex\n@article{wei2023uniir,\n  title={Uniir: Training and benchmarking universal multimodal information retrievers},\n  author={Wei, Cong and Chen, Yang and Chen, Haonan and Hu, Hexiang and Zhang, Ge and Fu, Jie and Ritter, Alan and Chen, Wenhu},\n  journal={arXiv preprint arXiv:2311.17136},\n  year={2023}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftiger-ai-lab%2Funiir","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftiger-ai-lab%2Funiir","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftiger-ai-lab%2Funiir/lists"}