{"id":18887127,"url":"https://github.com/thunlp-mt/brote","last_synced_at":"2025-04-14T22:33:45.438Z","repository":{"id":224491289,"uuid":"759860211","full_name":"THUNLP-MT/Brote","owner":"THUNLP-MT","description":null,"archived":false,"fork":false,"pushed_at":"2025-01-19T16:45:27.000Z","size":12744,"stargazers_count":11,"open_issues_count":1,"forks_count":0,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-03-28T10:54:12.849Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/THUNLP-MT.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-02-19T13:27:45.000Z","updated_at":"2025-01-19T16:45:29.000Z","dependencies_parsed_at":"2024-02-26T09:24:59.535Z","dependency_job_id":"694255fa-5eab-4324-8cc5-c32e46ee19c4","html_url":"https://github.com/THUNLP-MT/Brote","commit_stats":null,"previous_names":["thunlp-mt/brote"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/THUNLP-MT%2FBrote","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/THUNLP-MT%2FBrote/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/THUNLP-MT%2FBrote/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/THUNLP-MT%2FBrote/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/THUNLP-MT","download_url":"https://codeload.github.com/THUNLP-MT/Brote/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248973010,"owners_count":21191897,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-08T07:34:31.493Z","updated_at":"2025-04-14T22:33:40.428Z","avatar_url":"https://github.com/THUNLP-MT.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 👀 Browse and Concentrate: Comprehending Multimodal Content via prior-LLM Context Fusion\n\n[**🌐 Homepage**](https://thunlp-mt.github.io/Brote/) | [**📖 arXiv**](https://arxiv.org/pdf/2402.12195.pdf) | [**🤗 Models**](https://huggingface.co/wangphoebe/Brote-IM-XXL)\n\nThis repo includes codes and examples for paper [Browse and Concentrate: Comprehending Multimodal Content via prior-LLM Context Fusion](https://arxiv.org/pdf/2402.12195.pdf). \n## Activities\n\n1. [2024-05-16] This paper is accepted by ACL 2024 (main conference). Information for our training data is updated.\n2. [2024-04-18] Code and cases for data generation released. The generated data are used for pretraining.\n3. [2024-03-18] Brote-IM-XXL model released, please download from this [link](https://huggingface.co/wangphoebe/Brote-IM-XXL). \n4. [2024-02-26] Project released.\n\n## Framework\nWe propose a paradigm **Bro**wse and Concentra**te** (**Brote**) for incorporating multimodal context before feeding features into the LLM, together with two approaches to implement our paradigm, Brote-EX and Brote-IM. The model structures are shown in the following figure.\n\n\u003cimg src=\"./figures/model.png\" alt=\"Image\" width=\"500\"\u003e\n\n## Instructions For Training and Inference\n\n### Data\nPlease refer to the data format described in [MIC](https://github.com/HaozheZhao/MIC).\n\n1. Data for pretraining.\n\nWe create a dataset of 56k fewshot data samples, resulting in 191k training instances (one image per instance). These instances are supposed to contain question-aware and cross-image information. The data construction pipeline is illustrated in the following figure.\n\n\u003cimg src=\"./figures/pretrain_data.png\" alt=\"Image\" width=\"400\"\u003e\n\nIf you want plan to try our constructed pretraining data, please create an issue here. We will contact you ASAP.  \n\n2. Data for finetuning.\n\nWe sampled about 500k data from MIC for model finetuning.\n\n### Environment\n```\npip install -r requirements.txt\n```\n\n### Training\ncoming soon\n\n### Inference\nPlease refer to the test.py file; files under the **model** dir are for test only, and will be updated soon for training.\n\nTo run the test script (ensure the required libraries are properly installed):\n```\nexport CUDAID='please set you cuda id here'\nexport TASKID='please set the case id (from 1 to 5), or use the string 'all'(lowercase)'\nCUDA_VISIBLE_DEVICES=$CUDAID python test.py $TASKID \n```\n\n## Example\n\u003cimg src=\"./figures/git_showcase.png\" alt=\"Image\" width=\"600\"\u003e\n\n(🐱 in this figure is a 6-year-old cat, his name is Alan.)\n\n## Models\nPlease download our model from [**🤗 Models**](https://huggingface.co/wangphoebe/Brote-IM-XXL).\n\n## Reference\n\n📑 If you find our project helpful to your research, please consider citing:\n```\n@inproceedings{\nwang2024browse,\ntitle={Browse and Concentrate: Comprehending Multimodal Content via Prior-{LLM} Context Fusion},\nauthor={Wang, Ziyue and Chen, Chi and Zhu, Yiqi and Luo, Fuwen and Li, Peng and Yan, Ming and Zhang, Ji and Huang, Fei and Sun, Maosong and Liu, Yang},\nbooktitle={The 62nd Annual Meeting of the Association for Computational Linguistics},\nyear={2024},\n}\n```\n## Acknowledgement\nOur models are build upon [MMICL](https://github.com/HaozheZhao/MIC) and [InstructBLIP](https://github.com/salesforce/LAVIS/tree/main/projects/instructblip).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthunlp-mt%2Fbrote","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthunlp-mt%2Fbrote","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthunlp-mt%2Fbrote/lists"}