{"id":18322488,"url":"https://github.com/tencentarc/visft","last_synced_at":"2026-03-09T18:11:13.813Z","repository":{"id":218012141,"uuid":"743985154","full_name":"TencentARC/ViSFT","owner":"TencentARC","description":null,"archived":false,"fork":false,"pushed_at":"2024-01-20T15:44:48.000Z","size":1917,"stargazers_count":33,"open_issues_count":1,"forks_count":2,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-04-05T23:31:47.148Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/TencentARC.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-01-16T12:00:33.000Z","updated_at":"2025-02-26T05:02:11.000Z","dependencies_parsed_at":"2024-11-05T18:44:47.027Z","dependency_job_id":null,"html_url":"https://github.com/TencentARC/ViSFT","commit_stats":null,"previous_names":["tencentarc/visft"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/TencentARC/ViSFT","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TencentARC%2FViSFT","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TencentARC%2FViSFT/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TencentARC%2FViSFT/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TencentARC%2FViSFT/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/TencentARC","download_url":"https://codeload.github.com/TencentARC/ViSFT/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TencentARC%2FViSFT/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":268248533,"owners_count":24219555,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-01T02:00:08.611Z","response_time":67,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-05T18:24:49.826Z","updated_at":"2026-03-09T18:11:13.724Z","avatar_url":"https://github.com/TencentARC.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"This is the official repo for paper [Supervised Fine-tuning *in turn* Improves Visual Foundation Models](https://arxiv.org/abs/2401.10222).\n\n\u003cdiv align=\"center\"\u003e\n\n📃[**Paper (ArXiv)**](https://arxiv.org/abs/2401.10222) **|** [**Code**](https://github.com/TencentARC/ViSFT) **|** 🤗[**Huggingface**](https://huggingface.co/TencentARC/ViSFT)\n\n\n\n\u003c/div\u003e\n\n## News\n* [2024/01/19] We open source the [ViSFT]() including training scripts and weights. Evaluation codes will be released soon.\n\n## Introduction\nImage-text training like CLIP has dominated the pretraining of vision foundation models in recent years. Subsequent efforts have been made to introduce region-level visual learning into CLIP’s pretraining but face scalability challenges due to the lack of large-scale region-level datasets. Drawing inspiration from supervised fine-tuning (SFT) in natural language processing such as instruction tuning, we explore the potential of fine-grained SFT in enhancing the generation of vision foundation models after their pretraining. Thus a two-stage method **ViSFT** (**Vi**sion **SFT**) is proposed to unleash the fine-grained knowledge of vision foundation models. In ViSFT, the vision foundation model is enhanced by performing visual joint learning on some in-domain tasks and then tested on out-of-domain benchmarks. With updating using ViSFT on 8 V100 GPUs in less than 2 days, a vision transformer with over 4.4B parameters shows improvements across various out-of-domain benchmarks including vision and vision-linguistic scenarios.\n\n\u003cimg src=\"./assets/overview.png\" align=center /\u003e\n\n## Installation\n\n### creating a conda environment\n```\nconda create -n ViSFT python=3.8\n\nconda activate ViSFT\n```\n### install pytorch\nwe use torch1.12 with CUDA11.3 on 8 NVIDIA Volta V100- SXM2-32GB GPUs\n```\npip install --extra-index-url https://download.pytorch.org/whl/cu113 torch==1.12.0\n\npip install --extra-index-url https://download.pytorch.org/whl/cu113 torchvision==0.13.0\n\npip install --extra-index-url https://download.pytorch.org/whl/cu113 torchaudio==0.12.0 \n```\n\n\n### xformers installation\n\nFlash attention is required for running EVA-ViT-E.\nplease refer to [xformers](https://github.com/facebookresearch/xformers)\n\n### loralib installation\n\n```\npip install --user git+https://github.com/microsoft/LoRA\n```\n\n### compile MSDeform for Mask2former head\n```\ncd ./mmf/models/visft/ops\nsudo sh make.sh\n# back to root dir\ncd ../../../../\n```\n\n### other packages installation\n```\npip install -r requirements.txt\n```\n\n## Dataset Preparation\n\nexport DATA_PATH=your_data_path\n\n### image caption\nGenerating hdf5 files for image caption following [hdf5](https://github.com/sgrvinod/a-PyTorch-Tutorial-to-Image-Captioning/blob/master/create_input_files.py)\n\nfile strcture:\n\n```\nDATA_PATH/\n└── processed_datasets/\n    └─── coco_caption_hdf5_files\n        ├──TEST_CAPLENS_coco_5_cap_per_img_5_min_word_freq.json\n        ├──TEST_CAPTIONS_coco_5_cap_per_img_5_min_word_freq.json\n        ├──TEST_IMAGES_coco_5_cap_per_img_5_min_word_freq.hdf5\n        ├──TRAIN_CAPLENS_coco_5_cap_per_img_5_min_word_freq.json\n        ├──TRAIN_CAPTIONS_coco_5_cap_per_img_5_min_word_freq.json\n        ├──TRAIN_IMAGES_coco_5_cap_per_img_5_min_word_freq.hdf5\n        ├──VAL_CAPLENS_coco_5_cap_per_img_5_min_word_freq.json\n        ├──VAL_CAPTIONS_coco_5_cap_per_img_5_min_word_freq.json\n        ├──VAL_IMAGES_coco_5_cap_per_img_5_min_word_freq.hdf5\n        └───WORDMAP_coco_5_cap_per_img_5_min_word_freq.json\n```\n### detection \u0026 segmentation\n\nfile strcture:\n\n```\nDATA_PATH/\n└── public_datasets/\n    └─── coco\n        ├──train2017\n        ├──val2017\n        ├──test2017\n        └───annotations\n            ├──instances_train2017.json\n            ├──instances_val2017.json\n            └───image_info_test-dev2017.json\n```\n\n## Training\n### Stage1\nTo get compatible in-domain task heads. Using 8 NVIDIA Volta V100-SXM2-32GB GPUs for every in-domain task head.\n\n**For eva-vit-g**\n\nPreparing weights from [LAVIS](https://github.com/salesforce/LAVIS)\n```\nwget https://storage.googleapis.com/sfr-vision-language-research/LAVIS/models/BLIP2/eva_vit_g.pth\n```\nAdding your weights path to configs under dir:./projects/visft/configs/stage1/eva_g/\n```\nbackbone_dir: path/eva_vit_g.pth\n```\nImplementing training\n```\n# can be executed in parallel\nbash ./scripts/stage1_train/eva_g/caption.sh\nbash ./scripts/stage1_train/eva_g/detection.sh\nbash ./scripts/stage1_train/eva_g/segment.sh\n```\n\n**For eva-vit-e**\n\nPreparing EVA-CLIP weights from [EVA](https://huggingface.co/QuanSun/EVA-CLIP/blob/main/EVA02_CLIP_E_psz14_plus_s9B.pt)\n\nExtract ViT weights\n```\npython ./scripts/preprocess/extract_eva_e_vit.py\n```\nAdding your weights path to configs under dir:./projects/visft/configs/stage1/eva_e/\n```\nbackbone_dir: path/EVA02_CLIP_E_psz14_plus_s9B_Visual.pt\n```\nImplementing training\n```\n# can be executed in parallel\nbash ./scripts/stage1_train/eva_e/caption.sh\nbash ./scripts/stage1_train/eva_e/detection.sh\nbash ./scripts/stage1_train/eva_e/segment.sh\n```\n\nOr you can use the weights we provided.\n\n| In-domain Heads  |           | | \n|----------|:-------------:|:-------------:|\n| |       EVA-G     |  EVA-E|\n| Caption Head |  [weights](https://huggingface.co/TencentARC/ViSFT/blob/main/eva_g_caption_heads.ckpt) | [weights](https://huggingface.co/TencentARC/ViSFT/blob/main/eva_e_caption_heads.ckpt)|\n| Segment Head |    [weights](https://huggingface.co/TencentARC/ViSFT/blob/main/eva_g_segment_heads.ckpt)   |[weights](https://huggingface.co/TencentARC/ViSFT/blob/main/eva_e_segment_heads.ckpt)|\n| Detection Head | [weights](https://huggingface.co/TencentARC/ViSFT/blob/main/eva_g_detection_heads.ckpt) |[weights](https://huggingface.co/TencentARC/ViSFT/blob/main/eva_e_detection_heads.ckpt)|\n\n\n### Stage2\n\n**For eva-vit-g**\n\nAdding your weights path to configs under dir:./projects/visft/configs/stage2/eva_g/stage2.yaml\n```\nbackbone_dir: path/eva_vit_g.pth\ncaption_ckpt_path: 'path/eva_g_caption_heads.ckpt'\nsegment_ckpt_path:'path/eva_g_segment_heads.ckpt'\ndetection_ckpt_path: 'path/eva_g_detection_heads.ckpt'\n```\nImplementing training\n```\nbash ./scripts/stage2_train/eva_g/stage2.sh\n```\n\n**For eva-vit-e**\n\nAdding your weights path to configs under dir:./projects/visft/configs/stage2/eva_e/stage2.yaml\n```\nbackbone_dir: path/EVA02_CLIP_E_psz14_plus_s9B_Visual.pt\ncaption_ckpt_path: 'path/eva_e_caption_heads.ckpt'\nsegment_ckpt_path:'path/eva_e_segment_heads.ckpt'\ndetection_ckpt_path: 'path/eva_e_detection_heads.ckpt'\n```\nImplementing training\n```\nbash ./scripts/stage2_train/eva_e/stage2.sh\n```\n### Get LoRA Weights\nYou can extract expected LoRA weights by\n\n```\npython ./scripts/postprocess/extract_lora_weights.py\n```\n\nOr use the LoRA weights we provide:\n| LoRA weights |           | | \n|----------|:-------------:|:-------------:|\n|  Iters|     EVA-G     |  EVA-E|\n| 5k |  [weights](https://huggingface.co/TencentARC/ViSFT/blob/main/eva_g_lora_5000.pt) | [weights](https://huggingface.co/TencentARC/ViSFT/blob/main/eva_e_lora_5000.pt)|\n| 10k |    [weights](https://huggingface.co/TencentARC/ViSFT/blob/main/eva_g_lora_10000.pt)   |[weights](https://huggingface.co/TencentARC/ViSFT/blob/main/eva_e_lora_10000.pt)|\n| 15k |    [weights](https://huggingface.co/TencentARC/ViSFT/blob/main/eva_g_lora_15000.pt)   |[weights](https://huggingface.co/TencentARC/ViSFT/blob/main/eva_e_lora_15000.pt)|\n| 20k |    [weights](https://huggingface.co/TencentARC/ViSFT/blob/main/eva_g_lora_20000.pt)   |[weights](https://huggingface.co/TencentARC/ViSFT/blob/main/eva_e_lora_20000.pt)|\n| 50k | [weights](https://huggingface.co/TencentARC/ViSFT/blob/main/eva_g_lora_50000.pt) |[weights](https://huggingface.co/TencentARC/ViSFT/blob/main/eva_e_lora_50000.pt)|\n## Evaluation Benchmarks\n- [] Zero-shot Image Classification\n- [] Zero-shot Image-text Retrieval\n- [] OCR\n- [] Grounded Object Indentification\n- [] VQA\n- [] Image Captioning on NoCaps\n\n## Acknowledgement\nThe code of ViSFT is based on the official implementation of [mmf](https://github.com/facebookresearch/mmf), [EVA](https://github.com/baaivision/EVA/tree/master) and [LAVIS](https://github.com/salesforce/LAVIS/tree/main)\n\n## Citation\nIf you found our work valuable, please cite:\n```\n@misc{jiang2024supervised,\n      title={Supervised Fine-tuning in turn Improves Visual Foundation Models}, \n      author={Xiaohu Jiang and Yixiao Ge and Yuying Ge and Chun Yuan and Ying Shan},\n      year={2024},\n      eprint={2401.10222},\n      archivePrefix={arXiv},\n      primaryClass={cs.CV}\n}\n```\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftencentarc%2Fvisft","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftencentarc%2Fvisft","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftencentarc%2Fvisft/lists"}