{"id":13708405,"url":"https://github.com/TinyLLaVA/TinyLLaVA_Factory","last_synced_at":"2025-05-06T13:30:20.959Z","repository":{"id":223585749,"uuid":"760947966","full_name":"TinyLLaVA/TinyLLaVA_Factory","owner":"TinyLLaVA","description":"A Framework of Small-scale Large Multimodal Models","archived":false,"fork":false,"pushed_at":"2025-04-26T11:17:36.000Z","size":6297,"stargazers_count":801,"open_issues_count":95,"forks_count":83,"subscribers_count":11,"default_branch":"main","last_synced_at":"2025-04-26T12:27:53.146Z","etag":null,"topics":["large-multimodal-models","llama","llava","nlp","tinyllama","transformers","vision-language"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2402.14289","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/TinyLLaVA.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-02-21T00:28:14.000Z","updated_at":"2025-04-26T11:17:39.000Z","dependencies_parsed_at":"2024-06-21T22:37:28.833Z","dependency_job_id":"a5d03c58-d53c-4c22-9f4b-79ccec9a5537","html_url":"https://github.com/TinyLLaVA/TinyLLaVA_Factory","commit_stats":{"total_commits":192,"total_committers":20,"mean_commits":9.6,"dds":0.828125,"last_synced_commit":"5c806727538cb401e19bec5b3554fc3b5abd991b"},"previous_names":["dlcv-buaa/tinyllavabench","tinyllava/tinyllava_factory"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TinyLLaVA%2FTinyLLaVA_Factory","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TinyLLaVA%2FTinyLLaVA_Factory/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TinyLLaVA%2FTinyLLaVA_Factory/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TinyLLaVA%2FTinyLLaVA_Factory/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/TinyLLaVA","download_url":"https://codeload.github.com/TinyLLaVA/TinyLLaVA_Factory/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252693469,"owners_count":21789694,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["large-multimodal-models","llama","llava","nlp","tinyllama","transformers","vision-language"],"created_at":"2024-08-02T23:00:21.944Z","updated_at":"2025-05-06T13:30:20.953Z","avatar_url":"https://github.com/TinyLLaVA.png","language":"Python","funding_links":[],"categories":["微调 Fine-Tuning"],"sub_categories":[],"readme":"\u003ch2 align=\"center\"\u003e \u003ca href=\"https://arxiv.org/abs/2402.14289\"\u003eTinyLLaVA Factory\u003c/a\u003e\u003ch5 align=\"center\"\u003e\n\n[![hf_space](https://img.shields.io/badge/🤗-%20Open%20In%20HF-blue.svg)](https://huggingface.co/tinyllava) [![arXiv](https://img.shields.io/badge/Arxiv-2402.14289-b31b1b.svg?logo=arXiv)](https://arxiv.org/abs/2402.14289) [![arXiv](https://img.shields.io/badge/Arxiv-2405.11788-b31b1b.svg?logo=arXiv)](https://arxiv.org/abs/2405.11788)[![License](https://img.shields.io/badge/License-Apache%202.0-yellow)](https://github.com/TinyLLaVA/TinyLLaVA_Factory/blob/main/LICENSE) [![Doc](https://img.shields.io/badge/Doc-Document-logo=read%20the%20docs\u0026logoColor=white\u0026label=Doc)](https://tinyllava-factory.readthedocs.io/en/latest/) [![Demo](https://img.shields.io/badge/Demo-Demo-red.svg)](http://8843843nmph5.vicp.fun/#/)\n\n![architecture](./assets/architecture.jpg)\n\n## \u0026#x1F389; News\n* **[2025.01]**  Our new work [TinyLLaVA-Video](https://github.com/ZhangXJ199/TinyLLaVA-Video) is released.\n* **[2024.08.13]**  A simple [visualizaiton tool](https://github.com/TinyLLaVA/TinyLLaVA_Factory/tree/main/tinyllava_visualizer) for interpreting the prediction of TinyLLaVA is added.\n* **[2024.05.21]**  Our paper: [TinyLLaVA Factory: A Modularized Codebase for Small-scale Large Multimodal Models](https://arxiv.org/abs/2405.11788) is released!\n* **[2024.05.15]** [TinyLLaVA Factory](https://github.com/TinyLLaVA/TinyLLaVA_Factory), our new codebase, is released!  **Note that the old codebase, TinyLLaVABench, is moved to the [tinyllava_bench](https://github.com/TinyLLaVA/TinyLLaVA_Factory/tree/tinyllava_bench) branch.**\n* **[2024.05.04]**  [TinyLLaVA Demo](http://8843843nmph5.vicp.fun/#/) is released! (The password to access our demo is '1234'.)\n* **[2024.02.21]**  Our paper: [TinyLLaVA: A Framework of Small-scale Large Multimodal Models](https://arxiv.org/abs/2402.14289) is released!\n\n## \u0026#x1F525; Takeaways\n- Our best model, [TinyLLaVA-Phi-2-SigLIP-3.1B](https://huggingface.co/tinyllava/TinyLLaVA-Phi-2-SigLIP-3.1B), achieves better overall performance against existing 7B models such as LLaVA-1.5 and Qwen-VL.\n\n- TinyLLaVA Factory is an open-source modular codebase for small-scale large multimodal models (LMMs), implemented in PyTorch and HuggingFace, with a focus on simplicity of code implementations, extensibility of new features, and reproducibility of training results.\n\n- With TinyLLaVA Factory, you can customize your own large multimodal models with less coding effort and less coding mistakes.\n\n- TinyLLaVA Factory integrates a suite of cutting-edge models and methods. \n\n  - LLM currently supports **OpenELM**, **TinyLlama**, **StableLM**, **Qwen**, **Gemma**, and **Phi**. \n\n  - Vision tower currently supports **CLIP,** **SigLIP**, **Dino**, and **combination of CLIP and Dino**.\n    \n  - Connector currently supports **MLP**, **Qformer**, and **Resampler**.\n    \n  - Training Recipe currently supports **Frozen/Fully/Partially tuning** and **LoRA/QLoRA tuning**.\n\n## Contents\n\n- [🎉 News](#-news)\n- [🔥 Takeaways](#-takeaways)\n- [Contents](#contents)\n- [Installation and Requirements](#installation-and-requirements)\n    - [Upgrade to the latest code base](#upgrade-to-the-latest-code-base)\n- [Get Started](#get-started)\n    - [1. Data Preparation](#1-data-preparation)\n    - [2. Train](#2-train)\n    - [3. Evaluation](#3-evaluation)\n- [Model Zoo](#model-zoo)\n  - [Trained Models](#trained-models)\n    - [Model Performance](#model-performance)\n  - [Legacy Models](#legacy-models)\n- [Launch Demo Locally](#launch-demo-locally)\n  - [Gradio Web Demo](#gradio-web-demo)\n  - [CLI Inference](#cli-inference)\n  - [Quick Inference Scripts](#quick-inference-scripts)\n- [Custom Finetune](#custom-finetune)\n- [Customize Your Own Large Multimodel Models](#customize-your-own-large-multimodel-models)\n  - [LLM](#llm)\n  - [Vision Tower](#vision-tower)\n  - [Connector](#connector)\n- [Acknowledgement](#acknowledgement)\n- [Contact](#contact)\n- [✏ Citation](#-citation)\n- [❤️ Community efforts](#️-community-efforts)\n\n\n## Installation and Requirements\n\nPlease note that our environment requirements are different from LLaVA's environment requirements. We strongly recommend you create the environment from scratch as follows.\n\n1. Clone this repository and navigate to the folder\n```bash\ngit clone https://github.com/TinyLLaVA/TinyLLaVA_Factory.git\ncd TinyLLaVA_Factory\n```\n\n2. Create a conda environment, activate it and install Packages\n```Shell\nconda create -n tinyllava_factory python=3.10 -y\nconda activate tinyllava_factory\npip install --upgrade pip  # enable PEP 660 support\npip install -e .\n```\n\n3. Install additional packages\n```Shell\npip install flash-attn==2.5.7 --no-build-isolation\n```\n#### Upgrade to the latest code base\n\n```Shell\ngit pull\npip install -e .\n```\n\n## Get Started\n\n#### 1. Data Preparation\n\nPlease refer to the [Data Preparation](https://tinyllava-factory.readthedocs.io/en/latest/Prepare%20Datasets.html) section in our [Documenation](https://tinyllava-factory.readthedocs.io/en/latest/).\n\n#### 2. Train\n\nHere's an example for training a LMM using Phi-2.\n\n- Replace data paths with yours in `scripts/train/train_phi.sh`\n- Replace `output_dir` with yours in `scripts/train/pretrain.sh`\n- Replace `pretrained_model_path` and `output_dir` with yours in `scripts/train/finetune.sh`\n- Adjust your GPU ids (localhost) and `per_device_train_batch_size` in `scripts/train/pretrain.sh` and `scripts/train/finetune.sh`\n\n```bash\nbash scripts/train/train_phi.sh\n```\n\nImportant hyperparameters used in pretraining and finetuning are provided below.\n\n| Training Stage | Global Batch Size | Learning rate | conv_version |\n| -------------- | :---------------: | :-----------: | :----------: |\n| Pretraining    | 256               | 1e-3          | pretrain     |\n| Finetuning     | 128               | 2e-5          | phi          |\n\n**Tips:** \n\nGlobal Batch Size = num of GPUs * `per_device_train_batch_size` * `gradient_accumulation_steps`, we recommand you always keep global batch size and learning rate as above except for lora tuning your model.\n\n`conv_version` is a hyperparameter used for choosing different chat templates for different LLMs. In the pretraining stage, `conv_version` is the same for all LLMs, using `pretrain`. In the finetuning stage, we use\n\n`phi` for Phi-2, StableLM, Qwen-1.5\n\n`llama` for TinyLlama, OpenELM\n\n`gemma` for Gemma\n\n#### 3. Evaluation\n\nPlease refer to the [Evaluation](https://tinyllava-factory.readthedocs.io/en/latest/Evaluation.html) section in our [Documenation](https://tinyllava-factory.readthedocs.io/en/latest/Evaluation.html).\n\n## Model Zoo\n\n### Trained Models\n\nwhich are trained using TinyLLaVA Factory.\n\n- [TinyLLaVA-Phi-2-SigLIP-3.1B](https://huggingface.co/tinyllava/TinyLLaVA-Phi-2-SigLIP-3.1B)\n- [TinyLLaVA-Gemma-SigLIP-2.4B](https://huggingface.co/tinyllava/TinyLLaVA-Gemma-SigLIP-2.4B)\n- [TinyLLaVA-OpenELM-450M-SigLIP-0.89B](https://huggingface.co/jiajunlong/TinyLLaVA-0.89B)\n- [TinyLLaVA-Qwen2-0.5B-SigLIP](https://huggingface.co/Zhang199/TinyLLaVA-Qwen2-0.5B-SigLIP)\n- [TinyLLaVA-Qwen2.5-3B-SigLIP](https://huggingface.co/Zhang199/TinyLLaVA-Qwen2.5-3B-SigLIP)\n\n#### Model Performance\n\n| VT (HF Path)                      | LLM (HF Path)                      | Recipe    | VQA-v2 | GQA  | SQA-image | TextVQA | MM-Vet | POPE | MME    | MMMU-val |\n| --------------------------------- | ---------------------------------- | --------- | :----: | :--: | :-------: | :-----: | :----: | :--: | :----: | :------: |\n| openai/clip-vit-large-patch14-336 | apple/OpenELM-450M-Instruct        | base      | 69.5   | 52.1 | 50.6      | 40.4    | 20.0   | 83.6 | 1052.9 | 23.9     |\n| google/siglip-so400m-patch14-384  | apple/OpenELM-450M-Instruct        | base      | 71.7   | 53.9 | 54.1      | 44.0    | 20.0   | 85.4 | 1118.8 | 24.0     |\n| google/siglip-so400m-patch14-384  | Qwen/Qwen2-0.5B                    | base      | 72.3   | 55.8 | 60.1      | 45.2    | 19.5   | 86.6 | 1153.0 | 29.7     |\n| google/siglip-so400m-patch14-384  | Qwen/Qwen2.5-0.5B                  | base      | 75.3   | 59.5 | 60.3      | 48.3    | 23.9   | 86.1 | 1253.0 | 33.3     |\n| google/siglip-so400m-patch14-384  | Qwen/Qwen2.5-3B                    | base      | 79.4   | 62.5 | 74.1      | 58.3    | 34.8   | 87.4 | 1438.7 | 39.9     |\n| openai/clip-vit-large-patch14-336 | TinyLlama/TinyLlama-1.1B-Chat-v1.0 | base      | 73.7   | 58.0 | 59.9      | 46.3    | 23.2   | 85.5 | 1284.6 | 27.9     |\n| google/siglip-so400m-patch14-384  | TinyLlama/TinyLlama-1.1B-Chat-v1.0 | base      | 75.5   | 58.6 | 64.0      | 49.6    | 23.5   | 86.3 | 1256.5 | 28.3     |\n| openai/clip-vit-large-patch14-336 | stabilityai/stablelm-2-zephyr-1_6b | base      | 75.9   | 59.5 | 64.6      | 50.5    | 27.3   | 86.1 | 1368.1 | 31.8     |\n| google/siglip-so400m-patch14-384  | stabilityai/stablelm-2-zephyr-1_6b | base      | 78.2   | 60.7 | 66.7      | 56.0    | 29.4   | 86.3 | 1319.3 | 32.6     |\n| google/siglip-so400m-patch14-384  | google/gemma-2b-it                 | base      | 78.4   | 61.6 | 64.4      | 53.6    | 26.9   | 86.4 | 1339.0 | 31.7     |\n| openai/clip-vit-large-patch14-336 | microsoft/phi-2                    | base      | 76.8   | 59.4 | 71.2      | 53.4    | 31.7   | 86.8 | 1448.6 | 36.3     |\n| google/siglip-so400m-patch14-384  | microsoft/phi-2                    | base      | 79.2   | 61.6 | 71.9      | 57.4    | 35.0   | 87.2 | 1462.4 | 38.2     |\n| google/siglip-so400m-patch14-384  | microsoft/phi-2                    | base\u0026lora | 77.6   | 59.7 | 71.6      | 53.8    | 33.3   | 87.9 | 1413.2 | 35.6     |\n| google/siglip-so400m-patch14-384  | microsoft/phi-2                    | share     | 80.1   | 62.1 | 73.0      | 60.3    | 37.5   | 87.2 | 1466.4 | 38.4     |\n\n### Legacy Models\n\nwhich are trained using the old codebase TinyLLaVABench.\n\n- [TinyLLaVA-3.1B](https://huggingface.co/bczhou/TinyLLaVA-3.1B)\n- [TinyLLaVA-2.0B](https://huggingface.co/bczhou/TinyLLaVA-2.0B)\n- [TinyLLaVA-1.5B](https://huggingface.co/bczhou/TinyLLaVA-1.5B)\n- [tiny-llava-hf](https://huggingface.co/bczhou/tiny-llava-v1-hf)\n\nIf you have models trained by our old codebase TinyLLaVABench and you still want to use them, we provide an example of [TinyLLaVA-3.1B](https://huggingface.co/bczhou/TinyLLaVA-3.1B) for how to use legacy models.\n\n\u003cdetails\u003e\n\u003csummary\u003eExample of using legacy models\u003c/summary\u003e\n\n\n```Python\nfrom tinyllava.eval.run_tiny_llava import eval_model\nfrom tinyllava.model.convert_legecy_weights_to_tinyllavafactory import *\n\nmodel = convert_legecy_weights_to_tinyllavafactory('bczhou/TinyLLaVA-3.1B')\n\nprompt = \"What are the things I should be cautious about when I visit here?\"\nimage_file = \"https://llava-vl.github.io/static/images/view.jpg\"\n\nargs = type('Args', (), {\n    \"model_path\": None,\n    \"model\": model,\n    \"query\": prompt,\n    \"conv_mode\": \"phi\", # the same as conv_version in the training stage. Different LLMs have different conv_mode/conv_version, please replace it\n    \"image_file\": image_file,\n    \"sep\": \",\",\n    \"temperature\": 0,\n    \"top_p\": None,\n    \"num_beams\": 1,\n    \"max_new_tokens\": 512\n})()\n\neval_model(args)\n\n\"\"\"\nOutput: \nWhen visiting this serene lakeside location with a wooden dock, there are a few things to be cautious about. First, ensure that the dock is stable and secure before stepping onto it, as it might be slippery or wet, especially if it's a wooden structure. Second, be mindful of the surrounding water, as it can be deep or have hidden obstacles, such as rocks or debris, that could pose a risk. Additionally, be aware of the weather conditions, as sudden changes in weather can make the area more dangerous. Lastly, respect the natural environment and wildlife, and avoid littering or disturbing the ecosystem.\n\"\"\"\n```\n\n\u003c/details\u003e\n\n\n\n## Launch Demo Locally\n\n### Gradio Web Demo\nLaunch a local web demo by running:\n```bash\npython tinyllava/serve/app.py --model-path tinyllava/TinyLLaVA-Phi-2-SigLIP-3.1B\n```\n### CLI Inference\nWe also support running inference with CLI. To use our model, run:\n```bash\npython -m tinyllava.serve.cli \\\n   --model-path tinyllava/TinyLLaVA-Phi-2-SigLIP-3.1B \\\n   --image-file \"./tinyllava/serve/examples/extreme_ironing.jpg\" \n```\n### Quick Inference Scripts\nIf you want to launch the model trained by yourself or us locally, here's an example.\n\u003cdetails\u003e\n\u003csummary\u003eRun inference with the model trained by yourself or downloaded from HuggingFace\u003c/summary\u003e\n\n```Python\nfrom tinyllava.eval.run_tiny_llava import eval_model\n\nmodel_path = \"/absolute/path/to/your/model/\"\nprompt = \"What are the things I should be cautious about when I visit here?\"\nimage_file = \"https://llava-vl.github.io/static/images/view.jpg\"\nconv_mode = \"phi\" # or llama, gemma, etc\n\nargs = type('Args', (), {\n    \"model_path\": model_path,\n    \"model\": None,\n    \"query\": prompt,\n    \"conv_mode\": conv_mode,\n    \"image_file\": image_file,\n    \"sep\": \",\",\n    \"temperature\": 0,\n    \"top_p\": None,\n    \"num_beams\": 1,\n    \"max_new_tokens\": 512\n})()\n\neval_model(args)\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eRun inference with the model trained by us using huggingface transformers\u003c/summary\u003e\n\n```Python\nfrom transformers import AutoTokenizer, AutoModelForCausalLM\n\nhf_path = 'tinyllava/TinyLLaVA-Phi-2-SigLIP-3.1B'\nmodel = AutoModelForCausalLM.from_pretrained(hf_path, trust_remote_code=True)\nmodel.cuda()\nconfig = model.config\ntokenizer = AutoTokenizer.from_pretrained(hf_path, use_fast=False, model_max_length = config.tokenizer_model_max_length,padding_side = config.tokenizer_padding_side)\nprompt=\"What are these?\"\nimage_url=\"http://images.cocodataset.org/val2017/000000039769.jpg\"\noutput_text, genertaion_time = model.chat(prompt=prompt, image=image_url, tokenizer=tokenizer)\n\nprint('model output:', output_text)\nprint('runing time:', genertaion_time)\n```\n\u003c/details\u003e\n\n## Custom Finetune\nIf you want to finetune TinyLLaVA with your custom datasets, please refer to [here](https://github.com/TinyLLaVA/TinyLLaVA_Factory/blob/main/CUSTOM_FINETUNE.md).\n\n## Customize Your Own Large Multimodel Models\n\n### LLM\n\nIf you want to add a new LLM by yourself, you need to create two files: one for chat template and the other for language model, under the folders `tinyllava/data/template/` and `tinyllava/model/llm/`.\n\nHere is an example of adding the Gemma model.\n\nFirstly, create `tinyllava/data/template/gemma_template.py`, which will be used for the finetuning stage.\n\n```python\nfrom dataclasses import dataclass\nfrom typing import TYPE_CHECKING, Dict, List, Optional, Sequence, Tuple, Union\nfrom packaging import version\n\nfrom .formatter import EmptyFormatter, StringFormatter\nfrom .base import Template\nfrom .formatter import Formatter\nfrom . import register_template\nfrom ...utils.constants import *\n\nfrom transformers import PreTrainedTokenizer\nimport torch\nimport tokenizers\n\n    \nsystem = \"A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.\"\n\n@register_template('gemma') # Enable the TemplateFactory to obtain the added template by this string ('gemma').\n@dataclass\nclass GemmaTemplate(Template):\n    format_image_token: \"Formatter\" = StringFormatter(slot=\"\u003cimage\u003e\\n{{content}}\")\n    format_user: \"Formatter\" = StringFormatter(slot=\"USER\" + \": \" + \"{{content}}\" + \" \")\n    format_assistant: \"Formatter\" = StringFormatter(slot=\"ASSISTANT\" + \": \" + \"{{content}}\" + \"\u003ceos\u003e\") # to be modified according to the tokenizer you choose\n    system: \"Formatter\" = EmptyFormatter(slot=system+\" \")\n    separator: \"Formatter\" = EmptyFormatter(slot=[' ASSISTANT: ', '\u003ceos\u003e']) # to be modified according to the tokenizer you choose\n\n    def _make_masks(self, labels, tokenizer, sep, eos_token_length, rounds):\n        # your code here\n        return labels, cur_len\n```\n**Tips:**\n\nPlease ensure that the `labels` (returned by the `_make_masks` function) follows this format: answers and the eos token id are not masked, and the other tokens are masked with `-100`.\n\nSecondly, create `tinyllava/model/llm/gemma.py`.\n\n```python\nfrom transformers import GemmaForCausalLM, AutoTokenizer\n# The LLM you want to add along with its corresponding tokenizer.\n\nfrom . import register_llm\n\n# Add GemmaForCausalLM along with its corresponding tokenizer and handle special tokens.\n@register_llm('gemma') # Enable the LLMFactory to obtain the added LLM by this string ('gemma').\ndef return_gemmaclass(): \n    def tokenizer_and_post_load(tokenizer):\n        tokenizer.pad_token = tokenizer.unk_token\n        return tokenizer\n    return (GemmaForCausalLM, (AutoTokenizer, tokenizer_and_post_load))\n```\n\nFinally, create `scripts/train/train_gemma.sh` with the corresponding `LLM_VERSION` and `CONV_VERSION`.\n\n### Vision Tower\n\nIf you want to add a new vision tower, you need to implement a new vision tower class that should be inherited from the base class `VisionTower`. Here's an example of the MoF vision tower.\n\nFirst, create `tinyllava/model/vision_tower/mof.py`\n\n```python\n@register_vision_tower('mof')      \nclass MoFVisionTower(VisionTower):\n    def __init__(self, cfg):\n        super().__init__(cfg)\n\n        self._vision_tower = MoF(cfg)\n        self._image_processor = # your image processor\n  \n    def _load_model(self, vision_tower_name, **kwargs):\n        # your code here, make sure your model can be correctly loaded from pretrained parameters either by huggingface or pytorch loading\n\n    def forward(self, x, **kwargs):\n        # your code here\n```\n\nThen, modify your training scripts with the corresponding `VT_VERSION`.\n\n### Connector\n\nIf you want to add a new connector, you need to implement a new connector class that should be inherited from the base class `Connector`. Here's an example of the Linear connector.\n\nFirst, create `tinyllava/model/connector/linear.py`\n\n\n```python\nimport torch.nn as nn\n\nfrom . import register_connector\nfrom .base import Connector\n    \n@register_connector('linear') #Enable the ConnectorMFactory to obtain the added connector by this string ('linear').     \nclass LinearConnector(Connector):\n    def __init__(self, config):\n        super().__init__()\n        self._connector =  nn.Linear(config.vision_hidden_size, config.hidden_size) # define your connector model\n```\n\nThen, modify your training scripts with the corresponding `CN_VERSION`.\n\n## Acknowledgement\nWe give special thanks to Lei Zhao, Luche Wang, Kaijun Luo, and Junchen Wang for building the [Demo](http://8843843nmph5.vicp.fun/#/).\n\n## Contact\nIf you have any questions, feel free to either initiate an *Issue* or contact us by WeChat (WeChatID: *TinyLLaVA*).\n\n## \u0026#x270F; Citation\n\nIf you find our paper and code useful in your research, please consider giving a star :star: and citation :pencil:.\n\n```BibTeX\n@misc{zhou2024tinyllava,\n      title={TinyLLaVA: A Framework of Small-scale Large Multimodal Models}, \n      author={Baichuan Zhou and Ying Hu and Xi Weng and Junlong Jia and Jie Luo and Xien Liu and Ji Wu and Lei Huang},\n      year={2024},\n      eprint={2402.14289},\n      archivePrefix={arXiv},\n      primaryClass={cs.LG}\n}\n```\n```BibTeX\n@article{jia2024tinyllava,\n  title={TinyLLaVA Factory: A Modularized Codebase for Small-scale Large Multimodal Models},\n  author={Jia, Junlong and Hu, Ying and Weng, Xi and Shi, Yiming and Li, Miao and Zhang, Xingjian and Zhou, Baichuan and Liu, Ziyu and Luo, Jie and Huang, Lei and Wu, Ji},\n  journal={arXiv preprint arXiv:2405.11788},\n  year={2024}\n}\n```\n\n\n## ❤️ Community efforts\n* Our codebase is built upon the [LLaVA](https://github.com/haotian-liu/LLaVA) project. Great work!\n* Our project uses data from the [ShareGPT4V](https://github.com/InternLM/InternLM-XComposer/tree/main/projects/ShareGPT4V) project. Great work!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FTinyLLaVA%2FTinyLLaVA_Factory","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FTinyLLaVA%2FTinyLLaVA_Factory","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FTinyLLaVA%2FTinyLLaVA_Factory/lists"}