{"id":22767222,"url":"https://github.com/htanh2003/vistral-v","last_synced_at":"2025-07-08T01:39:42.892Z","repository":{"id":254062994,"uuid":"845115910","full_name":"HTAnh2003/Vistral-V","owner":"HTAnh2003","description":null,"archived":false,"fork":false,"pushed_at":"2024-08-21T06:41:02.000Z","size":16218,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-05T12:39:33.044Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/HTAnh2003.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-08-20T15:59:30.000Z","updated_at":"2024-08-21T06:41:06.000Z","dependencies_parsed_at":"2024-08-21T07:44:23.374Z","dependency_job_id":null,"html_url":"https://github.com/HTAnh2003/Vistral-V","commit_stats":null,"previous_names":["htanh2003/vistral-v"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HTAnh2003%2FVistral-V","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HTAnh2003%2FVistral-V/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HTAnh2003%2FVistral-V/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HTAnh2003%2FVistral-V/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/HTAnh2003","download_url":"https://codeload.github.com/HTAnh2003/Vistral-V/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246307931,"owners_count":20756478,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-11T13:17:38.239Z","updated_at":"2025-03-30T10:43:19.842Z","avatar_url":"https://github.com/HTAnh2003.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Vistral-V (Vistral-Vision) - Visual Instruction Tuning for Vistral - Vietnamese Large Vision-Language Model.\n\n## Contents\n- [Quick start Notebook](#quick-start-notebook)\n- [Install](#install)\n- [Train](#train)\n- [Demo](#demo)\n- [Examples](#examples)\n\n## Quick Start Notebook\n\nYou can follow the notebooks here to inference the Vistral-V model.\n\n[![Kaggle](https://kaggle.com/static/images/open-in-kaggle.svg)](https://kaggle.com/kernels/welcome?src=https://github.com/hllj/Vistral-V/blob/master/assets/quickstart_example.ipynb)\n\n\n## Install\n\nIf you are not using Linux, do *NOT* proceed, see instructions for [macOS](https://github.com/haotian-liu/LLaVA/blob/main/docs/macOS.md) and [Windows](https://github.com/haotian-liu/LLaVA/blob/main/docs/Windows.md).\n\n1. Clone this repository and navigate to LLaVA folder\n```bash\ngit clone https://github.com/hllj/Vistral-V.git\ncd Vistral-V\n```\n\n2. Install Package\n```Shell\nconda create -n llava python=3.10 -y\nconda activate llava\npip install --upgrade pip  # enable PEP 660 support\npip install -e .\n```\n\n```Shell\npip install -r requirements.txt\n```\n\n3. Install additional packages for training cases\n```\npip install -e \".[train]\"\npip install flash-attn --no-build-isolation\n```\n\n### Upgrade to latest code base\n\n```Shell\ngit pull\npip install -e .\n\n# if you see some import errors when you upgrade,\n# please try running the command below (without #)\n# pip install flash-attn --no-build-isolation --no-cache-dir\n```\n\n## Train\n\nYou can refer to the [Training model instructions](https://github.com/haotian-liu/LLaVA?tab=readme-ov-file#train) from the original LLaVA repository.\n\nBelow are the recipes to train the Vistral-V model through 2 stages: Pretrain - feature alignment and Finetune - visual instruction tuning. In this tutorial, we only use 1-2 device gpu NVIDIA A100 40Gb.\n\n### Pretrain (feature alignment)\n\nFor this phase, the model will train a projector to align features between the vision encoder and the language model.\n\nYou can retrain this stage using the instructions below, or use the [training results for the pretraining stage here](https://huggingface.co/Vi-VLM/llava-vistral-7b-pretrain).\n\n#### Data preparation\n\nPretrain dataset includes 2 subsets:\n- ShareGPT4V\n- WIT\n\nPretrain dataset includes image-text data pairs with requirements to describe the image with high quality caption.\n\nYou can refer to instructions from LLaVA to create more custom datasets [here](./docs/Data.md).\n\n```bash\npython data/create_dataset_images.py --local-dir \u003cIMAGE DIR\u003e --stage pretrain\n\nDATA_DIR=\u003cIMAGE DIR\u003e/images bash data/extract_pretrain_images.sh\n\npython data/create_dataset_json.py --local-dir \u003cIMAGE DIR\u003e/images --stage pretrain\n```\n\n#### Training\n\nAt this stage we scale data with 2 versions.\n\n- Version 1: Pretrain with ShareGPT4V dataset only. Dataset size for version 1 is around 100k samples.\n\n- Version 2: Pretrain with ShareGPT4V + WIT dataset. Dataset size for version 2 is around 340k samples.\n\nWe use the [multilingual SigLIP](https://huggingface.co/google/siglip-base-patch16-256-multilingual) model instead of CLIP in LLaVA because the ability to understand Vietnamese and features from SigLIP is much better than CLIP.\n\nYou can refer to the recipes we used for 2 versions here.\n\n[Version 1 script](./scripts/vistral_llava/pretrain_v1.sh):\n\n```bash\nbash scripts/vistral_llava/pretrain_v1.sh\n```\n\n[Version 2 script](./scripts/vistral_llava/pretrain_v2.sh):\n\n```bash\nbash scripts/vistral_llava/pretrain_v1.sh\n```\n\n#### Loss Curve\n\n![Pretrain Loss Curve](assets/images/pretrain_loss_curve.png)\n\nA few comments on the results:\n\n- Perform projector training so that the feature alignment between the vision feature and the language feature converges to a loss of about 0.9 - 1.1\n- After scaling more data (version 2) for training, it shows that the model has better convergence.\n\n### Finetune (visual instruction tuning)\n\nThis is the stage of finetuning the Language model and projector with the frozen Vision Encoder, to learn the visual instruction dataset.\n\n#### Data preparation\n\nFinetune dataset includes LLaVA-style data covering 3 different types of tasks:: conversation, complex reasoning, detail description.\n\n```bash\npython data/create_dataset_images.py --local-dir \u003cIMAGE DIR\u003e --stage finetune\n\nDATA_DIR=\u003cIMAGE DIR\u003e/images bash data/extract_finetune_images.sh\n\npython data/create_dataset_json.py --local-dir \u003cIMAGE DIR\u003e/images --stage finetune\n```\n\n#### Training\n\nFor the two versions of the pretrain phase, we also tried finetune to see the results.\n\nCurrently due to limitations in available hardware, we only implement finetune with LoRA. However, according to my guess, if you fully refine the Language Model, you can get better results.\n\nIn addition to changing the data for the pretrain/finetune, we also made adjustments to match the device's existing VRAM amount of 40Gb GPU VRAM: **adjust the batch size and scale down the learning rate**.\n\n[Version 1 script](./scripts/vistral_llava/finetune_lora_v1.sh):\n```bash\nbash scripts/vistral_llava/finetune_lora_v1.sh\n```\n\n[Version 2 script](./scripts/vistral_llava/finetune_lora_v2.sh):\n```bash\nbash scripts/vistral_llava/finetune_lora_v2.sh\n```\n\n#### Merge and Push to Hub (optional)\n\n```bash\npython upload_merge_model.py \\\n    --model-path \u003clora checkpoint dir\u003e \\\n    --model-base Viet-Mistral/Vistral-7B-Chat \\\n    --save-merged-model --save-dir \u003cmerged checkpoint dir\u003e \\\n    --push-to-hub --repo-id \u003crepo id HF\u003e\n```\n\nExample:\n\n```bash\npython upload_merge_model.py --model-path checkpoints/llava-vistral-7b-lora-2 --model-base Viet-Mistral/Vistral-7B-Chat --save-merged-model --save-dir checkpoints/Vistral-V-7B/ --push-to-hub --repo-id Vi-VLM/Vistral-V-7B\n```\n\n### Loss Curve\n\n![Finetune Loss Curve](assets/images/finetune_loss_curve.png)\n\nA few comments on the results:\n\n- Finetune on the LLaVA data set gives quite good loss results, about 0.7 - 0.9. Demonstrating the ability of the trained pretrain model to converge relatively quickly on LLaVA data.\n\n- Version 2 relies on pretraining on more data for better loss after training than version 1. \n\nIn fact, with version 2, we trained on 2 devices, the actual batch size will be double that of version 1, so that's it. Can also be a reason for better loss.\n\n## CLI\n\nYou can quickly test the model using the simple CLI as follows.\n\n```bash\npython -m llava.serve.cli --model-path \u003crepo_id or checkpoint dir\u003e --image-file ./assets/images/example.jpeg\n```\n\nExample:\n\n```bash\npython -m llava.serve.cli --model-path Vi-VLM/Vistral-V-7B --image-file ./assets/images/example.jpeg\n```\n\n## Demo\n\nYou can follow the instructions of the original LLaVA repository [here](https://github.com/haotian-liu/LLaVA?tab=readme-ov-file#demo) to start the models after training. Here we will show how to start a Vistral-V service for our current LoRA model.\n\n### Launch a controller\n\n```bash\npython -m llava.serve.controller --host 0.0.0.0 --port 10000\n```\n\n### Launch a gradio web server.\n\n```bash\npython -m llava.serve.gradio_web_server --controller http://localhost:10000 --model-list-mode reload\n```\n\n### Launch a model worker\n\nStart a model worker to start serving your model, the script supports multi device inference.\n\n#### LoRA, unmerged\n\nAfter training the model with LoRA you can start the model worker or download our lora checkpoint.\n\n```bash\npython -m llava.serve.model_worker --host 0.0.0.0 --controller http://localhost:10000 --port 40000 --worker http://localhost:40000 --model-path Vi-VLM/llava-vistral-7b-lora --model-base Viet-Mistral/Vistral-7B-Chat\n```\n\nor \n\n```bash\npython -m llava.serve.model_worker --host 0.0.0.0 --controller http://localhost:10000 --port 40000 --worker http://localhost:40000 --model-path \u003clora checkpoint dir\u003e --model-base Viet-Mistral/Vistral-7B-Chat\n```\n\n### Fully Finetuned Model / Merged Model\n\nYou can use a fully finetuned or merged model (if merging lora, see the [instructions above](#merge-and-push-to-hub-optional)), you can use our model checkpoint.\n\n```bash\npython -m llava.serve.model_worker --host 0.0.0.0 --controller http://localhost:10000 --port 40000 --worker http://localhost:40000 --model-path Vi-VLM/Vistral-V-7B\n```\n\nor\n\n```bash\npython -m llava.serve.model_worker --host 0.0.0.0 --controller http://localhost:10000 --port 40000 --worker http://localhost:40000 --model-path \u003cmodel checkpoint dir\u003e\n```\n\n### Examples\n\n![Example 1](assets/images/example_1.png)\n\n![Example 2](assets/images/example_2.png)\n\n![Example 3](assets/images/example_3.png)\n\n![Example 4](assets/images/example_4.png)\n\n![Example 5](assets/images/example_5.png)\n\n![Example 6](assets/images/example_6.png)\n\n![Example 7](assets/images/example_7.png)\n\n![Example 8](assets/images/example_8.png)\n\n## Evaluation\n\nCurrently, Vistral-V has not been run on any specific benchmark for the Vision-Language Benchmark. We will conduct an evaluation in some tasks and update as soon as possible.\n\n## Additional Information\n\n- Organization: [Vietnamese-VLM](https://huggingface.co/Vi-VLM)\n\n- Contact: [Bui Van Hop](https://github.com/hllj)\n\n## Acknowledgement\n\n- [LLaVA](https://github.com/haotian-liu/LLaVA): We used most of the source code and instructions of the LLaVA repository and made a few modifications on it to suit the model architecture.\n\n- [Vistral](https://huggingface.co/Viet-Mistral/Vistral-7B-Chat): We are very grateful to the Vistral development team for creating a great LLM for Vietnamese.\n\n- [SigLIP](https://huggingface.co/google/siglip-base-patch16-256-multilingual): We also thank Google and the SigLIP authors for creating a very good Image-Text model, which we took advantage of useful features from the multilingual SigLIP model.\n\n## Citation Information\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhtanh2003%2Fvistral-v","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhtanh2003%2Fvistral-v","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhtanh2003%2Fvistral-v/lists"}