https://github.com/roboflow/maestro
streamline the fine-tuning process for multimodal models: PaliGemma 2, Florence-2, and Qwen2.5-VL
https://github.com/roboflow/maestro
captioning fine-tuning florence-2 multimodal objectdetection paligemma phi-3-vision qwen2-vl transformers vision-and-language vqa
Last synced: about 2 months ago
JSON representation
streamline the fine-tuning process for multimodal models: PaliGemma 2, Florence-2, and Qwen2.5-VL
- Host: GitHub
- URL: https://github.com/roboflow/maestro
- Owner: roboflow
- License: apache-2.0
- Created: 2023-11-24T13:28:57.000Z (over 1 year ago)
- Default Branch: develop
- Last Pushed: 2025-05-12T17:23:03.000Z (2 months ago)
- Last Synced: 2025-05-13T09:12:10.533Z (2 months ago)
- Topics: captioning, fine-tuning, florence-2, multimodal, objectdetection, paligemma, phi-3-vision, qwen2-vl, transformers, vision-and-language, vqa
- Language: Python
- Homepage: https://maestro.roboflow.com
- Size: 10.6 MB
- Stars: 2,559
- Watchers: 35
- Forks: 204
- Open Issues: 19
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Citation: CITATION.cff
Awesome Lists containing this project
- awesome-ChatGPT-repositories - multimodal-maestro - Effective prompting for Large Multimodal Models like GPT-4 Vision, LLaVA or CogVLM. 🔥 (Prompts)
- awesome-yolo-object-detection - maestro - tuning for everyone. maestro is a streamlined tool to accelerate the fine-tuning of multimodal models. By encapsulating best practices from our core modules, maestro handles configuration, data loading, reproducibility, and training loop setup. It currently offers ready-to-use recipes for popular vision-language models such as [Florence-2](https://arxiv.org/abs/2311.06242), PaliGemma 2, and [Qwen2.5-VL](https://github.com/QwenLM/Qwen2.5-VL). [maestro.roboflow.com](https://maestro.roboflow.com/latest/) (Object Detection Applications)
- Awesome-LLMOps - maestro - tuning process for multimodal models: PaliGemma 2, Florence-2, and Qwen2.5-VL.    (Training / FineTune)
- awesome-hacking-lists - roboflow/maestro - streamline the fine-tuning process for multimodal models: PaliGemma 2, Florence-2, and Qwen2.5-VL (Python)
- StarryDivineSky - roboflow/maestro - 2和Qwen2.5-VL）的微调过程。该项目提供了一套工具和工作流程，帮助开发者更高效地调整这些模型，使其适应特定的任务和数据集。Maestro可能包含数据准备、模型训练、评估和部署等环节的优化。它可能利用Roboflow的现有平台或工具，例如数据标注和管理功能，来加速微调流程。该项目专注于提高多模态模型的性能和易用性，让开发者更容易地利用这些先进模型解决实际问题。具体实现细节和使用方法请参考项目文档。 (多模态大模型 / 资源传输下载)
README
maestro
[](https://badge.fury.io/py/maestro)
[](https://colab.research.google.com/github/roboflow/maestro/blob/develop/cookbooks/maestro_qwen2_5_vl_json_extraction.ipynb)
[](https://discord.gg/GbfgXGJ8Bk)
## Hello
**maestro** is a streamlined tool to accelerate the fine-tuning of multimodal models.
By encapsulating best practices from our core modules, maestro handles configuration,
data loading, reproducibility, and training loop setup. It currently offers ready-to-use
recipes for popular vision-language models such as **Florence-2**, **PaliGemma 2**, and
**Qwen2.5-VL**.
## Fine-tune VLMs for free
| model, task and acceleration | open in colab |
|:------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|
| Florence-2 (0.9B) object detection with LoRA (experimental) | [](https://colab.research.google.com/github/roboflow/maestro/blob/develop/cookbooks/maestro_florence_2_object_detection.ipynb) |
| PaliGemma 2 (3B) JSON data extraction with LoRA | [](https://colab.research.google.com/github/roboflow/maestro/blob/develop/cookbooks/maestro_paligemma_2_json_extraction.ipynb) |
| Qwen2.5-VL (3B) JSON data extraction with QLoRA | [](https://colab.research.google.com/github/roboflow/maestro/blob/develop/cookbooks/maestro_qwen2_5_vl_json_extraction.ipynb) |
| Qwen2.5-VL (7B) object detection with QLoRA (experimental) | [](https://colab.research.google.com/github/roboflow/maestro/blob/develop/cookbooks/maestro_qwen2_5_vl_object_detection.ipynb) |
## News
- `2025/02/05` (`1.0.0`): This release introduces support for Florence-2, PaliGemma 2, and Qwen2.5-VL and includes LoRA, QLoRA, and graph freezing to keep hardware requirements in check. It offers a single CLI/SDK to reduce code complexity, and a consistent JSONL format to streamline data handling.
## Quickstart
### Install
To begin, install the model-specific dependencies. Since some models may have clashing requirements,
we recommend creating a dedicated Python environment for each model.
```bash
pip install "maestro[paligemma_2]"
```
### CLI
Kick off fine-tuning with our command-line interface, which leverages the configuration
and training routines defined in each model’s core module. Simply specify key parameters such as
the dataset location, number of epochs, batch size, optimization strategy, and metrics.
```bash
maestro paligemma_2 train \
--dataset "dataset/location" \
--epochs 10 \
--batch-size 4 \
--optimization_strategy "qlora" \
--metrics "edit_distance"
```
### Python
For greater control, use the Python API to fine-tune your models.
Import the train function from the corresponding module and define your configuration
in a dictionary. The core modules take care of reproducibility, data preparation,
and training setup.
```python
from maestro.trainer.models.paligemma_2.core import train
config = {
"dataset": "dataset/location",
"epochs": 10,
"batch_size": 4,
"optimization_strategy": "qlora",
"metrics": ["edit_distance"]
}
train(config)
```
## Contribution
We appreciate your input as we continue refining Maestro. Your feedback is invaluable in guiding our improvements. To
learn how you can help, please check out our [Contributing Guide](https://github.com/roboflow/maestro/blob/develop/CONTRIBUTING.md).
If you have any questions or ideas, feel free to start a conversation in our [GitHub Discussions](https://github.com/roboflow/maestro/discussions).
Thank you for being a part of our journey!