{"id":51465402,"url":"https://github.com/ray-project/multimodal-training","last_synced_at":"2026-07-06T10:30:38.887Z","repository":{"id":322306498,"uuid":"1087424863","full_name":"ray-project/multimodal-training","owner":"ray-project","description":null,"archived":false,"fork":false,"pushed_at":"2026-01-23T05:24:53.000Z","size":484,"stargazers_count":25,"open_issues_count":0,"forks_count":3,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-01-23T17:33:03.606Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ray-project.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-10-31T22:18:47.000Z","updated_at":"2026-01-22T12:28:34.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/ray-project/multimodal-training","commit_stats":null,"previous_names":["ray-project/multimodal-training"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ray-project/multimodal-training","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ray-project%2Fmultimodal-training","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ray-project%2Fmultimodal-training/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ray-project%2Fmultimodal-training/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ray-project%2Fmultimodal-training/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ray-project","download_url":"https://codeload.github.com/ray-project/multimodal-training/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ray-project%2Fmultimodal-training/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35187624,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-06T02:00:07.184Z","response_time":106,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-07-06T10:30:38.279Z","updated_at":"2026-07-06T10:30:38.875Z","avatar_url":"https://github.com/ray-project.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Disaggregated Hybrid Parallelism with Ray\n\nA framework for training vision-language models using disaggregated hybrid parallelism, where each model component adopts its optimal parallelization strategy independently.\n\n## Overview\n\nModern vision-language models (VLMs) present a fundamental training challenge: they consist of **heterogeneous components** with vastly different computational and memory characteristics, making uniform parallelization strategies inefficient.\n\n### The Heterogeneity Problem\n\nVision-language models combine two components with fundamentally mismatched characteristics:\n\n**Vision Encoder** (Small Model, Long Sequences)\n- **Parameters**: ~500M-6B (relatively small)\n- **Sequences**: Up to 65k tokens per high-resolution image\n- **Computation**: Fast per iteration (smaller model, often sparse attention)\n- **Bottleneck**: Activation memory scales with sequence length (B×S×H)\n\n**Text Decoder** (Large Model)\n- **Parameters**: 7B-72B (10-100× larger than vision)\n- **Computation**: Slower per iteration (much larger model, dense attention)\n- **Bottleneck**: Model states scale with parameter count (M×20 bytes with mixed precision + AdamW)\n\nThis asymmetry creates a fundamental challenge: **the vision encoder is memory-bound by activations, while the text decoder is memory-bound by parameters**.\n\n### Why Uniform Parallelization Fails\n\nExisting distributed training frameworks (DeepSpeed, Megatron-LM) assume **uniform parallelization strategies** across all model components. This creates severe inefficiencies:\n\n**Tensor Parallelism (TP):**\n- ✅ Efficient for large text models (partitions parameters across GPUs)\n- ❌ **Inefficient for small vision encoders**: Small hidden dimensions make communication overhead dominate computation time, creating a bottleneck\n\n**Data Parallelism / ZeRO3 (parameter sharding):**\n- ✅ Works well for short sequences that fit in memory\n- ❌ **Critical limitation**: Minimum batch size per GPU = 1, causing OOM errors with long sequences (65k tokens)\n- ❌ **Real-world impact**: Researchers must downscale images (896×896 vs 3584×3584), resulting in a **16× reduction in trainable sequence length** compared to hardware capability\n\n### Our Solution: Disaggregated Hybrid Parallelism (DHP)\n\nThis repository implements **disaggregated hybrid parallelism (DHP)** using Ray, allowing each model component to adopt its optimal parallelization strategy:\n\n**✅ Vision Encoder: Sequence Parallelism (SP) + ZeRO1 (data parallel + optimizer states sharding)**\n- Implements DeepSpeed-Ulysses style sequence parallelism [1] to split long sequences across GPUs\n- Activation memory scales linearly with GPU count: B×S×H/N per GPU\n- Avoids tensor parallelism overhead for small models\n- Communication pattern: All-to-All operations enable efficient attention across partitioned sequences\n- Particularly effective with sparse attention patterns (see Qwen2.5-VL results below)\n\n**✅ Text Decoder: Tensor Parallelism (TP)**\n- Efficiently partitions large models: model memory = M×20/N per GPU (mixed precision + AdamW)\n- Dense computation amortizes TP communication cost\n- Essential for fitting 7B-72B parameter models in memory\n\n**Key Insight**: Heterogeneous models need heterogeneous parallelization strategies. By disaggregating components and choosing the parallelism of each independently, we achieve better resource utilization and enable previously infeasible training scenarios.\n\n## How Ray Enables Disaggregated Hybrid Parallelism\n\nTraditional distributed training frameworks (DeepSpeed, Megatron-LM) use a **single SPMD (Single Program Multiple Data) process group** where all model components must use the same parallelization strategy. This creates a fundamental constraint: **you cannot apply different parallelization strategies to different parts of the model**.\n\n**Traditional Single SPMD Group:**\n```\nGPU 0: Process(rank=0) → Full Model (Vision + Text)\nGPU 1: Process(rank=1) → Full Model (Vision + Text)\nGPU 2: Process(rank=2) → Full Model (Vision + Text)\nGPU 3: Process(rank=3) → Full Model (Vision + Text)\n           ↓\nSingle torch.distributed process group\nUniform parallelization strategy enforced across ALL components\n(e.g., if using TP, BOTH vision and text must use TP)\n```\n\nRay breaks this constraint by enabling **multiple independent SPMD groups**, each with its own parallelization strategy and resource allocation:\n\n**Ray-Orchestrated Multiple SPMD Groups:**\n```\nRay Driver (Central Orchestrator)\n├── Vision ActorGroup: Independent SPMD Group (SP)\n│   ├── VisionActor(rank=0) → Vision Encoder only\n│   ├── VisionActor(rank=1) → Vision Encoder only\n│   ├── VisionActor(rank=2) → Vision Encoder only\n│   └── VisionActor(rank=3) → Vision Encoder only\n│        ↓\n│   SP process group: All-to-All for sequence parallel attention\n│\n└── Text ActorGroup: Independent SPMD Group (TP)\n    ├── TextActor(rank=0) → Text Decoder only\n    ├── TextActor(rank=1) → Text Decoder only\n    ├── TextActor(rank=2) → Text Decoder only\n    └── TextActor(rank=3) → Text Decoder only\n         ↓\n    TP process group: All-Reduce for activations (forward) and gradients (backward)\n```\n\n### Key Components\n\n**ActorGroup**: Collection of Ray actors forming an independent SPMD group with its own `torch.distributed` process group. Supports GPU collocation (multiple actors per GPU) and provides APIs for synchronous (`execute_all`) and asynchronous (`execute_all_async`) execution across all actors.\n\n**VisionTrainer / TextTrainer**: Specialized trainer classes with decomposed methods (`forward_step`, `backward_step`, `optimizer_step`) instead of monolithic training loops. Each trainer initializes its component's model with the appropriate parallelization strategy (SP for vision, TP for text).\n\n**Ray Driver**: Orchestrates the training loop by invoking methods on ActorGroups. Coordinates cross-component data transfer via Ray ObjectRefs, enabling automatic data movement between vision and text actors.\n\n### Training Loop Example\n\n```python\n# Create independent actor groups with different parallelization strategies\nvision_group = ActorGroup(config, VisionTrainer, num_actors=4)  # SP\ntext_group = ActorGroup(config, TextTrainer, num_actors=4)      # TP\n\nfor iteration in range(num_iterations):\n    # Forward: Vision (SP) → Text (TP) via Ray ObjectRefs\n    vision_refs = vision_group.execute_all_async(\"forward_step\", iteration)\n    text_refs = text_group.execute_all_async(\"forward_step\", vision_refs, iteration)\n    losses = ray.get(text_refs)\n\n    # Backward: Text → Vision\n    text_grad_refs = text_group.execute_all_async(\"backward_step\")\n    vision_backward_refs = vision_group.execute_all_async(\"backward_step\", text_grad_refs)\n    ray.get(vision_backward_refs)\n\n    # Independent optimizer updates\n    vision_group.execute_all(\"optimizer_step\")\n    text_group.execute_all(\"optimizer_step\")\n```\n\n**Key Benefits**: Vision and text groups use different parallelization strategies (SP vs TP) on the same GPUs through collocation. Ray ObjectRefs handle cross-component data transfer automatically. Easy to reconfigure `num_actors` or strategies per component without framework changes.\n\n## Case Study: Qwen2.5-VL\n\n### Architecture Details\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/qwen2_5_architecture.svg\" alt=\"Qwen2.5-VL Architecture\" width=\"70%\"\u003e\n\u003c/p\u003e\n\n*Figure 1: Qwen2.5-VL architecture showing the vision encoder and the text decoder. The vision encoder processes high-resolution images (up to 3584×3584) into variable-length sequences (N = 1k-65k tokens). The text input contains `\u003cimage\u003e` placeholder tokens alongside 512 text tokens. Vision embeddings are scattered into these N placeholder positions, creating a combined sequence of N vision tokens + 512 text tokens processed by the language model.*\n\n**Vision Encoder** (Custom ViT - 670M parameters)\n- **Architecture**: 32 layers, 1280 hidden dimension, 16 attention heads\n- **Window Attention**: 112-token windows, full attention only on layers {7, 15, 23, 31}\n- **Sequences**: 1k-65k tokens from high-resolution images (up to 3584×3584) or videos\n- **Computation**: Fast (~0.2-0.7s per iteration for 4-16k tokens)\n\n**Text Decoder** (Qwen2.5 - 7B/32B/72B parameters)\n- **7B**: 28 layers, 3584 hidden dimension, 4 KV heads\n- **32B**: 64 layers, 5120 hidden dimension, 8 KV heads\n- **72B**: 80 layers, 8192 hidden dimension, 8 KV heads\n- **Sequences**: Vision tokens (up to 16k) + text tokens (~512) processed together\n- **Computation**: Slower (~1-3s per iteration) due to model size\n\n### Why Disaggregated Hybrid Parallelism Works for Qwen2.5-VL\n\nQwen2.5-VL's architecture is well-suited for disaggregated hybrid parallelism: the vision encoder's **small hidden size (1280)** makes TP inefficient, while **sparse window attention** (only 4 of 32 layers need cross-GPU All-to-All) makes SP highly efficient. The text decoder's **large parameter count (7B-72B)** requires TP's parameter partitioning to fit in memory.\n\n## Experimental Results\n\n### Throughput Comparison: Qwen2.5-VL-32B (8×H100)\n\nWe compare three parallelization strategies across varying sequence lengths:\n\n![Throughput Comparison](assets/qwen2_5_results.png)\n\n*Figure 2: Throughput comparison of three parallelization strategies (SP+TP, pure TP, ZeRO3) on Qwen2.5-VL-32B across varying vision sequence lengths (1k–65k tokens). SP+TP consistently outperforms pure TP and enables training at extreme sequence lengths where ZeRO3 hits OOM beyond 16k tokens. The batch size upper bound was set to 8 (corresponding to a micro batch size of 1, which is the minimum batch size supported by ZeRO3). When batch size 8 was not feasible due to memory limits, we used the largest batch size that could fit on the GPUs. ZeRO3 results were obtained using the implementation from the Qwen-VL repository, while the TP baseline was implemented in-house without using Ray.*\n\n**Key Findings:**\n\n1. **SP+TP achieves 1.26-1.37× speedup over pure TP** consistently across all sequence lengths (1k-65k tokens)\n\n2. **SP+TP enables training at extreme sequence lengths where ZeRO3 fails**: ZeRO3 hits OOM at 16k+ tokens due to replicated activations, while DHP distributes activation memory across GPUs\n\n3. **ZeRO3 has sequence-length dependent performance**: Optimal at ~9k tokens where it overlaps communication, but struggles with short sequences (insufficient computation) and long sequences (OOM). SP+TP maintains consistent performance across all lengths\n\n\n**Configuration**: Qwen2.5-VL-32B, 8× H100 GPUs, BF16 mixed precision, Flash Attention 2, activation checkpointing enabled.\n\n## How to use\n\n### Environment Setup\n\nThese are the environments we have tested and verified the system on. Other configurations may also work but are not officially tested.\n\n- Python 3.12\n- CUDA 12.6\n- NVIDIA A100/H100 GPUs\n  - Qwen2.5-VL 7B model: 4 GPUs\n  - Qwen2.5-VL 32B model: 8 GPUs\n\n#### Installation Steps\n\n1. **Clone the repository**\n\n```bash\ngit clone https://github.com/ray-project/multimodal-training\ncd multimodal-training\n```\n\n2. **Install Python dependencies**\n\n\n```bash\n# Install PyTorch with CUDA support (adjust CUDA version as needed)\npip install torch==2.8.0 --index-url https://download.pytorch.org/whl/cu126\n\n# Install Ray, DeepSpeed (custom branch), and other dependencies\npip install -r requirements.txt\n```\n\n3. **Start Ray cluster (Optional)**\n\nYou don't need to start a Ray cluster when running on your local machine. If you’d like to run on a Ray cluster, please refer to the [Ray documentation](https://docs.ray.io/en/latest/ray-core/starting-ray.html) for instructions on how to set one up.\n\n### Data Preparation\n\nThe framework uses a simplified, LLaVA-style data format that is widely adopted in the VLM community. This format requires two components:\n\n1. **Annotation file** (JSON or JSONL) containing conversation data and image references\n2. **Image directory** containing the actual image files\n\n#### Annotation File Format\n\nThe annotation file should contain conversation examples in the following structure:\n\n```json\n[\n  {\n    \"image\": \"relative/path/to/image.jpg\",\n    \"conversations\": [\n      {\n        \"from\": \"human\",\n        \"value\": \"\u003cimage\u003e\\nDescribe this image.\"\n      },\n      {\n        \"from\": \"gpt\",\n        \"value\": \"A man with a red helmet on a small moped on a dirt road.\"\n      }\n    ]\n  },\n  ...\n]\n```\n\n**Format Details:**\n- **File format**: Either JSON (array of objects) or JSONL (one JSON object per line)\n- **`image`**: Relative path to the image file (relative to the `data_path` specified in config)\n- **`conversations`**: List of conversation turns between human and model\n  - **`from`**: Either `\"human\"` (user) or `\"gpt\"` (assistant)\n  - **`value`**: Text content. Use `\u003cimage\u003e` token to indicate where the image should be inserted\n\n#### Tested Datasets\n\nWe have tested the codebase with:\n- [COCO](https://cocodataset.org/) - Image captioning dataset\n- [LAION POP](https://laion.ai/blog/laion-pop/) - High-resolution images with detailed descriptions\n\nYou'll need to convert the original annotation formats to the conversation format shown above.\n\n### Training Configuration\n\nThe model and training settings are specified using a YAML configuration file. See [example configuration](configs/sample.yaml) for full details.\n\n#### Key Configuration Sections\n\nBelow is a snippet illustrating the main configuration parameters:\n\n```yaml\n...\n# Vision model configuration\nvision:\n  model_type: \"qwen2_5_vl\"  # Options: \"qwen2_5_vl\"\n  model_name: \"Qwen/Qwen2.5-VL-32B-Instruct\"  # Path to pretrained model\n  parallelism: \"sequence\"  # Options: \"none\", \"tensor\", \"sequence\", \"deepspeed\"\n...\n\n# Text model configuration\ntext:\n  model_type: \"qwen2_5_vl\"  # Options: \"qwen2_5_vl\"\n  model_name: \"Qwen/Qwen2.5-VL-32B-Instruct\"  # Path to pretrained model\n  parallelism: \"tensor\"  # Options: \"none\", \"tensor\", \"deepspeed\", \"autotp\"\n  autotp_size: null  # Optional tensor parallel size when using AutoTP (defaults to world size)\n...\n\n# Training configuration\ntraining:\n  batch_size: 2\n  learning_rate: 5e-05\n  no_checkpoint: true  # Disable checkpointing (default: false)\n  checkpoint_dir: \"/tmp/checkpoints\"  # Directory to save/load\n...\n\n# Data configuration\ndata:\n  datasets:\n    - laion_pop_train\n  data_registry:\n    mscoco2017_train_captions:\n      annotation_path: \"/path/to/mscoco2017/annotations/coco2017_train.json\"\n      data_path: \"/path/to/mscoco2017\"\n    mscoco2017_val_captions:\n      annotation_path: \"/path/to/mscoco2017/annotations/coco2017_val.json\"\n      data_path: \"/path/to/mscoco2017\"\n    laion_pop_train:\n      annotation_path: \"/path/to/laion_pop/laion_pop_train.jsonl\"\n      data_path: \"/path/to/laion_pop/images\"\n    laion_pop_val:\n      annotation_path: \"/path/to/laion_pop/laion_pop_val.jsonl\"\n      data_path: \"/path/to/laion_pop/images\"\n  num_workers: 4\n  pin_memory: true\n  data_flatten: true\n  min_pixels: 802816\n  max_pixels: 802816\n```\n\n\n### Running Training\n\nYou can run training specifying the configuration file.\n\n```bash\npython -m python.train_ray \\\n    --config-path=../configs \\\n    --config-name=sample\n```\n\nTo save checkpoints during training, set `no_checkpoint` to `false` in the training configuration section. Checkpoints will be saved to the specified `checkpoint_dir`.\nCheckpoints are saved at the end of each epoch.\nIf you want to resume training from a checkpoint, specify the `checkpoint_dir` where the checkpoints are stored. The training script will automatically load the latest checkpoint from that directory.\n\nTo use the pretrained weights for Qwen2.5-VL, please refer to [Loading Pretrained Weights](LOAD_PRETRAINED_WEIGHTS.md).\n\nTo reproduce the experiments described above, please refer to [Experiments](experiments/EXPERIMENTS.md).\n\n## References\n\n[1] Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Reza Yazdani Aminabadi, Shuaiwen Leon Song, Samyam Rajbhandari, Yuxiong He. \"DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models.\" *arXiv:2309.14509*, 2023. https://arxiv.org/abs/2309.14509\n\n## Citation\n\nIf you use this codebase in your research, please cite:\n\n```bibtex\n@software{ray-multimodal-training,\n  title={Disaggregated Hybrid Parallelism with Ray},\n  author={Masahiro Tanaka, Amjad Almahairi, Nithin Chalapathi, Richard Liaw, Praveen Gorthy, Matthew Deng},\n  year={2025},\n  url={https://github.com/ray-project/multimodal-training}\n}\n```\n\n## Acknowledgments\n\n- Thanks to the Qwen team for their excellent open-source models\n- Built on top of DeepSpeed, PyTorch Distributed, and Ray\n\n## Contact\n\nFor questions or issues, please [open an issue](https://github.com/ray-project/multimodal-training/issues).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fray-project%2Fmultimodal-training","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fray-project%2Fmultimodal-training","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fray-project%2Fmultimodal-training/lists"}