{"id":127775,"url":"https://github.com/edujbarrios/awesome-vision-ai-stack","name":"awesome-vision-ai-stack","description":"A curated, builder-first list of Vision Language Models (VLMs), local runtimes, document AI tools, UI agents, robotics vision stacks, datasets, benchmarks, and production resources.","projects_count":136,"last_synced_at":"2026-10-07T07:00:19.199Z","repository":{"id":344834477,"uuid":"1183332129","full_name":"edujbarrios/awesome-vision-ai-stack","owner":"edujbarrios","description":"A curated, builder-first list of Vision Language Models (VLMs), local runtimes, document AI tools, UI agents, robotics vision stacks, datasets, benchmarks, and production resources.","archived":false,"fork":false,"pushed_at":"2026-03-16T14:18:05.000Z","size":17,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-08-28T17:08:52.057Z","etag":null,"topics":["ai","llm","vlm","vlm-benchmark","vlm-inference"],"latest_commit_sha":null,"homepage":"","language":"Markdown","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/edujbarrios.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-16T13:59:14.000Z","updated_at":"2026-03-16T14:18:09.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/edujbarrios/awesome-vision-ai-stack","commit_stats":null,"previous_names":["edujbarrios/awesome-vision-ai-stack"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/edujbarrios/awesome-vision-ai-stack","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/edujbarrios%2Fawesome-vision-ai-stack","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/edujbarrios%2Fawesome-vision-ai-stack/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/edujbarrios%2Fawesome-vision-ai-stack/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/edujbarrios%2Fawesome-vision-ai-stack/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/edujbarrios","download_url":"https://codeload.github.com/edujbarrios/awesome-vision-ai-stack/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/edujbarrios%2Fawesome-vision-ai-stack/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":341189360,"owners_count":37390576,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-08-22T15:14:58.755Z","status":"online","status_checked_at":"2026-09-17T02:00:07.197Z","response_time":111,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"created_at":"2026-04-10T10:33:17.331Z","updated_at":"2026-10-07T07:00:19.202Z","primary_language":"Python","list_of_lists":false,"displayable":true,"categories":["Local Inference and Serving","Datasets","Foundation VLMs","Benchmarks and Evaluation","Agents, Grounding, and Robotics","Document AI, OCR, and Chart Understanding","Training and Fine-Tuning","Learning Resources","Applications and Demos","UI Understanding and Computer Use","Video and Long-Context Multimodality"],"sub_categories":["Run in browser (zero-setup inference)","Image-text pretraining","Research landmarks","General VLM evaluation","General-purpose open models","Run locally","Grounding and detection data","Instruction and conversational multimodal data","Robotics-oriented projects","Models and tools","Libraries","Repositories and hubs","Serve at scale","Useful open projects","APIs and interfaces","Grounding and perception","Projects and references","Models and systems","Document and chart data"],"readme":"# Awesome Vision AI Stack [![Awesome](https://awesome.re/badge.svg)](https://awesome.re)\n\n\u003e A curated, builder-first list of **Vision Language Models (VLMs)**, local runtimes, document AI tools, UI agents, robotics vision stacks, datasets, benchmarks, and production resources.\n\u003e\n\u003e The goal is not to collect every paper. The goal is to help developers **build with vision**.\n\n## Why this list exists\n\nMost multimodal lists are strong on papers but weak on execution.\n\nThis one is optimized for builders who want to answer questions like:\n\n- Which VLMs can I run locally?\n- Which models are good for documents, charts, OCR, and screenshots?\n- What should I use for UI agents and computer-use workflows?\n- Which stacks help with robotics, grounding, and action?\n- How do I benchmark, fine-tune, and ship a multimodal product?\n\n## Contents\n\n- [Foundation VLMs](#foundation-vlms)\n- [Local Inference and Serving](#local-inference-and-serving)\n- [Document AI, OCR, and Chart Understanding](#document-ai-ocr-and-chart-understanding)\n- [UI Understanding and Computer Use](#ui-understanding-and-computer-use)\n- [Agents, Grounding, and Robotics](#agents-grounding-and-robotics)\n- [Video and Long-Context Multimodality](#video-and-long-context-multimodality)\n- [Training and Fine-Tuning](#training-and-fine-tuning)\n- [Benchmarks and Evaluation](#benchmarks-and-evaluation)\n- [Datasets](#datasets)\n- [Applications and Demos](#applications-and-demos)\n- [Learning Resources](#learning-resources)\n- [Contributing](#contributing)\n\n---\n\n## Foundation VLMs\n\n### General-purpose open models\n- [LLaVA](https://github.com/haotian-liu/LLaVA) - One of the most influential open visual instruction-tuned models.\n- [Qwen2-VL](https://github.com/QwenLM/Qwen2-VL) - Strong open multimodal family for image, document, and video understanding.\n- [Qwen-VL](https://github.com/QwenLM/Qwen-VL) - Earlier Qwen multimodal line with broad ecosystem support.\n- [InternVL](https://github.com/OpenGVLab/InternVL) - Strong family of open large vision-language models.\n- [CogVLM](https://github.com/THUDM/CogVLM) - Open visual language model family from THUDM.\n- [MiniGPT-4](https://github.com/Vision-CAIR/MiniGPT-4) - Early and influential image-chat system.\n- [InstructBLIP](https://github.com/salesforce/LAVIS/tree/main/projects/instructblip) - Instruction-tuned extension of BLIP-style architectures.\n- [BLIP-2](https://github.com/salesforce/LAVIS/tree/main/projects/blip2) - Efficient VLM architecture connecting frozen vision and language models.\n- [IDEFICS](https://github.com/huggingface/smollm/tree/main/vision) - Hugging Face open multimodal family.\n- [DeepSeek-VL](https://github.com/deepseek-ai/DeepSeek-VL) - Open multimodal reasoning models from DeepSeek.\n- [Molmo](https://github.com/allenai/molmo) - Open multimodal assistant from Ai2 with strong grounding focus.\n- [Phi-3 Vision](https://huggingface.co/microsoft/Phi-3-vision-128k-instruct) - Compact multimodal model useful for practical deployments.\n- [Fuyu](https://huggingface.co/adept/fuyu-8b) - Multimodal autoregressive model with a distinct design.\n- [Gemma Vision](https://huggingface.co/google/gemma-2-vision-9b) - Google's open multimodal model; efficient vision-language understanding with strong performance on document and image reasoning.\n- [SmolVLM](https://huggingface.co/collections/HuggingFaceTB/smolvlm-67531d47c514f6f87c3f448f) - Small multimodal models for lightweight use cases.\n- [Moondream](https://github.com/vikhyat/moondream) - Lightweight open-source VLM optimized for efficiency and local deployment.\n- [MedGEMMA](https://huggingface.co/google/medgemma) - Medical-focused vision language model from Google for healthcare applications.\n\n### Research landmarks\n- [Flamingo](https://arxiv.org/abs/2204.14198) - Landmark few-shot visual language model.\n- [Kosmos-1](https://arxiv.org/abs/2302.14045) - Early multimodal reasoning and grounding work.\n- [PaLI](https://arxiv.org/abs/2209.06794) - Scalable multilingual vision-language model.\n- [PaLI-X](https://arxiv.org/abs/2305.18565) - Larger multimodal extension of PaLI.\n- [Kosmos-2](https://arxiv.org/abs/2306.14824) - Grounded multimodal large language model.\n- [SEED-Bench ecosystem](https://github.com/AILab-CVC/SEED-Bench) - Useful benchmark family around multimodal reasoning.\n\n### Models people actually test in products\n- LLaVA-style models for rapid prototyping\n- Qwen2-VL family for stronger general multimodal performance\n- InternVL for competitive open performance\n- Phi-3 Vision and SmolVLM for lighter deployments\n- Molmo for grounding-heavy exploration\n\n---\n\n## Local Inference and Serving\n\n### Run locally\n- [Ollama](https://github.com/ollama/ollama) - Local model runtime with growing multimodal support.\n- [LM Studio](https://lmstudio.ai/) - Desktop app for running local models with a friendly UI.\n- [Jan](https://github.com/menloresearch/jan) - Open local AI runtime and desktop app.\n- [llama.cpp](https://github.com/ggerganov/llama.cpp) - Core local inference stack; important for lightweight experimentation.\n- [MLC-LLM](https://github.com/mlc-ai/mlc-llm) - Compile and deploy models on edge and mobile devices.\n- [OpenVINO](https://github.com/openvinotoolkit/openvino) - Useful for Intel-optimized deployments.\n\n### Run in browser (zero-setup inference)\n- [WebLLM](https://github.com/mlc-ai/web-llm) - Run VLMs natively in the browser without backend server.\n- [Transformers.js](https://github.com/xenova/transformers.js) - Hugging Face models (vision and audio) in JavaScript; enables client-side multimodal inference.\n- [ONNX Runtime Web](https://github.com/microsoft/onnxruntime) - Cross-platform model inference in browser; standardized format.\n- [TensorFlow.js](https://www.tensorflow.org/js) - TensorFlow models in browser and Node.js; useful for vision tasks and edge deployment.\n\n### Serve at scale\n- [vLLM](https://github.com/vllm-project/vllm) - High-throughput inference engine increasingly relevant for multimodal serving.\n- [SGLang](https://github.com/sgl-project/sglang) - Fast serving and structured generation framework.\n- [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM) - NVIDIA-optimized inference stack.\n- [TGI](https://github.com/huggingface/text-generation-inference) - Hugging Face serving stack.\n- [BentoML](https://github.com/bentoml/BentoML) - Production model serving and packaging.\n- [Ray Serve](https://github.com/ray-project/ray) - Scalable service orchestration for model workloads.\n\n### APIs and interfaces\n- [Open WebUI](https://github.com/open-webui/open-webui) - Popular self-hosted chat UI for local models.\n- [Lobe Chat](https://github.com/lobehub/lobe-chat) - Polished interface for model backends.\n- [LibreChat](https://github.com/danny-avila/LibreChat) - Open chat UI with multi-backend support.\n- [Flowise](https://github.com/FlowiseAI/Flowise) - Visual builder for LLM and multimodal pipelines.\n\n---\n\n## Document AI, OCR, and Chart Understanding\n\nThis is one of the biggest practical VLM opportunities.\n\n### Models and tools\n- [PaddleOCR](https://github.com/PaddlePaddle/PaddleOCR) - Strong OCR toolkit and a common baseline for document pipelines.\n- [docTR](https://github.com/mindee/doctr) - OCR for document text detection and recognition.\n- [Nougat](https://github.com/facebookresearch/nougat) - OCR-style document understanding for scientific PDFs.\n- [Donut](https://github.com/clovaai/donut) - OCR-free document understanding model.\n- [Pix2Struct](https://github.com/google-research/pix2struct) - Document understanding without OCR; excellent for structured layouts, charts, and complex documents.\n- [LayoutLM](https://github.com/microsoft/unilm/tree/master/layoutlm) - Important family for document layout understanding.\n- [DocLayout-YOLO](https://github.com/opendatalab/DocLayout-YOLO) - Modern layout detection and segmentation for complex documents.\n- [MinerU](https://github.com/opendatalab/MinerU) - Open document parsing and PDF extraction tooling.\n- [Marker](https://github.com/VikParuchuri/marker) - PDF-to-markdown/document extraction workflow.\n- [Surya](https://github.com/VikParuchuri/surya) - OCR and layout toolkit.\n- [PyMuPDF](https://pymupdf.readthedocs.io/) - Robust PDF handling and preprocessing; critical for document pipelines.\n- [ChartOCR](https://github.com/zmykevin/ChartOCR) - Chart understanding reference.\n- [ChartQA](https://github.com/vis-nlp/ChartQA) - Dataset and benchmark for chart reasoning.\n\n### Practical stacks\n- OCR + VLM reranking for robust document QA\n- PDF-to-markdown + VLM summarization\n- screenshot parsing for SaaS analytics and support tooling\n- chart extraction + VLM reasoning for BI workflows\n\n---\n\n## UI Understanding and Computer Use\n\nAnother high-value gap: builders want models that can read screens and act.\n\n### Projects and references\n- [Anthropic Computer Use](https://www.anthropic.com/news/introducing-computer-use) - Direct VLM-to-action API for web automation; Claude 3.5 Sonnet integration shipped late 2024.\n- [SeeAct](https://github.com/OSU-NLP-Group/SeeAct) - Visual web agent framework.\n- [OpenHands](https://github.com/All-Hands-AI/OpenHands) - Software agent platform; relevant for multimodal and browser-use workflows.\n- [Browser Use](https://github.com/browser-use/browser-use) - Browser automation with model control.\n- [Stagehand](https://github.com/browserbase/stagehand) - Browser automation framework aimed at AI-native workflows.\n- [OmniParser](https://github.com/microsoft/OmniParser) - Screen parsing for GUI grounding and action planning.\n- [UI-TARS](https://github.com/bytedance/UI-TARS) - UI-centric agent/model direction.\n- [GroundingDINO](https://github.com/IDEA-Research/GroundingDINO) - Key building block for screen and visual grounding.\n- [SAM 2](https://github.com/facebookresearch/sam2) - Segmentation backbone useful for visual agents and annotation loops.\n- [ScreenQA](https://github.com/niansuh/ScreenQA) - Benchmark for evaluating screenshot understanding and UI navigation.\n\n### Strong use cases\n- support copilots that understand screenshots\n- QA agents for web apps\n- workflow automation over dashboards and back-office tools\n- computer-use agents with grounding and action planning\n\n---\n\n## Agents, Grounding, and Robotics\n\n### Grounding and perception\n- [CLIP \u0026 OpenCLIP](https://github.com/openai/CLIP) - Foundation models for zero-shot detection, retrieval, and visual embeddings; critical for flexible grounding and cross-modal search.\n- [GroundingDINO](https://github.com/IDEA-Research/GroundingDINO) - Open-set detection from text prompts.\n- [YOLO-World](https://github.com/AILab-CVC/YOLO-World) - Open-vocabulary detection combining YOLO efficiency with open-set capabilities.\n- [Segment Anything](https://github.com/facebookresearch/segment-anything) - Foundational segmentation model.\n- [SAM 2](https://github.com/facebookresearch/sam2) - Video-capable segmentation and tracking direction.\n- [OWL-ViT](https://github.com/google-research/scenic/tree/main/scenic/projects/owl_vit) - Open-vocabulary detection.\n- [Detic](https://github.com/facebookresearch/Detic) - Detection with image-level supervision and open-vocabulary flavor.\n- [Florence-2](https://github.com/microsoft/Florence) - Dense prediction and structured understanding for documents, captions, and visual grounding.\n- [Depth-Anything](https://github.com/DepthAnything/Depth-Anything) - Monocular depth estimation; essential for 3D understanding, robotics, and spatial reasoning.\n\n### Robotics-oriented projects\n- [OpenVLA](https://github.com/openvla/openvla) - Vision-language-action direction for robotics.\n- [LeRobot](https://github.com/huggingface/lerobot) - Hugging Face robotics stack.\n- [Open X-Embodiment](https://github.com/google-deepmind/open_x_embodiment) - Cross-robot learning ecosystem.\n- [RT-1](https://robotics-transformer1.github.io/) - Robotics transformer reference.\n- [RT-2](https://robotics-transformer2.github.io/) - Vision-language-action robotics reference.\n\n### Why this section matters\nMany “awesome VLM” repos ignore the path from seeing to acting.\nThis is where vision goes from demo to system.\n\n---\n\n## Video and Long-Context Multimodality\n\n### Models and systems\n- [Video-LLaVA](https://github.com/PKU-YuanGroup/Video-LLaVA) - Video extension of LLaVA-style instruction tuning.\n- [VideoChat2](https://github.com/OpenGVLab/Ask-Anything) - Video multimodal conversation direction.\n- [LLaVA-NeXT-Video](https://github.com/LLaVA-VL/LLaVA-NeXT) - Video-capable branch of the LLaVA family.\n- [Qwen2-VL](https://github.com/QwenLM/Qwen2-VL) - Practical option for image and video understanding.\n- [LongVU](https://github.com/Vision-CAIR/LongVU) - Long video understanding direction.\n- [VideoMAE](https://github.com/MCG-NJU/VideoMAE) - Self-supervised masked autoencoder for video; strong backbone for video understanding and retrieval.\n- [Gemini 2.0 Video](https://deepmind.google/gemini/) - State-of-the-art multimodal video reasoning with native temporal understanding.\n\n### Product opportunities\n- meeting and call analysis\n- industrial inspection\n- sports video understanding\n- security review\n- agent memory over recorded workflows\n\n---\n\n## Training and Fine-Tuning\n\n### Libraries\n- [Transformers](https://github.com/huggingface/transformers) - The default ecosystem for many multimodal models.\n- [TRL](https://github.com/huggingface/trl) - Preference optimization and instruction-tuning workflows.\n- [PEFT](https://github.com/huggingface/peft) - Parameter-efficient fine-tuning.\n- [Axolotl](https://github.com/axolotl-ai-cloud/axolotl) - Popular fine-tuning framework.\n- [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory) - Large fine-tuning platform with multimodal support.\n- [DeepSpeed](https://github.com/microsoft/DeepSpeed) - Distributed training and optimization.\n- [PyTorch Lightning](https://github.com/Lightning-AI/pytorch-lightning) - Structured model training workflows.\n- [OpenFlamingo](https://github.com/mlfoundations/open_flamingo) - Open framework for Flamingo-style multimodal modeling.\n- [LAVIS](https://github.com/salesforce/LAVIS) - Vision-language research and training toolkit.\n\n### Common tuning patterns\n- LoRA on the language model head\n- projector tuning between vision encoder and language model\n- instruction tuning on screenshot or document tasks\n- synthetic data generation for niche workflows\n- DPO or preference tuning for agent outputs\n\n---\n\n## Benchmarks and Evaluation\n\n### General VLM evaluation\n- [MMMU](https://mmmu-benchmark.github.io/) - Massive multitask multimodal reasoning benchmark.\n- [MMBench](https://github.com/open-compass/MMBench) - Broad multimodal benchmark.\n- [MM-Vet](https://github.com/yuweihao/MM-Vet) - Challenging evaluation for integrated multimodal capabilities.\n- [SEED-Bench](https://github.com/AILab-CVC/SEED-Bench) - Foundation multimodal benchmark suite.\n- [POPE](https://github.com/RUCAIBox/POPE) - Evaluates object hallucination.\n- [HallusionBench](https://github.com/tianyi-lab/HallusionBench) - Hallucination-focused benchmark.\n- [ScienceQA](https://github.com/lupantech/ScienceQA) - Science reasoning benchmark with multimodal settings.\n- [TextVQA](https://textvqa.org/) - Text-centric visual QA.\n- [VQAv2](https://visualqa.org/) - Classic visual question answering benchmark.\n- [ChartQA](https://github.com/vis-nlp/ChartQA) - Chart reasoning benchmark.\n- [DocVQA](https://www.docvqa.org/) - Document visual question answering benchmark.\n\n### What to evaluate in real products\n- hallucination on screenshots and documents\n- OCR robustness on messy inputs\n- grounding consistency\n- latency and memory under real batch sizes\n- action reliability for UI agents\n- failure modes on out-of-domain images\n\n---\n\n## Datasets\n\n### Image-text pretraining\n- [LAION-5B](https://laion.ai/blog/laion-5b/) - Large-scale image-text resource.\n- [Conceptual Captions](https://ai.google.com/research/ConceptualCaptions/) - Image-caption dataset.\n- [COCO Captions](https://cocodataset.org/) - Standard image captioning dataset.\n- [Visual Genome](https://homes.cs.washington.edu/~ranjay/visualgenome/index.html) - Dense visual annotations and relationships.\n- [RedCaps](https://github.com/redcaps-dataset/redcaps) - Large image-text dataset from Reddit.\n- [DataComp](https://www.datacomp.ai/) - Dataset curation benchmark ecosystem.\n\n### Instruction and conversational multimodal data\n- [LLaVA-Instruct-150K](https://huggingface.co/datasets/liuhaotian/LLaVA-Instruct-150K) - Foundational multimodal instruction dataset.\n- [ShareGPT4V](https://huggingface.co/datasets/Lin-Chen/ShareGPT4V) - Large multimodal instruction-style dataset.\n- [M3IT](https://github.com/MMInstruction/M3IT) - Multimodal instruction tuning data collection.\n\n### Document and chart data\n- [DocVQA](https://www.docvqa.org/)\n- [InfographicVQA](https://www.docvqa.org/datasets/infographicvqa)\n- [ChartQA](https://github.com/vis-nlp/ChartQA)\n- [PubLayNet](https://github.com/ibm-aur-nlp/PubLayNet) - Document layout annotations.\n- [FUNSD](https://guillaumejaume.github.io/FUNSD/) - Form understanding benchmark.\n\n### Grounding and detection data\n- [RefCOCO / RefCOCO+ / RefCOCOg](https://github.com/lichengunc/refer) - Referring expression benchmarks.\n- [Objects365](https://www.objects365.org/) - Detection dataset.\n- [Open Images](https://storage.googleapis.com/openimages/web/index.html) - Large detection and localization dataset.\n\n---\n\n## Applications and Demos\n\n### Great categories to build in\n- screenshot QA\n- browser automation copilots\n- invoice and contract extraction\n- multimodal customer support\n- chart analysis copilots\n- accessibility tools\n- robotics perception and action\n- multimodal RAG for images, PDFs, and video\n\n### Useful open projects\n- [ComfyUI](https://github.com/comfyanonymous/ComfyUI) - Important ecosystem for visual model workflows.\n- [InvokeAI](https://github.com/invoke-ai/InvokeAI) - Image generation stack, useful alongside multimodal systems.\n- [Label Studio](https://github.com/HumanSignal/label-studio) - Data labeling for multimodal training loops.\n- [FiftyOne](https://github.com/voxel51/fiftyone) - Dataset inspection and vision evaluation.\n- [Unstructured](https://github.com/Unstructured-IO/unstructured) - Document preprocessing for multimodal pipelines.\n- [Haystack](https://github.com/deepset-ai/haystack) - RAG framework adaptable to multimodal retrieval.\n- [LlamaIndex](https://github.com/run-llama/llama_index) - Retrieval and agents, with multimodal extensions.\n- [LangChain](https://github.com/langchain-ai/langchain) - Orchestration stack with multimodal integrations.\n\n---\n\n## Learning Resources\n\n### Repositories and hubs\n- [Papers with Code - Vision Language](https://paperswithcode.com/task/visual-question-answering)\n- [Hugging Face Multimodal Tasks](https://huggingface.co/tasks)\n- [OpenCompass](https://github.com/open-compass/opencompass) - Evaluation ecosystem.\n- [LAVIS](https://github.com/salesforce/LAVIS) - Great for studying model families.\n\n### Topics worth learning deeply\n- projector-based multimodal alignment\n- OCR-free vs OCR-plus-VLM document stacks\n- visual grounding for action\n- hallucination detection in VLMs\n- long-context video reasoning\n- multimodal retrieval and indexing\n\n---\n\n## Opinionated Starter Paths\n\n### I want to build a local screenshot copilot\nStart with:\n- Qwen2-VL or LLaVA-style model\n- Ollama or vLLM\n- OmniParser or GroundingDINO for UI structure\n- Playwright, Browser Use, or Stagehand for actions\n\n### I want to build document AI\nStart with:\n- MinerU, Marker, Surya, PaddleOCR, or docTR\n- Donut or a strong general VLM for semantic reasoning\n- multimodal RAG over pages, tables, and extracted markdown\n\n### I want to explore robotics\nStart with:\n- OpenVLA\n- LeRobot\n- GroundingDINO + SAM 2\n- RT-1 and RT-2 papers for system design\n\n### I want to benchmark models before choosing one\nStart with:\n- MMMU\n- MMBench\n- MM-Vet\n- HallusionBench\n- task-specific evals on your own screenshots or PDFs\n\n---\n\n## What makes a resource a good fit for this repo?\n\nWe prioritize resources that are:\n- useful for builders, not only for citation graphs\n- open or accessible enough to try\n- relevant to local deployment, evaluation, or productization\n- focused on image, document, video, UI, or grounded multimodal workflows\n\nWe do **not** aim to be a giant paper dump.\n\n---\n\n## Contributing\n\nPlease read [CONTRIBUTING.md](CONTRIBUTING.md).\n\nGood additions include:\n- practical VLM repos\n- local runtimes with multimodal support\n- document AI tooling\n- UI agent frameworks\n- evaluation suites\n- strong tutorials and implementation guides\n\n---\n\n## License\n\n[MIT](LICENSE)\n","projects_url":"https://awesome.ecosyste.ms/api/v1/lists/edujbarrios%2Fawesome-vision-ai-stack/projects"}