{"id":34045355,"url":"https://github.com/vlm-run/vlmrun-hub","last_synced_at":"2026-04-02T01:43:04.855Z","repository":{"id":271914040,"uuid":"886899801","full_name":"vlm-run/vlmrun-hub","owner":"vlm-run","description":"A hub for various industry-specific schemas to be used with VLMs.","archived":false,"fork":false,"pushed_at":"2025-12-15T22:41:40.000Z","size":376,"stargazers_count":537,"open_issues_count":8,"forks_count":23,"subscribers_count":2,"default_branch":"main","last_synced_at":"2026-01-02T10:47:40.855Z","etag":null,"topics":["ai","computer-vision","etl","genai","json","multimodal","pydantic","pydantic-models","vlm","vlm-ocr"],"latest_commit_sha":null,"homepage":"https://docs.vlm.run/hub","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vlm-run.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"docs/CONTRIBUTING-SCHEMA.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-11-11T20:07:48.000Z","updated_at":"2025-12-27T13:45:59.000Z","dependencies_parsed_at":"2025-01-10T18:31:05.075Z","dependency_job_id":"f0f9f816-9a13-4f61-9f66-755c77ba4825","html_url":"https://github.com/vlm-run/vlmrun-hub","commit_stats":null,"previous_names":["vlm-run/vlmrun-hub"],"tags_count":31,"template":false,"template_full_name":null,"purl":"pkg:github/vlm-run/vlmrun-hub","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vlm-run%2Fvlmrun-hub","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vlm-run%2Fvlmrun-hub/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vlm-run%2Fvlmrun-hub/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vlm-run%2Fvlmrun-hub/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vlm-run","download_url":"https://codeload.github.com/vlm-run/vlmrun-hub/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vlm-run%2Fvlmrun-hub/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31294100,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-02T01:05:07.454Z","status":"ssl_error","status_checked_at":"2026-04-02T00:56:46.496Z","response_time":53,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","computer-vision","etl","genai","json","multimodal","pydantic","pydantic-models","vlm","vlm-ocr"],"created_at":"2025-12-13T23:06:41.088Z","updated_at":"2026-04-02T01:43:04.846Z","avatar_url":"https://github.com/vlm-run.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\u003cp align=\"center\" style=\"width: 100%;\"\u003e\n    \u003cimg src=\"https://raw.githubusercontent.com/vlm-run/.github/refs/heads/main/profile/assets/vlm-black.svg\" alt=\"VLM Run Logo\" width=\"80\" style=\"margin-bottom: -5px; color: #2e3138; vertical-align: middle; padding-right: 5px;\"\u003e\u003cbr\u003e\n\u003c/p\u003e\n\u003ch2\u003eVLM Run Hub\u003c/h2\u003e\n\u003cp align=\"center\"\u003e\n\u003ca href=\"https://vlm.run\"\u003e\u003cb\u003eWebsite\u003c/b\u003e\u003c/a\u003e |  \u003ca href=\"https://app.vlm.run/\"\u003e\u003cb\u003ePlatform\u003c/b\u003e\u003c/a\u003e | \u003ca href=\"https://docs.vlm.run/\"\u003e\u003cb\u003eDocs\u003c/b\u003e\u003c/a\u003e | \u003ca href=\"https://vlm.run/blog\"\u003e\u003cb\u003eBlog\u003c/b\u003e\u003c/a\u003e | \u003ca href=\"https://discord.gg/4jgyECY4rq\"\u003e\u003cb\u003eDiscord\u003c/b\u003e\u003c/a\u003e | \u003ca href=\"vlmrun/hub/catalog.yaml\"\u003e\u003cb\u003eCatalog\u003c/b\u003e\u003c/a\u003e\n\u003c/p\u003e\n\u003cp align=\"center\"\u003e\n\u003ca href=\"https://pypi.org/project/vlmrun-hub/\"\u003e\u003cimg alt=\"PyPI Version\" src=\"https://badge.fury.io/py/vlmrun-hub.svg\"\u003e\u003c/a\u003e\n\u003ca href=\"https://pypi.org/project/vlmrun-hub/\"\u003e\u003cimg alt=\"PyPI Version\" src=\"https://img.shields.io/pypi/pyversions/vlmrun-hub\"\u003e\u003c/a\u003e\n\u003ca href=\"https://www.pepy.tech/projects/vlmrun-hub\"\u003e\u003cimg alt=\"PyPI Downloads\" src=\"https://img.shields.io/pypi/dm/vlmrun-hub\"\u003e\u003c/a\u003e\u003cbr\u003e\n\u003ca href=\"https://github.com/vlm-run/vlmrun-hub/blob/main/LICENSE\"\u003e\u003cimg alt=\"PyPi Downloads\" src=\"https://img.shields.io/github/license/vlm-run/hub.svg\"\u003e\u003c/a\u003e\n\u003ca href=\"https://discord.gg/4jgyECY4rq\"\u003e\u003cimg alt=\"Discord\" src=\"https://img.shields.io/badge/discord-chat-purple?color=%235765F2\u0026label=discord\u0026logo=discord\"\u003e\u003c/a\u003e\n\u003ca href=\"https://twitter.com/vlmrun\"\u003e\u003cimg alt=\"PyPi Version\" src=\"https://img.shields.io/twitter/follow/vlmrun.svg?style=social\u0026logo=twitter\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\u003cbr\u003e\n\u003c/div\u003e\n\nWelcome to **VLM Run Hub**, a comprehensive repository of pre-defined [Pydantic](https://docs.pydantic.dev/latest/) schemas for extracting structured data from unstructured visual domains such as images, videos, and documents. Designed for [Vision Language Models (VLMs)](https://huggingface.co/blog/vlms) and optimized for real-world use cases, VLM Run Hub simplifies the integration of visual ETL into your workflows.\n\n\u003ctable\u003e\n\u003ctr\u003e\n\u003ctd\u003e \u003cb\u003eImage\u003c/b\u003e \u003c/td\u003e\n\u003ctd\u003e \u003cb\u003eJSON\u003c/b\u003e \u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd style=\"width: 40%;\"\u003e\n\u003cimg src=\"https://storage.googleapis.com/vlm-data-public-prod/hub/examples/document.us-drivers-license/dl3.jpg\"\u003e\n\u003c/td\u003e\n\n\u003ctd\u003e\n\n```json\n{\n  \"issuing_state\": \"MT\",\n  \"license_number\": \"0812319684104\",\n  \"first_name\": \"Brenda\",\n  \"middle_name\": \"Lynn\",\n  \"last_name\": \"Sample\",\n  \"address\": {\n    \"street\": \"123 MAIN STREET\",\n    \"city\": \"HELENA\",\n    \"state\": \"MT\",\n    \"zip_code\": \"59601\"\n  },\n  \"date_of_birth\": \"1968-08-04\",\n  \"gender\": \"F\",\n  \"height\": \"5'06\\\"\",\n  \"weight\": 150.0,\n  \"eye_color\": \"BRO\",\n  \"issue_date\": \"2015-02-15\",\n  \"expiration_date\": \"2023-08-04\",\n  \"license_class\": \"D\"\n}\n```\n\n\u003c/td\u003e\n\n\u003c/tr\u003e\n\u003c/table\u003e\n\u003c/details\u003e\n\n### 💡 Motivation\n\nWhile vision models like OpenAI's [GPT-4o](https://openai.com/index/hello-gpt-4o/) and Anthropic's [Claude Vision](https://www.anthropic.com/claude) excel in exploratory tasks like \"chat with images,\" they often lack practicality for automation and integration, where **strongly-typed**, **validated outputs** are crucial.\n\nThe **Structured Outputs API** (popularized by [GPT-4o](https://openai.com/index/introducing-structured-outputs-in-the-api/), [Gemini](https://ai.google.dev/gemini-api/docs/structured-output)) addresses this by constraining LLMs to return data in precise, strongly-typed formats such as [Pydantic](https://docs.pydantic.dev/latest/) models. This eliminates complex parsing and validation, ensuring outputs conform to expected types and structures. These schemas can be nested and include complex types like lists and dictionaries, enabling seamless integration with existing systems while leveraging the full capabilities of the model.\n\n### 🧰 Why use this hub of pre-defined Pydantic schemas?\n\n- 📚 **Easy to use:** [Pydantic](https://docs.pydantic.dev/latest/) is a well-understood and battle-tested data model for structured data.\n- 🔋 **Batteries included:** Each schema in this repo has been validated across real-world industry use cases—from healthcare to finance to media—saving you weeks of development effort.\n- 🔍 **Automatic Data-validation:** Built-in [Pydantic validation](https://docs.pydantic.dev/latest/concepts/validators/) ensures your extracted data is clean, accurate, and reliable, reducing errors and simplifying downstream workflows.\n- 🔌 **Type-safety:** With [Pydantic's type-safety](https://docs.pydantic.dev/latest/concepts/types/) and compatibility with tools like `mypy` and `pyright`, you can build composable, modular systems that are robust and maintainable.\n- 🧰 **Model-agnostic:** Use the same schema with multiple VLM providers, no need to rewrite prompts for different VLMs.\n- 🚀 **Optimized for Visual ETL:** Purpose-built for extracting structured data from images, videos, and documents, this repo bridges the gap between unstructured data and actionable insights.\n\n### 📖 Schema Catalog\n\nThe VLM Run Hub maintains a comprehensive catalog of all available schemas in the [`vlmrun/hub/catalog.yaml`](vlmrun/hub/catalog.yaml) file. The catalog is automatically validated to ensure consistency and completeness of schema documentation. We refer the developer to the [catalog-spec.yaml](docs/catalog-spec.yaml) for the full YAML specification.\n\n\u003ctable\u003e\n  \u003ctr\u003e\n    \u003cth\u003eCategory\u003c/th\u003e\n    \u003cth colspan=\"3\"\u003eDomains\u003c/th\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd rowspan=\"5\"\u003e\u003cstrong\u003e📄 Document Processing\u003c/strong\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"vlmrun/hub/schemas/document/bank_statement.py\"\u003edocument.bank-statement\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"vlmrun/hub/schemas/document/invoice.py\"\u003edocument.invoice\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"vlmrun/hub/schemas/document/receipt.py\"\u003edocument.receipt\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003e\u003ca href=\"vlmrun/hub/schemas/document/resume.py\"\u003edocument.resume\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"vlmrun/hub/schemas/document/us_drivers_license.py\"\u003edocument.us-drivers-license\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"vlmrun/hub/schemas/document/utility_bill.py\"\u003edocument.utility-bill\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003e\u003ca href=\"vlmrun/hub/schemas/contrib/document/us_passport.py\"\u003edocument.us-passport\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"vlmrun/hub/schemas/contrib/document/business_card.py\"\u003edocument.business-card\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"vlmrun/hub/schemas/contrib/document/insurance_claim.py\"\u003edocument.insurance-claim\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003e\u003ca href=\"vlmrun/hub/schemas/contrib/document/bank_check.py\"\u003edocument.bank-check\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"vlmrun/hub/schemas/contrib/document/request_for_proposal.py\"\u003edocument.request-for-proposal\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"vlmrun/hub/schemas/contrib/document/india/aadhaar_card.py\"\u003edocument.india.aadhaar-card\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003e\u003ca href=\"vlmrun/hub/schemas/contrib/document/india/pan_card.py\"\u003edocument.india.pan-card\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd rowspan=\"1\"\u003e\u003cstrong\u003e💰 Accounting \u0026 Finance\u003c/strong\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"vlmrun/hub/schemas/accounting/w2_form.py\"\u003eaccounting.form-w2\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"vlmrun/hub/schemas/contrib/accounting/form_payslip.py\"\u003eaccounting.form-payslip\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"vlmrun/hub/schemas/contrib/finance/balance_sheet.py\"\u003efinance.balance-sheet\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd rowspan=\"1\"\u003e\u003cstrong\u003e🏥 Healthcare\u003c/strong\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"vlmrun/hub/schemas/healthcare/medical_insurance_card.py\"\u003ehealthcare.medical-insurance-card\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"vlmrun/hub/schemas/healthcare/hipaa_release.py\"\u003ehealthcare.hipaa-release\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"vlmrun/hub/schemas/contrib/healthcare/pathology_report.py\"\u003ehealthcare.pathology-report\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd rowspan=\"1\"\u003e\u003cstrong\u003e🛒 Retail\u003c/strong\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"vlmrun/hub/schemas/retail/ecommerce_product_caption.py\"\u003eretail.ecommerce-product-caption\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"vlmrun/hub/schemas/retail/product_catalog.py\"\u003eretail.product-catalog\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"vlmrun/hub/schemas/contrib/food/nutrition_facts_label.py\"\u003efood.nutrition-facts-label\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd rowspan=\"1\"\u003e\u003cstrong\u003e📺 Media\u003c/strong\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"vlmrun/hub/schemas/media/tv_news.py\"\u003emedia.tv-news\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"vlmrun/hub/schemas/contrib/media/nba_game_state.py\"\u003emedia.nba-game-state\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"vlmrun/hub/schemas/contrib/media/nfl_game_state.py\"\u003emedia.nfl-game-state\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd rowspan=\"2\"\u003e\u003cstrong\u003e🏭 Other Industries\u003c/strong\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"vlmrun/hub/schemas/aerospace/remote_sensing.py\"\u003eaerospace.remote-sensing\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"vlmrun/hub/schemas/contrib/logistics/bill_of_lading.py\"\u003elogistics.bill-of-lading\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003ca href=\"vlmrun/hub/schemas/contrib/real_estate/lease_agreement.py\"\u003ereal-estate.lease-agreement\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003e\u003ca href=\"vlmrun/hub/schemas/contrib/social/twitter_card.py\"\u003esocial.twitter-card\u003c/a\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\nIf you have a new schema you want to add to the catalog, please refer to the [SCHEMA-GUIDELINES.md](docs/SCHEMA-GUIDELINES.md) for the full guidelines.\n\n### 🚀 Getting Started\n\nLet's say we want to extract invoice metadata from an [invoice image](https://storage.googleapis.com/vlm-data-public-prod/hub/examples/document.invoice/invoice_1.jpg). You can readily use our [`Invoice`](vlmrun/hub/schemas/document/invoice.py) schema we have defined under `vlmrun.hub.schemas.document.invoice` and use it with any VLM of your choosing.\n\nFor a comprehensive walkthrough of available schemas and their usage, check out our [Schema Showcase Notebook](https://github.com/vlm-run/vlmrun-cookbook/blob/main/notebooks/01_schema_showcase.ipynb).\n\n### 💾 Installation\n\n```python\npip install vlmrun-hub\n```\n\n#### With [VLM Run Python SDK](https://github.com/vlm-run/vlmrun-python-sdk)\n\n```python\nimport os\nfrom PIL import Image\nfrom vlmrun.client import VLMRun\nfrom vlmrun.client.types import PredictionResponse\nfrom vlmrun.common.utils import download_image\n\nVLMRUN_BASE_URL = os.getenv(\"VLMRUN_BASE_URL\", \"https://api.vlm.run/v1\")\nVLMRUN_API_KEY = os.getenv(\"VLMRUN_API_KEY\", None)\n\nclient = VLMRun(base_url=VLMRUN_BASE_URL, api_key=VLMRUN_API_KEY)\n\nIMAGE_URL = \"https://storage.googleapis.com/vlm-data-public-prod/hub/examples/document.invoice/invoice_1.jpg\"\nimage: Image.Image = download_image(IMAGE_URL)\n\nresponse: PredictionResponse = client.image.generate(\n    images=[image],\n    domain=\"document.invoice\",\n)\n```\n\n#### With [Instructor](https://github.com/jxnl/instructor) / OpenAI\n\n```python\nimport instructor\nfrom openai import OpenAI\n\nfrom vlmrun.hub.schemas.document.invoice import Invoice\n\nIMAGE_URL = \"https://storage.googleapis.com/vlm-data-public-prod/hub/examples/document.invoice/invoice_1.jpg\"\n\nclient = instructor.from_openai(\n    OpenAI(), mode=instructor.Mode.MD_JSON\n)\nresponse = client.chat.completions.create(\n    model=\"gpt-4o-mini\",\n    messages=[\n        { \"role\": \"user\", \"content\": [\n            {\"type\": \"text\", \"text\": \"Extract the invoice in JSON.\"},\n            {\"type\": \"image_url\", \"image_url\": {\"url\": IMAGE_URL}, \"detail\": \"auto\"}\n        ]}\n    ],\n    response_model=Invoice,\n    temperature=0,\n)\n```\n\n\u003cdetails\u003e\n\u003csummary\u003eJSON Response:\u003c/summary\u003e\n\n\u003ctable\u003e\n\u003ctr\u003e\n\u003ctd style=\"width: 40%;\"\u003e Image \u003c/td\u003e\n\u003ctd\u003e JSON Output 🔐 \u003c/td\u003e\n\u003c/tr\u003e\n\n\u003ctr\u003e\n\u003ctd style=\"width: 40%;\"\u003e\n\u003cimg src=\"https://storage.googleapis.com/vlm-data-public-prod/hub/examples/document.invoice/invoice_1.jpg\"\u003e\n\u003c/td\u003e\n\u003ctd\u003e\n\n```json\n{\n  \"invoice_id\": \"9999999\",\n  \"period_start\": null,\n  \"period_end\": null,\n  \"invoice_issue_date\": \"2023-11-11\",\n  \"invoice_due_date\": null,\n  \"order_id\": null,\n  \"customer_id\": null,\n  \"issuer\": \"Anytown, USA\",\n  \"issuer_address\": {\n    \"street\": \"123 Main Street\",\n    \"city\": \"Anytown\",\n    \"state\": \"USA\",\n    \"postal_code\": \"01234\",\n    \"country\": null\n  },\n  \"customer\": \"Fred Davis\",\n  \"customer_email\": \"email@invoice.com\",\n  \"customer_phone\": \"(800) 123-4567\",\n  \"customer_billing_address\": {\n    \"street\": \"1335 Martin Luther King Jr Ave\",\n    \"city\": \"Dunedin\",\n    \"state\": \"FL\",\n    \"postal_code\": \"34698\",\n    \"country\": null\n  },\n  \"customer_shipping_address\": {\n    \"street\": \"249 Windward Passage\",\n    \"city\": \"Clearwater\",\n    \"state\": \"FL\",\n    \"postal_code\": \"33767\",\n    \"country\": null\n  },\n  \"items\": [\n    {\n      \"description\": \"Service\",\n      \"quantity\": 1,\n      \"currency\": null,\n      \"unit_price\": 200.0,\n      \"total_price\": 200.0\n    },\n    {\n      \"description\": \"Parts AAA\",\n      \"quantity\": 1,\n      \"currency\": null,\n      \"unit_price\": 100.0,\n      \"total_price\": 100.0\n    },\n    {\n      \"description\": \"Parts BBB\",\n      \"quantity\": 2,\n      \"currency\": null,\n      \"unit_price\": 50.0,\n      \"total_price\": 100.0\n    }\n  ],\n  \"subtotal\": 400.0,\n  \"tax\": null,\n  \"total\": 400.0,\n  \"currency\": null,\n  \"notes\": \"\",\n  \"others\": null\n}\n```\n\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n\u003c/details\u003e\n\n#### With [OpenAI Structured Outputs API](https://platform.openai.com/docs/guides/structured-outputs)\n\n```python\nimport instructor\nfrom openai import OpenAI\n\nfrom vlmrun.hub.schemas.document.invoice import Invoice\n\nIMAGE_URL = \"https://storage.googleapis.com/vlm-data-public-prod/hub/examples/document.invoice/invoice_1.jpg\"\n\nclient = OpenAI()\ncompletion = client.beta.chat.completions.parse(\n    model=\"gpt-4o-mini\",\n    messages=[\n        {\"role\": \"user\", \"content\": [\n            {\"type\": \"text\", \"text\": \"Extract the invoice in JSON.\"},\n            {\"type\": \"image_url\", \"image_url\": {\"url\": IMAGE_URL}, \"detail\": \"auto\"}\n        ]},\n    ],\n    response_format=Invoice,\n    temperature=0,\n)\n```\n\n\u003e When working with the OpenAI Structured Outputs API, you need to ensure that the `response_format` is a valid Pydantic model with the [supported types](https://platform.openai.com/docs/guides/structured-outputs#supported-schemas).\n\n#### Locally with [Ollama](https://ollama.com)\n\nNote: For certain `vlmrun.common` utilities, you will need to install our main [Python SDK](https://github.com/vlm-run/vlmrun-python-sdk)\nvia `pip install vlmrun`.\n\n```python\nfrom ollama import chat\n\nfrom vlmrun.common.image import encode_image\nfrom vlmrun.common.utils import remote_image\nfrom vlmrun.hub.schemas.document.invoice import Invoice\n\n\nIMAGE_URL = \"https://storage.googleapis.com/vlm-data-public-prod/hub/examples/document.invoice/invoice_1.jpg\"\n\nimg = remote_image(IMAGE_URL)\nchat_response = chat(\n    model=\"llama3.2-vision:11b\",\n    format=Invoice.model_json_schema(),\n    messages=[\n        {\n            \"role\": \"user\",\n            \"content\": \"Extract the invoice in JSON.\",\n            \"images\": [encode_image(img, format=\"JPEG\").split(\",\")[1]],\n        },\n    ],\n    options={\n        \"temperature\": 0\n    },\n)\nresponse = Invoice.model_validate_json(\n    chat_response.message.content\n)\n```\n\n### 📖 Qualitative Results\n\nWe periodically run popular VLMs on each of the examples \u0026 schemas in the [catalog.yaml](vlmrun/hub/catalog.yaml) file and publish the results in the [benchmarks](tests/benchmarks/) directory.\n| Provider | Model | Date | Results |\n| --- | --- | --- | --- |\n| OpenAI | gpt-4o-2024-11-20 | 2025-01-09 | [link](tests/benchmarks/2025-01-09-gpt-4o-2024-11-20-instructor-results.md) |\n| OpenAI | gpt-4o-mini-2024-07-18 | 2025-01-09 | [link](tests/benchmarks/2025-01-09-gpt-4o-mini-2024-07-18-instructor-results.md) |\n| Gemini | gemini-2.0-flash-exp | 2025-01-10 | [link](tests/benchmarks/2025-01-10-gemini-2.0-flash-exp-instructor-results.md) |\n| Ollama | llama3.2-vision:11b | 2025-01-10 | [link](tests/benchmarks/2025-01-10-llama3.2-vision-11b-instructor-results.md) |\n| Ollama | Qwen2.5-VL-7B-Instruct:Q4_K_M_benxh | 2025-02-20 | [link](tests/benchmarks/2025-02-20-bsahane-Qwen2.5-VL-7B-Instruct-Q4_K_M_benxh-ollama-results.md) |\n| Ollama + Instructor | Qwen2.5-VL-7B-Instruct:Q4_K_M_benxh | 2025-02-20 | [link](tests/benchmarks/2025-02-20-bsahane-Qwen2.5-VL-7B-Instruct-Q4_K_M_benxh-instructor-results.md) |\n| Microsoft | phi-4 | 2025-01-10 | [link](tests/benchmarks/2025-01-11-phi4-instructor-results.md) |\n\n### 📂 Directory Structure\n\nSchemas are organized by industry for easy navigation:\n\n```\nvlmrun\n└── hub\n    ├── schemas\n    |   ├── \u003cindustry\u003e\n    |   |   ├── \u003cuse-case-1\u003e.py\n    |   |   ├── \u003cuse-case-2\u003e.py\n    |   |   └── ...\n    │   ├── aerospace\n    │   │   └── remote_sensing.py\n    │   ├── document  # all document schemas are here\n    |   |   ├── invoice.py\n    |   |   ├── us_drivers_license.py\n    |   |   └── ...\n    │   ├── healthcare\n    │   │   └── medical_insurance_card.py\n    │   └── retail\n    │   │   └── ecommerce_product_caption.py\n    │   └── contrib  # all contributions are welcome here!\n    │       └── \u003cschema-name\u003e.py\n    └── version.py\n```\n\n### ✨ How to Contribute\n\nWe're building this hub for the community, and contributions are always welcome! Follow the [CONTRIBUTING](docs/CONTRIBUTING.md) and [SCHEMA-GUIDELINES.md](docs/SCHEMA-GUIDELINES.md) to get started.\n\n### 🔗 Quick Links\n\n- 💬 Send us an email at [support@vlm.run](mailto:support@vlm.run) or join our [Discord](https://discord.gg/4jgyECY4rq) for help.\n- 📣 Follow us on [Twitter](https://x.com/vlmrun), and [LinkedIn](https://www.linkedin.com/company/vlm-run) to keep up-to-date on our products.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvlm-run%2Fvlmrun-hub","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvlm-run%2Fvlmrun-hub","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvlm-run%2Fvlmrun-hub/lists"}