{"id":27935860,"url":"https://github.com/jeremyarancio/vlm-batch-deployment","last_synced_at":"2025-10-25T02:25:08.151Z","repository":{"id":290317734,"uuid":"968083152","full_name":"jeremyarancio/VLM-Batch-Deployment","owner":"jeremyarancio","description":"Batch Deployment for Document Parsing with AWS Batch \u0026 Qwen-2.5-VL","archived":false,"fork":false,"pushed_at":"2025-04-28T06:33:29.000Z","size":408,"stargazers_count":14,"open_issues_count":0,"forks_count":3,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-28T07:33:39.420Z","etag":null,"topics":["aws","batch","llm","vllm","vlm"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jeremyarancio.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-04-17T13:31:37.000Z","updated_at":"2025-04-28T06:33:33.000Z","dependencies_parsed_at":"2025-04-28T07:44:14.588Z","dependency_job_id":null,"html_url":"https://github.com/jeremyarancio/VLM-Batch-Deployment","commit_stats":null,"previous_names":["jeremyarancio/vlm-batch-deployment"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jeremyarancio%2FVLM-Batch-Deployment","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jeremyarancio%2FVLM-Batch-Deployment/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jeremyarancio%2FVLM-Batch-Deployment/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jeremyarancio%2FVLM-Batch-Deployment/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jeremyarancio","download_url":"https://codeload.github.com/jeremyarancio/VLM-Batch-Deployment/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252831181,"owners_count":21810780,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","batch","llm","vllm","vlm"],"created_at":"2025-05-07T06:50:59.547Z","updated_at":"2025-10-25T02:25:08.071Z","avatar_url":"https://github.com/jeremyarancio.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# VLM for Document structured extraction\n\nIn this project, we build a Batch inference job to extract data from reports and invoices using Vision Language Models (VLM) with vLLM.\n\nThe Batch inference is deployed and orchestrated in [AWS Batch](https://aws.amazon.com/fr/batch/).\n\nThis project is part of the Webinar we presented with [Julien Hurault](https://www.linkedin.com/in/julienhuraultanalytics/).\n\n:microphone: Webinar (coming soon) \\\n:newspaper: [Article](https://medium.com/towards-artificial-intelligence/deploy-an-in-house-vision-language-model-to-parse-millions-of-documents-say-goodbye-to-gemini-and-cdac6f77aff5)\n\nSubscribe to the [Newsletter](https://medium.com/@jeremyarancio/subscribe).\n\n## Quick start\n\nThe repository is organized as such:\n\n```\n.\n├── src\n│   └── llm\n│       ├── __init__.py\n│       ├── __main__.py\n│       ├── parser           // Job module\n│       └── settings.py      // Settings and Env variables\n├── data\n│   └── docs                 // Downloaded documents for testing \n├── infra                    // AWS Batch insfrastructure deployment\n├── Dockerfile  \n├── Makefile\n├── NOTES.md                 // Technical notes\n├── README.md\n├── assets\n├── notebooks                // Experimentations\n├── scripts                  // Various scripts not used in package\n├── pyproject.toml\n└── uv.lock\n```\n\nThe module is packaged with [uv](https://github.com/astral-sh/uv).\nTo install all the dependencies, run:\n\n```bash\nuv sync\n```\n\nTo run the batch job:\n\n1. Use the `.env.template` to create your own `.env` file.\n2. You need to run the job within an environement with GPU such as L4, depending on the size of the model\n\nThen run:\n\n```bash\nuv run run-batch-job\n```\n\n## Run online Batch inference\n\nDeploy the module using Docker to AWS ECR with: \n\n```bash\nmake deploy ECR_ACCOUNT_ID=\u003cYOUR-ECR-ACCOUNT-ID\u003e \n```\n\nNOTE: You may want to change the ECR repository (ECR_REPO_NAME) or the AWS region (AWS_REGION)\n\nThen, deploy the Batch infrastructure on AWS using Terraform, run:\n\n```bash\nmake aws-batch-apply\n```\n\nNOTE: Be sure to have Terraform installed.\n\nOnce the infrastructure is set up, you can launch a job using the `aws batch` cli command.\n\n```bash\naws batch submit-job \\\n  --job-name \u003cYOUR-JOB-NAME\u003e \\\n  --job-queue demo-job-queue \\\n  --job-definition demo-job-definition\n```\n\n## Process overview\n\nThe Batch process looks like the following:\n\n* The documents are loaded from S3 as images. You need to indicates 3 environment variables:\n  * `S3_BUCKET`: the S3 bucket name\n  * `S3_PREPROCESSED_IMAGES_DIR_PREFIX`: the directory name where the invoices are stored. It should be images and not PDFs.\n  * `S3_PROCESSED_DATASET_PREFIX`: The path of the output dataset. Right now, the task only returns JSONL dataset (`.jsonl`).\n* The model `MODEL_NAME` is loaded using **vLLM**. By default, we load *\"Qwen/Qwen2.5-VL-3B-Instruct\"*. But feel free to get any larger models if they fit into memory.\n* vLLM is configured to return a structured output using `\"GuidedDecoding\"` by providing the expected schema with Pydantic. \n* Images are processed by vLLM and a `json` is extracted for each invoice. If the json decoding is not successful, an empty dict is returned instead.\n* NOT IMPLEMENTED YET: Pydantic is used to validate the extracted jsons and default values are returned if field validation fails.\n* The list of dicts, with an unique identifier (such as the S3 file path), is transformed into a usable dataset (here JSONL since there's no data type validation with Pydantic yet.)\n* The dataset is finally exported to S3. Indicate where with the environment variable `S3_PROCESSED_DATASET_PREFIX`. Be sure to indicate the proper file format (`.jsonl` in this case.)\n\n## Dataset\n\nFor this demo, we used synthetically generated invoices from this [dataset](https://huggingface.co/datasets/mathieu1256/FATURA2-invoices) on Hugging Face.\n\nTo download the full dataset: \n\n```bash\nmake download-data\n```\n\nThere's also a script in `scripts/` folder to load a sample of images.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjeremyarancio%2Fvlm-batch-deployment","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjeremyarancio%2Fvlm-batch-deployment","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjeremyarancio%2Fvlm-batch-deployment/lists"}