{"id":25903866,"url":"https://github.com/allenai/olmocr","last_synced_at":"2026-01-24T00:49:23.910Z","repository":{"id":277430813,"uuid":"858798469","full_name":"allenai/olmocr","owner":"allenai","description":"Toolkit for linearizing PDFs for LLM datasets/training","archived":false,"fork":false,"pushed_at":"2025-10-09T22:16:48.000Z","size":350567,"stargazers_count":14224,"open_issues_count":31,"forks_count":1065,"subscribers_count":75,"default_branch":"main","last_synced_at":"2025-10-10T14:52:10.238Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/allenai.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":".github/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-09-17T14:53:40.000Z","updated_at":"2025-10-10T14:34:38.000Z","dependencies_parsed_at":"2025-02-28T19:21:34.164Z","dependency_job_id":"520e6877-2ef6-454e-86a0-09e334fed47b","html_url":"https://github.com/allenai/olmocr","commit_stats":null,"previous_names":["allenai/olmocr"],"tags_count":29,"template":false,"template_full_name":"allenai/python-package-template","purl":"pkg:github/allenai/olmocr","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/allenai%2Folmocr","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/allenai%2Folmocr/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/allenai%2Folmocr/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/allenai%2Folmocr/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/allenai","download_url":"https://codeload.github.com/allenai/olmocr/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/allenai%2Folmocr/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279010256,"owners_count":26084719,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-12T02:00:06.719Z","response_time":53,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-03T04:02:20.189Z","updated_at":"2026-01-24T00:49:23.898Z","avatar_url":"https://github.com/allenai.png","language":"Python","funding_links":[],"categories":["Document Processing and OCR","Python","精选文章","Tools","数据 Data","A01_文本生成_文本对话","Industry Strength Natural Language Processing","Repos","Table of Contents","Librerías para usar NLP en español","📋 Scientific Documentation \u0026 Parsing","Parsers, OCR and extraction"],"sub_categories":["Economic Data Sources","OCR图像识别文字","大语言对话模型及数据","Information Processing","Preprocesamiento (de pdf u otro a markdown/json)","High-Performance Document Processing"],"readme":"\u003cdiv align=\"center\"\u003e\n  \u003cimg width=\"350\" alt=\"olmocr-2-full@2x\" src=\"https://github.com/user-attachments/assets/24f1b596-4059-46f1-8130-5d72dcc0b02e\" /\u003e\n\u003chr/\u003e\n\u003c/div\u003e\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://github.com/allenai/OLMo/blob/main/LICENSE\"\u003e\n    \u003cimg alt=\"GitHub License\" src=\"https://img.shields.io/github/license/allenai/OLMo\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://github.com/allenai/olmocr/releases\"\u003e\n    \u003cimg alt=\"GitHub release\" src=\"https://img.shields.io/github/release/allenai/olmocr.svg\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://arxiv.org/abs/2502.18443\"\u003e\n    \u003cimg alt=\"Tech Report v1\" src=\"https://img.shields.io/badge/Paper_v1-olmOCR-blue\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://arxiv.org/abs/2510.19817\"\u003e\n    \u003cimg alt=\"Tech Report v2\" src=\"https://img.shields.io/badge/Paper_v2-olmOCR-blue\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://olmocr.allenai.org\"\u003e\n    \u003cimg alt=\"Demo\" src=\"https://img.shields.io/badge/Ai2-Demo-F0529C\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://discord.gg/sZq3jTNVNG\"\u003e\n    \u003cimg alt=\"Discord\" src=\"https://img.shields.io/badge/Discord%20-%20blue?style=flat\u0026logo=discord\u0026label=Ai2\u0026color=%235B65E9\"\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\nA toolkit for converting PDFs and other image-based document formats into clean, readable, plain text format.\n\nTry the online demo: [https://olmocr.allenai.org/](https://olmocr.allenai.org/)\n\nFeatures:\n - Convert PDF, PNG, and JPEG based documents into clean Markdown\n - Support for equations, tables, handwriting, and complex formatting\n - Automatically removes headers and footers\n - Convert into text with a natural reading order, even in the presence of\n   figures, multi-column layouts, and insets\n - Efficient, less than $200 USD per million pages converted\n - (Based on a 7B parameter VLM, so it requires a GPU)\n\n### News\n - October 21, 2025 - v0.4.0 - [New model release](https://huggingface.co/allenai/olmOCR-2-7B-1025-FP8), boosts olmOCR-bench score by ~4 points using synthetic data and introduces RL training.\n - August 13, 2025 - v0.3.0 - [New model release](https://huggingface.co/allenai/olmOCR-7B-0825-FP8), fixes auto-rotation detection, and hallucinations on blank documents.\n - July 24, 2025 - v0.2.1 - [New model release](https://huggingface.co/allenai/olmOCR-7B-0725-FP8), scores 3 points higher on [olmOCR-Bench](https://github.com/allenai/olmocr/tree/main/olmocr/bench), also runs significantly faster because it's default FP8, and needs much fewer retries per document.\n - July 23, 2025 - v0.2.0 - New cleaned up [trainer code](https://github.com/allenai/olmocr/tree/main/olmocr/train), makes it much simpler to train olmOCR models yourself.\n - June 17, 2025 - v0.1.75 - Switch from sglang to vllm based inference pipeline, updated docker image to CUDA 12.8.\n - May 23, 2025 - v0.1.70 - Official docker support and images are now available! [See Docker usage](#using-docker)\n - May 19, 2025 - v0.1.68 - [olmOCR-Bench](https://github.com/allenai/olmocr/tree/main/olmocr/bench) launch, scoring 77.4. Launch includes 2 point performance boost in olmOCR pipeline due to bug fixes with prompts.\n - Mar 17, 2025 - v0.1.60 - Performance improvements due to better temperature selection in sampling.\n - Feb 25, 2025 - v0.1.58 -  Initial public launch and demo.\n\n### Benchmark\n\n[**olmOCR-Bench**](https://github.com/allenai/olmocr/tree/main/olmocr/bench):\nWe also ship a comprehensive benchmark suite covering over 7,000 test cases across 1,400 documents to help measure performance of OCR systems. \n\n\u003ctable\u003e\n    \u003cthead\u003e\n        \u003ctr\u003e\n            \u003cth\u003e\u003c/th\u003e\n            \u003cth\u003eArXiv\u003c/th\u003e\n            \u003cth\u003eOld\u003cbr\u003escans\u003cbr\u003emath\u003c/th\u003e\n            \u003cth\u003eTables\u003c/th\u003e\n            \u003cth\u003eOld\u003cbr\u003escans\u003c/th\u003e\n            \u003cth\u003eHeaders\u003cbr\u003e\u0026\u003cbr\u003efooters\u003c/th\u003e\n            \u003cth\u003eMulti\u003cbr\u003ecolumn\u003c/th\u003e\n            \u003cth\u003eLong\u003cbr\u003etiny\u003cbr\u003etext\u003c/th\u003e\n            \u003cth\u003eBase\u003c/th\u003e\n            \u003cth\u003eOverall\u003c/th\u003e\n        \u003c/tr\u003e\n    \u003c/thead\u003e\n    \u003ctbody\u003e\n        \u003ctr\u003e\n            \u003ctd\u003eMistral OCR API\u003c/td\u003e\n            \u003ctd\u003e77.2\u003c/td\u003e\n            \u003ctd\u003e67.5\u003c/td\u003e\n            \u003ctd\u003e60.6\u003c/td\u003e\n            \u003ctd\u003e29.3\u003c/td\u003e\n            \u003ctd\u003e93.6\u003c/td\u003e\n            \u003ctd\u003e71.3\u003c/td\u003e\n            \u003ctd\u003e77.1\u003c/td\u003e\n            \u003ctd\u003e99.4\u003c/td\u003e\n            \u003ctd\u003e72.0±1.1\u003c/td\u003e\n        \u003c/tr\u003e\n        \u003ctr\u003e\n            \u003ctd\u003eMarker 1.10.1\u003c/td\u003e\n            \u003ctd\u003e83.8\u003c/td\u003e\n            \u003ctd\u003e66.8\u003c/td\u003e\n            \u003ctd\u003e72.9\u003c/td\u003e\n            \u003ctd\u003e33.5\u003c/td\u003e\n            \u003ctd\u003e86.6\u003c/td\u003e\n            \u003ctd\u003e80.0\u003c/td\u003e\n            \u003ctd\u003e85.7\u003c/td\u003e\n            \u003ctd\u003e99.3\u003c/td\u003e\n            \u003ctd\u003e76.1±1.1\u003c/td\u003e\n        \u003c/tr\u003e\n        \u003ctr\u003e\n            \u003ctd\u003eMinerU 2.5.4*\u003c/td\u003e\n            \u003ctd\u003e76.6\u003c/td\u003e\n            \u003ctd\u003e54.6\u003c/td\u003e\n            \u003ctd\u003e84.9\u003c/td\u003e\n            \u003ctd\u003e33.7\u003c/td\u003e\n            \u003ctd\u003e96.6\u003c/td\u003e\n            \u003ctd\u003e78.2\u003c/td\u003e\n            \u003ctd\u003e83.5\u003c/td\u003e\n            \u003ctd\u003e93.7\u003c/td\u003e\n            \u003ctd\u003e75.2±1.1\u003c/td\u003e\n        \u003c/tr\u003e\n        \u003ctr\u003e\n            \u003ctd\u003eDeepSeek-OCR\u003c/td\u003e\n            \u003ctd\u003e77.2\u003c/td\u003e\n            \u003ctd\u003e73.6\u003c/td\u003e\n            \u003ctd\u003e80.2\u003c/td\u003e\n            \u003ctd\u003e33.3\u003c/td\u003e\n            \u003ctd\u003e96.1\u003c/td\u003e\n            \u003ctd\u003e66.4\u003c/td\u003e\n            \u003ctd\u003e79.4\u003c/td\u003e\n            \u003ctd\u003e99.8\u003c/td\u003e\n            \u003ctd\u003e75.7±1.0\u003c/td\u003e\n        \u003c/tr\u003e\n        \u003ctr\u003e\n            \u003ctd\u003eNanonets-OCR2-3B\u003c/td\u003e\n            \u003ctd\u003e75.4\u003c/td\u003e\n            \u003ctd\u003e46.1\u003c/td\u003e\n            \u003ctd\u003e86.8\u003c/td\u003e\n            \u003ctd\u003e40.9\u003c/td\u003e\n            \u003ctd\u003e32.1\u003c/td\u003e\n            \u003ctd\u003e81.9\u003c/td\u003e\n            \u003ctd\u003e93.0\u003c/td\u003e\n            \u003ctd\u003e99.6\u003c/td\u003e\n            \u003ctd\u003e69.5±1.1\u003c/td\u003e\n        \u003c/tr\u003e\n        \u003ctr\u003e\n            \u003ctd\u003ePaddleOCR-VL*\u003c/td\u003e\n            \u003ctd\u003e85.7\u003c/td\u003e\n            \u003ctd\u003e71.0\u003c/td\u003e\n            \u003ctd\u003e84.1\u003c/td\u003e\n            \u003ctd\u003e37.8\u003c/td\u003e\n            \u003ctd\u003e97.0\u003c/td\u003e\n            \u003ctd\u003e79.9\u003c/td\u003e\n            \u003ctd\u003e85.7\u003c/td\u003e\n            \u003ctd\u003e98.5\u003c/td\u003e\n            \u003ctd\u003e80.0±1.0\u003c/td\u003e\n        \u003c/tr\u003e\n        \u003ctr\u003e\n            \u003ctd\u003eInfinity-Parser 7B*\u003c/td\u003e\n            \u003ctd\u003e84.4\u003c/td\u003e\n            \u003ctd\u003e83.8\u003c/td\u003e\n            \u003ctd\u003e85.0\u003c/td\u003e\n            \u003ctd\u003e47.9\u003c/td\u003e\n            \u003ctd\u003e88.7\u003c/td\u003e\n            \u003ctd\u003e84.2\u003c/td\u003e\n            \u003ctd\u003e86.4\u003c/td\u003e\n            \u003ctd\u003e99.8\u003c/td\u003e\n            \u003ctd\u003e82.5±?\u003c/td\u003e\n        \u003c/tr\u003e\n        \u003ctr\u003e\n            \u003ctd\u003eChandra OCR 0.1.0*\u003c/td\u003e\n            \u003ctd\u003e82.2\u003c/td\u003e\n            \u003ctd\u003e80.3\u003c/td\u003e\n            \u003ctd\u003e88.0\u003c/td\u003e\n            \u003ctd\u003e50.4\u003c/td\u003e\n            \u003ctd\u003e90.8\u003c/td\u003e\n            \u003ctd\u003e81.2\u003c/td\u003e\n            \u003ctd\u003e92.3\u003c/td\u003e\n            \u003ctd\u003e99.9\u003c/td\u003e\n            \u003ctd\u003e83.1±0.9\u003c/td\u003e\n        \u003c/tr\u003e\n        \u003ctr\u003e\n            \u003ctd colspan=\"10\"\u003e\u003chr\u003e\u003c/td\u003e\n        \u003c/tr\u003e\n        \u003ctr\u003e\n            \u003ctd\u003e\u003cstrong\u003eolmOCR v0.4.0\u003c/strong\u003e\u003c/td\u003e\n            \u003ctd\u003e83.0\u003c/td\u003e\n            \u003ctd\u003e82.3\u003c/td\u003e\n            \u003ctd\u003e84.9\u003c/td\u003e\n            \u003ctd\u003e47.7\u003c/td\u003e\n            \u003ctd\u003e96.1\u003c/td\u003e\n            \u003ctd\u003e83.7\u003c/td\u003e\n            \u003ctd\u003e81.9\u003c/td\u003e\n            \u003ctd\u003e99.7\u003c/td\u003e\n            \u003ctd\u003e82.4±1.1\u003c/td\u003e\n        \u003c/tr\u003e\n    \u003c/tbody\u003e\n\u003c/table\u003e\n\n\n### Installation\n\nRequirements:\n - Recent NVIDIA GPU (tested on RTX 4090, L40S, A100, H100) with at least 12 GB of GPU RAM\n - 30GB of free disk space\n\nYou will need to install poppler-utils and additional fonts for rendering PDF images.\n\nInstall dependencies (Ubuntu/Debian)\n```bash\nsudo apt-get update\nsudo apt-get install poppler-utils ttf-mscorefonts-installer msttcorefonts fonts-crosextra-caladea fonts-crosextra-carlito gsfonts lcdf-typetools\n```\n\nSet up a conda environment and install olmocr. The requirements for running olmOCR\nare difficult to install in an existing python environment, so please do make a clean python environment to install into.\n```bash\nconda create -n olmocr python=3.11\nconda activate olmocr\n\n# For CPU-only operations, ex running the benchmark\npip install olmocr[bench]\n\n# For actually converting the files with your own GPU\npip install olmocr[gpu]  --extra-index-url https://download.pytorch.org/whl/cu128\n\n# Recommended: Install flash infer for faster inference on GPU\npip install https://download.pytorch.org/whl/cu128/flashinfer/flashinfer_python-0.2.5%2Bcu128torch2.7-cp38-abi3-linux_x86_64.whl\n```\n\n### Local Usage Example\n\nFor quick testing, try the [web demo](https://olmocr.allen.ai/). To run locally, a GPU is required, as inference is powered by [sglang](https://github.com/sgl-project/sglang) under the hood.\n\nConvert a Single PDF:\n```bash\n# Download a sample PDF\ncurl -o olmocr-sample.pdf https://olmocr.allenai.org/papers/olmocr_3pg_sample.pdf\n\n# Convert it to markdown\npython -m olmocr.pipeline ./localworkspace --markdown --pdfs olmocr-sample.pdf\n```\n\nConvert an Image file:\n```bash\npython -m olmocr.pipeline ./localworkspace --markdown --pdfs random_page.png\n```\n\nConvert Multiple PDFs:\n```bash\npython -m olmocr.pipeline ./localworkspace --markdown --pdfs tests/gnarly_pdfs/*.pdf\n```\n\nWith the addition of the `--markdown` flag, results will be stored as markdown files inside of `./localworkspace/markdown/`. \n\n#### Viewing Results\n\nThe `./localworkspace/` workspace folder will then have both [Dolma](https://github.com/allenai/dolma) and markdown files (if using `--markdown`).\n\n\n```bash\ncat localworkspace/markdown/olmocr-sample.md \n```\n\n```\nolmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Models\n...\n```\n\n### Using an Inference Provider or External Server\n\nIf you have a vLLM server already running elsewhere (or any inference platform implementing the OpenAI API), you can point olmOCR to use it instead of spawning a local instance:\n\n```bash\n# Use external vLLM server instead of local one\npython -m olmocr.pipeline ./localworkspace --server http://remote-server:8000/v1 --model allenai/olmOCR-2-7B-1025-FP8 --markdown --pdfs tests/gnarly_pdfs/*.pdf\n```\nThe served model name in VLLM needs to match the value provided in `--model`.\n\nAn example vLLM launch command would be:\n```bash\nvllm serve allenai/olmOCR-2-7B-1025-FP8 --max-model-len 16384\n```\n\n#### Verified External Providers\n\nWe have tested `olmOCR-2-7B-1025-FP8` on these external model providers and confirmed that they work\n\n|                                                                             | $/1M Input tokens | $/1M Output tokens | Example Command                                                                                                                                                                |\n|-----------------------------------------------------------------------------|-------------------|--------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| [Cirrascale](https://ai2endpoints.cirrascale.ai/models/overview)            | $0.07             | $0.15              | `python -m olmocr.pipeline ./localworkspace1 --server https://ai2endpoints.cirrascale.ai/api --api_key sk-XXXXXXX --workers 1 --max_concurrent_requests 20 --model olmOCR-2-7B-1025 --pdfs tests/gnarly_pdfs/*.pdf`     |\n| [DeepInfra](https://deepinfra.com/)                                         | $0.09             | $0.19              | `python -m olmocr.pipeline ./localworkspace1 --server https://api.deepinfra.com/v1/openai --api_key DfXXXXXXX --workers 1 --max_concurrent_requests 20 --model allenai/olmOCR-2-7B-1025 --pdfs tests/gnarly_pdfs/*.pdf` |\n| [Parasail](https://www.saas.parasail.io/serverless?name=olmocr-7b-1025-fp8) | $0.10             | $0.20              | `python -m olmocr.pipeline ./localworkspace1 --server https://api.parasail.io/v1 --api_key psk-XXXXX --workers 1 --max_concurrent_requests 20 --model allenai/olmOCR-2-7B-1025 --pdfs tests/gnarly_pdfs/*.pdf`          |\n\n\nNotes on arguments\n- `--server`: Defines the OpenAI-compatible endpoint: ex `https://api.deepinfra.com/v1/openai`\n- `--api_key`: Your API key, bassed in via Authorization Bearer HTTP header\n- `--max_concurrent_requests`: Max concurrent requests that will be in-flight to the inference provider at one time\n- `--workers`: Max number of page groups that will be processed at once. You may want to set this to `1` so that you finish one group of stuff before moving on.\n- `--pages_per_group`: You may want a smaller number of pages per group as many external provides have lower concurrent request limits\n- `--model`: The model identifier, ex. `allenai/olmOCR-2-7B-1025`, different providers have different names, and if you run locally, you can use `olmocr`\n- Other arguments work the same as with local inference\n\n\n### Multi-node / Cluster Usage\n\nIf you want to convert millions of PDFs, using multiple nodes running in parallel, then olmOCR supports\nreading your PDFs from AWS S3, and coordinating work using an AWS S3 output bucket.\n\nFor example, you can start this command on your first worker node, and it will set up\na simple work queue in your AWS bucket and start converting PDFs.\n\n```bash\npython -m olmocr.pipeline s3://my_s3_bucket/pdfworkspaces/exampleworkspace --pdfs s3://my_s3_bucket/jakep/gnarly_pdfs/*.pdf\n```\n\nNow on any subsequent nodes, just run this and they will start grabbing items from the same workspace queue.\n```bash\npython -m olmocr.pipeline s3://my_s3_bucket/pdfworkspaces/exampleworkspace\n```\n\nIf you are at Ai2 and want to linearize millions of PDFs efficiently using [beaker](https://www.beaker.org), just add the `--beaker`\nflag. This will prepare the workspace on your local machine, and then launch N GPU workers in the cluster to start\nconverting PDFs.\n\nFor example:\n```bash\npython -m olmocr.pipeline s3://my_s3_bucket/pdfworkspaces/exampleworkspace --pdfs s3://my_s3_bucket/jakep/gnarly_pdfs/*.pdf --beaker --beaker_gpus 4\n```\n\n\n### Using Docker\n\nPull the Docker image (large, includes the model, ~30GB):\n```bash\ndocker pull alleninstituteforai/olmocr:latest-with-model\n```\n\nFor advanced users who want to manage their own model downloads, we also provide a base image without the model:\n```bash\ndocker pull alleninstituteforai/olmocr:latest\n```\n\n#### Quick Start - Process PDFs\n\nProcess a single PDF in your current directory:\n```bash\ndocker run --gpus all \\\n  -v $(pwd):/workspace \\\n  alleninstituteforai/olmocr:latest-with-model \\\n  -c \"python -m olmocr.pipeline /workspace/output --markdown --pdfs /workspace/sample.pdf\"\n```\n\nProcess multiple PDFs:\n```bash\ndocker run --gpus all \\\n  -v /path/to/pdfs:/input \\\n  -v /path/to/output:/output \\\n  alleninstituteforai/olmocr:latest-with-model \\\n  -c \"python -m olmocr.pipeline /output --markdown --pdfs /input/*.pdf\"\n```\n\n#### Interactive Mode\n\nRun the container interactively for exploration and debugging:\n```bash\ndocker run -it --gpus all alleninstituteforai/olmocr:latest-with-model\n```\n\n\u003e Visit our Docker repository on [Docker Hub](https://hub.docker.com/r/alleninstituteforai/olmocr) for more information.\n\n### Full documentation for the pipeline\n\n```bash\npython -m olmocr.pipeline --help\nusage: pipeline.py [-h] [--pdfs [PDFS ...]] [--model MODEL] [--workspace_profile WORKSPACE_PROFILE] [--pdf_profile PDF_PROFILE] [--pages_per_group PAGES_PER_GROUP] [--max_page_retries MAX_PAGE_RETRIES] [--max_page_error_rate MAX_PAGE_ERROR_RATE] [--workers WORKERS]\n                   [--apply_filter] [--stats] [--markdown] [--target_longest_image_dim TARGET_LONGEST_IMAGE_DIM] [--target_anchor_text_len TARGET_ANCHOR_TEXT_LEN] [--guided_decoding] [--gpu-memory-utilization GPU_MEMORY_UTILIZATION] [--max_model_len MAX_MODEL_LEN]\n                   [--tensor-parallel-size TENSOR_PARALLEL_SIZE] [--data-parallel-size DATA_PARALLEL_SIZE] [--port PORT] [--server SERVER] [--beaker] [--beaker_workspace BEAKER_WORKSPACE] [--beaker_cluster BEAKER_CLUSTER] [--beaker_gpus BEAKER_GPUS] [--beaker_priority BEAKER_PRIORITY]\n                   workspace\n\nManager for running millions of PDFs through a batch inference pipeline\n\npositional arguments:\n  workspace             The filesystem path where work will be stored, can be a local folder, or an s3 path if coordinating work with many workers, s3://bucket/prefix/\n\noptions:\n  -h, --help            show this help message and exit\n  --pdfs [PDFS ...]     Path to add pdfs stored in s3 to the workspace, can be a glob path s3://bucket/prefix/*.pdf or path to file containing list of pdf paths\n  --model MODEL         Path where the model is located, allenai/olmOCR-7B-0725-FP8 is the default, can be local, s3, or hugging face.\n  --workspace_profile WORKSPACE_PROFILE\n                        S3 configuration profile for accessing the workspace\n  --pdf_profile PDF_PROFILE\n                        S3 configuration profile for accessing the raw pdf documents\n  --pages_per_group PAGES_PER_GROUP\n                        Aiming for this many pdf pages per work item group\n  --max_page_retries MAX_PAGE_RETRIES\n                        Max number of times we will retry rendering a page\n  --max_page_error_rate MAX_PAGE_ERROR_RATE\n                        Rate of allowable failed pages in a document, 1/250 by default\n  --workers WORKERS     Number of workers to run at a time\n  --apply_filter        Apply basic filtering to English pdfs which are not forms, and not likely seo spam\n  --stats               Instead of running any job, reports some statistics about the current workspace\n  --markdown            Also write natural text to markdown files preserving the folder structure of the input pdfs\n  --target_longest_image_dim TARGET_LONGEST_IMAGE_DIM\n                        Dimension on longest side to use for rendering the pdf pages\n  --target_anchor_text_len TARGET_ANCHOR_TEXT_LEN\n                        Maximum amount of anchor text to use (characters), not used for new models\n  --guided_decoding     Enable guided decoding for model YAML type outputs\n\nVLLM arguments:\n  --gpu-memory-utilization GPU_MEMORY_UTILIZATION\n                        Fraction of VRAM vLLM may pre-allocate for KV-cache (passed through to vllm serve).\n  --max_model_len MAX_MODEL_LEN\n                        Upper bound (tokens) vLLM will allocate KV-cache for, lower if VLLM won't start\n  --tensor-parallel-size TENSOR_PARALLEL_SIZE, -tp TENSOR_PARALLEL_SIZE\n                        Tensor parallel size for vLLM\n  --data-parallel-size DATA_PARALLEL_SIZE, -dp DATA_PARALLEL_SIZE\n                        Data parallel size for vLLM\n  --port PORT           Port to use for the VLLM server\n  --server SERVER       URL of external vLLM (or other compatible provider)\n                        server (e.g., http://hostname:port). If provided,\n                        skips spawning local vLLM instance\n\nbeaker/cluster execution:\n  --beaker              Submit this job to beaker instead of running locally\n  --beaker_workspace BEAKER_WORKSPACE\n                        Beaker workspace to submit to\n  --beaker_cluster BEAKER_CLUSTER\n                        Beaker clusters you want to run on\n  --beaker_gpus BEAKER_GPUS\n                        Number of gpu replicas to run\n  --beaker_priority BEAKER_PRIORITY\n                        Beaker priority level for the job\n```\n\n## Code overview\n\nThere are some nice reusable pieces of the code that may be useful for your own projects:\n - A prompting strategy to get really good natural text parsing using ChatGPT 4o - [buildsilver.py](https://github.com/allenai/olmocr/blob/main/olmocr/data/buildsilver.py)\n - Basic filtering by language and SEO spam removal - [filter.py](https://github.com/allenai/olmocr/blob/main/olmocr/filter/filter.py)\n - SFT Finetuning code for Qwen2.5-VL - [train.py](https://github.com/allenai/olmocr/blob/main/olmocr/train/train.py)\n - GRPO RL Trainer - [grpo_train.py](https://github.com/allenai/olmocr/blob/main/olmocr/train/grpo_train.py)\n - Synthetic data generation - [mine_html_templates.py](https://github.com/allenai/olmocr/blob/main/olmocr/bench/synth/mine_html_templates.py)\n - Processing millions of PDFs through a finetuned model using VLLM - [pipeline.py](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py)\n - Viewing [Dolma docs](https://github.com/allenai/dolma) created from PDFs - [dolmaviewer.py](https://github.com/allenai/olmocr/blob/main/olmocr/viewer/dolmaviewer.py)\n\n\n\n## Team\n\n\u003c!-- start team --\u003e\n\n**olmOCR** is developed and maintained by the AllenNLP team, backed by [the Allen Institute for Artificial Intelligence (AI2)](https://allenai.org/).\nAI2 is a non-profit institute with the mission to contribute to humanity through high-impact AI research and engineering.\nTo learn more about who specifically contributed to this codebase, see [our contributors](https://github.com/allenai/olmocr/graphs/contributors) page.\n\n\u003c!-- end team --\u003e\n\n## License\n\n\u003c!-- start license --\u003e\n\n**olmOCR** is licensed under [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0).\nA full copy of the license can be found [on GitHub](https://github.com/allenai/olmocr/blob/main/LICENSE).\n\n\u003c!-- end license --\u003e\n\n## Citing\n\nFor olmOCR v1 and OlmOCR-bench:\n```bibtex\n@misc{olmocrbench,\n      title={{olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Models}},\n      author={Jake Poznanski and Jon Borchardt and Jason Dunkelberger and Regan Huff and Daniel Lin and Aman Rangapur and Christopher Wilhelm and Kyle Lo and Luca Soldaini},\n      year={2025},\n      eprint={2502.18443},\n      archivePrefix={arXiv},\n      primaryClass={cs.CL},\n      url={https://arxiv.org/abs/2502.18443},\n}\n```\n\nFor olmOCR v2 Unit Testing Rewards with RL:\n```bibtex\n@misc{olmocr2,\n      title={olmOCR 2: Unit Test Rewards for Document OCR}, \n      author={Jake Poznanski and Luca Soldaini and Kyle Lo},\n      year={2025},\n      eprint={2510.19817},\n      archivePrefix={arXiv},\n      primaryClass={cs.CV},\n      url={https://arxiv.org/abs/2510.19817}, \n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fallenai%2Folmocr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fallenai%2Folmocr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fallenai%2Folmocr/lists"}