{"id":15432978,"url":"https://github.com/simonw/s3-ocr","last_synced_at":"2025-08-21T08:31:08.250Z","repository":{"id":41158361,"uuid":"508461227","full_name":"simonw/s3-ocr","owner":"simonw","description":"Tools for running OCR against files stored in S3","archived":false,"fork":false,"pushed_at":"2022-08-10T04:43:17.000Z","size":41,"stargazers_count":115,"open_issues_count":7,"forks_count":7,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-10-18T07:53:37.549Z","etag":null,"topics":["ocr","s3","textract"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/simonw.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-06-28T21:33:09.000Z","updated_at":"2024-08-05T02:48:44.000Z","dependencies_parsed_at":"2022-07-21T22:00:34.051Z","dependency_job_id":null,"html_url":"https://github.com/simonw/s3-ocr","commit_stats":null,"previous_names":[],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonw%2Fs3-ocr","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonw%2Fs3-ocr/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonw%2Fs3-ocr/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonw%2Fs3-ocr/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/simonw","download_url":"https://codeload.github.com/simonw/s3-ocr/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":230501172,"owners_count":18236061,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ocr","s3","textract"],"created_at":"2024-10-01T18:29:55.744Z","updated_at":"2024-12-19T21:10:02.928Z","avatar_url":"https://github.com/simonw.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# s3-ocr\n\n[![PyPI](https://img.shields.io/pypi/v/s3-ocr.svg)](https://pypi.org/project/s3-ocr/)\n[![Changelog](https://img.shields.io/github/v/release/simonw/s3-ocr?include_prereleases\u0026label=changelog)](https://github.com/simonw/s3-ocr/releases)\n[![Tests](https://github.com/simonw/s3-ocr/workflows/Test/badge.svg)](https://github.com/simonw/s3-ocr/actions?query=workflow%3ATest)\n[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/s3-ocr/blob/master/LICENSE)\n\nTools for running OCR against files stored in S3\n\nBackground on this project: [s3-ocr: Extract text from PDF files stored in an S3 bucket](https://simonwillison.net/2022/Jun/30/s3-ocr/)\n\n## Installation\n\nInstall this tool using `pip`:\n\n    pip install s3-ocr\n\n## Demo\n\nYou can see the results of running this tool against three PDFs from the Internet Archive ([one](https://archive.org/details/unmaskingrobert00houdgoog), [two](https://archive.org/details/practicalmagicia00harr), [three](https://archive.org/details/latestmagicbeing00hoff)) in [this example table](https://s3-ocr-demo.datasette.io/pages/pages?_facet=path#facet-path) hosted using [Datasette](https://datasette.io/).\n\n## Starting OCR against PDFs in a bucket\n\nThe `start` command takes a list of keys and submits them to [Textract](https://aws.amazon.com/textract/) for OCR processing.\n\nYou need to have AWS configured using environment variables, credentials file in your home directory or a JSON or INI file generated using [s3-credentials](https://datasette.io/tools/s3-credentials).\n\nYou can start the process running like this:\n\n    s3-ocr start name-of-your-bucket my-pdf-file.pdf\n\nThe paths you specify should be paths within the bucket. If you stored your PDF files in folders inside the bucket it should look like this:\n\n    s3-ocr start name-of-your-bucket path/to/one.pdf path/to/two.pdf\n\nOCR can take some time. The results of the OCR will be stored in `textract-output` in your bucket.\n\nTo process every file in the bucket with a `.pdf` extension use `--all`:\n\n    s3-ocr start name-of-bucket --all\n\nTo process every file with a `.pdf` extension within a specific folder, use `--prefix`:\n\n    s3-ocr start name-of-bucket --prefix path/to/folder\n\n### s3-ocr start --help\n\n\u003c!-- [[[cog\nimport cog\nfrom s3_ocr import cli\nfrom click.testing import CliRunner\nrunner = CliRunner()\nresult = runner.invoke(cli.cli, [\"start\", \"--help\"])\nhelp = result.output.replace(\"Usage: cli\", \"Usage: s3-ocr\")\ncog.out(\n    \"```\\n{}\\n```\".format(help)\n)\n]]] --\u003e\n```\nUsage: s3-ocr start [OPTIONS] BUCKET [KEYS]...\n\n  Start OCR tasks for PDF files in an S3 bucket\n\n      s3-ocr start name-of-bucket path/to/one.pdf path/to/two.pdf\n\n  To process every file with a .pdf extension:\n\n      s3-ocr start name-of-bucket --all\n\n  To process every .pdf in the PUBLIC/ folder:\n\n      s3-ocr start name-of-bucket --prefix PUBLIC/\n\nOptions:\n  --all                 Process all PDF files in the bucket\n  --prefix TEXT         Process all PDF files within this prefix\n  --dry-run             Show what this would do, but don't actually do it\n  --no-retry            Don't retry failed requests\n  --access-key TEXT     AWS access key ID\n  --secret-key TEXT     AWS secret access key\n  --session-token TEXT  AWS session token\n  --endpoint-url TEXT   Custom endpoint URL\n  -a, --auth FILENAME   Path to JSON/INI file containing credentials\n  --help                Show this message and exit.\n\n```\n\u003c!-- [[[end]]] --\u003e\n\n## Checking status\n\nThe `s3-ocr status \u003cbucket-name\u003e` command shows a rough indication of progress through the tasks:\n\n```\n% s3-ocr status sfms-history\n153 complete out of 532 jobs\n```\nIt compares the jobs that have been submitted, based on `.s3-ocr.json` files, to the jobs that have their results written to the `textract-output/` folder.\n\n### s3-ocr status --help\n\n\u003c!-- [[[cog\nresult = runner.invoke(cli.cli, [\"status\", \"--help\"])\nhelp = result.output.replace(\"Usage: cli\", \"Usage: s3-ocr\")\ncog.out(\n    \"```\\n{}\\n```\".format(help.split(\"--access-key\")[0] + \"--access-key ...\")\n)\n]]] --\u003e\n```\nUsage: s3-ocr status [OPTIONS] BUCKET\n\n  Show status of OCR jobs for a bucket\n\nOptions:\n  --access-key ...\n```\n\u003c!-- [[[end]]] --\u003e\n\n## Inspecting a job\n\nThe `s3-ocr inspect-job \u003cjob_id\u003e` command can be used to check the status of a specific job ID:\n```\n% s3-ocr inspect-job b267282745685226339b7e0d4366c4ff6887b7e293ed4b304dc8bb8b991c7864\n{\n  \"DocumentMetadata\": {\n    \"Pages\": 583\n  },\n  \"JobStatus\": \"SUCCEEDED\",\n  \"DetectDocumentTextModelVersion\": \"1.0\"\n}\n```\n\n### s3-ocr inspect-job --help\n\n\u003c!-- [[[cog\nresult = runner.invoke(cli.cli, [\"inspect-job\", \"--help\"])\nhelp = result.output.replace(\"Usage: cli\", \"Usage: s3-ocr\")\ncog.out(\n    \"```\\n{}\\n```\".format(help.split(\"--access-key\")[0] + \"--access-key ...\")\n)\n]]] --\u003e\n```\nUsage: s3-ocr inspect-job [OPTIONS] JOB_ID\n\n  Show the current status of an OCR job\n\n      s3-ocr inspect-job \u003cjob_id\u003e\n\nOptions:\n  --access-key ...\n```\n\u003c!-- [[[end]]] --\u003e\n\n## Fetching the results\n\nOnce an OCR job has completed you can download the resulting JSON using the `fetch` command:\n\n    s3-ocr fetch name-of-bucket path/to/file.pdf\n\nThis will save files in the current directory with names like this:\n\n- `4d9b5cf580e761fdb16fd24edce14737ebc562972526ef6617554adfa11d6038-1.json`\n- `4d9b5cf580e761fdb16fd24edce14737ebc562972526ef6617554adfa11d6038-2.json`\n\nThe number of files will vary depending on the length of the document.\n\nIf you don't want separate files you can combine them together using the `-c/--combine` option:\n\n    s3-ocr fetch name-of-bucket path/to/file.pdf --combine output.json\n\nThe `output.json` file will then contain data that looks something like this:\n\n```\n{\n  \"Blocks\": [\n    {\n      \"BlockType\": \"PAGE\",\n      \"Geometry\": {...}\n      \"Page\": 1,\n      ...\n    },\n    {\n      \"BlockType\": \"LINE\",\n      \"Page\": 1,\n      ...\n      \"Text\": \"Barry\",\n    },\n```\n### s3-ocr fetch --help\n\n\u003c!-- [[[cog\nresult = runner.invoke(cli.cli, [\"fetch\", \"--help\"])\nhelp = result.output.replace(\"Usage: cli\", \"Usage: s3-ocr\")\ncog.out(\n    \"```\\n{}\\n```\".format(help.split(\"--access-key\")[0] + \"--access-key ...\")\n)\n]]] --\u003e\n```\nUsage: s3-ocr fetch [OPTIONS] BUCKET KEY\n\n  Fetch the OCR results for a specified file\n\n      s3-ocr fetch name-of-bucket path/to/key.pdf\n\n  This will save files in the current directory called things like\n\n      a806e67e504fc15f...48314e-1.json     a806e67e504fc15f...48314e-2.json\n\n  To combine these together into a single JSON file with a specified name, use:\n\n      s3-ocr fetch name-of-bucket path/to/key.pdf --combine output.json\n\n  Use \"--output -\" to print the combined JSON to standard output instead.\n\nOptions:\n  -c, --combine FILENAME  Write combined JSON to file\n  --access-key ...\n```\n\u003c!-- [[[end]]] --\u003e\n\n## Fetching just the text of a page\n\nIf you don't want to deal with the JSON directly, you can use the `text` command to retrieve just the text extracted from a PDF:\n\n    s3-ocr text name-of-bucket path/to/file.pdf\n\nThis will output plain text to standard output.\n\nTo save that to a file, use this:\n\n    s3-ocr text name-of-bucket path/to/file.pdf \u003e text.txt\n\nSeparate pages will be separated by three newlines. To separate them using a `----` horizontal divider instead add `--divider`:\n\n    s3-ocr text name-of-bucket path/to/file.pdf --divider\n\n### s3-ocr text --help\n\n\u003c!-- [[[cog\nresult = runner.invoke(cli.cli, [\"text\", \"--help\"])\nhelp = result.output.replace(\"Usage: cli\", \"Usage: s3-ocr\")\ncog.out(\n    \"```\\n{}\\n```\".format(help.split(\"--access-key\")[0] + \"--access-key ...\")\n)\n]]] --\u003e\n```\nUsage: s3-ocr text [OPTIONS] BUCKET KEY\n\n  Retrieve the text from an OCRd PDF file\n\n      s3-ocr text name-of-bucket path/to/key.pdf\n\nOptions:\n  --divider             Add ---- between pages\n  --access-key ...\n```\n\u003c!-- [[[end]]] --\u003e\n\n## Avoiding processing duplicates\n\nIf you move files around within your S3 bucket `s3-ocr` can lose track of which files have already been processed. This can lead to additional Textract charges for processing should you run `s3-ocr start` against those new files.\n\nThe `s3-ocr dedupe` command addresses this by scanning your bucket for files that have a new name but have previously been processed. It does this by looking at the `ETag` for each file, which represents the MD5 hash of the file contents.\n\nThe command will write out new `.s3ocr.json` files for each detected duplicate. This will avoid those duplicates being run those duplicates through OCR a second time should yo run `s3-ocr start`.\n\n    s3-ocr dedupe name-of-bucket\n\nAdd `--dry-run` for a preview of the changes that will be made to your bucket.\n\n### s3-ocr dedupe --help\n\n\u003c!-- [[[cog\nresult = runner.invoke(cli.cli, [\"dedupe\", \"--help\"])\nhelp = result.output.replace(\"Usage: cli\", \"Usage: s3-ocr\")\ncog.out(\n    \"```\\n{}\\n```\".format(help.split(\"--access-key\")[0] + \"--access-key ...\")\n)\n]]] --\u003e\n```\nUsage: s3-ocr dedupe [OPTIONS] BUCKET\n\n  Scan every file in the bucket checking for duplicates - files that have not\n  yet been OCRd but that have the same contents (based on ETag) as a file that\n  HAS been OCRd.\n\n      s3-ocr dedupe name-of-bucket\n\nOptions:\n  --dry-run             Show output without writing anything to S3\n  --access-key ...\n```\n\u003c!-- [[[end]]] --\u003e\n\n## Changes made to your bucket\n\nTo keep track of which files have been submitted for processing, `s3-ocr` will create a JSON file for every file that it adds to the OCR queue.\n\nThis file will be called:\n\n    path-to-file/name-of-file.pdf.s3-ocr.json\n\nEach of these JSON files contains data that looks like this:\n\n```json\n{\n  \"job_id\": \"a34eb4e8dc7e70aa9668f7272aa403e85997364199a654422340bc5ada43affe\",\n  \"etag\": \"\\\"b0c77472e15500347ebf46032a454e8e\\\"\"\n}\n```\nThe recorded `job_id` can be used later to associate the file with the results of the OCR task in `textract-output/`.\n\nThe `etag` is the ETag of the S3 object at the time it was submitted. This can be used later to determine if a file has changed since it last had OCR run against it.\n\nThis design for the tool, with the `.s3-ocr.json` files tracking jobs that have been submitted, means that it is safe to run `s3-ocr start` against the same bucket multiple times without the risk of starting duplicate OCR jobs.\n\n## Creating a SQLite index of your OCR results\n\nThe `s3-ocr index \u003cbucket\u003e \u003cdatabase_file\u003e` command creates a SQLite database containing the results of the OCR, and configures SQLite full-text search against the text:\n\n```\n% s3-ocr index sfms-history index.db\nFetching job details  [####################################]  100%\nPopulating pages table  [####################----------------]   55%  00:03:18\n```\nThe schema of the resulting database looks like this (excluding the FTS tables):\n```sql\nCREATE TABLE [pages] (\n   [path] TEXT,\n   [page] INTEGER,\n   [folder] TEXT,\n   [text] TEXT,\n   PRIMARY KEY ([path], [page])\n);\nCREATE TABLE [ocr_jobs] (\n   [key] TEXT PRIMARY KEY,\n   [job_id] TEXT,\n   [etag] TEXT,\n   [s3_ocr_etag] TEXT\n);\nCREATE TABLE [fetched_jobs] (\n   [job_id] TEXT PRIMARY KEY\n);\n```\nThe database is designed to be used with [Datasette](https://datasette.io).\n\n### s3-ocr index --help\n\n\u003c!-- [[[cog\nresult = runner.invoke(cli.cli, [\"index\", \"--help\"])\nhelp = result.output.replace(\"Usage: cli\", \"Usage: s3-ocr\")\ncog.out(\n    \"```\\n{}\\n```\".format(help.split(\"--access-key\")[0] + \"--access-key ...\")\n)\n]]] --\u003e\n```\nUsage: s3-ocr index [OPTIONS] BUCKET DATABASE\n\n  Create a SQLite database with OCR results for files in a bucket\n\nOptions:\n  --access-key ...\n```\n\u003c!-- [[[end]]] --\u003e\n\n## Development\n\nTo contribute to this tool, first checkout the code. Then create a new virtual environment:\n\n    cd s3-ocr\n    python -m venv venv\n    source venv/bin/activate\n\nNow install the dependencies and test dependencies:\n\n    pip install -e '.[test]'\n\nTo run the tests:\n\n    pytest\n\nTo regenerate the README file with the latest `--help`:\n\n    cog -r README.md\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsimonw%2Fs3-ocr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsimonw%2Fs3-ocr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsimonw%2Fs3-ocr/lists"}