{"id":40115895,"url":"https://github.com/dlcs/pdf-to-alto","last_synced_at":"2026-01-19T12:10:08.942Z","repository":{"id":44921731,"uuid":"447199500","full_name":"dlcs/pdf-to-alto","owner":"dlcs","description":"Library to extract ALTO from PDF","archived":false,"fork":false,"pushed_at":"2024-02-02T09:11:46.000Z","size":16842,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-04-16T18:53:36.182Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dlcs.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2022-01-12T11:55:41.000Z","updated_at":"2022-01-12T18:01:52.000Z","dependencies_parsed_at":"2024-02-02T10:38:09.959Z","dependency_job_id":null,"html_url":"https://github.com/dlcs/pdf-to-alto","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/dlcs/pdf-to-alto","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dlcs%2Fpdf-to-alto","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dlcs%2Fpdf-to-alto/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dlcs%2Fpdf-to-alto/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dlcs%2Fpdf-to-alto/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dlcs","download_url":"https://codeload.github.com/dlcs/pdf-to-alto/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dlcs%2Fpdf-to-alto/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28567863,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-19T08:53:44.001Z","status":"ssl_error","status_checked_at":"2026-01-19T08:52:40.245Z","response_time":67,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-19T12:10:06.076Z","updated_at":"2026-01-19T12:10:08.928Z","avatar_url":"https://github.com/dlcs.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PDF to ALTO\n\nService that listens for incoming messages from SQS. On receipt of a message this service will download reference PDF,\nextract ALTO file per-page and upload to specified S3 bucket.\n\nIf `COMPLETED_TOPIC_ARN` env var specified a notification will be raised.\n\n## Messages Format\n\nThe incoming message is in the shape:\n\n```json\n{\n  \"pdfLocation\": \"https://www.hq.nasa.gov/alsj/a17/A17_FlightPlan.pdf\",\n  \"pdfIdentifier\": \"a17_flightplan\",\n  \"outputLocation\": \"s3://pdf-to-alto/a17_flightplan_alto\"\n}\n```\n\nWhere\n* `pdfLocation` - the URL where PDF can be downloaded from.\n* `pdfIdentifier` - unique identifier for PDF.\n  where `i` is 0-based page index. If omitted a random uuid will be used.\n* `outputLocation` - s3 location where final ALTO files will be output. With or without preceding `s3://` and no\n  trailing `/`.\n\n(See [sample.json](/sample.json))\n\nThe completed notification message echos back the original message with `\"numberOfFiles\"` property added.\n\nThe generated alto file will be placed in `outputLocation`. The format of each file will depend on value of `PREPEND_ID`\nenvvar. This format will be (where `i` is the page number):\n* If true: `f\"{pdfIdentifier}-{i:04d}.xml\"`\n* else `f\"{i:04d}.xml\"`,\n\n## Technology\n\nThis is a Python script that utilises the following libraries:\n\n* [pdfalto](https://github.com/kermitt2/pdfalto) - C lib used to generate ALTO files.\n* [PyMuPDF](https://pymupdf.readthedocs.io/en/latest/) - Python lib used to query PDF object for page dimensions.\n* [lxml](https://lxml.de/) - Used to parse ALTO files and update scaled values.\n* [requests](https://docs.python-requests.org/en/latest/) - Used to download PDF files.\n\n## Configuration\n\nThe following environment variables can be used to configure the app:\n\n| Env Var             | Description                                                          | Default               |\n|---------------------|----------------------------------------------------------------------|-----------------------|\n| DOWNLOAD_CHUNK_SIZE | Chunk size for downloading PDF                                       | 2048                  |\n| WORKING_FOLDER      | Local working folder for storing generated files                     | ./work                |\n| REMOVE_WORK_DIR     | Whether to clean up working dir on completion                        | True                  |\n| RESCALE_ALTO        | Whether to rescale generated ALTO to page                            | True                  |\n| MONITOR_SLEEP_SECS  | How long to sleep long polling operations if no messages received    | 30                    |\n| AWS_REGION          | AWS region being used                                                | eu-west-1             |\n| INCOMING_QUEUE      | The name of the SQS queue to monitor for incoming messages. Required |                       |\n| COMPLETED_TOPIC_ARN | The ARN of a topic to post completion notifications to               |                       |\n| LOCALSTACK          | If using LocalStack                                                  | False                 |\n| LOCALSTACK_ADDRESS  | Address for LocalStack instance                                      | http://localhost:4566 |\n\n(See [.env.dist](/.env.dist) for sample .env file)\n\n## Running Locally\n\nThere is a multi-stage Dockerfile that builds the `pdfalto` binary and copies it to a new stage.\n\n[docker-compose.yml](/docker-compose.yml) will build and start the main Python app alongside a LocalStack instance.\n\n```bash\n# build and start image using LocalStack\ndocker-compose up\n\n# build image\ndocker build --tag pdf-to-alto:local .\n\n# run docker image and listen to queue\ndocker run --env-file .env -it --rm --name pdf-to-alto pdf-to-alto:local\n\n# run docker image to process 1 single api\ndocker run -it --rm --name pdf-to-alto \\\n  pdf-to-alto:local \\\n  opt/app/app/pdf_processor.py https://text.example/test.pdf my-pdf-identifier s3://pdf-bucket/alto\n```\n\n_Note: Building pdfalto from source takes a few minutes_\n\n### LocalStack\n\nThe [`docker-compose.local.yml`](./docker-compose.local.yml) file will spin up a LocalStack instance and configure a few\nresource for local testing:\n\n* An S3 bucket titled \"pdf-to-alto\"\n* An SNS topic \"incoming-topic\" with an SQS subscription to \"incoming\"\n* An SNS topic \"completed-topic\" with an SQS subscription to \"completed\"\n\n```bash\ndocker-compose -f docker-compose.local.yml up\n```\n\nTo use LocalStack set the `LOCALSTACK` and `LOCALSTACK_ADDRESS` env vars (see above).\n\nWhen using the aws-cli with LocalStack the `--endpoint-url` needs to be specified. Below are some handy commands to use\nwhen testing:\n\n```bash\n# raise sample notification using sample.json\naws --endpoint-url=http://localhost:4566 sns publish --topic-arn arn:aws:sns:eu-west-1:000000000000:incoming-topic --message file://sample.json --region eu-west-1\n\n# clear incoming queue\naws --endpoint-url=http://localhost:4566 sqs purge-queue --queue-url \"http://localstack:4566/000000000000/incoming\" --region eu-west-1\n\n# check number of 'completed' notifications raised\naws --endpoint-url=http://localhost:4566 sqs get-queue-attributes --queue-url \"http://localstack:4566/000000000000/completed\" --attribute-names All --region eu-west-1\n\n# check contents of s3\naws --endpoint-url=http://localhost:4566 s3 ls pdf-to-alto --recursive --region eu-west-1\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdlcs%2Fpdf-to-alto","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdlcs%2Fpdf-to-alto","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdlcs%2Fpdf-to-alto/lists"}