{"id":51605226,"url":"https://github.com/mgifford/pdf-crawler","last_synced_at":"2026-07-12T01:31:12.717Z","repository":{"id":343279887,"uuid":"1176957442","full_name":"mgifford/pdf-crawler","owner":"mgifford","description":"An implementation of simplA11yPDFCrawler in GitHub Pages / Actions","archived":false,"fork":false,"pushed_at":"2026-07-06T04:21:41.000Z","size":79402,"stargazers_count":5,"open_issues_count":2,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-06T06:09:29.535Z","etag":null,"topics":["a11y","accessibility","pdf","pdf-accessibility"],"latest_commit_sha":null,"homepage":"https://mgifford.github.io/pdf-crawler/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mgifford.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2026-03-09T14:51:14.000Z","updated_at":"2026-07-06T04:21:45.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/mgifford/pdf-crawler","commit_stats":null,"previous_names":["mgifford/pdf-crawler"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mgifford/pdf-crawler","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mgifford%2Fpdf-crawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mgifford%2Fpdf-crawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mgifford%2Fpdf-crawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mgifford%2Fpdf-crawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mgifford","download_url":"https://codeload.github.com/mgifford/pdf-crawler/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mgifford%2Fpdf-crawler/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35379590,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-11T02:00:05.354Z","response_time":104,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["a11y","accessibility","pdf","pdf-accessibility"],"created_at":"2026-07-12T01:31:12.605Z","updated_at":"2026-07-12T01:31:12.690Z","avatar_url":"https://github.com/mgifford.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pdf-crawler\n\n**A free, zero-infrastructure PDF accessibility scanner.**\nPoint it at any website, it will begin a process that might take an hour to crawl \nyour site, discover most of the PDFs and then flag common accessibility issues. \nIt then posts a public report of the discoveries. All through a GitHub issue.  \nNo servers to deploy, no accounts to configure, no software to install.\n\n**How it works in three steps:**\n\n1. Fill out the [web form](https://mgifford.github.io/pdf-crawler/) — it\n   creates a GitHub issue titled `SCAN: https://…` with a single click.\n2. GitHub Actions crawls the site for up to one hour, analyses every PDF it\n   finds, and posts the full results as a comment on that issue.\n3. The issue is **automatically closed** once the report is ready.  The report\n   is **public**.  Reopen the issue any time to re-run the scan.\n\n---\n\n## What it does\n\n1. **Crawls** a website for PDF (and other document) files using\n   [Scrapy](https://scrapy.org).\n2. **Maintains a YAML manifest** (`reports/manifest.yaml`) with every\n   discovered file's URL, MD5 hash, and accessibility results.  Files whose\n   MD5 hash has not changed since the last run are **skipped** automatically.\n3. **Analyses** each pending PDF for the following accessibility issues\n   (based on WCAG 2.x / EN 301 549):\n\n   | Check | WCAG SC | Description |\n   |-------|---------|-------------|\n   | `TaggedTest` | – | Is the document tagged? |\n   | `EmptyTextTest` | 1.4.5 | Does it contain real text (not just images)? |\n   | `ProtectedTest` | – | Is it protected against assistive technologies? |\n   | `TitleTest` | 2.4.2 | Does it have a title with `DisplayDocTitle` set? |\n   | `LanguageTest` | 3.1.1 | Does it have a valid default language? |\n   | `BookmarksTest` | 2.4.1 | For documents \u003e 20 pages, does it have bookmarks? |\n   | `ImageAltTextTest` | 1.1.1 | Do all Figure structure elements carry alternate text? |\n\n4. **Generates reports** in Markdown and JSON.\n5. **Deletes the PDF files** after analysis to keep the repository small;\n   only the YAML manifest is committed.\n\n---\n\n## Interpreting results\n\nThe automated checks above are a **first step**, not a complete accessibility\naudit.\n\n| Level | What it tells you |\n|-------|-------------------|\n| All checks pass | The document meets a basic set of machine-testable criteria. |\n| [veraPDF](https://verapdf.org/) pass | The document also conforms to PDF/A or PDF/UA as verified by the leading open-source conformance checker (run separately). |\n| Manual review complete | The document has been tested by a person using assistive technology — the only way to confirm true accessibility. |\n\nPassing every automated check is good.  Passing [veraPDF](https://verapdf.org/)\nas well is better.\n**Manual testing is still required and will always be required.**  Automated\ntools cannot evaluate reading order, meaningful link text, appropriate use of\nheading levels, table header associations, or the accessibility of form fields,\namong other criteria.\n\n### Further reading\n\n- [PDF Accessibility Checklist – Canada.ca](https://a11y.canada.ca/en/pdf-accessibility-checklist/)\n  — a practical, human-centred checklist for evaluating PDF accessibility.\n- [Tagged PDF Q\u0026A – PDF Association](https://pdfa.org/resource/tagged-pdf-q-a/)\n  — authoritative answers on PDF tagging from the PDF standards body.\n- [PDF Accessibility (pdfa11y) – Drupal.org](https://www.drupal.org/project/pdfa11y)\n  — a Drupal module that validates PDFs for accessibility before they are published on Drupal sites.\n  Its checks (tagging, title, language, alternate text for figures) inform the approach taken by\n  this project.  We do not duplicate the module but draw on the same underlying standards\n  (WCAG 2.x / PDF/UA) to complement it.\n- [asap_pdf – Code for America](https://github.com/codeforamerica/asap_pdf)\n  — a Rails + Python application for navigating large-scale PDF accessibility audits.  It uses\n  NLP/ML to classify documents by category (agendas, reports, policies, forms, etc.) and\n  LLM-powered tools to summarise documents and suggest remediation priorities.  This project\n  draws on asap_pdf's document-category taxonomy — the same labels are used by the rule-based\n  `DocCategory` classifier added here — but does not require the Rails application, a trained\n  ML model, or LLM API keys.  For organisations that need ML-based classification, dashboard\n  workflows, and LLM summaries, asap_pdf is the recommended complementary tool.\n\n---\n\n## Frequently asked questions\n\n**How does this tool compare to commercial PDF accessibility checkers such as\nClarity by CommonLook?**\n\nThis project is a free, open-source proof-of-concept built on\n[simplA11yPDFCrawler](https://github.com/accessibility-luxembourg/simplA11yPDFCrawler)\n(MIT) and [veraPDF](https://verapdf.org/).  It is not intended to replace\ncommercial tools that offer deeper validation, remediation workflows, or\ndedicated support.  Commercial tools typically cover a wider range of checks,\nintegrate with document-authoring workflows, and carry vendor support\nagreements.  This tool is best suited for a quick, no-cost, first-pass audit\nacross many PDFs at once.\n\n**How accurate is the tool at detecting PDF accessibility problems?**\n\nThe automated checks cover a specific subset of machine-detectable criteria\n(tagging, language, title, bookmarks, image alt text, encryption — see\n*What it does* above).  They can surface common structural issues quickly and\nat scale, but they cannot replace a manual review.  Automated tools cannot\nevaluate reading order, meaningful link text, appropriate heading levels, table\nheader associations, or the accessibility of form fields, among other criteria.\nA passing result means the document meets this basic set of machine-testable\ncriteria — it does not certify full accessibility.  Running\n[veraPDF](https://verapdf.org/) separately provides a more rigorous conformance\ncheck against PDF/A and PDF/UA standards.\n\n**Can the tool scan multiple domains?**\n\nNot in its current form — each scan request targets a single domain.  The\narchitecture (GitHub Actions + YAML manifest) could be extended to support\nmulti-domain batching, and submitting multiple separate `SCAN:` issues is the\nsimplest workaround today.  This project is not set up to replace commercial\nproviders, but wider-scale scanning is a potential future direction.\n\n**Can WCAG or Section 508 conformance levels be configured?**\n\nNot yet.  The checks are mapped to specific WCAG 2.x success criteria (see the\ntable in *What it does*), but there is no option to restrict or expand checks\nby conformance level.  The tool exposes what the underlying open-source\ncheckers provide; targeted conformance-level filtering is a potential future\nenhancement.\n\n**Why are some PDFs on a site not found?**\n\nPDFs hosted on a different domain from the one being crawled may not be\ndiscovered.  The spider follows links within the target domain; PDF links that\npoint to an external domain are not followed in the current implementation.\n\n---\n\n## Quick start\n\n### 1 – Submit a crawl via the web form\n\nOpen the [PDF Crawler form](https://mgifford.github.io/pdf-crawler/#quick-start),\nenter a URL, and click **Submit Crawl Request**.  You will be taken to GitHub\nwith the issue title pre-filled as `SCAN: https://…` — just click\n*Submit new issue* to start the crawl.\n\nThe `SCAN:` prefix triggers the *Crawl Site for PDFs* workflow automatically.\nThe workflow will post a comment when the crawl starts and another comment with\nthe full accessibility report links when analysis is complete.\n\n\u003e **Note:** Issues are processed when **opened** or **reopened**.  Editing the\n\u003e issue body will not re-trigger a scan, so there is no risk of accidental\n\u003e recurring scans.  The legacy `PDF-CRAWL:` prefix is still accepted for\n\u003e backward compatibility.\n\n#### Restarting a failed scan\n\nIf a crawl fails (the issue is labelled `scan-failed`), you can restart it by\n**closing and then reopening** the issue.  The crawler will pick up the\n`reopened` event and start a fresh crawl.\n\n#### Abandoned scans\n\nOccasionally the analysis step does not start after a successful crawl.  This\ncan happen when many issues are opened at the same time and GitHub's\n`workflow_run` event queue becomes saturated.  The scan then appears stuck with\na `scan-in-progress` label indefinitely.\n\nThe **4 – Rescue Abandoned Scans** workflow detects these stuck issues\nautomatically:\n\n* It runs **every day at 06:00 UTC**.\n* It can also be triggered manually via\n  **Actions → 4 – Rescue Abandoned Scans → Run workflow**.\n* Any issue that has been in `scan-in-progress` for more than **3 hours**\n  without a `scan-complete` or `scan-failed` label is marked `scan-failed`\n  and a comment is posted explaining what happened and how to retry.\n* Issues that carry *both* `scan-in-progress` and `scan-complete` (a stale\n  label left over from an earlier run) are silently tidied up.\n\n#### Issue lifecycle\n\n| Label | Meaning |\n|-------|---------|\n| `scan-in-progress` | Crawl or analysis is currently running |\n| `scan-failed` | The crawl workflow failed; reopen the issue to retry |\n| `scan-complete` | Analysis finished and reports have been generated |\n\nIssues are **automatically closed** once the accessibility report is posted.\n\n### 2 – Submit a crawl manually\n\n[Open a new issue](https://github.com/mgifford/pdf-crawler/issues/new)\nand set the title to:\n\n```\nSCAN: https://example.com\n```\n\n### 3 – Trigger manually\n\nGo to **Actions → 1 – Crawl Site for PDFs → Run workflow** and enter the URL\nyou want to crawl.\n\nOnce the crawl finishes, the *2 – Analyse PDFs for Accessibility* workflow\nstarts automatically.  You can also trigger it manually.\n\n---\n\n## Limiting crawl scope\n\nLarge sites or sites that serve large PDFs can cause the crawl job to time out\n(the hard limit is **75 minutes**).  Use the options below to keep jobs within\nthat budget.\n\n### Setting a page cap via the issue body\n\nAdd a `Number:` line to the body of your `SCAN:` issue to cap the maximum\nnumber of pages (URLs) the spider will visit:\n\n```\nSCAN: https://example.com\n\nNumber: 200\n```\n\nThe default is **2,500 pages**.  For sites that are large or slow, start with\na lower value such as 200–500 and increase it on subsequent scans once you have\na feel for the site's size.\n\n\u003e **Tip:** The web form on the [PDF Crawler page](https://mgifford.github.io/pdf-crawler/#quick-start)\n\u003e has a *Max pages* field that inserts the `Number:` line for you.\n\n### Setting a page cap via workflow dispatch\n\nWhen triggering the workflow manually (**Actions → 1 – Crawl Site for PDFs →\nRun workflow**) you can set:\n\n| Input | Default | Description |\n|-------|---------|-------------|\n| `max_pages` | `2500` | Maximum number of URLs/pages to visit |\n| `timeout` | `3600` | Maximum crawl time in seconds (1 hour) |\n\n### What happens when a crawl times out\n\nIf the job is cancelled because it exceeded the 75-minute limit:\n\n1. The workflow automatically **halves the page cap** (minimum: 100 pages) and\n   writes the new `Number:` value back into the issue body.\n2. A comment is posted on the issue explaining what happened and showing the\n   new cap.\n3. **Close and reopen** the issue to retry the crawl with the smaller batch.\n\nRepeat this process until the crawl completes within the time limit.\n\n### Tips for large or slow sites\n\n* **Start small** – use `Number: 100` for an initial probe, then increase.\n* **Check the workflow logs** – the *Run PDF crawler* step shows how many pages\n  were visited and how many PDFs were found before the timeout.\n* **Large PDFs slow analysis** – even a crawl of 50 pages can time out during\n  the *Analyse PDFs* step if the individual files are very large.  The\n  `scan-failed` label signals an analysis failure; reopen the issue to retry.\n* **Sequential queue** – if multiple scans are queued, use\n  **Actions → 3 – Process Scan Queue** to run them one-at-a-time instead of\n  triggering them all simultaneously.\n\n---\n\n## Workflows\n\n| Workflow | File | Trigger |\n|----------|------|---------|\n| Crawl Site for PDFs | `.github/workflows/crawl.yml` | Manual dispatch or issue opened/reopened with `SCAN:` title (legacy: `PDF-CRAWL:`) |\n| Analyse PDFs for Accessibility | `.github/workflows/analyse.yml` | After crawl succeeds, or manual dispatch |\n| Process Scan Queue | `.github/workflows/process_scan_queue.yml` | Manual dispatch |\n| Rescue Abandoned Scans | `.github/workflows/rescue_abandoned_scans.yml` | Daily at 06:00 UTC, or manual dispatch |\n\n---\n\n## Output files\n\n| File | Description |\n|------|-------------|\n| `reports/manifest.yaml` | YAML tracking file – one entry per PDF |\n| `reports/report.md` | Human-readable Markdown report |\n| `reports/report.json` | Machine-readable JSON report |\n| `reports/report_structured.json` | Structured JSON report with per-rule categories and compatibility mode |\n\nSee [`reports/README.md`](reports/README.md) for the full manifest schema.\n\n---\n\n## Local development\n\n```bash\n# Clone the repo\ngit clone https://github.com/mgifford/pdf-crawler.git\ncd pdf-crawler\n\n# Install uv (recommended: standalone installer)\ncurl -LsSf https://astral.sh/uv/install.sh | sh\n\n# Create/update virtual environment from lockfile\nuv sync --frozen --dev\n\n# Install dependencies\n# (Legacy fallback)\n# pip install -r requirements.txt\n\n# Crawl a site (runs for up to 1 hour by default)\nuv run python scripts/crawl.py --url https://example.com\n\n# Analyse the downloaded PDFs\nuv run python scripts/pdf_analyser.py\n\n# Generate reports\nuv run python scripts/generate_report.py\n\n# Validate a single PDF (structured JSON output by default)\nuv run python scripts/validate_pdf.py path/to/file.pdf\n\n# Validate a single PDF (raw analyser output)\nuv run python scripts/validate_pdf.py path/to/file.pdf --mode raw\n```\n\n---\n\n## Behavior-driven development (BDD)\n\nBehavior expectations are documented in:\n\n- `FEATURES.md` (user stories, acceptance criteria, governance, traceability)\n- `features/` (canonical Gherkin scenarios)\n\n### BDD command set\n\n```bash\n# Install BDD dependencies\nnpm install\n\n# Run UI smoke behavior checks (for PR-level confidence)\nnpm run bdd:ui:smoke\n\n# Run full UI behavior checks\nnpm run bdd:ui:all\n\n# Ensure non-UI Gherkin specs remain parseable\nnpm run bdd:spec:lint\n\n# Run full domain regression suite\nuv run python -m pytest tests/ -v\n```\n\nSustainability alignment:\n- PR checks run smoke-tagged UI scenarios only (`@ui @smoke`).\n- Full matrix runs on schedule/manual and relevant-path main-branch updates.\n- Existing pytest coverage remains the source of executable domain behavior.\n\n---\n\n## Architecture\n\n```\npdf-crawler/\n├── .github/\n│   └── workflows/\n│       ├── crawl.yml          # Step 1: crawl a site for PDFs\n│       └── analyse.yml        # Step 2: analyse PDFs for accessibility\n├── docs/\n│   └── index.html             # GitHub Pages submission form\n├── scripts/\n│   ├── pdf_spider.py          # Scrapy spider (downloads PDF files)\n│   ├── crawl.py               # Crawl wrapper + manifest update\n│   ├── manifest.py            # YAML manifest management (MD5 dedup)\n│   ├── pdf_analyser.py        # Accessibility checks (pikepdf-based)\n│   ├── generate_report.py     # Markdown + JSON report generator\n│   └── validate_pdf.py        # Single-PDF validator (raw/structured JSON)\n├── reports/\n│   ├── README.md              # Manifest schema docs\n│   ├── manifest.yaml          # ← committed; grows over time\n│   ├── report.md              # ← committed; regenerated each run\n│   ├── report.json            # ← committed; regenerated each run\n│   └── report_structured.json # ← committed; regenerated each run\n├── pyproject.toml             # Python project metadata and dependency groups\n├── uv.lock                    # Locked Python dependencies for reproducible installs\n├── requirements.txt           # Compatibility export of Python dependencies\n└── README.md\n```\n\n---\n\n## AI Disclosure\n\nThis section documents all AI tools used in this project.  Transparency about\nAI involvement is a core commitment — see [SUSTAINABILITY.md](./SUSTAINABILITY.md)\nfor the full AI usage policy.\n\n### Building the project\n\nThe following LLMs were used during development and are the only AI tools known\nto have been applied to this repository:\n\n| LLM / tool | Provider | Used for |\n|------------|----------|----------|\n| GitHub Copilot (GPT-4-class) | GitHub / OpenAI | Code suggestions, CI workflow improvements, PR support |\n| GPT-4-class models via Copilot Chat | GitHub / OpenAI | Content drafting, structural editing, documentation |\n| Claude (Anthropic) | Anthropic via GitHub Copilot Coding Agent | Automated issue resolution and code changes via the GitHub Copilot coding agent |\n\nEach use involved human review and editing before the output was merged.\n\n### Runtime AI usage\n\n**No AI runs automatically at runtime.**  When a crawl or analysis job executes,\nall processing is performed by deterministic Python scripts\n(`pdf_spider.py`, `pdf_analyser.py`, `generate_report.py`).  No LLM is called\nduring a scan.\n\n### Browser-based AI\n\n**No browser-based AI is enabled.**  The `docs/index.html` submission form is\na static HTML page with no runtime AI features.  Browser built-in AI APIs (if\nsupported by the visitor's browser) are not activated by this page.  Any future\nuse of browser AI would require explicit user opt-in per the AI usage policy in\n[SUSTAINABILITY.md](./SUSTAINABILITY.md).\n\n---\n\n## Credits\n\n- Accessibility checks are based on\n  [simplA11yPDFCrawler](https://github.com/accessibility-luxembourg/simplA11yPDFCrawler)\n  by [SIP Luxembourg](https://sip.gouvernement.lu/en.html) (MIT licence).\n- Architecture inspired by [mgifford/open-scans](https://github.com/mgifford/open-scans).\n\n---\n\n## Licence\n\nBuilding on simplA11yPDFCrawler, this is released under the MIT.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmgifford%2Fpdf-crawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmgifford%2Fpdf-crawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmgifford%2Fpdf-crawler/lists"}