{"id":51861536,"url":"https://github.com/lancedb/liteparse-lancedb-pdf-qa","last_synced_at":"2026-07-24T08:01:17.607Z","repository":{"id":367650347,"uuid":"1281728314","full_name":"lancedb/liteparse-lancedb-pdf-qa","owner":"lancedb","description":"Turn messy, visual PDF reports into a local, inspectable multimodal evidence store in LanceDB","archived":false,"fork":false,"pushed_at":"2026-06-30T18:11:32.000Z","size":221,"stargazers_count":8,"open_issues_count":0,"forks_count":3,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-12T00:18:11.183Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lancedb.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2026-06-26T21:29:32.000Z","updated_at":"2026-07-09T17:23:42.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/lancedb/liteparse-lancedb-pdf-qa","commit_stats":null,"previous_names":["lancedb/liteparse-lancedb-pdf-qa"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/lancedb/liteparse-lancedb-pdf-qa","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lancedb%2Fliteparse-lancedb-pdf-qa","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lancedb%2Fliteparse-lancedb-pdf-qa/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lancedb%2Fliteparse-lancedb-pdf-qa/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lancedb%2Fliteparse-lancedb-pdf-qa/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lancedb","download_url":"https://codeload.github.com/lancedb/liteparse-lancedb-pdf-qa/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lancedb%2Fliteparse-lancedb-pdf-qa/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35832970,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-07-20T02:08:10.276Z","status":"online","status_checked_at":"2026-07-24T02:00:07.870Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-07-24T08:01:16.737Z","updated_at":"2026-07-24T08:01:17.570Z","avatar_url":"https://github.com/lancedb.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# LiteParse + LanceDB: a local multimodal PDF evidence store\n\nA small, fully reproducible demo that turns messy, visual ESG reports into a local,\ninspectable evidence store and then benchmarks retrieval over it. It is the companion code\nfor the blog post *Building a Local Multimodal PDF Evidence Store with LiteParse and LanceDB*.\n\nThe pipeline runs entirely on your machine:\n\n```text\nPDF reports\n  -\u003e LiteParse (Python SDK)   text + extracted figures + page screenshots\n  -\u003e normalized records       documents / pages / chunks / assets, keyed by page\n  -\u003e LanceDB                  text, blobs, vectors, and metadata in one store\n  -\u003e retrieval modes          chunks / pages / assets / images / hybrid_bundle\n  -\u003e optional answer agent    PydanticAI answer + LLM judge, reading image bytes from LanceDB\n```\n\n[LiteParse](https://github.com/run-llama/liteparse) (by LlamaIndex) parses each PDF through its\nnative Python SDK — a Rust core with no cloud calls, no LLMs, and no API keys. [LanceDB](https://github.com/lancedb/lancedb)\nstores the text, the page screenshots and extracted figures (as blobs), the embeddings, and the\nprovenance in a single multimodal store, with indexes and versioning alongside the data.\n\n## The dataset\n\nA six-report subset of [Climate Finance Bench](https://github.com/Pladifes/climate_finance_bench),\nwith 50 page-labeled questions:\n\n| Company | Report | Pages | Questions |\n|---|---|---:|---:|\n| Ali Baba Group | 2024 ESG Report | 200 | 10 |\n| Google | 2024 Environmental Report | 86 | 9 |\n| NVIDIA | FY2024 Corporate Sustainability Report | 41 | 8 |\n| Nestle | 2023 Creating Shared Value \u0026 Sustainability | 89 | 8 |\n| Samsung | 2024 Sustainability Report | 83 | 7 |\n| Total Energies | 2024 Sustainability \u0026 Climate Progress | 112 | 8 |\n\nFor fast iteration the experiments parse only the benchmark-labeled pages (70 of 611). To run on\nfull documents, drop the `--pages labeled` filter (use `--pages all`); the pipeline is otherwise\nidentical.\n\n## What lands in LanceDB\n\nFive tables, all keyed by a stable `page_id` (`{doc_id}:p{page_num}`):\n\n| Table | Rows | Holds |\n|---|---:|---|\n| `documents` | 6 | report metadata, checksum, parse config, timings |\n| `pages` | 70 | full page text, page screenshot (blob), text + CLIP image vectors |\n| `chunks` | 252 | page-bounded text spans and their text vectors |\n| `assets` | 77 | extracted figures, their bytes (blob), text + CLIP image vectors |\n| `eval_questions` | 50 | normalized questions and their expected pages |\n\nIndexes are scalar `BTREE` on the prefilter columns (`company`, `source_pdf`) and an `FTS` index\non `text`. At this scale LanceDB runs an exact vector search; an ANN index is only built once a\ntable grows past a few hundred rows.\n\nRetrieval exposes five modes: `chunks`, `pages`, and `assets` (text-vector search), `images`\n(CLIP text-to-image search over the page and figure image vectors), and `hybrid_bundle` (pools the\nthree text searches and merges candidates by `page_id`).\n\n## Setup\n\nRequires Python 3.13+ and [uv](https://docs.astral.sh/uv/).\n\n**1. Install the environment.** `uv sync` reads `pyproject.toml`/`uv.lock` and creates a `.venv`\nwith every dependency (`liteparse`, `lancedb`, `pylance`, `open-clip-torch`, `openai`,\n`pydantic-ai`, …):\n\n```bash\nuv sync\n```\n\nEach `uv run …` below also installs on first use, so this step is optional — but it makes the\nfirst run explicit and surfaces any install issues up front. No system packages are needed for\nthese born-digital PDFs: LiteParse bundles PDFium and OCR is off.\n\n**2. Add your OpenAI API key.** The OpenAI text embeddings and the answer/judge models need a key.\n(Skip this if you only want the offline smoke test described at the end.)\n\n```bash\ncp .env.example .env          # then edit .env and set OPENAI_API_KEY\nset -a \u0026\u0026 source .env \u0026\u0026 set +a\n```\n\n## Reproduce the results\n\nEach step writes its outputs and metrics to disk. Generated data under `data/` is gitignored.\n\n**1. Download the dataset.** Pulls the six reports and the benchmark questions live from the public\n[Climate Finance Bench](https://github.com/Pladifes/climate_finance_bench) GitHub repo — no account,\nAPI token, or manual step needed (the script uses only the Python standard library, so it even runs\nbefore `uv sync`):\n\n```bash\nuv run python scripts/download_climate_finance_bench.py \\\n  --companies Samsung NVIDIA Google \"Ali Baba Group\" Nestle \"Total Energies S.A\" \\\n  --out data/raw/climate_finance_bench\n```\n\nIt fetches the benchmark JSON and normalizes the matching questions into\n`data/eval/selected_questions.jsonl` (the 50 page-labeled questions), discovers each company's report\nPDF through the GitHub API and downloads it under `data/raw/climate_finance_bench/`, then writes a\n`manifest.json` recording the source repo, ref, and a SHA256 for every file. The download is\nidempotent — existing files are skipped unless you pass `--force`. Pass `--ref \u003ccommit-sha\u003e` to pin an\nexact upstream commit; the default is `main`, so pin the SHA if you need bit-identical reproduction.\n\nOptionally sanity-check the download — counts of reports, pages, and labeled questions:\n\n```bash\nuv run python scripts/inspect_benchmark.py\n```\n\n**2. Parse with LiteParse** → normalized `documents`/`pages`/`chunks`/`assets` records, page\nscreenshots, and extracted figures:\n\n```bash\nuv run python scripts/parse_reports.py \\\n  --pages labeled \\\n  --out data/parsed/liteparse \\\n  --metrics results/extraction_performance.json\n```\n\n**3. Build the LanceDB store** with OpenAI text embeddings + OpenCLIP image embeddings:\n\n```bash\nuv run python scripts/build_lancedb.py \\\n  --parsed-dir data/parsed/liteparse \\\n  --db data/lancedb/esg_pdf_qa_openai_small_clip.lancedb \\\n  --overwrite \\\n  --embedding-backend openai --text-model text-embedding-3-small \\\n  --image-embedding-backend open-clip \\\n  --metrics results/storage_metrics_openai_small_clip.json\n```\n\n**4. Run the retrieval benchmark** across all five modes:\n\n```bash\nuv run python scripts/run_retrieval_eval.py \\\n  --db data/lancedb/esg_pdf_qa_openai_small_clip.lancedb \\\n  --embedding-backend openai --text-model text-embedding-3-small \\\n  --image-embedding-backend open-clip \\\n  --top-k 5 \\\n  --out data/eval/retrieval_results_openai_small_clip.jsonl \\\n  --metrics results/metrics_openai_small_clip.json\n```\n\n**5. (Optional) Answer-correctness eval** — a capstone that retrieves the `hybrid_bundle`, reads\nthe page screenshots straight from LanceDB blobs, sends them to an answer agent, and grades the\noutput with an LLM judge:\n\n```bash\nuv run python scripts/run_answer_eval.py \\\n  --db data/lancedb/esg_pdf_qa_openai_small_clip.lancedb \\\n  --answer-model gpt-5.4 --judge-model gpt-5.4-mini \\\n  --embedding-backend openai --text-model text-embedding-3-small \\\n  --out data/eval/answer_eval_results.jsonl \\\n  --metrics results/answer_eval_metrics.json\n```\n\nFor an offline smoke test (no API key), omit the OpenAI flags: the build and retrieval scripts fall\nback to a deterministic local hash embedder. Use `--limit N` on the eval scripts to run a subset.\n\n## Results\n\nAll numbers below are from the commands above on the 70 labeled pages.\n\n**Extraction** (LiteParse, labeled-page run):\n\n| Stage | Time | Throughput |\n|---|---:|---:|\n| `parse()` | 0.51 s | ~136 pages/s |\n| `screenshot()` | 1.77 s | ~40 pages/s |\n| End-to-end | 2.35 s | ~30 pages/s |\n\n**Storage** (OpenAI `text-embedding-3-small` + OpenCLIP `ViT-B-32`): write 0.20 s, index 0.10 s,\ntotal **114 MB**. Almost all of it is page screenshots in `pages` (101 MB); extracted figures in\n`assets` add 10 MB, and everything else is a couple of MB. For comparison, the six source PDFs are\n64 MB — the store is larger because it keeps full-resolution page renders on hand.\n\n**Retrieval** (top-5, page-label hits):\n\n| Mode | Any@5 | Cov@5 | All@5 | Modality@5 | P50 |\n|---|---:|---:|---:|---:|---:|\n| `hybrid_bundle` | 0.82 | 0.733 | 0.66 | 0.68 | 4.7 ms |\n| `pages` | 0.76 | 0.672 | 0.60 | 0.94 | 1.7 ms |\n| `images` | 0.76 | 0.609 | 0.48 | 0.90 | 17.9 ms |\n| `chunks` | 0.72 | 0.588 | 0.48 | 0.58 | 1.7 ms |\n| `assets` | 0.38 | 0.277 | 0.20 | 0.66 | 1.7 ms |\n\nThe `hybrid_bundle` leads on page recall because the page-keyed layout lets it pool chunk, page,\nand figure hits and let each page win on its best signal. The single visual modes (`pages`,\n`images`) lead on `modality_hit`, since they reliably return page-level or image evidence for the\ntable and figure questions.\n\n## Repository layout\n\n```text\nsrc/            config, benchmark, reports, parse, schema, embeddings, index (build), retrieval, eval, answer_eval\nscripts/        download / inspect / parse / build / run_retrieval_eval / summarize_retrieval_results / run_answer_eval\ndata/           raw PDFs, parsed records, LanceDB store, eval sets (gitignored)\nresults/        extraction, storage, and retrieval metrics (JSON)\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flancedb%2Fliteparse-lancedb-pdf-qa","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flancedb%2Fliteparse-lancedb-pdf-qa","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flancedb%2Fliteparse-lancedb-pdf-qa/lists"}