https://github.com/retospect/acatome-extract
PDF extraction pipeline for acatome — Marker/fitz, metadata, block chunking
https://github.com/retospect/acatome-extract
document-parsing extraction pdf python scientific-papers
Last synced: 3 months ago
JSON representation
PDF extraction pipeline for acatome — Marker/fitz, metadata, block chunking
- Host: GitHub
- URL: https://github.com/retospect/acatome-extract
- Owner: retospect
- License: other
- Created: 2026-03-10T18:33:06.000Z (5 months ago)
- Default Branch: main
- Last Pushed: 2026-05-07T17:56:44.000Z (3 months ago)
- Last Synced: 2026-05-11T10:56:56.760Z (3 months ago)
- Topics: document-parsing, extraction, pdf, python, scientific-papers
- Language: Python
- Size: 248 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
- Security: SECURITY.md
Awesome Lists containing this project
README
# acatome-extract
PDF extraction and enrichment pipeline for scientific papers. Converts PDFs into structured, searchable bundles with block-level summaries and embeddings.
## Features
- **Marker PDF extraction** — structured block extraction with headings, tables, figures
- **Fitz fallback** — recursive character chunking when Marker is unavailable
- **LLM enrichment** — block and paper summaries via Ollama or litellm
- **Embeddings** — sentence-transformer embeddings for semantic search
- **File watcher** — `acatome-extract watch` monitors an inbox folder
- **Bundle format** — `.acatome` companion files for sharing pre-built extractions
- **CLI** — `acatome-extract` command for extract, enrich, and watch workflows
## Installation
```bash
uv pip install -e .
```
On **macOS/Linux** this includes Marker for structured PDF extraction.
On **Windows** it installs with the lighter pymupdf (fitz) backend by default.
To add Marker on Windows (requires C build tools):
```bash
uv pip install -e ".[marker]"
```
With GPU acceleration (embeddings + torch):
```bash
uv pip install -e ".[gpu]"
```
Everything at once:
```bash
uv pip install -e ".[full]"
```
## Usage
```python
from acatome_extract.pipeline import extract
bundle = extract("/path/to/paper.pdf")
```
## CLI
```bash
# Extract (RAKE summaries included automatically, no LLM needed)
acatome-extract extract paper.pdf
acatome-extract extract --type datasheet TI_LM317.pdf # non-article types
# Enrich — embeddings only by default; add --summarize for LLM summaries
acatome-extract enrich /path/to/bundle
acatome-extract enrich --summarize /path/to/bundle # enable LLM summaries
acatome-extract enrich --summarize --skip-existing dir/ # incremental LLM pass
# Watch — extract + embed + ingest; LLM summaries off by default
acatome-extract watch ~/papers/inbox
acatome-extract watch ~/papers/inbox --summarize # enable LLM summaries
# Migrate old bundles to new summaries dict format + add RAKE
acatome-extract migrate ~/.acatome/papers
acatome-extract migrate ~/.acatome/papers --dry-run # preview changes
# Supplements
acatome-extract attach parent-slug supplement.pdf --name s1
```
### Summaries
Extraction always generates **RAKE** (extractive keyword) summaries — instant, no LLM required. LLM-based summaries are opt-in via `--summarize` and require an Ollama or litellm-compatible model.
RAKE summaries are used as the default for search and display. To add LLM summaries later:
```bash
acatome-extract enrich --summarize --skip-existing ~/.acatome/papers
```
### Sidecar metadata
Place a `.meta.json` alongside any PDF to override metadata:
```json
{"type": "datasheet", "title": "LM317 Regulator", "author": "Texas Instruments", "year": 2022}
```
Supported fields: `type`, `title`, `author` (string or list), `year`, `doi`,
`abstract`, `journal`, `s2_id`, `arxiv_id`, `verified`.
Use **explicit `null`** to clear a field that the upstream Crossref/S2 lookup
got wrong — empty strings are ignored for backward compatibility:
```json
{"doi": "10.1002/9781118519301.ch5", "s2_id": null, "verified": true}
```
When `"verified": true` is set, the fuzzy-title verification gate is
bypassed — useful for PDFs where the real title doesn't appear on page 1
(Elsevier header strips, ACS internal production PDFs, abstract collections).
## Dependencies
- **acatome-meta** — metadata lookup and verification
- **marker-pdf** — structured PDF extraction
- **litellm** / **Ollama** — LLM-based enrichment
## Testing
```bash
uv run python -m pytest tests/ -v
```
## License
GPL-3.0-or-later — see [LICENSE](LICENSE).