{"id":34617118,"url":"https://github.com/hummat/paperpipe","last_synced_at":"2026-05-21T09:01:09.594Z","repository":{"id":330183138,"uuid":"1120503046","full_name":"hummat/paperpipe","owner":"hummat","description":"Extract equations and context from research papers for LLM coding assistants (arXiv, LaTeX, RAG)","archived":false,"fork":false,"pushed_at":"2026-04-15T06:59:16.000Z","size":685,"stargazers_count":9,"open_issues_count":5,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-04-15T07:31:16.852Z","etag":null,"topics":["ai-coding-assistant","arxiv","cli","developer-tools","equation-extraction","latex","litellm","llm","paper-management","paperqa","python","rag","research-papers","scientific-research"],"latest_commit_sha":null,"homepage":"https://hummat.github.io/paperpipe/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hummat.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":".github/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2025-12-21T11:08:11.000Z","updated_at":"2026-04-15T06:59:22.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/hummat/paperpipe","commit_stats":null,"previous_names":["hummat/paperpipe"],"tags_count":31,"template":false,"template_full_name":null,"purl":"pkg:github/hummat/paperpipe","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hummat%2Fpaperpipe","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hummat%2Fpaperpipe/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hummat%2Fpaperpipe/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hummat%2Fpaperpipe/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hummat","download_url":"https://codeload.github.com/hummat/paperpipe/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hummat%2Fpaperpipe/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33295225,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-21T02:57:32.698Z","status":"ssl_error","status_checked_at":"2026-05-21T02:57:31.990Z","response_time":62,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-coding-assistant","arxiv","cli","developer-tools","equation-extraction","latex","litellm","llm","paper-management","paperqa","python","rag","research-papers","scientific-research"],"created_at":"2025-12-24T14:46:24.124Z","updated_at":"2026-05-21T09:01:09.586Z","avatar_url":"https://github.com/hummat.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# paperpipe\n\n![image](https://repository-images.githubusercontent.com/1120503046/8ba4e2ed-30ef-4d0d-996f-68a48962cb9b)\n\n**The problem:** You're implementing a paper. You need the exact equations, want to verify your code matches the math, and your coding agent keeps hallucinating details. Reading PDFs is slow; copy-pasting LaTeX is tedious.\n\n**The solution:** paperpipe maintains a local paper database with PDFs, LaTeX source (when available), extracted equations, and coding-oriented summaries. It integrates with coding agents (Claude Code, Codex, Gemini CLI) so they can ground their responses in actual paper content.\n\n## Typical workflow\n\n```bash\n# 1. Add papers you're implementing (multiple at once, mixed sources OK)\npapi add 2303.08813 1706.03762 \"Attention Is All You Need\"\n\n# Or one at a time\npapi add 2303.08813                    # LoRA paper\npapi add https://arxiv.org/abs/1706.03762  # URL\npapi add \"Attention Is All You Need\"   # Search by title\n\n# 2. Check what equations you need to implement\npapi show lora --level eq             # prints equations to stdout\n\n# 3. Verify your code matches the paper\n#    (or let your coding agent do this via the /papi skill)\npapi show lora --level tex            # exact LaTeX definitions\n\n# 4. Ask cross-paper questions (requires RAG backend)\npapi ask \"How does LoRA differ from full fine-tuning in terms of parameter count?\"\n\n# 5. Keep implementation notes\npapi notes lora                       # opens notes.md in $EDITOR\n```\n\n## Installation\n\n```bash\n# Basic (uv recommended)\nuv tool install paperpipe\n\n# With features\nuv tool install paperpipe --with \"paperpipe[llm]\"      # better summaries via LLMs\nuv tool install paperpipe --with \"paperpipe[paperqa]\"  # RAG via PaperQA2\nuv tool install paperpipe --with \"paperpipe[leann]\"    # local RAG via LEANN\nuv tool install paperpipe --with \"paperpipe[figures]\"  # figure extraction from LaTeX/PDF\nuv tool install paperpipe --with \"paperpipe[mcp]\"      # MCP server integrations (Python 3.11+)\nuv tool install paperpipe --with \"paperpipe[all]\"      # everything\n```\n\n\u003cdetails markdown=\"1\"\u003e\n\u003csummary\u003eAlternative: pip install\u003c/summary\u003e\n\n```bash\npip install paperpipe\npip install 'paperpipe[llm]'\npip install 'paperpipe[paperqa]'  # PaperQA2 + multimodal PDF parsing\npip install 'paperpipe[leann]'\npip install 'paperpipe[figures]'        # figure extraction from LaTeX/PDF\npip install 'paperpipe[mcp]'\npip install 'paperpipe[all]'\n```\n\u003c/details\u003e\n\n\u003cdetails markdown=\"1\"\u003e\n\u003csummary\u003eFrom source\u003c/summary\u003e\n\n```bash\ngit clone https://github.com/hummat/paperpipe \u0026\u0026 cd paperpipe\npip install -e \".[all]\"\n```\n\u003c/details\u003e\n\n## What paperpipe stores\n\n```\n~/.paperpipe/                         # override with PAPER_DB_PATH\n├── index.json\n├── .pqa_papers/                      # staged PDFs for RAG (created on first `papi ask`)\n├── .pqa_index/                       # PaperQA2 index cache\n├── .leann/                           # LEANN index cache\n├── papers/\n│   └── lora/\n│       ├── paper.pdf                 # for RAG backends\n│       ├── source.tex                # full LaTeX (if available from arXiv)\n│       ├── equations.md              # extracted equations with context\n│       ├── summary.md                # coding-oriented summary\n│       ├── tldr.md                   # one-paragraph TL;DR\n│       ├── meta.json                 # metadata + tags\n│       ├── notes.md                  # your implementation notes\n│       └── figures/                  # extracted figures (if available)\n│           ├── figure1.png\n│           └── figure2.pdf\n```\n\n**Why this structure matters:**\n- `equations.md` — Key equations with variable definitions. Use for code verification.\n- `source.tex` — Original LaTeX. Use when you need exact notation or the equation extraction missed something.\n- `summary.md` — High-level overview focused on implementation (not literature review). Use for understanding the approach.\n- `tldr.md` — Quick 2-3 sentence overview of the paper's contribution.\n- `figures/` — Architecture diagrams, network structures, and result plots extracted from LaTeX source or PDF.\n- `.pqa_papers/` — Staged PDFs only (no markdown) so RAG backends don't index generated content.\n\n## Core commands\n\n| Command | Purpose |\n|---------|---------|\n| `papi add \u003cid-or-url-or-title\u003e...` | Add one or more papers (downloads PDF + LaTeX, generates summary/equations/TL;DR) |\n| `papi add --pdf file.pdf` | Add a local PDF or URL |\n| `papi add --from-file list.json` | Import papers from a JSON list or text file |\n| `papi list` | List papers (filter with `--tag`) |\n| `papi search \"query\"` | Search across titles, tags, summaries, equations (`--rg` for grep-style, `-p paper1,paper2` to limit scope) |\n| `papi index --backend search` | Build/update ranked search index (`search.db`) |\n| `papi show \u003cpaper\u003e --level eq` | Print equations (best for agent sessions) |\n| `papi show \u003cpaper\u003e --level tex` | Print LaTeX source |\n| `papi show \u003cpaper\u003e --level summary` | Print summary |\n| `papi show \u003cpaper\u003e --level tldr` | Print TL;DR |\n| `papi export \u003cpapers...\u003e --to ./dir` | Export context files into a repo (`--level summary\\|equations\\|full`) |\n| `papi notes \u003cpaper\u003e` | Open/print implementation notes |\n| `papi regenerate \u003cpapers...\u003e` | Regenerate summary/equations/tags/TL;DR |\n| `papi remove \u003cpapers...\u003e` | Remove papers |\n| `papi ask \"question\"` | Cross-paper RAG query (requires PaperQA2 or LEANN) |\n| `papi index` | Build/update the retrieval index |\n| `papi tags` | List all tags (`--audit` to find duplicates, `--merge OLD NEW`, `--delete TAG`) |\n| `papi path` | Print database location |\n| `papi docs` | Print agent integration snippet (for CLAUDE.md/AGENTS.md) |\n| `papi rebuild-index` | Rebuild index.json from on-disk paper directories (recovery) |\n\nRun `papi --help` or `papi \u003ccommand\u003e --help` for full options.\n\n## Import/Export\n\nShare your paper collection with others or back it up.\n\n**Export:**\n```bash\n# Export full list to JSON\npapi list --json \u003e my_papers.json\n\n# Export specific tag\npapi list --tag \"computer-vision\" --json \u003e cv_papers.json\n```\n\n**Import:**\n```bash\n# Import from JSON (preserves custom names and tags)\npapi add --from-file my_papers.json\n\n# Import from text file (one arXiv ID per line)\npapi add --from-file paper_ids.txt --tags \"imported\"\n\n# Import from BibTeX file (requires bibtexparser)\npapi add --from-file papers.bib\n# or install with BibTeX support:\n# uv tool install paperpipe --with \"paperpipe[bibtex]\"\n```\n\n**Title Search:**\n```bash\n# Add papers by title (auto-selects if high confidence match)\npapi add \"Attention Is All You Need\"\npapi add \"NeRF: Representing Scenes as Neural Radiance Fields\"\n```\n\n**Semantic Scholar Support:**\n```bash\n# Add papers from Semantic Scholar\npapi add https://www.semanticscholar.org/paper/...\npapi add 0123456789abcdef0123456789abcdef01234567  # S2 paper ID\n```\n\n**Multiple papers at once** (mixed sources OK):\n```bash\npapi add 2303.08813 1706.03762 \"Attention Is All You Need\"\npapi add 2303.08813 https://www.semanticscholar.org/paper/... \"NeRF\"\n```\n\nExact text search (fast, no LLM required):\n\n```bash\npapi search --rg \"AdamW\"              # case-insensitive, literal string (default)\npapi search --rg --case-sensitive \"NeRF\"  # match exact case\npapi search --rg --regex \"Eq\\\\. [0-9]+\"   # regex mode (opt-in)\n```\n\nRanked search (BM25 via SQLite FTS5, no LLM required):\n\n```bash\npapi index --backend search --search-rebuild    # builds \u003cpaper_db\u003e/search.db\npapi search \"surface reconstruction\"             # uses FTS if available (default)\npapi search --no-fts \"surface reconstruction\"    # force in-memory scan (disables FTS, uses fuzzy matching)\npapi search --no-fts --exact \"exact phrase\"      # force scan with exact matching only\n```\n\nHybrid ranked+exact search:\n\n```bash\npapi search --hybrid \"surface reconstruction\"\npapi search --hybrid --show-grep-hits \"surface reconstruction\"\n```\n\nLimit search to specific papers:\n\n```bash\npapi search \"attention\" -p attention-is-all-you-need\npapi search \"loss\" -p paper1,paper2,paper3\n```\n\n### What are FTS and BM25?\n\n- **FTS** = *Full-Text Search*. Here it means SQLite’s FTS5 extension, which builds an inverted index so searches don’t\n  have to rescan every file on every query.\n- **BM25** = *Okapi BM25*, a standard relevance-ranking function used by many search engines. It ranks results based on\n  term frequency, inverse document frequency, and document length normalization.\n\nReferences (external):\n```text\nhttps://sqlite.org/fts5.html\nhttps://en.wikipedia.org/wiki/Okapi_BM25\n```\n\n\u003cdetails markdown=\"1\"\u003e\n\u003csummary\u003eGlossary (RAG, embeddings, MCP, LiteLLM)\u003c/summary\u003e\n\n- **RAG** = retrieval‑augmented generation: retrieve relevant paper passages first, then generate an answer grounded in\n  those passages.\n- **Embedding model** = turns text into vectors for semantic search; changing it usually requires rebuilding an index.\n- **LiteLLM model id** = the model string you pass to LiteLLM (provider/model routing), e.g. `gpt-4o`, `gemini/...`,\n  `ollama/...`.\n- **MCP** = Model Context Protocol: lets tools/agents call into paperpipe’s retrieval helpers (e.g. “retrieve chunks”)\n  without copying PDFs into the chat.\n- **Staging dir** (`.pqa_papers/`) = PDF-only mirror used so RAG backends don’t index generated Markdown.\n\n\u003c/details\u003e\n\n\u003cdetails markdown=\"1\"\u003e\n\u003csummary\u003eConfig: default search mode\u003c/summary\u003e\n\nSet a default for `papi search` (CLI flags still win):\n\n```bash\nexport PAPERPIPE_SEARCH_MODE=auto   # auto|fts|scan|hybrid\n```\n\nOr in `config.toml`:\n\n```toml\n[search]\nmode = \"auto\" # auto|fts|scan|hybrid\n```\n\n\u003c/details\u003e\n\n## Agent integration\n\npaperpipe is designed to work with coding agents. Install the skill and MCP servers:\n\n```bash\npapi install                          # installs skill + MCP for detected CLIs\n# or be specific:\npapi install skill --claude --codex --gemini\npapi install mcp --claude --codex --gemini\n```\n\nAfter installation, your agent can:\n- Use `/papi` to get paper context (skill)\n- Call MCP tools like `retrieve_chunks` for RAG retrieval\n- Verify code against paper equations\n\n### Custom skills\n\n| Skill | Description |\n|--------|-------------|\n| `/papi` | Route questions to the cheapest papi command |\n| `/papi-init` | Add/update PaperPipe integration in your project's AGENTS.md/CLAUDE.md |\n| `/verify-with-paper` | Verify code against paper equations |\n| `/ground-with-paper` | Ground responses in paper excerpts |\n| `/compare-papers` | Compare multiple papers for a decision |\n| `/curate-paper-note` | Create a project note from paper excerpts |\n\nFor a ready-to-paste snippet for your repo's agent instructions, run `papi docs` or see [AGENT_INTEGRATION.md](AGENT_INTEGRATION.md).\n\n### What the agent sees\n\nWhen you (or your agent) run `papi show \u003cpaper\u003e --level eq`, you get structured output like:\n\n```markdown\n## Equation 1: LoRA Update\n$$h = W_0 x + \\Delta W x = W_0 x + BA x$$\nwhere:\n- $W_0 \\in \\mathbb{R}^{d \\times k}$: pretrained weight matrix (frozen)\n- $B \\in \\mathbb{R}^{d \\times r}$, $A \\in \\mathbb{R}^{r \\times k}$: low-rank matrices\n- $r \\ll \\min(d, k)$: the rank (typically 1-64)\n```\n\nThis is what makes verification possible — the agent can compare your code symbol-by-symbol.\n\n\u003cdetails markdown=\"1\"\u003e\n\u003csummary\u003eMCP server setup (manual)\u003c/summary\u003e\n\n### MCP servers\n\npaperpipe provides MCP servers for retrieval-only workflows:\n- **PaperQA2 retrieval**: raw chunks + citations (via `paperqa_mcp`)\n- **LEANN search**: fast semantic search over papers (via `leann_mcp`)\n\nMCP servers are configured automatically when you run `papi install mcp`. The install command creates the appropriate configuration files for your agent (Claude Code, Codex CLI, or Gemini CLI).\n\n**Installation**:\n```bash\n# Install MCP servers for all supported agents (user scope)\npapi install mcp\n\n# Install for specific agents\npapi install mcp --claude\npapi install mcp --codex\npapi install mcp --gemini\n\n# Install repo-local MCP configs (Claude + Gemini) and Codex globally\npapi install mcp --repo\n\n# Customize embedding model\npapi install mcp --embedding text-embedding-3-small\n```\n\nThe MCP servers are automatically launched by your agent when needed. You don't need to manually start them.\n\n### MCP environment variables\n\n| Variable | Default | Description |\n|----------|---------|-------------|\n| `PAPERPIPE_PQA_INDEX_DIR` | `~/.paperpipe/.pqa_index` | Root directory for PaperQA2 indices |\n| `PAPERPIPE_PQA_INDEX_NAME` | `paperpipe_\u003cembedding\u003e` | Index name (subfolder under index dir) |\n| `PAPERQA_EMBEDDING` | (from config) | Embedding model (must match index for PaperQA2) |\n\n### MCP tools\n\n| Tool | Backend | Description |\n|------|---------|-------------|\n| `retrieve_chunks` | PaperQA2 | Retrieve raw chunks + citations (no LLM answering) |\n| `list_pqa_indexes` | PaperQA2 | List available PaperQA2 indices with embedding model metadata |\n| `get_pqa_index_status` | PaperQA2 | Show index stats (files, failures) |\n| `leann_search` | LEANN | Semantic search over papers (faster, simpler output) |\n| `leann_list` | LEANN | List available LEANN indexes |\n\n### MCP usage\n\n1. Build indexes: `papi index --backend pqa --pqa-embedding text-embedding-3-small`\n2. In your agent: `leann_search()` (fast) or `retrieve_chunks()` (with citations)\n3. For PaperQA2: embedding model is **automatically inferred** from index metadata (or index name for backward compatibility)\n\n\u003c/details\u003e\n\n## RAG backends (`papi ask`)\n\npaperpipe supports two RAG backends for cross-paper questions:\n\n| Backend | Install | Best for |\n|---------|---------|----------|\n| [PaperQA2](https://github.com/Future-House/paper-qa) | `paperpipe[paperqa]` | Agentic synthesis with citations (cloud LLMs) |\n| [LEANN](https://github.com/yichuan-w/LEANN) | `paperpipe[leann]` | Local retrieval (Ollama) |\n\n```bash\n# PaperQA2 (default if installed)\npapi ask \"What regularization techniques do these papers use?\"\n\n# LEANN (local)\npapi ask \"...\" --backend leann\n```\n\nThe first query builds an index (cached under `.pqa_index/` or `.leann/`). Use `papi index` to pre-build.\n\n\u003cdetails markdown=\"1\"\u003e\n\u003csummary\u003ePaperQA2 configuration\u003c/summary\u003e\n\n### Common options\n\n| Flag | Description |\n|------|-------------|\n| `--pqa-llm MODEL` | LLM for answer generation (LiteLLM id) |\n| `--pqa-summary-llm MODEL` | LLM for evidence summarization (often cheaper) |\n| `--pqa-embedding MODEL` | Embedding model for text chunks |\n| `--pqa-temperature FLOAT` | LLM temperature (0.0-1.0) |\n| `--pqa-verbosity INT` | Logging level (0-3; 3 = log all LLM calls) |\n| `--pqa-agent-type TEXT` | Agent type (e.g., `fake` for deterministic low-token retrieval) |\n| `--pqa-answer-length TEXT` | Target answer length (e.g., \"about 200 words\") |\n| `--pqa-evidence-k INT` | Number of evidence pieces to retrieve (default: 10) |\n| `--pqa-max-sources INT` | Max sources to cite in answer (default: 5) |\n| `--pqa-timeout FLOAT` | Agent timeout in seconds (default: 500) |\n| `--pqa-concurrency INT` | Indexing concurrency (default: 1) |\n| `--pqa-rebuild-index` | Force full index rebuild |\n| `--pqa-retry-failed` | Retry previously failed documents |\n| `--format evidence-blocks` | Output JSON with `{answer, evidence[]}` (requires PaperQA2 Python package) |\n| `--pqa-raw` | Show raw PaperQA2 output (streaming logs + answer); disables `papi ask` output filtering (also enabled by global `-v/--verbose`) |\n\nAny additional arguments are passed through to `pqa` (e.g., `--agent.search_count 10`).\n\n### Model combinations\n\n\u003cdetails markdown=\"1\"\u003e\n\u003csummary\u003e\u003cstrong\u003eModel combination examples\u003c/strong\u003e\u003c/summary\u003e\n\n**Indexing:**\n\n```bash\n# API keys should be in env\nexport OPENAI_API_KEY=...\nexport ANTHROPIC_API_KEY=...\nexport GEMINI_API_KEY=...\nexport VOYAGE_API_KEY=...\nexport OPENROUTER_API_KEY=...\n\n# Ollama (local) + Ollama embeddings\npapi index --backend pqa --pqa-llm ollama/olmo-3:7b --pqa-embedding ollama/nomic-embed-text\n\n# GPT + OpenAI Embeddings\npapi index --backend pqa --pqa-llm gpt-4.1 --pqa-summary-llm gpt-4.1-mini --pqa-embedding text-embedding-3-small\n\n# Gemini + Google Embeddings\npapi index --backend pqa --pqa-llm gemini/gemini-3-flash-preview --pqa-embedding gemini/gemini-embedding-001\n\n# Claude + Voyage Embeddings\npapi index --backend pqa --pqa-llm claude-sonnet-4-5 --pqa-summary-llm claude-haiku-4-5 --pqa-embedding voyage/voyage-4\n\n# OpenRouter + Voyage Embeddings\npapi index --backend pqa --pqa-llm openrouter/google/gemini-3.5-flash --pqa-embedding voyage/voyage-4\n```\n\n**Asking:**\n\n```bash\n# Ollama (local)\npapi ask \"how is neus different from nerf?\" --backend pqa --pqa-llm ollama/olmo-3:7b --pqa-embedding ollama/nomic-embed-text\n\n# GPT\npapi ask \"how is neus different from nerf?\" --backend pqa --pqa-llm gpt-4.1 --pqa-summary-llm gpt-4.1-mini --pqa-embedding text-embedding-3-small\n\n# Gemini\npapi ask \"how is neus different from nerf?\" --backend pqa --pqa-llm gemini/gemini-3-flash-preview --pqa-embedding gemini/gemini-embedding-001\n\n# Claude\npapi ask \"how is neus different from nerf?\" --backend pqa --pqa-llm claude-sonnet-4-5 --pqa-summary-llm claude-haiku-4-5 --pqa-embedding voyage/voyage-4\n\n# OpenRouter\npapi ask \"how is neus different from nerf?\" --backend pqa --pqa-llm openrouter/google/gemini-3.5-flash --pqa-embedding voyage/voyage-4\n```\n\n\u003c/details\u003e\n\n\u003cdetails markdown=\"1\"\u003e\n\u003csummary\u003e\u003cstrong\u003eEmbedding provider examples (indexing)\u003c/strong\u003e\u003c/summary\u003e\n\n#### OpenAI\n\n```bash\nexport OPENAI_API_KEY=...\npapi index --backend pqa --pqa-embedding text-embedding-3-small\n```\n\n#### Gemini (native LiteLLM id)\n\n```bash\nexport GEMINI_API_KEY=...\npapi index --backend pqa --pqa-embedding gemini/gemini-embedding-001\n```\n\n#### Voyage (native LiteLLM id)\n\n```bash\nexport VOYAGE_API_KEY=...\npapi index --backend pqa --pqa-embedding voyage/voyage-4\n```\n\n#### OpenAI-compatible endpoints (advanced)\n\nIf you want to hit an OpenAI-compatible endpoint directly (instead of a native LiteLLM provider id), set\n`OPENAI_API_BASE` and `OPENAI_API_KEY` and use an `openai/...` embedding id.\n\n```bash\nexport OPENAI_API_BASE=https://api.voyageai.com/v1\nexport OPENAI_API_KEY=\"$VOYAGE_API_KEY\"\npapi index --backend pqa --pqa-embedding openai/voyage-4\n```\n\n\u003c/details\u003e\n\n### Index/caching notes\n\n- First run builds an index under `\u003cpaper_db\u003e/.pqa_index/` and stages PDFs under `\u003cpaper_db\u003e/.pqa_papers/`.\n- Override index location with `PAPERPIPE_PQA_INDEX_DIR`.\n- If you indexed wrong content (or changed embeddings), delete `.pqa_index/` to force rebuild.\n- If PDFs failed indexing (recorded as `ERROR`), re-run with `--pqa-retry-failed` or `--pqa-rebuild-index`.\n- By default, `papi ask` uses `--settings default` to avoid stale user settings; pass `-s/--settings \u003cname\u003e` to override.\n- Managed PaperQA2 indexing uses a CSV manifest from `meta.json` and defaults to text-only PDF parsing\n  (`--parsing.multimodal OFF`, `--parsing.use_doc_details false`) so embedding updates do not invoke PaperQA2\n  metadata/enrichment LLM calls. Pass explicit `--parsing...` args to opt into PaperQA2 multimodal enrichment.\n\n\u003c/details\u003e\n\n\u003cdetails markdown=\"1\"\u003e\n\u003csummary\u003eLEANN configuration\u003c/summary\u003e\n\n### Common options\n\n```bash\npapi ask \"...\" --backend leann --leann-provider ollama --leann-model qwen3:8b\npapi ask \"...\" --backend leann --leann-host http://localhost:11434\npapi ask \"...\" --backend leann --leann-top-k 12 --leann-complexity 64\n```\n\nNotes:\n- If you use `--leann-provider anthropic`, your `leann` install must include the `anthropic` Python package\n  (`pip install anthropic` in the same environment that runs `leann`).\n- You can pass through extra `leann` CLI flags after `--` (useful for debugging), e.g.:\n  `papi -v ask \"...\" --backend leann -- ...`\n\n### Model combinations\n\n\u003cdetails markdown=\"1\"\u003e\n\u003csummary\u003e\u003cstrong\u003eModel combination examples\u003c/strong\u003e\u003c/summary\u003e\n\n**Indexing:**\n\n```bash\n# API keys should be in env\nexport OPENAI_API_KEY=...\nexport ANTHROPIC_API_KEY=...\nexport GEMINI_API_KEY=...\nexport VOYAGE_API_KEY=...\nexport OPENROUTER_API_KEY=...\n\n# Ollama (local) + Ollama embeddings\npapi index --backend leann --leann-embedding-mode ollama --leann-embedding-model nomic-embed-text\n\n# OpenAI + OpenAI embeddings\npapi index --backend leann --leann-embedding-mode openai --leann-embedding-model text-embedding-3-small --leann-embedding-api-key $OPENAI_API_KEY\n\n# Gemini + Gemini embeddings (OpenAI-compatible)\npapi index --backend leann --leann-embedding-mode openai --leann-embedding-model gemini-embedding-001 --leann-embedding-api-base https://generativelanguage.googleapis.com/v1beta/openai/ --leann-embedding-api-key $GEMINI_API_KEY\n\n# Voyage embeddings (OpenAI-compatible)\npapi index --backend leann --leann-embedding-mode openai --leann-embedding-model voyage-4 --leann-embedding-api-base https://api.voyageai.com/v1 --leann-embedding-api-key $VOYAGE_API_KEY\n```\n\n**Asking:**\n\n```bash\n# Ollama (local)\npapi ask \"how is neus different from nerf?\" --backend leann --leann-provider ollama --leann-model olmo-3:7b --leann-index papers_ollama_nomic-embed-text\n\n# OpenAI\npapi ask \"how is neus different from nerf?\" --backend leann --leann-provider openai --leann-model gpt-4.1 --leann-api-key $OPENAI_API_KEY --leann-index papers_openai_text-embedding-3-small\n\n# Anthropic + Voyage embeddings\npapi ask \"how is neus different from nerf?\" --backend leann --leann-provider anthropic --leann-model claude-sonnet-4-5 --leann-api-key $ANTHROPIC_API_KEY --leann-index papers_openai_voyage-4\n\n# OpenRouter + Voyage embeddings\npapi ask \"how is neus different from nerf?\" --backend leann --leann-provider openai --leann-model google/gemini-3.5-flash --leann-api-base https://openrouter.ai/api/v1 --leann-api-key $OPENROUTER_API_KEY --leann-index papers_openai_voyage-4\n\n# Gemini (OpenAI-compatible)\npapi ask \"how is neus different from nerf?\" --backend leann --leann-provider openai --leann-model gemini-3-flash-preview --leann-api-base https://generativelanguage.googleapis.com/v1beta/openai/ --leann-api-key $GEMINI_API_KEY --leann-index papers_openai_gemini-embedding-001\n```\n\n\u003c/details\u003e\n\n\u003cdetails markdown=\"1\"\u003e\n\u003csummary\u003e\u003cstrong\u003eEmbedding provider examples\u003c/strong\u003e\u003c/summary\u003e\n\n**Note:** For `--leann-embedding-mode openai`, LEANN defaults the API key to `OPENAI_API_KEY` unless you pass `--leann-embedding-api-key`.\n\n```bash\n# Ollama (local)\npapi index --backend leann --leann-embedding-mode ollama --leann-embedding-model nomic-embed-text\n\n# OpenAI\nexport OPENAI_API_KEY=...\npapi index --backend leann --leann-embedding-mode openai --leann-embedding-model text-embedding-3-small --leann-embedding-api-key $OPENAI_API_KEY\n\n# Gemini (OpenAI-compatible)\nexport GEMINI_API_KEY=...\npapi index --backend leann --leann-embedding-mode openai --leann-embedding-model gemini-embedding-001 --leann-embedding-api-base https://generativelanguage.googleapis.com/v1beta/openai/ --leann-embedding-api-key $GEMINI_API_KEY\n\n# Voyage (OpenAI-compatible)\nexport VOYAGE_API_KEY=...\npapi index --backend leann --leann-embedding-mode openai --leann-embedding-model voyage-4 --leann-embedding-api-base https://api.voyageai.com/v1 --leann-embedding-api-key $VOYAGE_API_KEY\n```\n\n**Gemini notes:**\n- May hit quota/rate limits (HTTP 429). Retry after suggested delay.\n- Some LEANN versions batch too many inputs per request for Gemini (hard limit: 100 inputs/request) and fail with HTTP 400; update LEANN or reduce chunk counts (e.g., larger `--leann-doc-chunk-size`).\n\n\u003c/details\u003e\n\n### Defaults\n\nBy default, paperpipe derives LEANN's defaults from your global `[llm]` / `[embedding]` model settings when they are\nLEANN-compatible:\n- `ollama/...` → `--llm ollama` / `--embedding-mode ollama`\n- `gpt-*` / `text-embedding-*` → `--llm openai` / `--embedding-mode openai`\n- `gemini/...` → `--llm openai` (Gemini OpenAI-compatible endpoint)\n\nFor Gemini, paperpipe defaults `--leann-api-base` to `https://generativelanguage.googleapis.com/v1beta/openai/` and uses\n`GEMINI_API_KEY`/`GOOGLE_API_KEY` if set.\n\nNote: LEANN's current CLI batches OpenAI-compatible embeddings in chunks of up to ~500-800 texts per request; Gemini's\nembedding endpoint hard-limits batches to 100, so paperpipe does *not* auto-map `gemini/...` embeddings to LEANN by\ndefault. Use `PAPERPIPE_LEANN_EMBEDDING_*` / `[leann]` to override (and expect to tune batch behavior upstream in LEANN).\n\n### Multiple indices\n\nLEANN supports multiple index names under `\u003cpaper_db\u003e/.leann/indexes/`.\n\nBy default, paperpipe auto-derives the LEANN index name from the embedding mode/model (similar to PaperQA2).\n\nTo disable and always use a single LEANN index named `papers`, set:\n\n```toml\n[leann]\nindex_by_embedding = false\n```\n\nor `export PAPERPIPE_LEANN_INDEX_BY_EMBEDDING=0`.\n\nWhen enabled, the default LEANN index name becomes `papers_\u003cmode\u003e_\u003cmodel\u003e` (with `/` and `:` replaced by `_`).\n\nIf model ids are not recognized as compatible, it falls back to `ollama` with `olmo-3:7b` (LLM) and `nomic-embed-text`\n(embeddings).\n\nOverride via `config.toml`:\n```toml\n[leann]\nllm_provider = \"ollama\"\nllm_model = \"qwen3:8b\"\nembedding_model = \"nomic-embed-text\"\nembedding_mode = \"ollama\"\n```\n\nOr env vars: `PAPERPIPE_LEANN_LLM_PROVIDER`, `PAPERPIPE_LEANN_LLM_MODEL`, `PAPERPIPE_LEANN_EMBEDDING_MODEL`, `PAPERPIPE_LEANN_EMBEDDING_MODE`.\n\n### Index builds\n\n```bash\npapi index --backend leann\n\n# Override common LEANN build knobs (maps to `leann build ...`):\npapi index --backend leann --leann-embedding-mode ollama --leann-embedding-model nomic-embed-text\npapi index --backend leann --leann-embedding-mode ollama --leann-embedding-host http://localhost:11434\npapi index --backend leann --leann-doc-chunk-size 350 --leann-doc-chunk-overlap 128\n```\n\nBy default, `papi ask --backend leann` auto-builds the index if missing (disable with `--leann-no-auto-index`).\nFor explicit derived names such as `papers_openai_voyage-4`, auto-build infers the embedding mode/model from the name.\n\n\u003c/details\u003e\n\n## LLM configuration\n\npaperpipe uses LLMs for generating summaries, extracting equations, and tagging. Without an LLM, it falls back to regex extraction and metadata-based summaries.\n\n```bash\n# Set your API key (pick one)\nexport GEMINI_API_KEY=...       # default provider\nexport OPENAI_API_KEY=...\nexport ANTHROPIC_API_KEY=...\nexport VOYAGE_API_KEY=...       # for Voyage embeddings (recommended with Claude)\nexport OPENROUTER_API_KEY=...   # 200+ models\n\n# Override the default model\nexport PAPERPIPE_LLM_MODEL=gpt-4o\nexport PAPERPIPE_LLM_TEMPERATURE=0.3  # default: 0.3\n```\n\n### Local-only via Ollama\n\n```bash\nexport PAPERPIPE_LLM_MODEL=ollama/qwen3:8b\nexport PAPERPIPE_EMBEDDING_MODEL=ollama/nomic-embed-text\n\n# Either env var name works (paperpipe normalizes both):\nexport OLLAMA_HOST=http://localhost:11434\n# export OLLAMA_API_BASE=http://localhost:11434\n```\n\nCheck which models work with your keys:\n```bash\npapi models                    # probe default models for your configured keys\npapi models latest             # probe latest model candidates (gpt-5, Gemini via OpenRouter/Gemini, Claude, Voyage 4)\npapi models last-gen           # probe previous generation\npapi models all                # probe broader superset\npapi models --verbose          # show underlying provider errors\n```\n\n## Tagging\n\nPapers are auto-tagged from:\n1. arXiv categories (cs.CV → computer-vision)\n2. LLM-generated semantic tags (biased toward existing tags for consistency)\n3. Your `--tags` flag\n\n```bash\npapi add 1706.03762 --tags my-project,priority\npapi list --tag attention\npapi tags --audit               # find duplicate/similar tags\npapi tags --merge old-tag new-tag  # rename a tag across all papers\npapi tags --delete junk-tag     # remove a tag from all papers\n```\n\n## Non-arXiv papers\n\n```bash\npapi add ./paper.pdf                                       # local PDF (auto-detected)\npapi add \"https://example.com/paper.pdf\"                   # PDF URL (auto-detected)\npapi add --pdf ./paper.pdf --title \"My Paper\" --no-llm     # --pdf for explicit metadata options\npapi add --pdf \"https://example.com/paper.pdf\" --tags siggraph\n```\n\n## Configuration file\n\nFor persistent settings, create `~/.paperpipe/config.toml` (override location with `PAPERPIPE_CONFIG_PATH`):\n\n```toml\n[llm]\nmodel = \"gemini/gemini-2.5-flash\"\ntemperature = 0.3\n\n[embedding]\nmodel = \"gemini/gemini-embedding-001\"\n\n[paperqa]\nsettings = \"default\"\nindex_dir = \"~/.paperpipe/.pqa_index\"\nsummary_llm = \"gpt-4o-mini\"\nenrichment_llm = \"gpt-4o-mini\"\n\n# Optional: override LEANN separately (otherwise it follows [llm]/[embedding] for openai/ollama model ids)\n[leann]\nllm_provider = \"ollama\"\nllm_model = \"qwen3:8b\"\nembedding_model = \"nomic-embed-text\"\nembedding_mode = \"ollama\"\n\n[tags.aliases]\ncv = \"computer-vision\"\nnlp = \"natural-language-processing\"\n```\n\nPrecedence: **CLI flags \u003e env vars \u003e config.toml \u003e built-in defaults**.\n\n## Development\n\n```bash\ngit clone https://github.com/hummat/paperpipe \u0026\u0026 cd paperpipe\npip install -e \".[dev]\"\nmake check                            # format + lint + typecheck + test\n```\n\n\u003cdetails markdown=\"1\"\u003e\n\u003csummary\u003eRelease (maintainers)\u003c/summary\u003e\n\nThis repo publishes to PyPI from release tags, with a manual workflow fallback (see `.github/workflows/publish.yml`).\n\n```bash\n# Bump version in pyproject.toml, then:\nmake release\n```\n\n\u003c/details\u003e\n\n## Credits\n\n- [PaperQA2](https://github.com/Future-House/paper-qa) by Future House — RAG backend.\n  *Skarlinski et al., \"Language Agents Achieve Superhuman Synthesis of Scientific Knowledge\", 2024.*\n  [arXiv:2409.13740](https://arxiv.org/abs/2409.13740)\n- [LEANN](https://github.com/yichuan-w/LEANN) — (local) RAG backend.\n  *Wang et al., \"LEANN: A Low-Storage Vector Index\", 2025.*\n  [arXiv:2506.08276](https://arxiv.org/abs/2506.08276)\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhummat%2Fpaperpipe","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhummat%2Fpaperpipe","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhummat%2Fpaperpipe/lists"}