{"id":49903793,"url":"https://github.com/lucasastorian/llmwiki","last_synced_at":"2026-05-16T09:38:16.709Z","repository":{"id":349443724,"uuid":"1201432708","full_name":"lucasastorian/llmwiki","owner":"lucasastorian","description":"Open Source Implementation of Karpathy's LLM Wiki. Upload documents, connect your Claude account via MCP, and have it write your wiki ! ","archived":false,"fork":false,"pushed_at":"2026-05-11T02:25:27.000Z","size":1927,"stargazers_count":853,"open_issues_count":7,"forks_count":135,"subscribers_count":2,"default_branch":"master","last_synced_at":"2026-05-11T02:27:41.863Z","etag":null,"topics":["agents","ai-agents","claude","karpathy","knowledge-base","llm","llm-wiki","mcp","mcp-server","rag","supabase"],"latest_commit_sha":null,"homepage":"https://llmwiki.app","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lucasastorian.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-04T17:10:02.000Z","updated_at":"2026-05-11T02:25:37.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/lucasastorian/llmwiki","commit_stats":null,"previous_names":["lucasastorian/llmwiki"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/lucasastorian/llmwiki","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucasastorian%2Fllmwiki","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucasastorian%2Fllmwiki/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucasastorian%2Fllmwiki/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucasastorian%2Fllmwiki/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lucasastorian","download_url":"https://codeload.github.com/lucasastorian/llmwiki/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucasastorian%2Fllmwiki/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33097284,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-16T04:41:52.686Z","status":"ssl_error","status_checked_at":"2026-05-16T04:41:52.009Z","response_time":115,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agents","ai-agents","claude","karpathy","knowledge-base","llm","llm-wiki","mcp","mcp-server","rag","supabase"],"created_at":"2026-05-16T09:38:16.046Z","updated_at":"2026-05-16T09:38:16.702Z","avatar_url":"https://github.com/lucasastorian.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# LLM Wiki\n\n[![License](https://img.shields.io/badge/license-Apache%202.0-green)](https://opensource.org/licenses/Apache-2.0)\n\nOpen-source implementation of [Karpathy's LLM Wiki](https://x.com/karpathy/status/2039805659525644595) ([spec](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f)).\n\nI built this because research folders accumulate useful material faster than I can keep summaries, links, and citations current by hand. LLM Wiki offloads that editing work to Claude so I can focus on source selection and analysis instead.\n\nPoint it at a folder, start the local app, and connect Claude over MCP. From there, Claude reads your sources, writes wiki pages, and keeps links and citations in sync.\n\n![LLM Wiki — a compiled wiki page with citations and table of contents](wiki-page.png)\n\n## What actually happens\n\n1. **You have a folder** — PDFs, notes, articles, spreadsheets. Your existing research.\n2. **LLM Wiki indexes it** — extracts text, chunks for search, builds a local SQLite index. Source files stay where they are.\n3. **Claude connects via MCP** — reads sources, writes wiki pages under `wiki/`, maintains cross-references and footnote citations.\n4. **The wiki improves** as Claude reads more of the workspace and writes more pages. Summaries, entity pages, and cross-references accumulate instead of being re-derived from scratch each conversation.\n\n## Quick Start\n\n**Requirements:** Python 3.11+, Node.js 20+\n\n```bash\ngit clone https://github.com/lucasastorian/llmwiki.git\ncd llmwiki\n\n# Install Python deps\ncd api \u0026\u0026 python -m venv .venv \u0026\u0026 source .venv/bin/activate\npip install -r requirements.txt\ncd ..\n\n# Install web deps\ncd web \u0026\u0026 npm install \u0026\u0026 cd ..\n\n# Initialize a workspace (point at any folder with your files)\n./llmwiki init ~/research\n\n# Start the app\n./llmwiki serve ~/research\n```\n\nOpen [localhost:3000](http://localhost:3000). Your files are indexed, wiki is scaffolded, ready to go.\n\n### Connect Claude\n\n```bash\n./llmwiki mcp-config ~/research\n```\n\nThis prints a JSON snippet for `claude_desktop_config.json` (Claude Desktop) or `.claude/settings.json` (Claude Code). One workspace runs as one MCP server entry, so if you have multiple research folders, add one entry per folder.\n\nThen tell Claude: *\"Read the guide, then ingest my sources and start building the wiki.\"*\n\n### One-command start\n\n```bash\n./llmwiki open ~/research\n```\n\nDoes everything: init if needed, start servers, open browser, print MCP config hint.\n\n## CLI\n\n| Command | What it does |\n|---------|-------------|\n| `llmwiki open \u003cfolder\u003e` | Init + serve + open browser |\n| `llmwiki init \u003cfolder\u003e` | Create `.llmwiki/` + `wiki/`, index existing files |\n| `llmwiki serve \u003cfolder\u003e` | Start API on :8000 + web on :3000 |\n| `llmwiki mcp \u003cfolder\u003e` | Run stdio MCP server (for Claude config) |\n| `llmwiki mcp-config \u003cfolder\u003e` | Print `claude_desktop_config.json` snippet |\n| `llmwiki reindex \u003cfolder\u003e` | Rebuild the index from disk |\n\n## What happens on disk\n\nLLM Wiki adds two things to your folder. Source files are not moved or modified.\n\n```\n~/research/                  # Your existing files (untouched)\n  papers/paper.pdf\n  notes.md\n  data.xlsx\n  wiki/                      # Generated pages (created by LLM Wiki)\n    overview.md\n    log.md\n    concepts/\n      attention.md\n  .llmwiki/                  # Index + cache (hidden, rebuildable)\n    index.db\n    cache/\n```\n\n- `wiki/` — ordinary markdown files. Edit them in any editor. Claude writes and updates them via MCP.\n- `.llmwiki/` — SQLite search index and processed artifacts. Delete it anytime; `llmwiki reindex` rebuilds from the source files.\n\nBy default, indexing, storage, and file writes happen on your machine. No cloud services required.\n\n## How Claude interacts with the workspace\n\nOnce connected, Claude has these tools:\n\n| Tool | Description |\n|------|-------------|\n| `guide` | Explains how the wiki works, lists what's in the workspace |\n| `search` | Browse files (`list`) or full-text search (`search`) |\n| `read` | Read documents — PDFs with page ranges, glob batch reads |\n| `write` | Create wiki pages, edit with `str_replace`, append. SVG/CSV assets |\n| `delete` | Delete documents by path or glob pattern |\n\nAll writes go to disk first, then update the search index. If Claude creates `/wiki/concepts/attention.md`, that file appears on disk immediately.\n\n## Architecture\n\n```\n┌──────────────┐     ┌──────────────┐     ┌──────────────┐\n│   Next.js    │────▶│   FastAPI    │────▶│   SQLite     │\n│   Frontend   │     │   Backend    │     │   (local)    │\n└──────────────┘     └──────┬───────┘     └──────────────┘\n                            │\n                     ┌──────┴───────┐\n                     │  MCP Server  │◀──── Claude Desktop / Code\n                     │   (stdio)    │\n                     └──────────────┘\n                            │\n                     ┌──────┴───────┐\n                     │  Filesystem  │  ← source of truth\n                     └──────────────┘\n```\n\nThe filesystem is the source of truth. SQLite is a derived index — it accelerates search and stores extracted page data, but it can always be rebuilt from the files. A background file watcher picks up changes you make outside the app.\n\n## Document processing\n\nAll processing runs locally. No API keys required for basic usage.\n\n| Format | Parser | Notes |\n|--------|--------|-------|\n| PDF | pdf-oxide | Rust-based text extraction. Works well for text-heavy papers. Scanned PDFs still benefit from real OCR. |\n| Markdown/Text | native | Indexed and chunked directly |\n| HTML | webmd | Strips nav/ads, extracts clean markdown |\n| Excel/CSV | openpyxl | Sheet-by-sheet extraction |\n| Images | native | Stored as-is, viewable inline |\n| Word/PowerPoint | LibreOffice | Optional. Install LibreOffice for office conversion; without it, these formats are stored but not extracted. |\n\nSet `MISTRAL_API_KEY` for higher-quality PDF OCR with better table and layout detection. pdf-oxide is the free default and handles most text-heavy documents well enough.\n\n## Limitations and tradeoffs\n\n- **One workspace = one MCP server.** If you work across multiple research projects, each gets its own folder and its own MCP entry. This is intentional — it keeps context and file access scoped.\n- **PDF table extraction is rough.** pdf-oxide extracts prose reliably but tables come through as messy text. For financial filings or data-heavy PDFs, Mistral OCR is significantly better.\n- **LibreOffice adds setup friction.** Office file conversion requires a local LibreOffice install. If you mostly work with PDFs and markdown, you can skip it entirely.\n- **No vector search in local mode.** Full-text search uses SQLite FTS5 (porter stemming). It works well for keyword queries but does not do semantic/embedding search. The hosted version at llmwiki.app uses PGroonga for ranked search.\n\n## Self-hosting the multi-tenant version\n\nIf you want to run the hosted version (like [llmwiki.app](https://llmwiki.app)) with Postgres, Supabase auth, and S3:\n\n\u003cdetails\u003e\n\u003csummary\u003eHosted setup instructions\u003c/summary\u003e\n\n### Prerequisites\n\n- Python 3.11+\n- Node.js 20+\n- A [Supabase](https://supabase.com) project\n- An S3-compatible bucket\n\n### Database\n\n```bash\npsql $DATABASE_URL -f supabase/migrations/001_initial.sql\n```\n\n### API\n\n```bash\ncd api\npip install -r requirements.txt\nMODE=hosted DATABASE_URL=postgresql://... uvicorn main:app --port 8000\n```\n\n### MCP Server\n\n```bash\ncd mcp\npip install -r requirements.txt\nMODE=hosted DATABASE_URL=postgresql://... uvicorn server:app --port 8080\n```\n\n### Web\n\n```bash\ncd web\nnpm install\nNEXT_PUBLIC_MODE=hosted \\\nNEXT_PUBLIC_SUPABASE_URL=https://your-ref.supabase.co \\\nNEXT_PUBLIC_SUPABASE_ANON_KEY=your-anon-key \\\nNEXT_PUBLIC_API_URL=http://localhost:8000 \\\nnpm run dev\n```\n\n### Environment Variables\n\n**API**\n```\nMODE=hosted\nDATABASE_URL=postgresql://...\nSUPABASE_URL=https://your-ref.supabase.co\nAWS_ACCESS_KEY_ID=...\nAWS_SECRET_ACCESS_KEY=...\nS3_BUCKET=your-bucket\nMISTRAL_API_KEY=              # optional, for better PDF OCR\nCONVERTER_URL=                # optional, for office conversion\n```\n\n**Web**\n```\nNEXT_PUBLIC_MODE=hosted\nNEXT_PUBLIC_SUPABASE_URL=https://your-ref.supabase.co\nNEXT_PUBLIC_SUPABASE_ANON_KEY=your-anon-key\nNEXT_PUBLIC_API_URL=http://localhost:8000\n```\n\n\u003c/details\u003e\n\n## Why this beats a static notes folder\n\nPersonal wikis usually fail on maintenance, not intent. Someone has to update links, fix stale summaries, merge overlapping pages, and keep citations aligned with the source material. That work scales with the number of sources, and people stop doing it.\n\nLLM Wiki offloads that editing work. You choose the source material and direct the analysis. Claude handles the repetitive bookkeeping — updating cross-references, keeping summaries current, flagging contradictions, touching the 15 pages that a single new source affects.\n\n## License\n\nApache 2.0\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flucasastorian%2Fllmwiki","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flucasastorian%2Fllmwiki","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flucasastorian%2Fllmwiki/lists"}