https://github.com/robertld/libscope
AI-powered knowledge base with MCP integration — query library docs, internal wikis, and topics with semantic search
https://github.com/robertld/libscope
ai context document-store llm mcp rag
Last synced: 4 months ago
JSON representation
AI-powered knowledge base with MCP integration — query library docs, internal wikis, and topics with semantic search
- Host: GitHub
- URL: https://github.com/robertld/libscope
- Owner: RobertLD
- License: other
- Created: 2026-02-28T21:59:45.000Z (4 months ago)
- Default Branch: development
- Last Pushed: 2026-03-06T03:48:08.000Z (4 months ago)
- Last Synced: 2026-03-06T08:46:41.064Z (4 months ago)
- Topics: ai, context, document-store, llm, mcp, rag
- Language: TypeScript
- Homepage: https://libscope.com
- Size: 966 KB
- Stars: 1
- Watchers: 0
- Forks: 0
- Open Issues: 5
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Agents: agents.md
Awesome Lists containing this project
README
# LibScope
[](https://www.npmjs.com/package/libscope)
[](https://github.com/RobertLD/libscope/actions/workflows/ci.yml)
[](LICENSE)
[](https://nodejs.org)
[](https://libscope.com)
LibScope is a local knowledge base that makes your documentation searchable by AI assistants. Point it at markdown files, URLs, or connect it to Obsidian/Notion/Confluence/Slack, and it chunks, embeds, and indexes everything into a local SQLite database. Your AI tools query it through [MCP](https://modelcontextprotocol.io/) (Model Context Protocol) or a REST API.
Everything runs on your machine. No cloud services required for basic usage — just `npm install` and go.
## Getting Started
```bash
npm install -g libscope
# Set up the database
libscope init
# Index some docs
libscope add ./docs/getting-started.md --library my-lib
libscope add https://docs.example.com/guide
libscope import ./docs/ --library my-lib --extensions .md,.mdx
# Search
libscope search "how to authenticate"
# Start the MCP server so your AI assistant can query it
libscope serve
```
On first run with the default embedding provider, LibScope downloads the [all-MiniLM-L6-v2](https://huggingface.co/Xenova/all-MiniLM-L6-v2) model (~80 MB). Subsequent runs use the cached model.
### Document Format Support
LibScope supports **Markdown** (`.md`, `.mdx`) and **plain text** natively. Additional formats are available via optional dependencies:
| Format | Extension | Optional Dependency | Node.js Requirement |
| ------------ | --------- | ------------------- | ------------------- |
| PDF | `.pdf` | `pdf-parse` (v2) | ≥ 20.16 or ≥ 22.3 |
| Word | `.docx` | `mammoth` | Any |
| EPUB | `.epub` | `epub2` | Any |
| PowerPoint | `.pptx` | `pizzip` | Any |
| CSV | `.csv` | Built-in | Any |
The `pdf-parse`, `mammoth`, `epub2`, and `pizzip` packages are listed as `optionalDependencies` and install automatically when the Node.js version is compatible. Note: binary `.ppt` files are not supported — only `.pptx`.
## Using with AI Assistants
LibScope exposes an MCP server over stdio. Point your MCP-compatible client at it:
**Cursor** — add to `~/.cursor/mcp.json`:
```json
{
"mcpServers": {
"libscope": {
"command": "npx",
"args": ["-y", "libscope", "serve"]
}
}
}
```
**Claude Code:**
```bash
claude mcp add --scope user libscope -- npx -y libscope serve
```
Once connected, your assistant can search docs, submit new documents, rate content quality, and ask RAG-powered questions against your knowledge base.
Full list of MCP tools
| Tool | What it does |
| ---------------------- | --------------------------------------------------------- |
| `search-docs` | Semantic search with topic/library/version/rating filters |
| `ask-question` | RAG question-answering with source citations |
| `get-document` | Retrieve a document by ID |
| `list-documents` | List docs with filters |
| `list-topics` | Browse the topic hierarchy |
| `submit-document` | Index new content (raw text or a URL to fetch) |
| `update-document` | Update a document's title, content, or metadata |
| `delete-document` | Remove a document |
| `rate-document` | Rate a doc 1–5 with optional feedback and corrections |
| `suggest-tags` | Auto-suggest tags based on content analysis |
| `save-search` | Save a named search query with filters |
| `list-saved-searches` | List all saved searches |
| `run-saved-search` | Execute a saved search by name or ID |
| `delete-saved-search` | Delete a saved search |
| `link-documents` | Create a cross-reference between two documents |
| `get-document-links` | List all incoming and outgoing links for a document |
| `delete-link` | Remove a cross-reference link |
| `reindex-documents` | Re-embed chunks (useful after switching providers) |
| `health-check` | DB status, doc/chunk counts |
| `sync-obsidian-vault` | Sync an Obsidian vault |
| `sync-onenote` | Sync OneNote notebooks via Microsoft Graph |
| `sync-notion` | Sync Notion pages and databases |
| `sync-confluence` | Sync Confluence spaces |
| `sync-slack` | Sync Slack channels and threads |
| `install-pack` | Install a knowledge pack |
| `list-packs` | List installed or registry packs |
## Connectors
LibScope can pull documentation from several platforms. Each connector handles incremental syncing so re-runs only process what changed.
```bash
# Obsidian — parses wikilinks, frontmatter, embeds, tags
libscope connect obsidian /path/to/vault
libscope connect obsidian /path/to/vault --sync # incremental re-sync
# Notion
libscope connect notion --token secret_abc123
# Confluence
libscope connect confluence \
--url https://acme.atlassian.net \
--email user@acme.com \
--token $CONFLUENCE_TOKEN
# Slack — index channel messages and threads
libscope connect slack --token xoxb-... --channels general,engineering
# OneNote — device code auth via Microsoft Graph
export ONENOTE_CLIENT_ID=your-client-id
libscope connect onenote
# GitHub / GitLab repos
libscope add-repo https://github.com/org/repo --branch main --path docs/
# Remove a connector's data
libscope disconnect obsidian /path/to/vault
```
Connector options reference
**Obsidian:** `--topic-mapping frontmatter`, `--exclude "templates/*" "daily/*"`, `--sync`
**Notion:** `--exclude page-id-1 db-id-2`, `--sync`
**Confluence:** `--spaces ENG,DEVOPS`, `--exclude-spaces ARCHIVE`
**Slack:** `--thread-mode aggregate|separate`, `--sync`
**OneNote:** `--notebook "Work Notes"`, `--sync`
**GitHub/GitLab:** `--token`, `--branch`, `--path`, `--extensions .md,.mdx,.rst`
## Search and RAG
```bash
# Semantic search
libscope search "authentication best practices"
libscope search "API rate limiting" --library my-lib --topic security --limit 10
# Ask questions (needs an LLM provider configured — see Configuration below)
libscope ask "How do I configure OAuth2?" --library my-lib
# Interactive REPL for iterative searching
libscope repl
```
Search uses sqlite-vec for vector similarity when available, with FTS5 full-text search as a fallback. Results automatically get a 1.5× title boost when the document title matches query words. You can also pass `--diversity 0.5` (0–1) for MMR-based diversity reranking.
### Programmatic SDK
LibScope also exports a `LibScope` class for use as a library:
```ts
import { LibScope } from "libscope";
const scope = LibScope.create();
await scope.index({ title: "My Doc", content: "..." });
const results = await scope.search("query");
scope.close();
```
See the [Programmatic Usage](/guide/programmatic-usage) guide for details on the SDK, batch search, and document TTL/expiry.
## Organizing Content
**Topics** give your docs a hierarchy. **Tags** give them flexible labels. **Workspaces** give you isolated databases.
```bash
# Topics
libscope topics create "backend"
libscope topics create "auth" --parent backend --description "Auth & identity"
# Tags
libscope tag add typescript,api,v2
# Workspaces — separate knowledge bases entirely
libscope workspace create my-project
libscope workspace use my-project
libscope --workspace my-project search "deploy steps"
```
Documents also keep version history, so you can roll back if a re-index goes wrong:
```bash
libscope docs history
libscope docs rollback 3
```
## REST API
For programmatic access outside of MCP:
```bash
libscope serve --api --port 3378
```
OpenAPI 3.0 spec at `GET /openapi.json`. Key endpoints:
| Method | Endpoint | Description |
| --------------- | --------------------------------- | ---------------------------------- |
| `GET` | `/api/v1/search?q=...` | Semantic search |
| `POST` | `/api/v1/batch-search` | Batch search (up to 20 queries) |
| `POST` | `/api/v1/ask` | RAG question-answering |
| `GET/POST` | `/api/v1/documents` | List or create documents |
| `GET/PATCH/DELETE` | `/api/v1/documents/:id` | Get, update, or delete a document |
| `POST` | `/api/v1/documents/url` | Index from a URL |
| `POST` | `/api/v1/documents/:id/tags` | Add tags |
| `GET` | `/api/v1/documents/:id/suggest-tags` | Auto-suggest tags |
| `GET/POST` | `/api/v1/documents/:id/links` | List or create cross-references |
| `DELETE` | `/api/v1/links/:id` | Delete a cross-reference |
| `GET/POST` | `/api/v1/topics` | List or create topics |
| `GET` | `/api/v1/tags` | List tags |
| `GET/POST` | `/api/v1/searches` | List or create saved searches |
| `POST` | `/api/v1/searches/:id/run` | Run a saved search |
| `DELETE` | `/api/v1/searches/:id` | Delete a saved search |
| `POST` | `/api/v1/bulk/delete` | Bulk delete documents |
| `POST` | `/api/v1/bulk/retag` | Bulk add/remove tags |
| `POST` | `/api/v1/bulk/move` | Bulk move to a topic |
| `GET/POST` | `/api/v1/webhooks` | List or create webhooks |
| `DELETE` | `/api/v1/webhooks/:id` | Delete a webhook |
| `POST` | `/api/v1/webhooks/:id/test` | Send a test ping to a webhook |
| `GET` | `/api/v1/analytics/searches` | Search analytics and knowledge gaps|
| `GET` | `/api/v1/connectors/status` | Connector sync status and history |
| `GET` | `/api/v1/stats` | Usage statistics |
| `GET` | `/api/v1/health` | Health check |
## Configuration
LibScope reads config from (highest priority first): **environment variables** → **`.libscope.json`** in your project → **`~/.libscope/config.json`** → built-in defaults.
### Embedding Providers
| Provider | Default? | Requirements | Model |
| -------- | -------- | ----------------------------------------- | ---------------------- |
| `local` | Yes | None (~80 MB model download on first run) | all-MiniLM-L6-v2 |
| `ollama` | | Ollama running locally | nomic-embed-text |
| `openai` | | `LIBSCOPE_OPENAI_API_KEY` | text-embedding-3-small |
```bash
libscope config set embedding.provider ollama
# or
export LIBSCOPE_EMBEDDING_PROVIDER=openai
export LIBSCOPE_OPENAI_API_KEY=sk-...
```
### LLM for RAG
The `ask` command and `ask-question` MCP tool need an LLM. Configure one with:
```bash
export LIBSCOPE_LLM_PROVIDER=openai # or ollama, anthropic
export LIBSCOPE_LLM_MODEL=gpt-4o-mini # optional
```
All environment variables
| Variable | Description | Default |
| ---------------------------------- | ---------------------------------------- | ------------------------ |
| `LIBSCOPE_EMBEDDING_PROVIDER` | `local`, `ollama`, or `openai` | `local` |
| `LIBSCOPE_OPENAI_API_KEY` | OpenAI API key | — |
| `LIBSCOPE_OLLAMA_URL` | Ollama server URL | `http://localhost:11434` |
| `LIBSCOPE_LLM_PROVIDER` | LLM for RAG (`openai` / `ollama` / `anthropic`) | — |
| `LIBSCOPE_LLM_MODEL` | LLM model override | — |
| `LIBSCOPE_ANTHROPIC_API_KEY` | Anthropic API key (for Claude models) | — |
| `LIBSCOPE_ALLOW_PRIVATE_URLS` | Allow fetching from private/internal IPs | `false` |
| `LIBSCOPE_ALLOW_SELF_SIGNED_CERTS` | Accept self-signed TLS certificates | `false` |
| `ONENOTE_CLIENT_ID` | Microsoft app registration client ID | — |
| `ONENOTE_TENANT_ID` | Microsoft tenant ID | `common` |
| `NOTION_TOKEN` | Notion integration token | — |
| `CONFLUENCE_URL` | Confluence base URL | — |
| `CONFLUENCE_EMAIL` | Confluence user email | — |
| `CONFLUENCE_TOKEN` | Confluence API token | — |
Example config file (~/.libscope/config.json)
```json
{
"embedding": {
"provider": "local",
"ollamaUrl": "http://localhost:11434",
"ollamaModel": "nomic-embed-text",
"openaiModel": "text-embedding-3-small"
},
"llm": {
"provider": "openai",
"model": "gpt-4o-mini",
"anthropicApiKey": "sk-ant-..."
},
"database": {
"path": "~/.libscope/libscope.db"
},
"logging": {
"level": "info"
},
"indexing": {
"allowPrivateUrls": false,
"allowSelfSignedCerts": false
}
}
```
### Corporate / Internal Networks
If you're indexing docs from internal servers (Confluence, wikis, etc.), you may need:
```bash
# Allow fetching from private/internal IP addresses
libscope config set indexing.allowPrivateUrls true
# Accept self-signed or corporate TLS certificates
libscope config set indexing.allowSelfSignedCerts true
```
Or via environment variables:
```bash
export LIBSCOPE_ALLOW_PRIVATE_URLS=true
export LIBSCOPE_ALLOW_SELF_SIGNED_CERTS=true
```
## Webhooks
LibScope can push events to any HTTP endpoint. Useful for triggering CI pipelines, Slack notifications, or custom workflows whenever documents are created or updated.
```bash
libscope serve --api # webhooks require the REST API
```
```bash
# Create a webhook
curl -X POST http://localhost:3378/api/v1/webhooks \
-H "Content-Type: application/json" \
-d '{"url": "https://hooks.example.com/libscope", "events": ["document.created", "document.updated"], "secret": "my-hmac-secret"}'
# Send a test ping
curl -X POST http://localhost:3378/api/v1/webhooks//test
```
Webhook payloads are signed with HMAC-SHA256 when a secret is set. The signature is in the `X-LibScope-Signature` header.
Supported events: `document.created`, `document.updated`, `document.deleted`.
## Other Tools
LibScope ships with a few more utilities beyond the core index-and-search loop:
```bash
# Watch a directory and auto-reindex on changes
libscope watch ./docs/
# Re-embed everything after switching embedding providers
libscope reindex
# Find duplicate documents
libscope dedupe
# Export / import the whole knowledge base
libscope export ./backup.json
libscope import-backup ./backup.json
# Usage analytics
libscope stats # overview
libscope stats popular # most-searched docs
libscope stats stale --days 90 # docs nobody searches for
# Knowledge packs — portable document bundles
libscope pack create --name "react-docs" --topic react
libscope pack install ./react-docs.json
```
There's also a web dashboard at `http://localhost:3377` when you run `libscope serve`, with search, document browsing, topic navigation, and a knowledge graph visualization at `/graph`.
Full CLI reference
**Core**
| Command | Description |
| ----------------------------------- | ----------------------------------- |
| `libscope init` | Initialize the database |
| `libscope add ` | Index a file or URL |
| `libscope import ` | Bulk import from a directory |
| `libscope import-batch ` | Parallel batch import |
| `libscope search ` | Search |
| `libscope ask ` | RAG question-answering |
| `libscope repl` | Interactive search REPL |
| `libscope serve` | Start MCP server (`--api` for REST) |
**Documents**
| Command | Description |
| --------------------------------------- | ---------------------------- |
| `libscope docs list` | List documents |
| `libscope docs show ` | Show a document |
| `libscope docs update ` | Update title/content/metadata|
| `libscope docs delete ` | Delete a document |
| `libscope docs history ` | Version history |
| `libscope docs rollback ` | Roll back to a prior version |
**Organization**
| Command | Description |
| ------------------------------------ | -------------------------------- |
| `libscope topics list` | List topics |
| `libscope topics create ` | Create a topic |
| `libscope tag add ` | Add tags |
| `libscope tag remove ` | Remove a tag |
| `libscope tag list` | List tags |
| `libscope workspace create ` | Create workspace |
| `libscope workspace list` | List workspaces |
| `libscope workspace use ` | Switch workspace |
| `libscope workspace delete ` | Delete workspace |
**Saved Searches**
| Command | Description |
| ------------------------------------ | ----------------------------- |
| `libscope searches list` | List all saved searches |
| `libscope searches run ` | Re-run a saved search |
| `libscope searches delete ` | Delete a saved search |
| `libscope search --save ` | Save a search while running it|
**Document Links**
| Command | Description |
| ------------------------------------- | -------------------------------- |
| `libscope link ` | Create a cross-reference |
| `libscope links ` | Show all links for a document |
| `libscope unlink ` | Remove a link |
| `libscope prereqs ` | Show prerequisite reading chain |
**Bulk Operations**
| Command | Description |
| ---------------------------- | ----------------------------------- |
| `libscope bulk delete` | Delete all matching documents |
| `libscope bulk retag` | Add/remove tags on matching docs |
| `libscope bulk move` | Move matching docs to a topic |
**Connectors**
| Command | Description |
| ---------------------------------- | -------------------------- |
| `libscope connect obsidian ` | Sync Obsidian vault |
| `libscope connect onenote` | Sync OneNote |
| `libscope connect notion` | Sync Notion |
| `libscope connect confluence` | Sync Confluence |
| `libscope connect slack` | Sync Slack |
| `libscope add-repo ` | Index a GitHub/GitLab repo |
| `libscope disconnect ` | Remove connector data |
**Utilities**
| Command | Description |
| --------------------------------- | ---------------------------- |
| `libscope watch ` | Auto-reindex on file changes |
| `libscope reindex` | Re-embed all chunks |
| `libscope dedupe` | Find duplicates |
| `libscope export ` | Export to JSON |
| `libscope import-backup ` | Import from backup |
| `libscope stats` | Usage overview |
| `libscope pack install ` | Install a knowledge pack |
| `libscope pack create` | Create a knowledge pack |
| `libscope config set ` | Set config |
| `libscope config show` | Show config |
**Global flags:** `--verbose`, `--log-level `, `--workspace `
## How It Works
LibScope stores everything in a local SQLite database (at `~/.libscope/libscope.db` by default):
- Documents are split into chunks by heading boundaries
- Each chunk is embedded into a vector using the configured provider
- Vector search is done via [sqlite-vec](https://github.com/asg017/sqlite-vec); FTS5 full-text search is used as a fallback
- The MCP server reads from this same database over stdio
- Connectors fetch content from external platforms and feed it through the same indexing pipeline
The stack: [better-sqlite3](https://github.com/WiseLibs/better-sqlite3) + [sqlite-vec](https://github.com/asg017/sqlite-vec) + [@modelcontextprotocol/sdk](https://github.com/modelcontextprotocol/typescript-sdk) + [@xenova/transformers](https://github.com/xenova/transformers.js) for local embeddings.
## Contributing
See [CONTRIBUTING.md](CONTRIBUTING.md). The short version:
```bash
git clone https://github.com/RobertLD/libscope.git
cd libscope && npm install
npm run dev # watch mode
npm test # run tests
npm run typecheck # type check
npm run lint # lint
```
## License
[Business Source License 1.1](LICENSE) — see [LICENSE](LICENSE) for full terms.