{"id":50253707,"url":"https://github.com/zaxbysauce/ragapp","last_synced_at":"2026-05-27T03:14:56.076Z","repository":{"id":339367178,"uuid":"1150230797","full_name":"zaxbysauce/ragapp","owner":"zaxbysauce","description":null,"archived":false,"fork":false,"pushed_at":"2026-03-27T11:56:28.000Z","size":1629,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2026-03-27T22:56:07.917Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zaxbysauce.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-02-05T03:14:17.000Z","updated_at":"2026-03-27T11:56:32.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/zaxbysauce/ragapp","commit_stats":null,"previous_names":["zaxbysauce/ragapp"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/zaxbysauce/ragapp","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zaxbysauce%2Fragapp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zaxbysauce%2Fragapp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zaxbysauce%2Fragapp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zaxbysauce%2Fragapp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zaxbysauce","download_url":"https://codeload.github.com/zaxbysauce/ragapp/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zaxbysauce%2Fragapp/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33548875,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-27T02:00:06.184Z","response_time":53,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-05-27T03:14:55.275Z","updated_at":"2026-05-27T03:14:56.045Z","avatar_url":"https://github.com/zaxbysauce.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# KnowledgeVault\n\nA self-hosted web service for ingesting thousands of technical documents and interacting with them through natural language chat powered by RAG (Retrieval-Augmented Generation).\n\n## Overview\n\nKnowledgeVault enables you to:\n- Upload and index documents in formats: docx, xlsx, pptx, pdf, csv, sql, txt, and code files\n- Chat with your documents using AI-powered RAG responses\n- Store and retrieve memories for persistent knowledge across sessions\n- Search your knowledge base with semantic similarity\n- Self-host everything on your own infrastructure with local LLMs\n\n## Features\n\n| Feature | Description |\n|---------|-------------|\n| **Multi-Format Support** | Process Word, Excel, PowerPoint, PDF, CSV, SQL, and text documents |\n| **Semantic Chunking** | Structure-aware document processing preserves tables and code blocks |\n| **Vector Search** | LanceDB-powered semantic search with relevance scoring |\n| **Memory System** | SQLite FTS5-backed memory storage with natural language retrieval |\n| **Streaming Chat** | Real-time AI responses with source citations |\n| **File Watcher** | Automatic detection and processing of new documents |\n| **Email Ingestion** | Ingest documents via email with IMAP polling and vault routing |\n| **Web UI** | Modern React interface with Material 3 design |\n| **API Access** | Full REST API with OpenAPI documentation |\n\n## Architecture\n\n### Development Architecture\n\n```\n+------------------+     +------------------+     +------------------+\n|   React Frontend |----\u003e|  FastAPI Backend |----\u003e|   LanceDB Vector |\n|   (Port 5173*)   |     |   (Port 8080)    |     |   Store          |\n\n*Port 5173 is for development only. Production access is via port 8080.\n+------------------+     +------------------+     +------------------+\n                               |                           |\n                               |                    +------v------+\n                               |                    |  SQLite     |\n                               |                    |  Memories   |\n                               |                    +-------------+\n                               |\n                        +------v---------------------------+\n                        |  Ollama (External)               |\n                         |  - Embeddings (bge-m3) |\n                        |  - Chat (your choice of model)   |\n                        +----------------------------------+\n```\n\n### Production Architecture\n\nIn production, the frontend is built as static files and served directly by the backend container on port 8080. Both the React frontend and FastAPI backend are accessible through a single port (8080).\n\n### Technology Stack\n\n| Component | Technology |\n|-----------|------------|\n| Frontend | React 18, TypeScript, Vite, shadcn/ui, Tailwind CSS, assistant-ui |\n| Backend | Python 3.11, FastAPI, Pydantic |\n| Vector DB | LanceDB (embedded) |\n| Memory DB | SQLite with FTS5 |\n| Document Processing | Unstructured.io |\n| LLM Integration | Ollama API (OpenAI-compatible) |\n| Deployment | Docker Compose |\n\n## Quick Start\n\n### Prerequisites\n\n- Docker and Docker Compose installed\n- Ollama installed and running (see Ollama Setup below)\n- At least 8GB RAM (16GB+ recommended)\n\n### 1. Clone and Configure\n\n```bash\ngit clone \u003crepository-url\u003e\ncd RAGAPPv2\ncp .env.example .env\n```\n\nEdit `.env` to match your setup:\n```bash\n# Required: Set your data directory\nHOST_DATA_DIR=/path/to/your/data\n\n# Optional: Change default models\nCHAT_MODEL=llama3.2:latest\n```\n\n### 2. Start Ollama\n\nEnsure Ollama is running on your host machine:\n\n```bash\n# macOS/Linux\nollama serve\n\n# Windows (Ollama runs as a service by default)\n# Verify with:\nollama list\n```\n\n### 3. Pull Required Models\n\n```bash\n# Required: Embedding model\nollama pull bge-m3\n\n# Required: Chat model (choose one)\nollama pull qwen2.5:32b    # Recommended for technical content\nollama pull llama3.2:latest # Lighter alternative\n```\n\n### 4. Start KnowledgeVault\n\n```bash\ndocker compose up -d\n```\n\n### 5. Access the Application\n\nOpen your browser to: `http://localhost:8080`\n\n## Environment Setup\n\n### Environment Variables\n\n| Variable | Default | Description |\n|----------|---------|-------------|\n| `PORT` | 8080 | Web server port |\n| `HOST_DATA_DIR` | ./data | Host path for data persistence |\n| `DATA_DIR` | /app/data | Container data path |\n| `OLLAMA_EMBEDDING_URL` | http://host.docker.internal:11434 | Ollama embedding endpoint |\n| `OLLAMA_CHAT_URL` | http://host.docker.internal:11434 | Ollama chat endpoint |\n| `EMBEDDING_MODEL` | bge-m3 | Embedding model name |\n| `CHAT_MODEL` | qwen2.5:32b | Chat model name |\n| `CHUNK_SIZE` | 512 | Document chunk size (tokens) |\n| `CHUNK_OVERLAP` | 50 | Chunk overlap (tokens) |\n| `MAX_CONTEXT_CHUNKS` | 10 | Max chunks in RAG context |\n| `RAG_RELEVANCE_THRESHOLD` | 0.1 | Minimum relevance score (0.0-1.0) |\n| `LOG_LEVEL` | INFO | Logging level |\n| `AUTO_SCAN_ENABLED` | true | Enable auto-scanning |\n| `AUTO_SCAN_INTERVAL_MINUTES` | 60 | Scan interval |\n| `IMAP_ENABLED` | false | Enable email ingestion |\n| `IMAP_HOST` | - | IMAP server hostname |\n| `IMAP_PORT` | 993 | IMAP server port (993 for SSL, 143 for non-SSL) |\n| `IMAP_USE_SSL` | true | Use SSL/TLS for IMAP connection |\n| `IMAP_USERNAME` | - | IMAP account username |\n| `IMAP_PASSWORD` | - | IMAP account password |\n| `IMAP_POLL_INTERVAL` | 60 | Email poll interval (seconds) |\n\n### Data Directory Structure\n\n```\ndata/\n├── knowledgevault/       # Root data directory\n│   ├── uploads/          # [LEGACY] Legacy flat uploads directory (deprecated)\n│   ├── vaults/           # Vault-specific data directories\n│   │   ├── 1/            # Vault 1 (default/orphan vault)\n│   │   │   └── uploads/  # Uploads for vault 1\n│   │   ├── 2/            # Vault 2\n│   │   │   └── uploads/  # Uploads for vault 2\n│   │   └── ...           # Additional vaults\n│   ├── documents/        # Documents (legacy, kept for compatibility)\n│   ├── library/          # Library files\n│   ├── lancedb/          # Vector database\n│   │   └── chunks.lance/\n│   ├── app.db            # SQLite database\n│   └── logs/\n│       └── app.log\n```\n\n**Note:** The system now stores uploads in vault-specific directories (`/data/knowledgevault/vaults/{vault_id}/uploads/`). On first startup, the system automatically migrates files from the legacy flat `uploads/` directory to the appropriate vault-specific directories. Files are renamed with `.migrated` suffix to create a safe backup. If a file cannot be associated with a specific vault, it defaults to the orphan vault (vault 1).\n\n## Ollama Models\n\n### Recommended Models\n\n#### Embedding Model\n\n**bge-m3** (Required)\n- 768 dimensions\n- 8192 token context\n- ~0.5GB VRAM\n- Excellent for technical content\n\n```bash\nollama pull bge-m3\n```\n\n#### Chat Models\n\n| Model | Size | RAM | Speed | Best For |\n|-------|------|-----|-------|----------|\n| qwen2.5:32b | 32B | ~22GB | ~15 tok/s | Technical reasoning |\n| qwen2.5:72b | 72B | ~45GB | ~10 tok/s | Complex analysis |\n| llama3.2:latest | 3B | ~4GB | ~30 tok/s | General use, fast |\n| mistral:latest | 7B | ~8GB | ~25 tok/s | Balanced performance |\n\n```bash\n# Pull your preferred chat model\nollama pull qwen2.5:32b\n```\n\n### Verifying Ollama Connection\n\n```bash\n# Test Ollama is running\ncurl http://localhost:11434/api/tags\n\n# Test embedding model\ncurl http://localhost:11434/api/embeddings -d '{\n  \"model\": \"bge-m3\",\n  \"prompt\": \"test\"\n}'\n```\n\n## Troubleshooting\n\n### Container Won't Start\n\n**Problem:** `docker compose up` fails\n\n**Solutions:**\n```bash\n# Check Docker is running\ndocker info\n\n# Check port availability\nlsof -i :8080  # macOS/Linux\nnetstat -ano | findstr :8080  # Windows\n\n# View logs\ndocker compose logs knowledgevault\n```\n\n### LLM Unavailable Error\n\n**Problem:** Health check shows \"LLM unavailable\"\n\n**Solutions:**\n1. Verify Ollama is running: `ollama list`\n2. Check Ollama URL in `.env` matches your setup\n3. For Linux, use host IP instead of `host.docker.internal`:\n   ```bash\n   OLLAMA_CHAT_URL=http://192.168.1.100:11434\n   ```\n\n### Documents Not Processing\n\n**Problem:** Uploaded files stay in \"pending\" status\n\n**Solutions:**\n1. Check logs: `docker compose logs -f knowledgevault`\n2. Verify file format is supported\n3. Check disk space in data directory\n4. Restart container: `docker compose restart`\n\n### Out of Memory\n\n**Problem:** Container crashes during document processing\n\n**Solutions:**\n1. Reduce `CHUNK_SIZE` in `.env` (e.g., 256)\n2. Process fewer files at once\n3. Increase Docker memory limit\n4. Use smaller chat model\n\n### Slow Responses\n\n**Problem:** Chat responses are very slow\n\n**Solutions:**\n1. Use a smaller/faster chat model\n2. Reduce `MAX_CONTEXT_CHUNKS` in `.env`\n3. Increase `RAG_RELEVANCE_THRESHOLD` to filter more chunks\n4. Ensure Ollama has GPU access if available\n\n## API Endpoints\n\n### Health \u0026 Status\n\n| Method | Endpoint | Description |\n|--------|----------|-------------|\n| GET | `/health` | Service health status |\n\n### Authentication\n\n| Method | Endpoint | Description |\n|--------|----------|-------------|\n| POST | `/api/v1/auth/register` | Register new user (first = superadmin) |\n| POST | `/api/v1/auth/login` | Login with username/password |\n| POST | `/api/v1/auth/refresh` | Refresh access token |\n| POST | `/api/v1/auth/logout` | Logout and revoke refresh token |\n| GET | `/api/v1/auth/me` | Get current user profile (includes `must_change_password` flag) |\n| PATCH | `/api/v1/auth/me` | Update current user profile |\n| POST | `/api/v1/auth/change-password` | Change user password (validates strength policy) |\n| GET | `/api/v1/auth/sessions` | List active sessions for current user |\n| DELETE | `/api/v1/auth/sessions/{id}` | Revoke a specific session |\n| DELETE | `/api/v1/auth/sessions` | Revoke all other sessions |\n\n### Vaults\n\n| Method | Endpoint | Description |\n|--------|----------|-------------|\n| GET | `/api/v1/vaults` | List all vaults with counts |\n| GET | `/api/v1/vaults/accessible` | List vault IDs user has access to |\n| GET | `/api/v1/vaults/{id}` | Get vault details |\n| POST | `/api/v1/vaults` | Create new vault |\n| PUT | `/api/v1/vaults/{id}` | Update vault |\n| DELETE | `/api/v1/vaults/{id}` | Delete vault |\n\n### Chat\n\n| Method | Endpoint | Description |\n|--------|----------|-------------|\n| POST | `/api/v1/chat` | Non-streaming chat (requires vault_id with read access) |\n| POST | `/api/v1/chat/stream` | Streaming chat (SSE, requires vault_id with read access) |\n\n### Documents\n\n| Method | Endpoint | Description |\n|--------|----------|-------------|\n| GET | `/api/v1/documents` | List all documents |\n| GET | `/api/v1/documents/stats` | Document statistics |\n| POST | `/api/v1/documents/upload` | Upload file(s) |\n| POST | `/api/v1/documents/scan` | Trigger directory scan |\n| DELETE | `/api/v1/documents/{id}` | Delete document |\n\n### Search\n\n| Method | Endpoint | Description |\n|--------|----------|-------------|\n| POST | `/api/v1/search` | Semantic search |\n| POST | `/api/v1/search/chunks` | Search document chunks |\n\n### Memories\n\n| Method | Endpoint | Description |\n|--------|----------|-------------|\n| GET | `/api/v1/memories` | List all memories |\n| GET | `/api/v1/memories/search` | Search memories |\n| POST | `/api/v1/memories` | Create memory |\n| PUT | `/api/v1/memories/{id}` | Update memory |\n| DELETE | `/api/v1/memories/{id}` | Delete memory |\n\n### Settings\n\n| Method | Endpoint | Description |\n|--------|----------|-------------|\n| GET | `/api/v1/settings` | Get settings |\n| PUT | `/api/v1/settings` | Update settings |\n\n### API Documentation\n\nInteractive API docs available at: `http://localhost:8080/docs`\n\nOpenAPI schema: `http://localhost:8080/openapi.json`\n\n## Frontend Usage\n\n### Navigation\n\nThe web interface uses a navigation rail with five sections:\n\n1. **Chat** - Ask questions about your documents using the streaming AI interface\n2. **Search** - Find specific content in your knowledge base\n3. **Documents** - Upload and manage documents\n4. **Memory** - View and manage stored memories\n5. **Settings** - Configure application settings (includes password change for forced updates)\n\n### Authentication Flow\n\nThe frontend implements secure route protection with the following features:\n\n- **JWT Authentication**: Access tokens with automatic refresh on expiration\n- **Password Policy Enforcement**: Users with `must_change_password=true` are automatically redirected to change their password\n- **Role-Based Access**: Admin-only routes are protected via `AdminRoute` component\n- **Session Management**: Users can view and revoke active sessions from Settings\n\n### Chat Interface\n\n1. Type your question in the input field\n2. Press Enter or click Send\n3. Watch the AI response stream in real-time (powered by SSE)\n4. Click \"Sources\" to see which documents were referenced\n5. Say \"Remember that...\" to save information to memory\n\n**Streaming**: The chat interface uses Server-Sent Events (SSE) for real-time response streaming with automatic token refresh on 401 errors.\n\n### Document Upload\n\n**Method 1: Web Upload**\n1. Go to Documents page\n2. Click \"Upload\" or drag files onto the drop zone\n3. Files are automatically processed and indexed\n\n**Method 2: Direct File Placement**\n1. Place files in `data/knowledgevault/vaults/{vault_id}/uploads/` (e.g., `data/knowledgevault/vaults/1/uploads/`)\n2. Click \"Scan Directory\" on Documents page\n3. Or wait for auto-scan (if enabled)\n\n### Search\n\n1. Go to Search page\n2. Enter search query\n3. Use filters to narrow results:\n   - File type\n   - Date range\n   - Relevance threshold\n4. Click results to view source context\n\n### Memory Management\n\n1. Go to Memory page to view all memories\n2. Use search to find specific memories\n3. Click edit icon to modify\n4. Click delete icon to remove\n5. Memories are automatically used in chat context\n\n## Development\n\n### Backend Development\n\n```bash\n# Run with hot-reload (includes frontend dev service)\ndocker compose -f docker-compose.yml -f docker-compose.override.yml up -d\n\n# View logs\ndocker compose logs -f backend\n\n# Run tests\ndocker compose exec backend pytest tests/\n```\n\n### Frontend Development\n\n```bash\ncd frontend\nnpm install\nnpm run dev\n```\n\n#### Frontend API Library\n\nThe frontend includes a streaming-capable API client (`frontend/src/lib/api.ts`):\n\n- `apiRequest\u003cT\u003e(method, path, body?)` - Standard REST requests with JWT auth\n- `apiStream(path, body, callbacks)` - SSE streaming with automatic token refresh\n\nThe auth store (`frontend/src/stores/authStore.ts`) tracks:\n- User session state\n- `must_change_password` flag from `/api/v1/auth/me` endpoint\n- Password change enforcement via `useRequirePasswordChange` hook\n\n### Building Production Images\n\n```bash\ndocker compose -f docker-compose.yml build\ndocker compose -f docker-compose.yml up -d\n```\n\n## Documentation\n\n### Feature Guides\n\n- **[Email Ingestion](docs/email-ingestion.md)** - Ingest documents via email with IMAP polling and automatic vault routing\n\n### Administration\n\n- **[Admin Guide](docs/admin-guide.md)** - Administrative tasks and configuration\n- **[Release Process](docs/release.md)** - Deployment and release procedures\n- **[Non-Technical Setup](docs/non-technical-setup.md)** - Setup guide for non-technical users\n\n## License\n\nNo license file present. Add LICENSE file or update this section as needed.\n\n## Support\n\n- Documentation: See `docs/` directory\n- Issues: Create an issue in the repository\n- Admin Guide: See `docs/admin-guide.md`\n- Non-Technical Setup: See `docs/non-technical-setup.md`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzaxbysauce%2Fragapp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzaxbysauce%2Fragapp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzaxbysauce%2Fragapp/lists"}