{"id":42189904,"url":"https://github.com/stoerr/openwebui-knowledgesync","last_synced_at":"2026-01-26T22:40:10.891Z","repository":{"id":306626721,"uuid":"1026789196","full_name":"stoerr/openwebui-knowledgesync","owner":"stoerr","description":"synchronizes a directory with e.g. markdown files with a knowledge base in OpenWebUI","archived":false,"fork":false,"pushed_at":"2025-07-26T16:18:59.000Z","size":23,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-07-26T21:25:00.719Z","etag":null,"topics":["ai","knowledge-management","openwebui","rag"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/stoerr.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-07-26T16:05:36.000Z","updated_at":"2025-07-26T16:19:02.000Z","dependencies_parsed_at":"2025-07-26T21:36:00.412Z","dependency_job_id":null,"html_url":"https://github.com/stoerr/openwebui-knowledgesync","commit_stats":null,"previous_names":["stoerr/openwebui-knowledgesync"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/stoerr/openwebui-knowledgesync","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stoerr%2Fopenwebui-knowledgesync","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stoerr%2Fopenwebui-knowledgesync/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stoerr%2Fopenwebui-knowledgesync/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stoerr%2Fopenwebui-knowledgesync/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/stoerr","download_url":"https://codeload.github.com/stoerr/openwebui-knowledgesync/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stoerr%2Fopenwebui-knowledgesync/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28790563,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-26T21:49:50.245Z","status":"ssl_error","status_checked_at":"2026-01-26T21:48:29.455Z","response_time":59,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","knowledge-management","openwebui","rag"],"created_at":"2026-01-26T22:40:09.266Z","updated_at":"2026-01-26T22:40:10.747Z","avatar_url":"https://github.com/stoerr.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# OpenWebUI Knowledge Sync\n\nA command-line tool to synchronize local directories with [OpenWebUI](https://docs.openwebui.com/)\nknowledge bases: use an AI for your local knowledge without uploading files to a public service.\n\nEach directory can be synchronized to its own knowledge base (one on one relationship!), \nkeeping your knowledge collections in sync with your local file system automatically. \nI've got several directories with knowledge files where I write down\nstuff, and a cron job runs this tool periodically to keep the corresponding knowledge bases up to date.\n\nI developed that since I need to query knowledge I have in markdown files which\nI don't want or cannot upload into public services like an OpenAI GPT / assistant but rather want to keep only locally.\nFor that OpenWebUI is great, but having to drag and drop files into the web interface is cumbersome, but, as of\n7/25, OpenWebUI does not have tool to support synchronizing directories with knowledge bases anymore.\nI'm using OpenWebUI together with local models running on [LM Studio](https://lmstudio.ai/).\n\nCaution: this is vibe-coded in the sense that I wrote the specifications but hardly even studied the code, so this is not\nrepresentative for my professional coding skills. :-) I just needed something quick that works good enough. Even most of\nthis README is vibe-coded.\n\n## Features\n\n- 🔄 **Directory-to-Knowledge Base Sync**: Each directory synchronizes to its own knowledge base\n- 📁 **Multi-directory Support**: Synchronize different directories to different knowledge bases\n- 🎯 **Regex filtering**: Include only files matching your specified pattern\n- 🔍 **Hash verification**: Uses SHA256 to detect file changes and avoid duplicate uploads\n- 💾 **Smart deduplication**: Reuses existing files with matching content instead of re-uploading\n- ⚡ **Zero dependencies**: Pure Node.js implementation, no external packages required\n- 🛡️ **Error handling**: Comprehensive validation and cleanup on failures\n\n## Installation\n\nClone the repository and link the script to your PATH for easy access,\nassuming you have a `bin` directory in your home folder:\n\n```bash\ngit clone https://github.com/your-username/openwebui-knowledgesync.git\nln -s openwebui-knowledgesync/openwebui-knowledgesync $HOME/bin/openwebui-knowledgesync\n```\n\n## Setup Process\n\n### 1. Find Your Knowledge Base ID\n\nFirst, discover the ID of the knowledge base you want to sync with:\n\n```bash\nopenwebui-knowledgesync listkb\n```\n\nThis will show all available knowledge bases with their IDs, names, and descriptions. Copy the ID of the knowledge base\nyou want to use.\n\n### 2. Create Configuration File\n\nIn each directory you want to synchronize, create a `.open-webui-knowledgesync` configuration file:\n\n```bash\nknowledge_base_id=your-knowledge-base-id-from-step-1\nopen_webui_api_key=your-api-key\nopen_webui_api_url=http://localhost:3000\nkbdir_id=unique-directory-identifier\nfileregex=\\.(md|txt|pdf|docx?)$\n```\n\nAn open_webui_api_key can be created in OpenWebUI under users settings -\u003e account -\u003e API keys.\n\n### 3. Choose a Unique Directory Identifier\n\nThe `kbdir_id` must be unique for each directory you synchronize. This identifier:\n\n- Distinguishes files from different directories in the same knowledge base\n- Should be descriptive (e.g., `project-docs`, `user-manual`, `api-specs`)\n- Can contain letters, numbers, hyphens, and underscores\n- Must be different for each directory, even if syncing to different knowledge bases\n\n## Configuration Parameters\n\n| Parameter            | Required | Description                             | Example                                |\n|----------------------|----------|-----------------------------------------|----------------------------------------|\n| `knowledge_base_id`  | Yes      | Target knowledge base ID from OpenWebUI | `a836f4b9-2ce6-4d30-bd28-ebd56c59eab3` |\n| `open_webui_api_key` | Yes      | Your OpenWebUI API authentication key   | `sk-1234567890abcdef...`               |\n| `open_webui_api_url` | No       | OpenWebUI instance URL                  | `http://localhost:3000` (default)      |\n| `kbdir_id`           | Yes      | Unique identifier for this directory    | `docs`, `project-x`, etc.              |\n| `fileregex`          | No       | Regex pattern for files to include      | `\\.(md\\|txt\\|pdf)$`                    |\n\n## File Path Encoding\n\nFiles are uploaded with encoded paths that preserve the directory structure:\n\n- Directory separators (`/` or `\\`) are replaced with `%%`\n- The `kbdir_id` is prefixed to identify the source directory\n- Example: `docs/api/endpoints.md` becomes `project-docs%%docs%%api%%endpoints.md`\n\nThis encoded format is visible in the OpenWebUI interface, making it easy to identify which directory and path each file\ncame from.\n\n## Usage\n\n### Commands\n\n```bash\n# Synchronize current directory with knowledge base. This is the main command, the rest is for setup / debugging.\nopenwebui-knowledgesync sync\n\n# List all available knowledge bases\nopenwebui-knowledgesync listkb\n\n# List all uploaded files\nopenwebui-knowledgesync listfiles\n\n# List files in a specific knowledge base\nopenwebui-knowledgesync listkbfiles\n\n# Download a file by ID to stdout\nopenwebui-knowledgesync download \u003cfile-id\u003e\n```\n\n### Basic Workflow\n\n1. **Discover**: Use `listkb` to find your knowledge base ID\n2. **Setup**: Create configuration file in your directory with unique `kbdir_id`\n3. **Sync**: Run `openwebui-knowledgesync sync` to synchronize files. Can be run periodically (e.g. with cron) or manually.\n4. **Iterate**: Modify files locally and re-run sync as needed\n\n### Example Setup\n\n```bash\n# Navigate to your documentation directory\ncd /path/to/your/project-docs\n\n# Find knowledge base ID\nopenwebui-knowledgesync listkb\n\n# Create configuration with unique kbdir_id\ncat \u003e .open-webui-knowledgesync \u003c\u003c EOF\nknowledge_base_id=a836f4b9-2ce6-4d30-bd28-ebd56c59eab3\nopen_webui_api_key=sk-your-api-key-here\nopen_webui_api_url=http://localhost:3000\nkbdir_id=project-docs\nfileregex=\\.(md|txt)$\nEOF\n\n# Perform initial sync\nopenwebui-knowledgesync sync\n```\n\nFor a second directory:\n\n```bash\n# Navigate to another directory\ncd /path/to/your/api-specs\n\n# Create configuration with different kbdir_id\ncat \u003e .open-webui-knowledgesync \u003c\u003c EOF\nknowledge_base_id=b947c3a8-8e71-5f41-ce39-f68ef7c4fb6d\nopen_webui_api_key=sk-your-api-key-here\nopen_webui_api_url=http://localhost:3000\nkbdir_id=api-specs\nfileregex=\\.(json|yaml|md)$\nEOF\n\n# Sync this directory to a different knowledge base\nopenwebui-knowledgesync sync\n```\n\n## How It Works\n\n### Multi-Directory Support\n\n- Each directory maintains its own configuration file\n- Files are prefixed with `kbdir_id` to avoid conflicts\n- Different directories can sync to the same or different knowledge bases\n- The tool only manages files with its specific `kbdir_id` prefix\n\n### Synchronization Process\n\n1. **Scan local files** matching the regex pattern\n2. **Fetch knowledge base** content and filter by `kbdir_id`\n3. **Compare hashes** to identify changes\n4. **Remove outdated files** belonging to this directory\n5. **Upload new/changed files** with deduplication\n6. **Add files to collection** and verify success\n\n## Requirements\n\n- Node.js (version 12 or higher)\n- Access to an OpenWebUI instance\n- Valid API key with knowledge base permissions\n\n## Troubleshooting\n\n### Common Issues\n\n**Configuration not found**\n\n```bash\nConfiguration file not found: /path/to/.open-webui-knowledgesync\n```\n\nEnsure the configuration file exists in your current directory.\n\n**Hash mismatch warnings**\n\n```bash\nHash mismatch for uploaded file: local vs uploaded hash different\n```\n\nThis may indicate file encoding issues. Ensure files are valid UTF-8.\n\n**Upload failures**\n\n```bash\nUpload failed: No file ID in response\n```\n\nCheck your API key permissions and OpenWebUI instance connectivity.\n\n### Debug Mode\n\nEnable debug output by modifying the script:\n\n```javascript\nconst debug = true; // Set to true for debug output\n```\n\n## License\n\nMIT License - see [LICENSE](LICENSE) file for details.\n\n## Related Projects\n\n- [OpenWebUI](https://github.com/open-webui/open-webui) - The main OpenWebUI project\n- [OpenWebUI Documentation](https://docs.openwebui.com/) - Official documentation\n\n## Bugs\n\nIt seems the sha256 calculation of OpenWebUI is somehow different than\nthis in some cases. That means the file will be uploaded again each time.\nI don't know how that can be fixed at the moment.\n\n---\n\n**Note**: This tool is not affiliated with the OpenWebUI project in any way.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstoerr%2Fopenwebui-knowledgesync","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstoerr%2Fopenwebui-knowledgesync","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstoerr%2Fopenwebui-knowledgesync/lists"}