{"id":27521170,"url":"https://github.com/alizdavoodi/mcpdocsearch","last_synced_at":"2026-05-13T20:32:56.764Z","repository":{"id":287685865,"uuid":"965486658","full_name":"alizdavoodi/MCPDocSearch","owner":"alizdavoodi","description":"This project provides a toolset to crawl websites, generate Markdown documentation, and make that documentation searchable via a Model Context Protocol (MCP) server, designed for integration with tools like Cursor.","archived":false,"fork":false,"pushed_at":"2025-04-13T10:20:19.000Z","size":159,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-13T10:34:32.778Z","etag":null,"topics":["crawler","mcp","mcp-server"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/alizdavoodi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-04-13T09:20:27.000Z","updated_at":"2025-04-13T10:20:22.000Z","dependencies_parsed_at":"2025-04-13T10:44:45.588Z","dependency_job_id":null,"html_url":"https://github.com/alizdavoodi/MCPDocSearch","commit_stats":null,"previous_names":["alizdavoodi/mcpdocsearch"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alizdavoodi%2FMCPDocSearch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alizdavoodi%2FMCPDocSearch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alizdavoodi%2FMCPDocSearch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alizdavoodi%2FMCPDocSearch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/alizdavoodi","download_url":"https://codeload.github.com/alizdavoodi/MCPDocSearch/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249458964,"owners_count":21275807,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","mcp","mcp-server"],"created_at":"2025-04-18T07:59:19.348Z","updated_at":"2026-05-13T20:32:56.732Z","avatar_url":"https://github.com/alizdavoodi.png","language":"Python","funding_links":[],"categories":["Search \u0026 Data Extraction"],"sub_categories":["How to Submit"],"readme":"# Documentation Crawler \u0026 MCP Server\n\nThis project provides a toolset to crawl websites, generate Markdown documentation, and make that documentation searchable via a Model Context Protocol (MCP) server, designed for integration with tools like Cursor.\n\n## Features\n\n- **Web Crawler (`crawler_cli`)**:\n  - Crawls websites starting from a given URL using `crawl4ai`.\n  - Configurable crawl depth, URL patterns (include/exclude), content types, etc.\n  - Optional cleaning of HTML before Markdown conversion (removes nav links, headers, footers).\n  - Generates a single, consolidated Markdown file from crawled content.\n  - Saves output to `./storage/` by default.\n- **MCP Server (`mcp_server`)**:\n  - Loads Markdown files from the `./storage/` directory.\n  - Parses Markdown into semantic chunks based on headings.\n  - Generates vector embeddings for each chunk using `sentence-transformers` (`multi-qa-mpnet-base-dot-v1`).\n  - **Caching:** Utilizes a cache file (`storage/document_chunks_cache.pkl`) to store processed chunks and embeddings.\n    - **First Run:** The initial server startup after crawling new documents may take some time as it needs to parse, chunk, and generate embeddings for all content.\n    - **Subsequent Runs:** If the cache file exists and the modification times of the source `.md` files in `./storage/` haven't changed, the server loads directly from the cache, resulting in much faster startup times.\n    - **Cache Invalidation:** The cache is automatically invalidated and regenerated if any `.md` file in `./storage/` is modified, added, or removed since the cache was last created.\n  - Exposes MCP tools via `fastmcp` for clients like Cursor:\n    - `list_documents`: Lists available crawled documents.\n    - `get_document_headings`: Retrieves the heading structure for a document.\n    - `search_documentation`: Performs semantic search over document chunks using vector similarity.\n- **Cursor Integration**: Designed to run the MCP server via `stdio` transport for use within Cursor.\n\n## Workflow\n\n1. **Crawl:** Use the `crawler_cli` tool to crawl a website and generate a `.md` file in `./storage/`.\n2. **Run Server:** Configure and run the `mcp_server` (typically managed by an MCP client like Cursor).\n3. **Load \u0026 Embed:** The server automatically loads, chunks, and embeds the content from the `.md` files in `./storage/`.\n4. **Query:** Use the MCP client (e.g., Cursor Agent) to interact with the server's tools (`list_documents`, `search_documentation`, etc.) to query the crawled content.\n\n## Setup\n\nThis project uses [`uv`](https://github.com/astral-sh/uv) for dependency management and execution.\n\n1. **Install `uv`**: Follow the instructions on the [uv website](https://github.com/astral-sh/uv).\n2. **Clone the repository:**\n\n   ```bash\n   git clone https://github.com/alizdavoodi/MCPDocSearch.git\n   cd MCPDocSearch\n   ```\n\n3. **Install dependencies:**\n\n   ```bash\n   uv sync\n   ```\n\n   This command creates a virtual environment (usually `.venv`) and installs all dependencies listed in `pyproject.toml`.\n\n## Usage\n\n### 1. Crawling Documentation\n\nRun the crawler using the `crawl.py` script or directly via `uv run`.\n\n**Basic Example:**\n\n```bash\nuv run python crawl.py https://docs.example.com\n```\n\nThis will crawl `https://docs.example.com` with default settings and save the output to `./storage/docs.example.com.md`.\n\n**Example with Options:**\n\n```bash\nuv run python crawl.py https://docs.another.site --output ./storage/custom_name.md --max-depth 2 --keyword \"API\" --keyword \"Reference\" --exclude-pattern \"*blog*\"\n```\n\n**View all options:**\n\n```bash\nuv run python crawl.py --help\n```\n\nKey options include:\n\n- `--output`/`-o`: Specify output file path.\n- `--max-depth`/`-d`: Set crawl depth (must be between 1 and 5).\n- `--include-pattern`/`--exclude-pattern`: Filter URLs to crawl.\n- `--keyword`/`-k`: Keywords for relevance scoring during crawl.\n- `--remove-links`/`--keep-links`: Control HTML cleaning.\n- `--cache-mode`: Control `crawl4ai` caching (`DEFAULT`, `BYPASS`, `FORCE_REFRESH`).\n- `--wait-for`: Wait for a specific time (seconds) or CSS selector before capturing content (e.g., `5` or `'css:.content'`). Useful for pages with delayed loading.\n- `--js-code`: Execute custom JavaScript on the page before capturing content.\n- `--page-load-timeout`: Set the maximum time (seconds) to wait for a page to load.\n- `--wait-for-js-render`/`--no-wait-for-js-render`: Enable a specific script to better handle JavaScript-heavy Single Page Applications (SPAs) by scrolling and clicking potential \"load more\" buttons. Automatically sets a default wait time if `--wait-for` is not specified.\n\n#### Refining Crawls with Patterns and Depth\n\nSometimes, you might want to crawl only a specific subsection of a documentation site. This often requires some trial and error with `--include-pattern` and `--max-depth`.\n\n- **`--include-pattern`**: Restricts the crawler to only follow links whose URLs match the given pattern(s). Use wildcards (`*`) for flexibility.\n- **`--max-depth`**: Controls how many \"clicks\" away from the starting URL the crawler will go. A depth of 1 means it only crawls pages directly linked from the start URL. A depth of 2 means it crawls those pages _and_ pages linked from them (if they also match include patterns), and so on.\n\n**Example: Crawling only the Pulsar Admin API section**\n\nSuppose you want only the content under `https://pulsar.apache.org/docs/4.0.x/admin-api-*`.\n\n1. **Start URL:** You could start at the overview page: `https://pulsar.apache.org/docs/4.0.x/admin-api-overview/`.\n2. **Include Pattern:** You only want links containing `admin-api`: `--include-pattern \"*admin-api*\"`.\n3. **Max Depth:** You need to figure out how many levels deep the admin API links go from the starting page. Start with `2` and increase if needed.\n4. **Verbose Mode:** Use `-v` to see which URLs are being visited or skipped, which helps debug the patterns and depth.\n\n```bash\nuv run python crawl.py https://pulsar.apache.org/docs/4.0.x/admin-api-overview/ -v --include-pattern \"*admin-api*\" --max-depth 2\n```\n\nCheck the output file (`./storage/pulsar.apache.org.md` by default in this case). If pages are missing, try increasing `--max-depth` to `3`. If too many unrelated pages are included, make the `--include-pattern` more specific or add `--exclude-pattern` rules.\n\n### 2. Running the MCP Server\n\nThe MCP server is designed to be run by an MCP client like Cursor via the `stdio` transport. The command to run the server is:\n\n```bash\npython -m mcp_server.main\n```\n\nHowever, it needs to be run from the project's root directory (`MCPDocSearch`) so that Python can find the `mcp_server` module.\n\n## ⚠️ Caution: Embedding Time\n\nThe MCP server generates embeddings locally the first time it runs or whenever the source Markdown files in `./storage/` change. This process involves loading a machine learning model and processing all the text chunks.\n\n- **Time Varies:** The time required for embedding generation can vary significantly based on:\n  - **Hardware:** Systems with a compatible GPU (CUDA or Apple Silicon/MPS) will be much faster than CPU-only systems.\n  - **Data Size:** The total number of Markdown files and their content length directly impacts processing time.\n- **Be Patient:** For large documentation sets or on slower hardware, the initial startup (or startup after changes) might take several minutes. Subsequent startups using the cache will be much faster. ⏳\n\n\n### 3. Configuring Cursor/Claude for Desktop\n\nTo use this server with Cursor, create a `.cursor/mcp.json` file in the root of this project (`MCPDocSearch/.cursor/mcp.json`) with the following content:\n\n```json\n{\n  \"mcpServers\": {\n    \"doc-query-server\": {\n      \"command\": \"uv\",\n      \"args\": [\n        \"--directory\",\n        // IMPORTANT: Replace with the ABSOLUTE path to this project directory on your machine\n        \"/path/to/your/MCPDocSearch\",\n        \"run\",\n        \"python\",\n        \"-m\",\n        \"mcp_server.main\"\n      ],\n      \"env\": {}\n    }\n  }\n}\n```\n\n**Explanation:**\n\n- `\"doc-query-server\"`: A name for the server within Cursor.\n- `\"command\": \"uv\"`: Specifies `uv` as the command runner.\n- `\"args\"`:\n  - `\"--directory\", \"/path/to/your/MCPDocSearch\"`: **Crucially**, tells `uv` to change its working directory to your project root before running the command. **Replace `/path/to/your/MCPDocSearch` with the actual absolute path on your system.**\n  - `\"run\", \"python\", \"-m\", \"mcp_server.main\"`: The command `uv` will execute within the correct directory and virtual environment.\n\nAfter saving this file and restarting Cursor, the \"doc-query-server\" should become available in Cursor's MCP settings and usable by the Agent (e.g., `@doc-query-server search documentation for \"how to install\"`).\n\nFor Claude for Desktop, you can use this [official documentation](https://modelcontextprotocol.io/quickstart/server#mac-os-linux) to set up the MCP server\n\n\n## Dependencies\n\nKey libraries used:\n\n- `crawl4ai`: Core web crawling functionality.\n- `fastmcp`: MCP server implementation.\n- `sentence-transformers`: Generating text embeddings.\n- `torch`: Required by `sentence-transformers`.\n- `typer`: Building the crawler CLI.\n- `uv`: Project and environment management.\n- `beautifulsoup4` (via `crawl4ai`): HTML parsing.\n- `rich`: Enhanced terminal output.\n\n## Architecture\n\nThe project follows this basic flow:\n\n1. **`crawler_cli`**: You run this tool, providing a starting URL and options.\n2. **Crawling (`crawl4ai`)**: The tool uses `crawl4ai` to fetch web pages, following links based on configured rules (depth, patterns).\n3. **Cleaning (`crawler_cli/markdown.py`)**: Optionally, HTML content is cleaned (removing navigation, links) using BeautifulSoup.\n4. **Markdown Generation (`crawl4ai`)**: Cleaned HTML is converted to Markdown.\n5. **Storage (`./storage/`)**: The generated Markdown content is saved to a file in the `./storage/` directory.\n6. **`mcp_server` Startup**: When the MCP server starts (usually via Cursor's config), it runs `mcp_server/data_loader.py`.\n7. **Loading \u0026 Caching**: The data loader checks for a cache file (`.pkl`). If valid, it loads chunks and embeddings from the cache. Otherwise, it reads `.md` files from `./storage/`.\n8. **Chunking \u0026 Embedding**: Markdown files are parsed into chunks based on headings. Embeddings are generated for each chunk using `sentence-transformers` and stored in memory (and saved to cache).\n9. **MCP Tools (`mcp_server/mcp_tools.py`)**: The server exposes tools (`list_documents`, `search_documentation`, etc.) via `fastmcp`.\n10. **Querying (Cursor)**: An MCP client like Cursor can call these tools. `search_documentation` uses the pre-computed embeddings to find relevant chunks based on semantic similarity to the query.\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n## Contributing\n\nContributions are welcome! Please feel free to open an issue or submit a pull request.\n\n## Security Notes\n\n- **Pickle Cache:** This project uses Python's `pickle` module to cache processed data (`storage/document_chunks_cache.pkl`). Unpickling data from untrusted sources can be insecure. Ensure that the `./storage/` directory is only writable by trusted users/processes.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falizdavoodi%2Fmcpdocsearch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falizdavoodi%2Fmcpdocsearch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falizdavoodi%2Fmcpdocsearch/lists"}