https://github.com/pinkpixel-dev/web-scout-mcp
A powerful MCP server extension providing web search and content extraction capabilities. Integrates DuckDuckGo search functionality and URL content extraction into your MCP environment, enabling AI assistants to search the web and extract webpage content programmatically.
https://github.com/pinkpixel-dev/web-scout-mcp
ai-assistant ai-tools cheerio content-extraction crawler duckduckgo duckduckgo-search google-search mcp mcp-server web-content web-crawler web-scraper web-scraping web-search web-search-agent
Last synced: 3 months ago
JSON representation
A powerful MCP server extension providing web search and content extraction capabilities. Integrates DuckDuckGo search functionality and URL content extraction into your MCP environment, enabling AI assistants to search the web and extract webpage content programmatically.
- Host: GitHub
- URL: https://github.com/pinkpixel-dev/web-scout-mcp
- Owner: pinkpixel-dev
- License: apache-2.0
- Created: 2025-05-10T18:29:09.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2026-02-13T22:18:28.000Z (4 months ago)
- Last Synced: 2026-02-14T04:44:06.408Z (4 months ago)
- Topics: ai-assistant, ai-tools, cheerio, content-extraction, crawler, duckduckgo, duckduckgo-search, google-search, mcp, mcp-server, web-content, web-crawler, web-scraper, web-scraping, web-search, web-search-agent
- Language: JavaScript
- Homepage: https://pinkpixel.dev
- Size: 8.55 MB
- Stars: 116
- Watchers: 1
- Forks: 11
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project
README
[](https://mseep.ai/app/pinkpixel-dev-web-scout-mcp)
Web Scout MCP Server
An MCP server for web search using DuckDuckGo and content extraction, with support for multiple URLs and memory optimizations.
## ✨ Features
- 🔍 **DuckDuckGo Search**: Fast and privacy-focused web search capability
- 📄 **Content Extraction**: Clean, readable text extraction from web pages
- 🚀 **Parallel Processing**: Support for extracting content from multiple URLs simultaneously
- 💾 **Memory Optimization**: Smart memory management to prevent application crashes
- ⏱️ **Rate Limiting**: Intelligent request throttling to avoid API blocks
- 🛡️ **Error Handling**: Robust error handling for reliable operation
## 📦 Installation
### Installing via Smithery
To install Web Scout for Claude Desktop automatically via [Smithery](https://smithery.ai/server/@pinkpixel-dev/web-scout-mcp):
```bash
npx -y @smithery/cli install @pinkpixel-dev/web-scout-mcp --client claude
```
### Global Installation
```bash
npm install -g @pinkpixel/web-scout-mcp
```
### Local Installation
```bash
npm install @pinkpixel/web-scout-mcp
```
## 🚀 Usage
### Command Line
After installing globally, run:
```bash
web-scout-mcp
```
### With MCP Clients
Add this to your MCP client's `config.json` (Claude Desktop, Cursor, etc.):
```json
{
"mcpServers": {
"web-scout": {
"command": "npx",
"args": [
"-y",
"@pinkpixel/web-scout-mcp@latest"
]
}
}
}
```
### Environment Variables
Set the `WEB_SCOUT_DISABLE_AUTOSTART=1` environment variable when embedding the package and calling `createServer()` yourself. By default running the published entrypoint (for example `node dist/index.js` or `npx @pinkpixel/web-scout-mcp`) automatically bootstraps the stdio transport.
## 🧰 Tools
The server provides the following MCP tools:
### 🔍 DuckDuckGoWebSearch
Initiates a web search query using the DuckDuckGo search engine and returns a well-structured list of findings.
**Input:**
- `query` (string): The search query string
- `maxResults` (number, optional): Maximum number of results to return (default: 10)
**Example:**
```json
{
"query": "latest advancements in AI",
"maxResults": 5
}
```
**Output:**
A formatted list of search results with titles, URLs, and snippets.
### 📄 UrlContentExtractor
Fetches and extracts clean, readable content from web pages by removing unnecessary elements like scripts, styles, and navigation.
**Input:**
- `url`: Either a single URL string or an array of URL strings
**Example (single URL):**
```json
{
"url": "https://example.com/article"
}
```
**Example (multiple URLs):**
```json
{
"url": [
"https://example.com/article1",
"https://example.com/article2"
]
}
```
**Output:**
Extracted text content from the specified URL(s).
## 🛠️ Development
```bash
# Clone the repository
git clone https://github.com/pinkpixel-dev/web-scout-mcp.git
cd web-scout-mcp
# Install dependencies
npm install
# Build
npm run build
# Run
npm start
```
## 📚 Documentation
For more detailed information about the project, check out these resources:
- [OVERVIEW.md](OVERVIEW.md) - Technical overview and architecture
- [CONTRIBUTING.md](CONTRIBUTING.md) - Guidelines for contributors
- [CHANGELOG.md](CHANGELOG.md) - Version history and changes
## 📋 Requirements
- Node.js >= 18.0.0
- npm or yarn
## 📄 License
This project is licensed under the [Apache 2.0 License](LICENSE).
Made with ❤️ by Pink Pixel
✨ Dream it, Pixel it ✨