{"id":47359227,"url":"https://github.com/Zjianru/web-search-pro","last_synced_at":"2026-04-01T17:01:15.109Z","repository":{"id":342257127,"uuid":"1153390977","full_name":"Zjianru/web-search-pro","owner":"Zjianru","description":"Multi-engine web search skill for OpenClaw with full parameter control. Supports Tavily, Exa, Serper, SerpAPI with domain filtering, date ranges, deep search, news mode, and content extraction. 多引擎精细化搜索 Skill。","archived":false,"fork":false,"pushed_at":"2026-03-12T03:08:42.000Z","size":19,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-03-12T08:40:49.745Z","etag":null,"topics":["ai-agent","exa","multi-engine","openclaw","openclaw-skill","serpapi","serper","tavily","web-search"],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Zjianru.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-02-09T08:45:06.000Z","updated_at":"2026-03-12T03:08:45.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/Zjianru/web-search-pro","commit_stats":null,"previous_names":["zjianru/web-search-pro"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/Zjianru/web-search-pro","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Zjianru%2Fweb-search-pro","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Zjianru%2Fweb-search-pro/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Zjianru%2Fweb-search-pro/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Zjianru%2Fweb-search-pro/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Zjianru","download_url":"https://codeload.github.com/Zjianru/web-search-pro/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Zjianru%2Fweb-search-pro/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31290538,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-01T13:12:26.723Z","status":"ssl_error","status_checked_at":"2026-04-01T13:12:25.102Z","response_time":53,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-agent","exa","multi-engine","openclaw","openclaw-skill","serpapi","serper","tavily","web-search"],"created_at":"2026-03-18T12:00:34.520Z","updated_at":"2026-04-01T17:01:15.056Z","avatar_url":"https://github.com/Zjianru.png","language":"JavaScript","funding_links":[],"categories":["Skills \u0026 Plugins"],"sub_categories":["Notable Skills"],"readme":"[English](README.md) | [中文](README_zh.md)\n\n# Web Search Pro\n\n`web-search-pro` is an OpenClaw search skill and local Node retrieval runtime for agents. It can\nsearch the live web, fetch and extract pages, crawl sites, map docs, and assemble research-ready\nevidence packs with explainable routing.\n\n- ClawHub: [web-search-pro](https://clawhub.ai/Zjianru/web-search-pro)\n- GitHub: [Zjianru/web-search-pro](https://github.com/Zjianru/web-search-pro)\n- OpenClaw archive: [openclaw/skills/tree/main/skills/zjianru/web-search-pro](https://github.com/openclaw/skills/tree/main/skills/zjianru/web-search-pro)\n\n## What It Is\n\nThis project sits between a lightweight web-search skill and a full hosted scraping product.\n\nThe simplest mental model is:\n\n- a search skill for agents\n- a local retrieval runtime\n- a bridge from `search` to `extract`, `crawl`, `map`, and `research`\n\nUse it when search is only the beginning of the task and the agent may need to keep collecting,\nstructuring, and handing off evidence.\n\n## Choose This If\n\nChoose `web-search-pro` if you need:\n\n- live web search and current-events lookup\n- news search with explainable routing and visible fallback behavior\n- official docs, API docs, and code lookup\n- company, product, and competitor research\n- site crawl, site map, and docs discovery\n- a no-key baseline first, then premium providers only when needed\n- structured outputs that an upstream model can keep using\n\nIn short: this is for developers who want one skill to cover search, retrieval, and evidence prep.\n\n## Do Not Choose This If\n\nDo not choose `web-search-pro` if you primarily want:\n\n- the lightest possible single-purpose web search wrapper\n- a hosted remote scraping SaaS\n- a browser-first crawler by default\n- a narrative report writer that hides retrieval details\n- an unlimited no-key search guarantee\n\nIf all you need is lightweight one-shot search, a smaller skill will usually be a better fit.\n\n## Why Developers Pick It\n\nCompared with a plain search skill, the differentiators are:\n\n- **Explainable routing**\n  `routingSummary` exposes why a provider was selected, how confident the planner is, and what the\n  top signals were.\n- **Visible federated gains**\n  `federated.value` shows what multi-provider fanout actually recovered, corroborated, or deduped.\n- **Search-to-research chain**\n  One surface can move from `search` into `extract`, `crawl`, `map`, and `research`.\n- **No-key baseline**\n  You can evaluate the skill without buying into a provider stack first.\n- **Agent-readable diagnostics**\n  `doctor.mjs`, `bootstrap.mjs`, `capabilities.mjs`, and `review.mjs` expose runtime state and\n  boundaries instead of leaving the model to guess.\n\n## Quick Start\n\nThere are two honest ways to approach this project:\n\n- **Install and use it as a skill**\n  Start from the ClawHub page or the OpenClaw archive entry if you want to use it inside OpenClaw.\n- **Run it from source**\n  Use the commands below if you want to evaluate the local runtime directly, inspect outputs, or\n  contribute to the repo.\n\nThe shortest successful path is:\n\n- start with the no-key baseline\n- add one premium provider only when you need stronger recall or fresher results\n- then try docs, news, and research flows\n\n### Option A: No-key baseline\n\nNo API key is required for the first successful run.\n\nThe baseline is:\n\n- `ddg` for best-effort web search\n- `fetch` for extract, crawl, and map fallback\n\n```bash\nnode scripts/doctor.mjs --json\nnode scripts/bootstrap.mjs --json\nnode scripts/search.mjs \"OpenAI Responses API docs\" --json\n```\n\nWhat these commands tell you:\n\n- `doctor.mjs` tells you whether the runtime is usable right now\n- `bootstrap.mjs` gives an agent-readable runtime snapshot\n- `search.mjs` proves the baseline path works before you add premium providers\n\n### Option B: Add one premium provider\n\nIf you only add one premium provider, start with `TAVILY_API_KEY`.\n\nThat is the shortest upgrade path because one credential improves:\n\n- general web search\n- news search\n- extract quality\n\n```bash\nexport TAVILY_API_KEY=tvly-xxxxx\nnode scripts/doctor.mjs --json\nnode scripts/search.mjs \"latest OpenAI news\" --type news --json\n```\n\n### First successful searches\n\n```bash\nnode scripts/search.mjs \"OpenClaw web search\" --json\nnode scripts/search.mjs \"OpenAI Responses API docs\" --preset docs --plan --json\nnode scripts/extract.mjs \"https://platform.openai.com/docs\" --json\n```\n\n### Then try docs, news, and research\n\n```bash\nnode scripts/search.mjs \"OpenAI Responses API docs\" --preset docs --json\nnode scripts/search.mjs \"latest OpenAI news\" --type news --json\nnode scripts/research.mjs \"OpenClaw search skill landscape\" --plan --json\n```\n\n## Core Commands\n\n| Command | Purpose |\n| --- | --- |\n| `search.mjs` | Multi-provider search with explainable routing |\n| `extract.mjs` | Single-page readable extraction |\n| `render.mjs` | Forced browser-backed extraction through the local render lane |\n| `crawl.mjs` | Safe BFS crawl |\n| `map.mjs` | Site-structure discovery |\n| `research.mjs` | Structured `plan + evidence pack` generation |\n| `doctor.mjs` | Runtime diagnostics |\n| `bootstrap.mjs` | Agent-readable runtime bootstrap contract |\n| `capabilities.mjs` | Provider capability snapshot |\n| `review.mjs` | Review and moderation summary |\n| `cache.mjs` | Cache inspection |\n| `health.mjs` | Provider health inspection |\n| `eval.mjs` | Retrieval and benchmark harness |\n\n## Why Federated Search Matters\n\nFederation is not just \"more providers\". It makes multi-provider value visible with compact,\nmachine-readable gain metrics.\n\nKey fields:\n\n- `federated.providersUsed`\n  Providers that actually returned results.\n- `federated.value.additionalProvidersUsed`\n  How many non-primary providers really contributed.\n- `federated.value.resultsRecoveredByFanout`\n  Final results that would disappear in primary-only mode.\n- `federated.value.resultsCorroboratedByFanout`\n  Final results supported by both the primary and at least one fanout provider.\n- `federated.value.duplicateSavings`\n  Exact or near-duplicate results removed by merge.\n- `routingSummary.federation.value`\n  The compact federation gain summary exposed alongside route explanation.\n\nExample:\n\n```json\n{\n  \"federated\": {\n    \"providersUsed\": [\"serper\", \"tavily\"],\n    \"value\": {\n      \"additionalProvidersUsed\": 1,\n      \"resultsWithFanoutSupport\": 2,\n      \"resultsRecoveredByFanout\": 1,\n      \"resultsCorroboratedByFanout\": 1,\n      \"duplicateSavings\": 1,\n      \"answerProvider\": \"tavily\",\n      \"primarySucceeded\": true\n    }\n  }\n}\n```\n\nInterpretation:\n\n- `resultsRecoveredByFanout=1` means federation recovered one final result that primary-only\n  search would have missed\n- `resultsCorroboratedByFanout=1` means another final result got multi-provider support\n- `duplicateSavings=1` means the merge removed one duplicate instead of wasting result slots\n\n## Routing And Output Contract\n\nThe router combines five layers of truth:\n\n1. provider capability facts\n2. structured query signals\n3. runtime policy from `config.json`\n4. local health state\n5. optional federation\n\nImportant fields:\n\n- `selectedProvider`\n  The primary route. It is not the same thing as \"the only provider used\".\n- `routingSummary`\n  Compact route explanation with `selectionMode`, `confidence`, `topSignals`, alternatives, and\n  federation context.\n- `routing.diagnostics`\n  Full route diagnostics exposed by `--explain-routing` or `--plan`.\n- `federated.providersUsed`\n  The provider set that actually returned results when fanout is active.\n- `federated.value`\n  Compact federation gain summary: added providers, recovered results, corroborated results, and\n  duplicate savings.\n- `cached` / `cache`\n  Cache hit plus age / TTL telemetry for agents.\n- `renderLane`\n  Runtime availability and policy summary for the browser-backed render lane.\n- `meta.searchType`\n  User-facing result surface selector. Current shipped values are `web | news`.\n- `meta.intentPreset`\n  User-facing intent preset. Current shipped values are\n  `general | code | company | docs | research`.\n\nThese are product-layer inputs, not provider ids.\n\n## Providers And Upgrade Paths\n\nNo API key is required for the baseline. Optional provider credentials or endpoints unlock stronger\ncoverage.\n\n```bash\nTAVILY_API_KEY=tvly-xxxxx\nEXA_API_KEY=exa-xxxxx\nQUERIT_API_KEY=xxxxx\nSERPER_API_KEY=xxxxx\nBRAVE_API_KEY=xxxxx\nSERPAPI_API_KEY=xxxxx\nYOU_API_KEY=xxxxx\nSEARXNG_INSTANCE_URL=https://searx.example.com\n\n# Perplexity / Sonar: choose one transport path\nPERPLEXITY_API_KEY=xxxxx\nOPENROUTER_API_KEY=xxxxx\nOPENROUTER_BASE_URL=https://openrouter.ai/api/v1\nKILOCODE_API_KEY=xxxxx\n\n# Or use a custom OpenAI-compatible gateway\nPERPLEXITY_GATEWAY_API_KEY=xxxxx\nPERPLEXITY_BASE_URL=https://gateway.example.com/v1\nPERPLEXITY_MODEL=perplexity/sonar-pro\n```\n\nProvider roles:\n\n- Tavily: strongest premium default for general search, news, and extract\n- Exa: semantic retrieval and extract fallback\n- Querit: multilingual AI search with native geo and language filters\n- Serper: Google-like search with strong news and locale coverage\n- Brave: structured general web search\n- SerpAPI: multi-engine routing including Baidu and Yandex\n- You.com: LLM-ready web search with freshness and mixed web/news coverage\n- SearXNG: self-hosted privacy-first metasearch fallback\n- Perplexity / Sonar: answer-first grounded search via native or gateway transport\n- DDG: best-effort no-key baseline search\n- Fetch: no-key extract / crawl / map baseline\n- Render: optional local browser lane\n\n## Distribution Surfaces\n\nThe project has two honest distribution surfaces:\n\n- **GitHub / local source tree**\n  The full surface, including `render.mjs`, `eval.mjs`, tests, and the deeper research toolchain.\n- **ClawHub publish package**\n  A generated core profile built by `scripts/build-clawhub-package.mjs`.\n\nWhy this split exists:\n\n- local developers need the full runtime and benchmark surface\n- ClawHub moderation benefits from a narrower, more honest publish boundary\n- the registry package is still a code-backed Node runtime, not an instruction-only bundle\n\nDetailed notes:\n\n- [docs/clawhub-package.md](/Users/codez/develop/web-search-pro/docs/clawhub-package.md)\n- [docs/clawhub-compliance.md](/Users/codez/develop/web-search-pro/docs/clawhub-compliance.md)\n\n## Boundaries\n\n`web-search-pro` is strong at:\n\n- capability-aware retrieval\n- explainable routing\n- safe extract / crawl / map\n- structured research packs\n- local diagnostics and review surfaces\n\nIt is intentionally not:\n\n- a hosted remote scraping service\n- a final report writer\n- a browser-first crawler by default\n- an unlimited no-key search guarantee\n\n## Positioning vs Other Skills\n\n- **vs lightweight `web-search` skills**\n  `web-search-pro` is heavier, but it gives you explainable routing, federation visibility, and a\n  path into `extract`, `crawl`, `map`, and `research`.\n- **vs search-router-first skills such as `web-search-plus`**\n  `web-search-pro` is broader as a retrieval stack. The differentiator is not only where to\n  search, but what happens after search.\n- **vs hosted scrape-first products**\n  `web-search-pro` stays local-first, more inspectable, and more explicit about safety boundaries\n  and runtime behavior.\n\n## Safety\n\nSafe fetch:\n\n- allows only `http` / `https`\n- blocks credential-bearing URLs\n- blocks localhost, private, and metadata targets\n- revalidates redirects\n- keeps JavaScript disabled\n\nBrowser render:\n\n- is off by default\n- uses a local headless browser only when enabled\n- revalidates navigations\n- can enforce same-origin-only navigation\n\nChallenge and anti-bot interstitial pages are reported as failures, not silent successes.\n\n## Discovery Keywords\n\n`web search`, `news search`, `latest updates`, `current events`, `docs search`, `API docs`,\n`code search`, `company research`, `competitor analysis`, `site crawl`, `site map`,\n`multilingual search`, `Baidu search`, `Google-like search`, `answer-first search`,\n`cited answers`, `explainable routing`, `no-key baseline`\n\n## Versioning\n\n- Product / docs version: `2.1`\n- JSON schema version: `1.0`\n\n`2.x` is the retrieval-stack product line. Machine-readable payloads remain additive and\ncompatible, so schema stays `1.0`.\n\n## Docs\n\n- [README_zh.md](/Users/codez/develop/web-search-pro/README_zh.md)\n- [CHANGELOG.md](/Users/codez/develop/web-search-pro/CHANGELOG.md)\n- [docs/search-routing-model.md](/Users/codez/develop/web-search-pro/docs/search-routing-model.md)\n- [docs/search-ux-model.md](/Users/codez/develop/web-search-pro/docs/search-ux-model.md)\n- [docs/research-layer.md](/Users/codez/develop/web-search-pro/docs/research-layer.md)\n- [docs/head-to-head-eval.md](/Users/codez/develop/web-search-pro/docs/head-to-head-eval.md)\n- [docs/clawhub-package.md](/Users/codez/develop/web-search-pro/docs/clawhub-package.md)\n- [docs/clawhub-compliance.md](/Users/codez/develop/web-search-pro/docs/clawhub-compliance.md)\n- [docs/marketing-launch-kit.md](/Users/codez/develop/web-search-pro/docs/marketing-launch-kit.md)\n- [docs/agent-contract-p0.md](/Users/codez/develop/web-search-pro/docs/agent-contract-p0.md)\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FZjianru%2Fweb-search-pro","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FZjianru%2Fweb-search-pro","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FZjianru%2Fweb-search-pro/lists"}