https://github.com/khamel83/argus
Multi-provider web search broker for AI agents with budget-aware routing and 5,000+ free monthly queries.
https://github.com/khamel83/argus
ai-agents brave-search cli content-extraction duckduckgo fastapi llm-tools mcp mcp-registry mcp-server python search-api search-broker searxng tavily web-search
Last synced: about 2 months ago
JSON representation
Multi-provider web search broker for AI agents with budget-aware routing and 5,000+ free monthly queries.
- Host: GitHub
- URL: https://github.com/khamel83/argus
- Owner: Khamel83
- License: mit
- Created: 2026-03-31T21:50:02.000Z (3 months ago)
- Default Branch: main
- Last Pushed: 2026-04-21T06:05:26.000Z (2 months ago)
- Last Synced: 2026-04-21T07:30:17.490Z (2 months ago)
- Topics: ai-agents, brave-search, cli, content-extraction, duckduckgo, fastapi, llm-tools, mcp, mcp-registry, mcp-server, python, search-api, search-broker, searxng, tavily, web-search
- Language: Python
- Homepage: https://pypi.org/project/argus-search/
- Size: 633 KB
- Stars: 1
- Watchers: 0
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Security: SECURITY.md
Awesome Lists containing this project
README
# Argus
[](https://www.python.org/downloads/)
[](https://pypi.org/project/argus-search/)
[](https://pepy.tech/projects/argus-search)
[](LICENSE)
[](https://github.com/Khamel83/argus/actions/workflows/ci.yml)
[](https://registry.modelcontextprotocol.io/servers/io.github.Khamel83/argus)
[](https://github.com/Khamel83/argus/pkgs/container/argus)
Multi-provider web search broker for AI agents. 14 providers, budget-aware routing, content extraction — one API so your agent doesn't need to stitch search results together.
**Features at a glance:**
- **14 providers, one API** — free-first tier routing, budget-exhausted providers skipped automatically
- **Zero-key start** — `pip install argus-search` gives you DuckDuckGo + Yahoo immediately, no accounts needed
- **SearXNG self-host = 70+ engines** — Google, Bing, Yahoo, Startpage, Ecosia, Qwant and more via one Docker container
- **9-step content extraction** — returns full page text with quality gates, not just links
- **Multi-turn sessions** — pass `session_id` for conversational context across searches
- **4 search modes** — discovery, research, recovery, grounding
- **Dead URL recovery** — `/recover-url` with Wayback Machine and archive fallbacks
- **4 integration paths** — HTTP API, CLI, MCP server, Python SDK
_Built for AI agent builders, RAG pipelines, and ops teams who need reliable search without stitching APIs together._
## Contents
- [Quickstart](#quickstart)
- [Providers](#providers)
- [HTTP API](#http-api)
- [Integration](#integration)
- [CLI](#cli)
- [MCP](#mcp)
- [Python](#python)
- [Content Extraction](#content-extraction)
- [Architecture](#architecture)
- [Configuration](#configuration)
- [FAQ](#faq)
## Quickstart
### Mode 1: Local CLI (zero config)
```bash
pip install argus-search && argus search -q "python web frameworks"
```
That's it. DuckDuckGo handles the search — no accounts, no keys, no containers. You get unlimited free search from your laptop right now. Add API keys whenever you want more providers, or don't.
```bash
argus extract -u "https://example.com/article" # extract clean text from any URL
```
Works on any machine with Python 3.11+ — laptop, Mac Mini, Raspberry Pi, cloud VM. Nothing to host.
**For MCP (Claude Code, Cursor, VS Code):**
```bash
pipx install argus-search[mcp] && argus mcp serve
```
Then add to your MCP config:
```json
{"mcpServers": {"argus": {"command": "argus", "args": ["mcp", "serve"]}}}
```
Or install from the [MCP Registry](https://registry.modelcontextprotocol.io/servers/io.github.Khamel83/argus):
```json
{
"mcpServers": {
"argus": {
"registryType": "pypi",
"identifier": "argus-search",
"runtimeHint": "uvx"
}
}
}
```
One command to install, one JSON block to connect. No server to run, no keys to configure.
### Mode 2: Full Stack Server
Got a Raspberry Pi running Pi-hole? A Mac Mini on your desk? An old laptop? That's enough to run the full stack — SearXNG (your own private search engine) plus local JS-rendering content extraction.
```bash
docker compose up -d # SearXNG + Argus
```
| What you have | What you get |
|--------------|-------------|
| **Any machine with Python 3.11+** | DuckDuckGo + API providers (no server) |
| **Raspberry Pi 4 / old laptop** (4GB+) | Everything — SearXNG, all providers, Crawl4AI |
| **Mac Mini M1+** (8GB+) | Full stack with headroom |
| **Free cloud VM** (1GB) | SearXNG + search providers (skip Crawl4AI) |
SearXNG takes 512MB of RAM and gives you a private Google-style search engine that nobody can rate-limit, block, or charge for. It runs alongside Pi-hole on hardware millions of people already own.
## Providers
| Provider | Credit type | Free capacity | Setup |
|----------|------------|---------------|-------|
| DuckDuckGo | Free (scraped) | Unlimited | None |
| Yahoo | Free (scraped) | Unlimited | None — fragile, auto-skipped if broken |
| SearXNG | Free (self-hosted) | Unlimited — 70+ engines¹ | Docker |
| GitHub | Free (API) | Unlimited | None (token for higher rate limit) |
| WolframAlpha | Free (API key) | 2,000 queries/month | [free key](https://developer.wolframalpha.com/) |
| Brave Search | Monthly recurring | 2,000 queries/month | [dashboard](https://brave.com/search/api/) |
| Tavily | Monthly recurring | 1,000 queries/month | [signup](https://app.tavily.com/sign-up) |
| Exa | Monthly recurring | 1,000 queries/month | [signup](https://dashboard.exa.ai/signup) |
| Linkup | Monthly recurring | 1,000 queries/month | [signup](https://linkup.so) |
| Serper | One-time signup | 2,500 credits | [signup](https://serper.dev/signup) |
| Parallel AI | One-time signup | 4,000 credits | [signup](https://parallel.ai) |
| You.com | One-time signup | $20 credit | [platform](https://you.com/platform) |
| Valyu | One-time signup | $10 credit | [platform](https://platform.valyu.ai) |
¹ SearXNG aggregates Google, Bing, Yahoo, Startpage, Ecosia, Qwant, Wikipedia, and 60+ more — all behind a single self-hosted endpoint. Run `docker compose up -d` on any machine with 512MB of free RAM.
² WolframAlpha returns **computed answers** (math, unit conversions, factual lookups), not web search results. It only activates in `grounding` and `research` modes. Queries it can't compute (general web searches) return empty — no error, no health penalty.
**7,000+ free queries/month** from recurring free-tier providers alone (WolframAlpha 2k + Brave 2k + Tavily 1k + Exa 1k + Linkup 1k). DuckDuckGo, Yahoo, SearXNG, and GitHub have no monthly cap. Routing priority: **Tier 0** (free: SearXNG, DuckDuckGo, Yahoo, GitHub, WolframAlpha) → **Tier 1** (monthly recurring: Brave, Tavily, Exa, Linkup) → **Tier 3** (one-time: Serper, Parallel, You.com, Valyu, SearchAPI). Budget-exhausted providers are skipped automatically.
## HTTP API
All endpoints prefixed with `/api`. OpenAPI docs at `http://localhost:8000/docs`.
```bash
# Search
curl -X POST http://localhost:8000/api/search \
-H "Content-Type: application/json" \
-d '{"query": "python web frameworks", "mode": "discovery", "max_results": 5}'
# Multi-turn search (conversational refinement)
curl -X POST http://localhost:8000/api/search \
-H "Content-Type: application/json" \
-d '{"query": "what about async?", "session_id": "my-session"}'
# Extract content from a working URL
curl -X POST http://localhost:8000/api/extract \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com/article"}'
# Recover a dead or moved URL
curl -X POST http://localhost:8000/api/recover-url \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com/old-page", "title": "Example Article"}'
# Health & budgets
curl http://localhost:8000/api/health/detail
curl http://localhost:8000/api/budgets
```
#### Search modes
| Mode | Use for | Example |
|------|---------|---------|
| `discovery` | Related pages, canonical sources | "Find the official docs for X" |
| `research` | Broad exploratory retrieval | "Latest approaches to Y?" |
| `recovery` | Finding moved/dead content | "This URL is 404" |
| `grounding` | Fact-checking with live sources | "Verify this claim about Z" |
Tier-based routing always applies first. Within each tier, the mode selects provider order.
#### Response format
```json
{
"query": "python web frameworks",
"mode": "discovery",
"results": [
{"url": "https://fastapi.tiangolo.com", "title": "FastAPI", "snippet": "Modern Python web framework", "score": 0.942}
],
"total_results": 1,
"cached": false,
"traces": [
{"provider": "duckduckgo", "status": "success", "results_count": 5, "latency_ms": 312}
]
}
```
Each result includes `url`, `title`, `snippet`, `domain`, `provider`, and `score`. The `traces` array shows which providers were called and their outcomes.
#### Budgets
```json
{
"budgets": {
"brave": {"remaining": 1847, "monthly_usage": 153, "usage_count": 153, "exhausted": false},
"duckduckgo": {"remaining": 0, "monthly_usage": 0, "usage_count": 42, "exhausted": false}
},
"token_balances": {"jina": 9833638}
}
```
Each provider tracks usage. Tier 1 (monthly) uses a 30-day rolling window; tier 3 (one-time) uses a lifetime counter that never resets. When a provider hits its budget, Argus skips it and moves to the next. Free providers (SearXNG, DuckDuckGo, GitHub) have no limit. Set `ARGUS_*_MONTHLY_BUDGET_USD` to enforce custom limits per provider.
## Integration
### CLI
```bash
argus search -q "python web framework" # zero-config, uses DuckDuckGo
argus search -q "python web framework" --mode research -n 20
argus search -q "fastapi" --session my-session # multi-turn context
argus extract -u "https://example.com/article" # extract clean text
argus extract -u "https://example.com/article" -d nytimes.com # auth extraction
argus recover-url -u "https://dead.link" -t "Title"
argus health # provider status
argus budgets # budget + token balances
argus set-balance -s jina -b 9833638 # track token balance
argus test-provider -p brave # smoke-test a provider
argus serve # start API server
argus mcp serve # start MCP server
```
All commands support `--json` for structured output.
How sessions work
Pass `session_id` to any search call. Argus stores each query and extracted URL in a SQLite-backed session. Reusing the same `session_id` gives the broker context from prior queries — follow-up searches are automatically refined using earlier conversation context. Sessions persist across restarts. Omit `session_id` for stateless, one-shot searches.
### MCP
Add to your MCP client config:
```json
{
"mcpServers": {
"argus": {
"command": "argus",
"args": ["mcp", "serve"]
}
}
}
```
Works with **Claude Code**, **Cursor**, **VS Code**, and any MCP-compatible client.
**Option B — Self-hosted server (homelab / always-on machine)**
Run Argus once on a server, connect every client to it over the network. No local install needed on client machines.
On the server (`docker compose up -d` starts both):
```bash
argus mcp serve --transport sse --host 0.0.0.0 --port 8001
```
On each client machine, add to `~/.claude/claude_desktop_config.json` (or equivalent):
```json
{
"mcpServers": {
"argus": {
"url": "http://:8271/sse"
}
}
}
```
With [Tailscale](https://tailscale.com), `` is your machine's Tailscale hostname (e.g. `homelab-ts`). One server, every machine on your network gets search.
Available tools: `search_web`, `extract_content`, `recover_url`, `expand_links`, `search_health`, `search_budgets`, `test_provider`, `cookie_health`, `valyu_answer`
### Python
```python
from argus.broker.router import create_broker
from argus.models import SearchQuery, SearchMode
from argus.extraction import extract_url
broker = create_broker()
response = await broker.search(
SearchQuery(query="python web frameworks", mode=SearchMode.DISCOVERY, max_results=10)
)
for r in response.results:
print(f"{r.title}: {r.url} (score: {r.score:.3f})")
content = await extract_url(response.results[0].url)
print(content.title)
print(content.text)
```
## Content Extraction
Argus tries up to nine methods to extract content from any URL: first local (trafilatura, Crawl4AI, Playwright), then external APIs (Jina, Valyu Contents, Firecrawl, You.com, Wayback, archive.is). Each attempt is quality-checked for garbage output. See [docs/providers.md](docs/providers.md) for the full extractor comparison.
**Extract** gets the full text of a working URL. **Recover-URL** finds alternatives when a URL is dead, paywalled, or radically changed by querying archival sources (Wayback, archive.is) and running a question-guided extraction loop.
## Architecture
```
Caller (CLI/HTTP/MCP/Python) → SearchBroker → tier-sorted providers → RRF ranking → response
↕ SessionStore (optional)
Extractor (on demand) → 9-step fallback chain with quality gates
```
| Module | Responsibility |
|--------|---------------|
| `argus/broker/` | Tier-based routing, ranking, dedup, caching, health, budgets |
| `argus/providers/` | Provider adapters (one per search API) |
| `argus/extraction/` | 9-step URL extraction fallback chain with quality gates |
| `argus/sessions/` | Multi-turn session store and query refinement |
| `argus/api/` | FastAPI HTTP endpoints |
| `argus/cli/` | Click CLI commands |
| `argus/mcp/` | MCP server for LLM integration |
| `argus/persistence/` | PostgreSQL query/result storage |
Add new providers or extractors with a single adapter file. See [CONTRIBUTING.md](CONTRIBUTING.md) for the interface.
### How a Query Works
```
query arrives → cache? → build provider queue → execute sequentially → RRF fuse → dedup → respond
```
1. **Cache check.** `SearchCache` hashes `normalized_query:mode` (SHA256). Hit returns immediately with a TTL of 168 hours (7 days).
2. **Provider queue.** `resolve_routing()` takes the mode-specific preference list and stable-sorts by tier: tier 0 (free) first, tier 1 (monthly) next, tier 3 (one-time) last. Example for discovery mode:
```
searxng → duckduckgo → yahoo → github → brave → exa → tavily → linkup → serper → parallel → you → valyu
```
3. **Sequential execution with gates.** Each provider is checked in order. Four gates must pass before an API call:
- **Config** — is the provider enabled and configured (API key present)?
- **Health** — has it failed 5+ consecutive times (triggers 60-minute cooldown)?
- **Budget** — for tier 1+: is the budget exhausted? For tier 1 (monthly), pacing checks if the 7-day usage rate would drain the remaining budget in under a week — empty days bank headroom. For tier 3 (one-time), a lifetime counter gates access — exhaustion is the sole check.
- **Execute** — the actual HTTP call. Successes reset failure counters; failures increment them.
4. **RRF fusion.** Results from all queried providers are merged using Reciprocal Rank Fusion (`k=60`). Each result's score is the sum of `1/(k + rank)` across every provider that returned it. Results appearing in multiple providers rank higher.
5. **Dedup and truncate.** URLs are normalized (stripped `www.`, tracking params like `utm_*`, trailing slashes) and deduplicated. The merged list is truncated to `max_results` (default 10).
6. **Cache and persist.** The final response is written to the in-memory cache and persisted to PostgreSQL. Provider traces (which were called, which were skipped and why) are included in the response for observability.
## Configuration
All config via environment variables. See `.env.example` for the full list. Missing keys degrade gracefully — providers are skipped, not errors.
| Variable | Default | Description |
|----------|---------|-------------|
| `ARGUS_SEARXNG_BASE_URL` | `http://127.0.0.1:8080` | SearXNG endpoint |
| `ARGUS_BRAVE_API_KEY` | — | Brave Search API key |
| `ARGUS_SERPER_API_KEY` | — | Serper API key |
| `ARGUS_TAVILY_API_KEY` | — | Tavily API key |
| `ARGUS_EXA_API_KEY` | — | Exa API key |
| `ARGUS_LINKUP_API_KEY` | — | Linkup API key |
| `ARGUS_PARALLEL_API_KEY` | — | Parallel AI API key |
| `ARGUS_YOU_API_KEY` | — | You.com API key |
| `ARGUS_VALYU_API_KEY` | — | Valyu API key (search, contents, answer) |
| `ARGUS_FIRECRAWL_API_KEY` | — | Firecrawl API key (content extraction) |
| `ARGUS_GITHUB_API_KEY` | — | GitHub token (higher rate limit) |
| `ARGUS_*_MONTHLY_BUDGET_USD` | 0 (unlimited) | Query-count budget per provider |
| `ARGUS_CRAWL4AI_ENABLED` | false | Enable Crawl4AI extraction step |
| `ARGUS_YOU_CONTENTS_ENABLED` | false | Enable You.com Contents API extraction |
| `ARGUS_CACHE_TTL_HOURS` | 168 | Result cache TTL |
## FAQ
**How is this different from calling Tavily/Serper directly?**
Argus calls them for you — plus 13 other providers. You get one ranked, deduplicated result set instead of managing multiple API keys and stitching results together. Free providers are tried first, so you only burn credits when needed.
**Can I run only one provider?**
Yes. Set only the API key for the provider you want. All others are silently skipped. For zero-config, just install and go — DuckDuckGo + Yahoo handle search with no keys.
**Do I need Docker?**
No. `pip install argus-search` works immediately on any machine with Python 3.11+. Docker is only needed for SearXNG (self-hosted search, aggregates 70+ engines) or Crawl4AI (local JS rendering).
## License
MIT — see [CHANGELOG.md](CHANGELOG.md) for release history.