{"id":30594948,"url":"https://github.com/serpapi/web-research-agent","last_synced_at":"2025-08-29T20:33:21.083Z","repository":{"id":310455692,"uuid":"1035889678","full_name":"serpapi/web-research-agent","owner":"serpapi","description":"AI Research agent that combines OpenAI and SerpApi Search API to research open web","archived":false,"fork":false,"pushed_at":"2025-08-11T09:01:27.000Z","size":7,"stargazers_count":1,"open_issues_count":0,"forks_count":2,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-08-29T07:03:44.705Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://serpapi.com/blog/building-a-fast-self-hosted-research-agent-with-openai-models-serpapi/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/serpapi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-08-11T08:44:40.000Z","updated_at":"2025-08-22T12:32:07.000Z","dependencies_parsed_at":"2025-08-18T08:55:22.437Z","dependency_job_id":"4f2749c0-09ac-44b2-b92b-116c9a87e5f5","html_url":"https://github.com/serpapi/web-research-agent","commit_stats":null,"previous_names":["serpapi/web-research-agent"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/serpapi/web-research-agent","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/serpapi%2Fweb-research-agent","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/serpapi%2Fweb-research-agent/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/serpapi%2Fweb-research-agent/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/serpapi%2Fweb-research-agent/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/serpapi","download_url":"https://codeload.github.com/serpapi/web-research-agent/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/serpapi%2Fweb-research-agent/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272758436,"owners_count":24988239,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-29T02:00:10.610Z","response_time":87,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-08-29T20:33:20.599Z","updated_at":"2025-08-29T20:33:21.072Z","avatar_url":"https://github.com/serpapi.png","language":"Python","readme":"# Research Agent\n\nLLM-powered researcher that combines OpenAI chat models with Google results via SerpAPI. The agent asks the model to emit all needed searches at once, runs them concurrently, feeds snippets back, and returns a well‑cited answer. Includes a simple CLI.\n\n## Features\n- OpenAI Chat Completions with function calling\n- Batches 2–50 `search_web` tool calls in one turn\n- Concurrent Google searches via SerpAPI\n- Optional JSON trace (`--outfile`) with steps and final answer\n\n## Requirements\n- Python 3.9+ (3.10+ recommended)\n- OPENAI_API_KEY\n- SERPAPI_API_KEY\n\n## Install\n\nAssuming you have Python 3.9+ and virtual env installed:\n\n```bash\ngit clone https://github.com/vladm-serpapi/web-research-agent\ncd web-research-agent\npip install -r requirements.txt\n```\n\n## Setup API keys\nOption A — export in your shell (recommended):\n```bash\nexport OPENAI_API_KEY=\"sk-...\"\nexport SERPAPI_API_KEY=\"...\"\n```\nOption B — .env file (don’t commit this file):\n```bash\n# .env\nexport OPENAI_API_KEY=\"sk-...\"\nexport SERPAPI_API_KEY=\"...\"\n# load it\nsource .env\n```\nSecurity: Never share or commit your keys.\n\n## Quick start\n```bash\npython research_agent.py -q \"What are the latest approaches to retrieval‑augmented generation in 2025?\"\n# Save full JSON trace\npython research_agent.py -q \"State of LLM reasoning benchmarks in 2025\" --outfile trace.json\n```\n\n## CLI\n```bash\npython research_agent.py -h\n# usage: research_agent.py [-h] -q QUERY [-m {o3,o4-mini,gpt-4o}] [-n TOPN] [-o OUTFILE] [-d]\n#   -q, --query        Research question (required)\n#   -m, --model        o3 (default) | o4-mini | gpt-4o\n#   -n, --topn         Organic results per search (default: 10)\n#   -o, --outfile      Write JSON trace to file\n#   -d, --debug        Print debug logs\n```\n\n## How it works (brief)\n- System prompt asks the model to emit all `search_web` calls first\n- Agent executes all requested Google searches concurrently (SerpAPI)\n- Results are passed back as tool messages; model produces a final, cited answer\n- Note on model tool behavior: o3 / o4-mini reasoning models prefer to output single tool call per prompt, so gpt-4o is preferred when many queries are required\n\n## Examples\n```bash\npython research_agent.py -q \"Compare FAISS vs. Milvus vs. Qdrant for RAG (2025)\" -m o3 -n 8 -o rag_db_trace.json\n```\n\n### With debug mode\n```bash\n python research_agent.py -q \"airlines industry trend 2025, compare multiple trends by impact and research each deeper to provide a comphrehensive picture\" --outfile trace.json --debug --model gpt-4o\n```\n\n### Sample output\n\n```bash\n python research_agent.py -q \"research the nuclear energy sector in 2025 and build a comprehensive thesis / report on it. I want this \nreport to cover AI, uranium, energy, etc. Financial projections, key players, companies, etc. Do the research in iterative fashion, after each round of searches and getting new info\nrmation, do another round of searches to dive deeper into each specific topic. Don't stop on surface findings. Think and analyze what data are you missing, and proceed to research it deeper.\" --outfile trace.json --debug --model gpt-4o\n[DEBUG] → OpenAI chat.completions.create request …\n[DEBUG] → SerpAPI query: 'nuclear energy sector 2025 overview'\n[DEBUG] → SerpAPI query: 'AI in nuclear energy 2025'\n[DEBUG] → SerpAPI query: 'uranium market 2025'\n[DEBUG] → SerpAPI query: 'key companies in nuclear energy 2025'\n[DEBUG] → OpenAI chat.completions.create request …\n[DEBUG] → SerpAPI query: 'financial projections nuclear energy 2025'\n[DEBUG] → SerpAPI query: 'nuclear energy policies 2025'\n[DEBUG] → SerpAPI query: 'AI-driven nuclear technologies 2025'\n[DEBUG] → SerpAPI query: 'key innovations in nuclear technology 2025'\n[DEBUG] → OpenAI chat.completions.create request …\n…\n```\n\n## JSON trace example (with --outfile)\n```json\n{\n  \"question\": \"...\",\n  \"answer\": \"...\",\n  \"steps\": [\n    { \"type\": \"tool_call\", \"query\": \"first search\" },\n    { \"type\": \"tool_result\", \"content\": \"- Title: snippet ...\" },\n    { \"type\": \"assistant_answer\", \"content\": \"final answer text\" }\n  ]\n}\n```\n\n## Programmatic use\n```python\nfrom research_agent import ResearchAgent\n\nagent = ResearchAgent(model=\"o3\", topn=10, debug=False)\nresult = agent.run(\"Summarize the most cited papers on RAG.\")\nprint(result[\"answer\"])  # final answer\nprint(len(result[\"steps\"]))\n```\n\n## Troubleshooting\n- \"OPENAI_API_KEY and SERPAPI_API_KEY must be set.\" → export both keys or source your .env\n- Model not available → switch to a supported one (o3, o4-mini, gpt-4o)\n- Empty/failed searches → check SerpAPI key/quota and network settings\n\n## License\nMIT License — see LICENSE.","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fserpapi%2Fweb-research-agent","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fserpapi%2Fweb-research-agent","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fserpapi%2Fweb-research-agent/lists"}