{"id":31690255,"url":"https://github.com/balaji1233/deep_research_agent","last_synced_at":"2026-05-18T19:03:21.339Z","repository":{"id":314470195,"uuid":"1055652597","full_name":"balaji1233/DEEP_RESEARCH_AGENT","owner":"balaji1233","description":"A Deep research agent that will pull live data from sources like Google, Bing, and Reddit for answering user queries","archived":false,"fork":false,"pushed_at":"2025-09-29T15:23:32.000Z","size":76,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-29T17:26:28.078Z","etag":null,"topics":["brightdata","langchain","langgraph-agents"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/balaji1233.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-12T15:31:51.000Z","updated_at":"2025-09-29T15:23:35.000Z","dependencies_parsed_at":"2025-09-12T18:20:38.938Z","dependency_job_id":"b0f417c4-bbfe-4468-a9a8-cb65b44b5251","html_url":"https://github.com/balaji1233/DEEP_RESEARCH_AGENT","commit_stats":null,"previous_names":["balaji1233/deep_research_agent"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/balaji1233/DEEP_RESEARCH_AGENT","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/balaji1233%2FDEEP_RESEARCH_AGENT","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/balaji1233%2FDEEP_RESEARCH_AGENT/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/balaji1233%2FDEEP_RESEARCH_AGENT/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/balaji1233%2FDEEP_RESEARCH_AGENT/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/balaji1233","download_url":"https://codeload.github.com/balaji1233/DEEP_RESEARCH_AGENT/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/balaji1233%2FDEEP_RESEARCH_AGENT/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278947970,"owners_count":26073736,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-08T02:00:06.501Z","response_time":56,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["brightdata","langchain","langgraph-agents"],"created_at":"2025-10-08T12:42:41.838Z","updated_at":"2025-10-08T12:42:43.061Z","avatar_url":"https://github.com/balaji1233.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DEEP_RESEARCH_AGENT\nA Deep research agent that will pull live data from sources like Google, Bing, and Reddit for answering user queries\n# Multi-Source Research AI Agent — LangGraph + Bright Data + OpenAI\n\n**One-line:** I built a production-grade, multi-step AI research agent in Python that runs parallel live searches (Google, Bing, Reddit), scrapes and normalizes results via Bright Data, analyzes them with OpenAI/GPT, and returns a structured, cited synthesis — all orchestrated as a LangGraph workflow.\n\n---\n\n## Why this project (problem → solution)\n**Problem:** Most LLM assistants rely on single APIs or cached corpora and thus miss fresh, diverse, real-world evidence (SERPs, social sentiment, niche forum discussions). This leads to incomplete, biased, or outdated answers for real research tasks (e.g., competitive analysis, market research, relocation decisions).\n\n**My solution:** a graph-based research agent that:\n- queries multiple live sources in parallel,\n- reliably scrapes and normalizes raw results (including Reddit posts/comments),\n- applies layered LLM analysis (source-level → synthesis),\n- enforces typed outputs and citations for traceability.\n\nThis is framed as a **research assistant** (not a toy chat), designed to reduce manual research time and increase answer fidelity.\n\n---\n\n## Highlights / Key capabilities\n- 🚀 **Parallel multi-source search:** Google, Bing, Reddit (via Bright Data).\n- 🔄 **Asynchronous snapshot handling:** trigger/crawl → poll for readiness → download parsed results.\n- 🧠 **Layered LLM analysis:** per-source analysis + cross-source synthesis using OpenAI.\n- 🧩 **LangGraph orchestration:** nodes with typed inputs/outputs, deterministic wiring, retries and partial state updates.\n- 📐 **Structured outputs:** Pydantic (typed) schemas ensure consistent, machine-readable results (URLs, snippets, sentiment).\n- 🔍 **Citations \u0026 provenance:** every claim links back to source results.\n- 🛠️ **Production-grade scaffolding:** modular functions, prompt templates, logging, error handling.\n\n---\n\n## Business value / Impact\n- **Faster, better research:** replaces hours of manual searching with a ~minute response (depends on crawl latency), increasing analyst throughput ×5–10.  \n- **Broader coverage:** combining search engines + Reddit reduces blind spots and improves recall for niche topics.  \n- **Auditability:** citations \u0026 typed outputs reduce hallucination risk and improve stakeholder trust.  \n- **Operational readiness:** modular design makes it easy to extend to other sources (Twitter, news APIs) and integrate into production pipelines.\n\n---\n\n## Architecture (high level)\nUser Query\n│\n▼\nLangGraph Controller (State)\n├─ Parallel Nodes: GoogleSearch, BingSearch, RedditSearch (calls Bright Data)\n├─ Fetch / Snapshot Node: polls until data ready, downloads parsed JSON\n├─ SourceAnalysis Node(s): LLM analyzes each source separately (prompt templates)\n├─ Synthesizer Node: LLM synthesizes final answer, merging source analyses\n└─ Output Formatter: enforces Pydantic schema + citations\n\n## Architecture — Description\n\n### User Query  \n**User-provided question or prompt** that starts the pipeline.\n\n### LangGraph Controller (State)  \nCentral orchestrator that holds the typed state object (Pydantic), schedules nodes (parallel \u0026 sequential), manages retries/backoff, and collects partial outputs.\n\n### Parallel Nodes: `GoogleSearch`, `BingSearch`, `RedditSearch` (calls Bright Data)  \nNodes that simultaneously issue search/scrape requests via Bright Data to gather SERPs and social content. Each node writes raw results into the shared state.\n\n### Fetch / Snapshot Node  \nFor asynchronous sources (e.g., Reddit snapshots), this node polls for crawl readiness, downloads parsed JSON results, and stores them in state.\n\n### SourceAnalysis Node(s)  \nPer-source LLM analysis nodes that consume raw results, run source-specific prompt templates, and extract summaries, ranked URLs, and metadata (sentiment, notable quotes).\n\n### Synthesizer Node  \nAggregates the per-source analyses and runs a higher-capacity LLM to merge insights, resolve contradictions, and produce the consolidated answer.\n\n### Output Formatter  \nValidates and formats the final output using Pydantic schemas, attaches citations and confidence scores, and serializes the structured JSON result for downstream use.\n\n---\n\n### Compact flow (pasteable)\n\n\n\nKey design points:\n- **State object:** dict-like (user_query, raw_results, filtered_urls, reddit_posts, analyses, final_answer).\n- **LangGraph nodes:** pure functions that `read → update → return` state slices.\n- **Bright Data:** unified API for SERP \u0026 social scraping; handles CAPTCHA/IP issues at scale.\n- **Pedantic/Pydantic models:** enforce output structure (lists of URLs, score fields, summaries).\n\n---\n\n## Tradeoffs \u0026 Challenges\n- **Latency vs Coverage:** deep scraping + Reddit snapshots increase latency. I mitigate with parallelism and partial early results, but there’s an inherent tradeoff when you need exhaustive retrieval.  \n- **Cost vs Freshness:** Bright Data + high-capacity LLM calls incur costs — tune sampling depth and model choice for use-case (GPT-4 for final synth, GPT-3.5 for source summaries).  \n- **Hallucination risk:** even with RAG, LLMs can hallucinate. I reduce this via explicit citation requirements, structured outputs, and a hallucination detector/QA stage.  \n- **Scraping reliability:** Bright Data helps, but snapshots and polling are required (complexity in async handling).  \n- **Rate limits \u0026 retries:** robust error handling and backoff strategies are needed for production.\n\n---\n\n## Metrics I track\n- **Recall of relevant sources (%):** manual sampling vs agent results.  \n- **Precision / factuality score:** human-annotated ranking of claims.  \n- **End-to-end latency:** median and 95th percentile.  \n- **Cost per query:** API + scraping cost.  \n- **User satisfaction / accuracy lift:** AB tests vs human-only research.\n\n---\n\n## Quickstart (run locally)\n\u003e **Prereqs:** Python 3.10+, virtualenv, Bright Data account (API key), OpenAI API key.\n\n```bash\ngit clone https://github.com/yourname/langgraph-research-agent.git\ncd langgraph-research-agent\npython -m venv .venv \u0026\u0026 source .venv/bin/activate\npip install -r requirements.txt\ncp .env.example .env\n# edit .env to add OPENAI_API_KEY, BRIGHTDATA_API_KEY, etc.\npython run_agent.py --query \"Should I move to Lisbon in 2025?\"\n```\n## How I built it — Implementation notes \u0026 best practices\n\n### Node design\n- Keep nodes **deterministic** and **side-effect free**: accept state in → return updated state out.  \n- Benefits: easier unit testing, retries, and reproducibility.\n\n### Prompt layering\n- **Summarize each source first**, then run a separate synthesis prompt over those summaries.  \n- This reduces LLM cognitive load and improves factual grounding.\n\n### Typed outputs\n- Use **Pydantic models** for outputs such as `SelectedURLs`, `SourceSummary`, `FinalSynthesis`.  \n- Validate LLM outputs against schemas; **retry or re-prompt** when structure is violated.\n\n### Snapshot polling\n- For crawls/snapshots implement robust polling with:\n  - exponential backoff,\n  - sensible timeouts,\n  - support for **partial retrievals** when some snapshots finish earlier.\n\n### Cost control\n- Use smaller, cheaper models for intermediate summarization tasks.  \n- Reserve higher-cost models for the **final synthesis** step only.\n\n### Logging \u0026 observability\n- Emit **structured JSON logs** and request traces.  \n- Expose metrics/dashboards for latency, error rates, and cost-per-query.\n\n---\n\n## Tests \u0026 validation\n- **Unit tests** for every node (mock Bright Data and OpenAI responses).  \n- **Integration smoke tests** to validate end-to-end flow (use canned datasets).  \n- **Human evaluation pipeline** to measure factuality, precision, and recall; use these metrics to iterate on prompts and filters.\n\n---\n\n## Roadmap / Extensions\n- Add more sources: **Twitter/X**, news APIs, academic sources (CrossRef, Semantic Scholar).  \n- Support **multi-turn conversation** and incremental updating of the state.  \n- Add a **knowledge-update node** to feed validated claims into a short-term cache or knowledge store.  \n- Build a **lightweight UI** for running queries, inspecting state \u0026 sources, and approving final outputs.\n\n---\n\n## Contribution\nI welcome PRs that:\n- add new source integrations,  \n- improve prompt templates,  \n- harden snapshot/poll handling, or  \n- add CI tests and sample datasets.\n\nPlease open issues for feature requests or bugs.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbalaji1233%2Fdeep_research_agent","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbalaji1233%2Fdeep_research_agent","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbalaji1233%2Fdeep_research_agent/lists"}