{"id":51063634,"url":"https://github.com/diffbot/langchain-diffbot","last_synced_at":"2026-06-23T04:30:48.819Z","repository":{"id":362375457,"uuid":"1251722104","full_name":"diffbot/langchain-diffbot","owner":"diffbot","description":"A thin LangChain integration over the official diffbot-python SDK","archived":false,"fork":false,"pushed_at":"2026-06-10T22:46:23.000Z","size":959,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-11T00:18:51.940Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/diffbot.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-27T21:17:23.000Z","updated_at":"2026-06-10T22:46:27.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/diffbot/langchain-diffbot","commit_stats":null,"previous_names":["diffbot/langchain-diffbot"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/diffbot/langchain-diffbot","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/diffbot%2Flangchain-diffbot","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/diffbot%2Flangchain-diffbot/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/diffbot%2Flangchain-diffbot/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/diffbot%2Flangchain-diffbot/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/diffbot","download_url":"https://codeload.github.com/diffbot/langchain-diffbot/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/diffbot%2Flangchain-diffbot/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34675970,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-23T02:00:07.161Z","response_time":65,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-23T04:30:48.172Z","updated_at":"2026-06-23T04:30:48.807Z","avatar_url":"https://github.com/diffbot.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# langchain-diffbot\n\n[![CI](https://github.com/diffbot/langchain-diffbot/actions/workflows/ci.yml/badge.svg)](https://github.com/diffbot/langchain-diffbot/actions/workflows/ci.yml)\n\n**LangChain docs:** [https://docs.langchain.com/oss/python/integrations/providers/diffbot](https://docs.langchain.com/oss/python/integrations/providers/diffbot)\n\nA thin LangChain integration over the official [`diffbot-python`](https://github.com/diffbot/diffbot-python) SDK. Every Diffbot API gets the closest LangChain primitive:\n\n| Diffbot API | LangChain class(es) |\n| --- | --- |\n| Knowledge Graph (DQL) | `DiffbotKnowledgeGraphRetriever`, `DiffbotKnowledgeGraphTool` |\n| Web Search | `DiffbotWebSearchRetriever`, `DiffbotWebSearchTool` |\n| Extract (Analyze) | `DiffbotExtractTool`, `DiffbotExtractLoader` |\n| NLP entities | `DiffbotEntitiesTool` |\n| Crawl | `DiffbotCrawlLoader` |\n| LLM RAG (`ask`) | `ChatDiffbot` (with native streaming), `DiffbotAskTool` |\n\n## Installation\n\n```bash\npip install langchain-diffbot\n```\n\n## Authentication \u0026 clients\n\nGet an API token at https://app.diffbot.com/get-started/.\n\nEvery component takes a pre-built SDK client — you build a `diffbot.Diffbot` (sync) and/or `diffbot.DiffbotAsync` (async) and pass it via `client=` / `async_client=`. That's the only way to give a component HTTP access, and it keeps configuration in one place: customize the client (token, `timeout`, `transport=`, custom URLs) however the SDK allows, and share one client across many components to reuse a single connection pool. The component uses the client as-is and never closes it — you own its lifecycle.\n\n```python\nimport os\nfrom diffbot import Diffbot\n\ndb = Diffbot(token=os.environ[\"DIFFBOT_API_TOKEN\"])\n```\n\nPick your execution mode by which client you build: `Diffbot` for the sync surface (`invoke`, `stream`, `load`), `DiffbotAsync` for the async surface (`ainvoke`, `astream`, `alazy_load`). Pass both if a component is used both ways.\n\n## Quickstart — Knowledge Graph retriever\n\n```python\nimport os\nfrom diffbot import Diffbot\nfrom langchain_diffbot import DiffbotKnowledgeGraphRetriever\n\ndb = Diffbot(token=os.environ[\"DIFFBOT_API_TOKEN\"])\nretriever = DiffbotKnowledgeGraphRetriever(client=db, k=5)\ndocs = retriever.invoke(\"type:Organization industries:\\\"Artificial Intelligence\\\" location.city.name:\\\"Boston\\\"\")\nfor d in docs:\n    print(d.metadata[\"name\"], \"—\", d.page_content[:120])\n```\n\nThe query string is a [DQL (Diffbot Query Language)](https://docs.diffbot.com/reference/dql-quickstart) expression.\n\n## Shaping the output\n\nDiffbot KG entities and web-search results are large. Dumping them straight into an LLM prompt can blow past per-minute input-token limits in a single call. Both retrievers expose three shaping knobs:\n\n```python\nimport os\nfrom diffbot import Diffbot\nfrom langchain_core.documents import Document\nfrom langchain_diffbot import DiffbotKnowledgeGraphRetriever\n\ndb = Diffbot(token=os.environ[\"DIFFBOT_API_TOKEN\"])\n\n# 1. Project only the top-level fields you care about. Drops everything else\n#    from `metadata`. Recommended for agent / tool-use scenarios.\nretriever = DiffbotKnowledgeGraphRetriever(\n    client=db,\n    k=5,\n    fields=[\"id\", \"type\", \"name\", \"homepageUri\", \"nbEmployees\"],\n)\n\n# 2. Choose which field becomes `page_content`. First non-empty value wins.\nretriever = DiffbotKnowledgeGraphRetriever(\n    client=db,\n    content_fields=[\"summary\", \"description\", \"name\"],\n)\n\n# 3. For total control, pass a `document_mapper` that turns a raw entity\n#    dict into whatever Document shape you want.\ndef mapper(entity: dict) -\u003e Document:\n    return Document(\n        page_content=entity.get(\"summary\", \"\"),\n        metadata={\"id\": entity[\"id\"], \"name\": entity[\"name\"]},\n    )\n\nretriever = DiffbotKnowledgeGraphRetriever(client=db, document_mapper=mapper)\n```\n\n`fields` and `content_fields` are ignored when `document_mapper` is set. The same knobs work on `DiffbotWebSearchRetriever`.\n\n## Web search\n\n```python\nimport os\nfrom diffbot import Diffbot\nfrom langchain_diffbot import DiffbotWebSearchRetriever\n\ndb = Diffbot(token=os.environ[\"DIFFBOT_API_TOKEN\"])\nweb = DiffbotWebSearchRetriever(client=db, k=5, fields=[\"title\", \"pageUrl\", \"score\"])\ndocs = web.invoke(\"diffbot knowledge graph llm grounding\")\n```\n\n## Extract a URL\n\n```python\nimport os\nfrom diffbot import Diffbot\nfrom langchain_diffbot import DiffbotExtractTool, DiffbotExtractLoader\n\ndb = Diffbot(token=os.environ[\"DIFFBOT_API_TOKEN\"])\n\n# Single URL\ntool = DiffbotExtractTool(client=db)\npage = tool.invoke({\"url\": \"https://www.diffbot.com/products/extract/\"})\n\n# Batch — yields one Document per URL (the same client is reused)\nloader = DiffbotExtractLoader(client=db, urls=[\"https://example.com\", \"https://diffbot.com\"])\nfor doc in loader.lazy_load():\n    print(doc.metadata[\"title\"], doc.page_content[:200])\n```\n\n`DiffbotExtractTool` returns a structured `{\"error\": ..., \"errorCode\": ...}` dict when Diffbot reports an extraction failure (200 with `errorCode`), so agents can react and try another URL instead of catching an exception. Auth / rate-limit errors propagate as `diffbot.errors.AuthError` / `RateLimitError`.\n\n## Crawl a site\n\n`DiffbotCrawlLoader` drives a Diffbot crawl job and yields one `Document` per crawled URL. The `page_content` is the URL itself (the crawl API surfaces URLs, not page contents) — chain it with `DiffbotExtractLoader` to fetch the content of each URL.\n\n```python\nimport os\nfrom diffbot import Diffbot\nfrom langchain_diffbot import DiffbotCrawlLoader\n\ndb = Diffbot(token=os.environ[\"DIFFBOT_API_TOKEN\"])\nloader = DiffbotCrawlLoader(client=db, site=\"https://www.diffbot.com\")\nfor doc in loader.lazy_load():\n    print(doc.metadata[\"url\"], doc.metadata[\"status\"])\n```\n\n## ChatDiffbot\n\n```python\nimport os\nfrom diffbot import Diffbot\nfrom langchain.messages import HumanMessage\nfrom langchain_diffbot import ChatDiffbot\n\nllm = ChatDiffbot(client=Diffbot(token=os.environ[\"DIFFBOT_API_TOKEN\"]))\n\nfor chunk in llm.stream([HumanMessage(content=\"What is the Diffbot Knowledge Graph?\")]):\n    print(chunk.content, end=\"\", flush=True)\n```\n\n`_stream` / `_astream` are native — no thread-pool fallback. `.invoke()` aggregates the stream into a single message.\n\nTo let a tool-calling agent *consult* Diffbot's LLM (rather than use it as the primary model), hand it `DiffbotAskTool` instead — it answers a natural-language question from the Knowledge Graph + live web and returns a synthesized string:\n\n```python\nimport os\nfrom diffbot import Diffbot\nfrom langchain_diffbot import DiffbotAskTool\n\nask = DiffbotAskTool(client=Diffbot(token=os.environ[\"DIFFBOT_API_TOKEN\"]))\nprint(ask.invoke({\"question\": \"Who founded Diffbot, and when?\"}))\n```\n\n## Agent tools\n\nEvery Diffbot API is also exposed as an agent-callable `BaseTool`. Hand a tool-calling agent only the tools you want — they all share whatever client you pass. `DiffbotExtractTool` and `DiffbotAskTool` are shown above; the rest:\n\n### DiffbotWebSearchTool\n\nRuns a [Diffbot web search](https://docs.diffbot.com/reference/web-search-get) and returns the result list — each item with `title`, `pageUrl`, `score`, and `content`. New accounts include 100,000 free web searches per month. (Use `DiffbotWebSearchRetriever` when you want `Document` output instead.)\n\n```python\nimport os\nfrom diffbot import Diffbot\nfrom langchain_diffbot import DiffbotWebSearchTool\n\ntool = DiffbotWebSearchTool(client=Diffbot(token=os.environ[\"DIFFBOT_API_TOKEN\"]))\nresults = tool.invoke({\"text\": \"diffbot knowledge graph\"})\n```\n\n### DiffbotKnowledgeGraphTool\n\nRuns a DQL query against the Knowledge Graph from within an agent and returns the raw response dict. (Use `DiffbotKnowledgeGraphRetriever` when you want `Document` output instead.)\n\n```python\nimport os\nfrom diffbot import Diffbot\nfrom langchain_diffbot import DiffbotKnowledgeGraphTool\n\ntool = DiffbotKnowledgeGraphTool(client=Diffbot(token=os.environ[\"DIFFBOT_API_TOKEN\"]))\nbody = tool.invoke({\"query\": 'type:Organization name:\"Diffbot\"'})\n```\n\n### DiffbotEntitiesTool\n\nIdentifies named entities and sentiment in text via Diffbot's NLP API. The returned entity IDs can be looked up in the Knowledge Graph (e.g. `id:or(\"E1\",\"E2\")`).\n\n```python\nimport os\nfrom diffbot import Diffbot\nfrom langchain_diffbot import DiffbotEntitiesTool\n\ntool = DiffbotEntitiesTool(client=Diffbot(token=os.environ[\"DIFFBOT_API_TOKEN\"]))\nresult = tool.invoke({\"text\": \"Diffbot was founded in Menlo Park.\"})\n```\n\n### Authoring DQL on the fly: DiffbotOntologyTool + DiffbotDQLProbeTool\n\nSo an agent can build valid DQL instead of guessing field names, two tools wrap Diffbot's DQL-authoring helpers. The intended loop is **introspect (ontology) → probe → run (`DiffbotKnowledgeGraphTool`) → refine**.\n\n`DiffbotOntologyTool` navigates the KG ontology — discover real entity types, field paths, taxonomy, and enum values before querying. The ontology is fetched once over HTTP and cached on the tool instance for the rest of its lifetime.\n\n```python\nimport os\nfrom diffbot import Diffbot\nfrom langchain_diffbot import DiffbotOntologyTool\n\ntool = DiffbotOntologyTool(client=Diffbot(token=os.environ[\"DIFFBOT_API_TOKEN\"]))\ntypes = tool.invoke({\"op\": \"types\"})\n```\n\n`DiffbotDQLProbeTool` probes query variants at `size=0` (hit counts only), so an agent can check selectivity — not zero, not millions — before committing to a full query.\n\n```python\nimport os\nfrom diffbot import Diffbot\nfrom langchain_diffbot import DiffbotDQLProbeTool\n\ntool = DiffbotDQLProbeTool(client=Diffbot(token=os.environ[\"DIFFBOT_API_TOKEN\"]))\ncounts = tool.invoke({\"queries\": ['type:Organization name:\"Diffbot\"', \"type:Person\"]})\n```\n\n## Using a retriever in a chain\n\nThe retrievers are standard `BaseRetriever`s, so they slot into LCEL like any other:\n\n```python\nimport os\nfrom diffbot import Diffbot\nfrom langchain_anthropic import ChatAnthropic\nfrom langchain_core.output_parsers import StrOutputParser\nfrom langchain_core.prompts import ChatPromptTemplate\nfrom langchain_core.runnables import RunnablePassthrough\nfrom langchain_diffbot import DiffbotKnowledgeGraphRetriever\n\nretriever = DiffbotKnowledgeGraphRetriever(\n    client=Diffbot(token=os.environ[\"DIFFBOT_API_TOKEN\"]),\n    k=5,\n    fields=[\"id\", \"name\", \"homepageUri\", \"nbEmployees\", \"industries\"],\n)\n\nprompt = ChatPromptTemplate.from_template(\n    \"Answer using only this Diffbot KG context:\\n\\n{context}\\n\\nQuestion: {question}\"\n)\n\n\ndef _format(docs):\n    return \"\\n---\\n\".join(\n        f\"{d.metadata.get('name')} (id={d.metadata.get('id')}): {d.page_content}\"\n        for d in docs\n    )\n\n\nchain = (\n    {\"context\": retriever | _format, \"question\": RunnablePassthrough()}\n    | prompt\n    | ChatAnthropic(model=\"claude-sonnet-4-6\")\n    | StrOutputParser()\n)\n\nchain.invoke('type:Organization location.city.name:\"Boston\" industries:\"Biotech\"')\n```\n\n## Sharing a client across components\n\nBecause every component takes a client, you configure the SDK once and hand the same client to as many components as you like — they share its connection pool, and there's no per-call pool churn. Build the tools/retrievers you actually want and add only those to your agent; the client is the shared resource, not a bundle.\n\n```python\nimport os\nfrom diffbot import Diffbot\nfrom langchain_diffbot import (\n    DiffbotKnowledgeGraphTool,\n    DiffbotAskTool,\n    DiffbotWebSearchRetriever,\n)\n\n# One client, configured once (timeout, transport, custom URLs, ...),\n# shared across every component.\ndb = Diffbot(token=os.environ[\"DIFFBOT_API_TOKEN\"], timeout=60.0)\n\nkg = DiffbotKnowledgeGraphTool(client=db)\nask = DiffbotAskTool(client=db)\nweb = DiffbotWebSearchRetriever(client=db, k=5)\n\n# `db.close()` when you're done — the components never close it for you.\n```\n\nAnything the SDK supports (custom URLs, `transport=`, headers via a custom transport) is configured on the client you build — there's no second configuration surface to learn. For async, build a `diffbot.DiffbotAsync` and pass `async_client=` instead (or both, if a component is used both ways).\n\n## Components reference\n\n| Class | Abstraction | Import path |\n|-------|-------------|-------------|\n| `ChatDiffbot` | Chat model | `from langchain_diffbot import ChatDiffbot` |\n| `DiffbotKnowledgeGraphRetriever` | Retriever | `from langchain_diffbot import DiffbotKnowledgeGraphRetriever` |\n| `DiffbotWebSearchRetriever` | Retriever | `from langchain_diffbot import DiffbotWebSearchRetriever` |\n| `DiffbotExtractLoader` | Document loader | `from langchain_diffbot import DiffbotExtractLoader` |\n| `DiffbotCrawlLoader` | Document loader | `from langchain_diffbot import DiffbotCrawlLoader` |\n| `DiffbotExtractTool` | Tool | `from langchain_diffbot import DiffbotExtractTool` |\n| `DiffbotWebSearchTool` | Tool | `from langchain_diffbot import DiffbotWebSearchTool` |\n| `DiffbotKnowledgeGraphTool` | Tool | `from langchain_diffbot import DiffbotKnowledgeGraphTool` |\n| `DiffbotEntitiesTool` | Tool | `from langchain_diffbot import DiffbotEntitiesTool` |\n| `DiffbotAskTool` | Tool | `from langchain_diffbot import DiffbotAskTool` |\n| `DiffbotOntologyTool` | Tool | `from langchain_diffbot import DiffbotOntologyTool` |\n| `DiffbotDQLProbeTool` | Tool | `from langchain_diffbot import DiffbotDQLProbeTool` |\n\n## Examples\n\nThe [`examples/`](./examples) folder has runnable demos:\n\n- [`examples/quickstart/`](./examples/quickstart) — full tour: every public class, output shaping, async, and a multi-tool research agent.\n- [`examples/company_research/`](./examples/company_research) — the same multi-tool agent as a one-shot CLI: `cd examples \u0026\u0026 python -m company_research \"your question\"`. The agent combines KG search + web search + URL extract.\n\nBoth need `langchain` + `langchain-anthropic` on top of the base package — install the extra:\n\n```bash\npip install \"langchain-diffbot[examples]\"\n```\n\n## Development\n\n```bash\nuv sync --all-groups\nuv run pytest tests/unit_tests\n```\n\nIntegration tests hit the live Diffbot API and require `DIFFBOT_API_TOKEN`:\n\n```bash\nuv run pytest tests/integration_tests\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdiffbot%2Flangchain-diffbot","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdiffbot%2Flangchain-diffbot","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdiffbot%2Flangchain-diffbot/lists"}