{"id":48337446,"url":"https://github.com/us/langchain-crw","last_synced_at":"2026-04-05T03:04:20.229Z","repository":{"id":347407713,"uuid":"1193990615","full_name":"us/langchain-crw","owner":"us","description":"LangChain document loader for CRW — high-performance web scraping","archived":false,"fork":false,"pushed_at":"2026-04-04T17:48:34.000Z","size":179,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-04T20:22:24.202Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/us.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-27T19:53:23.000Z","updated_at":"2026-04-04T17:48:24.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/us/langchain-crw","commit_stats":null,"previous_names":["us/langchain-crw"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/us/langchain-crw","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/us%2Flangchain-crw","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/us%2Flangchain-crw/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/us%2Flangchain-crw/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/us%2Flangchain-crw/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/us","download_url":"https://codeload.github.com/us/langchain-crw/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/us%2Flangchain-crw/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31422901,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-05T02:22:46.605Z","status":"ssl_error","status_checked_at":"2026-04-05T02:22:33.263Z","response_time":75,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-04-05T03:04:16.306Z","updated_at":"2026-04-05T03:04:20.225Z","avatar_url":"https://github.com/us.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# langchain-crw\n\n[![PyPI version](https://img.shields.io/pypi/v/langchain-crw)](https://pypi.org/project/langchain-crw/)\n[![Python](https://img.shields.io/pypi/pyversions/langchain-crw)](https://pypi.org/project/langchain-crw/)\n[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)\n\nLangChain document loader for [CRW](https://github.com/us/crw) — a high-performance, Firecrawl-compatible web scraper written in Rust.\n\n## Installation\n\n```bash\npip install langchain-crw\n# or\nuv add langchain-crw\n```\n\nThat's it. No server to install, no `cargo install`, no Docker. The `crw` SDK automatically downloads and manages the CRW binary for you.\n\n## Quick Start — Zero Config (Subprocess Mode)\n\n```python\nfrom langchain_crw import CrwLoader\n\n# Just works — crw SDK handles everything locally\nloader = CrwLoader(url=\"https://example.com\", mode=\"scrape\")\ndocs = loader.load()\nprint(docs[0].page_content)  # clean markdown\n```\n\n## Cloud Mode ([fastcrw.com](https://fastcrw.com))\n\nNo local binary needed. [Sign up at fastcrw.com](https://fastcrw.com) and get **500 free credits**:\n\n```python\nfrom langchain_crw import CrwLoader\n\nloader = CrwLoader(\n    url=\"https://example.com\",\n    mode=\"scrape\",\n    api_url=\"https://fastcrw.com/api\",\n    api_key=\"crw_live_...\",  # or set CRW_API_KEY env var\n)\ndocs = loader.load()\n```\n\n## Advanced: Self-hosted Server\n\nIf you prefer running a persistent CRW server (e.g., shared across services):\n\n```bash\n# Option A: Install binary\ncurl -fsSL https://raw.githubusercontent.com/us/crw/main/install.sh | sh\ncrw  # starts on http://localhost:3000\n\n# Option B: Docker\ndocker run -d -p 3000:3000 ghcr.io/us/crw:latest\n```\n\n```python\nloader = CrwLoader(url=\"https://example.com\", api_url=\"http://localhost:3000\")\n```\n\n## Usage\n\n### Scrape a single page\n\n```python\nloader = CrwLoader(url=\"https://example.com\", mode=\"scrape\")\ndocs = loader.load()\n\nprint(docs[0].page_content)    # clean markdown\nprint(docs[0].metadata)        # {'title': '...', 'sourceURL': '...', 'statusCode': 200}\n```\n\n### Crawl an entire site\n\n```python\nloader = CrwLoader(\n    url=\"https://docs.example.com\",\n    mode=\"crawl\",\n    params={\"max_depth\": 3, \"max_pages\": 50},\n)\ndocs = loader.load()\nprint(f\"Crawled {len(docs)} pages\")\n```\n\n### Discover URLs (map mode)\n\n```python\nloader = CrwLoader(url=\"https://example.com\", mode=\"map\")\nurls = [doc.page_content for doc in loader.load()]\n```\n\n### Search the web (Cloud Only)\n\n\u003e **Cloud-only feature.** Search requires a fastcrw.com API key or a CRW server with search configured.\n\n```python\nfrom langchain_crw import CrwLoader\n\nloader = CrwLoader(\n    query=\"web scraping tools 2026\",\n    mode=\"search\",\n    api_url=\"https://fastcrw.com/api\",\n    api_key=\"YOUR_KEY\",\n    params={\"limit\": 5},\n)\ndocs = loader.load()\n\nfor doc in docs:\n    print(doc.metadata[\"title\"], doc.metadata[\"url\"])\n    print(doc.page_content[:200])\n```\n\n### Scrape with JS rendering\n\n```python\nloader = CrwLoader(\n    url=\"https://spa-app.example.com\",\n    mode=\"scrape\",\n    params={\n        \"render_js\": True,\n        \"wait_for\": 3000,\n        \"css_selector\": \"article.main-content\",\n    },\n)\ndocs = loader.load()\n```\n\n### RAG pipeline\n\n```python\nfrom langchain_crw import CrwLoader\nfrom langchain_openai import OpenAIEmbeddings\nfrom langchain_community.vectorstores import FAISS\nfrom langchain_text_splitters import RecursiveCharacterTextSplitter\n\n# Crawl docs (self-hosted or cloud — same code)\nloader = CrwLoader(url=\"https://docs.example.com\", mode=\"crawl\", params={\"max_depth\": 3, \"max_pages\": 50})\ndocs = loader.load()\n\n# Split and embed\nsplitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)\nchunks = splitter.split_documents(docs)\nvectorstore = FAISS.from_documents(chunks, OpenAIEmbeddings())\n\n# Query\nresults = vectorstore.similarity_search(\"how to authenticate\")\n```\n\n## Configuration\n\n### Constructor\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `url` | `str` | `\"\"` | URL to scrape, crawl, or map. Not required for search mode |\n| `api_key` | `str \\| None` | `None` | Bearer token. Falls back to `CRW_API_KEY` env var |\n| `api_url` | `str \\| None` | `None` | CRW server URL. Falls back to `CRW_API_URL`. If unset, uses subprocess mode (no server needed) |\n| `mode` | `\"scrape\" \\| \"crawl\" \\| \"map\" \\| \"search\"` | `\"scrape\"` | Operation mode |\n| `query` | `str \\| None` | `None` | Search query string. Required for search mode |\n| `params` | `dict \\| None` | `None` | Additional API parameters |\n\n### Params (snake_case, auto-converted to camelCase)\n\n| Param | Modes | Description |\n|-------|-------|-------------|\n| `render_js` | scrape | Enable JavaScript rendering |\n| `wait_for` | scrape | Wait time in ms after page load |\n| `css_selector` | scrape | CSS selector to extract |\n| `only_main_content` | scrape, crawl | Extract main content only |\n| `max_depth` | crawl, map | Maximum crawl depth |\n| `max_pages` | crawl | Maximum pages to crawl |\n| `use_sitemap` | map | Use sitemap for URL discovery |\n| `poll_interval` | crawl | Poll interval in seconds (default: 2) |\n| `timeout` | crawl | Crawl timeout in seconds (default: 300) |\n\n## Migrating from FireCrawlLoader\n\n`CrwLoader` supports the same `scrape`, `crawl`, and `map` modes, plus a `search` mode. Note that `CrwLoader` defaults to `mode=\"scrape\"` while `FireCrawlLoader` defaults to `mode=\"crawl\"` — set the mode explicitly when migrating.\n\n```python\n# Before\nfrom langchain_community.document_loaders import FireCrawlLoader\nloader = FireCrawlLoader(url=\"https://example.com\", api_key=\"fc-...\", mode=\"scrape\")\n\n# After — pip install langchain-crw, zero config, no server needed\nfrom langchain_crw import CrwLoader\nloader = CrwLoader(url=\"https://example.com\", mode=\"scrape\")\n```\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fus%2Flangchain-crw","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fus%2Flangchain-crw","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fus%2Flangchain-crw/lists"}