{"id":25774497,"url":"https://github.com/ericc-ch/crawldown","last_synced_at":"2025-07-05T23:33:35.030Z","repository":{"id":275375756,"uuid":"921819788","full_name":"ericc-ch/crawldown","owner":"ericc-ch","description":"Crawl websites and convert their pages into clean, readable Markdown content using Mozilla's Readability and Turndown.","archived":false,"fork":false,"pushed_at":"2025-02-09T15:31:34.000Z","size":178,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-19T13:46:38.492Z","etag":null,"topics":["crawler","markdown","scraper"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ericc-ch.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-01-24T17:16:58.000Z","updated_at":"2025-02-09T15:31:37.000Z","dependencies_parsed_at":"2025-02-07T00:31:41.387Z","dependency_job_id":null,"html_url":"https://github.com/ericc-ch/crawldown","commit_stats":null,"previous_names":["ericc-ch/html2md","ericc-ch/web2md","ericc-ch/crawldown"],"tags_count":10,"template":false,"template_full_name":"ericc-ch/starter-ts","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ericc-ch%2Fcrawldown","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ericc-ch%2Fcrawldown/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ericc-ch%2Fcrawldown/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ericc-ch%2Fcrawldown/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ericc-ch","download_url":"https://codeload.github.com/ericc-ch/crawldown/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240983716,"owners_count":19888742,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","markdown","scraper"],"created_at":"2025-02-27T05:30:09.730Z","updated_at":"2025-02-27T05:30:10.277Z","avatar_url":"https://github.com/ericc-ch.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# crawldown\n\nCrawl websites and convert their pages into clean, readable Markdown content using Mozilla's Readability and Turndown.\n\nThis package combines website crawling with Mozilla's Readability (the same technology behind Firefox's Reader View) and Turndown to:\n\n1. Crawl websites and follow links up to a specified depth\n2. Extract the main content from each page, removing clutter and ads\n3. Convert the cleaned content into well-formatted Markdown\n4. Preserve the site structure in the output files\n\n## Features\n\n- Intelligent website crawling with configurable depth\n- Clean extraction of main content from web pages using Mozilla's Readability\n- Conversion of readable HTML to Markdown using Turndown\n- Removes ads, navigation, sidebars, and other distracting elements\n- Preserves important content structure and formatting\n- Maintains website hierarchy in output directory structure\n- Flexible output options including single file or directory structure\n- Smart URL handling and deduplication during crawling\n\n## Installation\n\n```bash\nnpm install crawldown\n# or\npnpm add crawldown\n# or\nyarn add crawldown\n```\n\n## CLI Usage\n\nThe package provides a command-line interface for easy conversion:\n\n```bash\n# Basic usage\nnpx crawldown https://example.com\n\n# Specify crawl depth (default: 0)\nnpx crawldown https://example.com -d 2\n\n# Custom output directory (when not using --single-file)\nnpx crawldown https://example.com -o my-docs\n\n# Output to a single file (will create result.md)\nnpx crawldown https://example.com --single-file -o result\n\n# Output to a single file with explicit .md extension\nnpx crawldown https://example.com --single-file -o result.md\n\n# Force scraping even if page hasn't loaded completely\nnpx crawldown https://example.com --force\n\n# Force scraping with short flag\nnpx crawldown https://example.com -f\n\n# Set custom navigation timeout (default: 10000ms)\nnpx crawldown https://example.com --timeout 30000\n\n# Set timeout with short flag\nnpx crawldown https://example.com -t 30000\n\n# Enable verbose logging\nnpx crawldown https://example.com -v\n\n# Specify custom browser path\nnpx crawldown https://example.com --browser-path /path/to/chrome\n\n# Set number of concurrent pages (default: 4)\nnpx crawldown https://example.com -c 8\n\n# Define a specific scope URL for crawling\nnpx crawldown https://example.com --scope-url https://example.com/docs\n```\n\n### CLI Options\n\n- `url`: URL to scrape (required, positional argument)\n- `-d, --depth`: Number of levels to crawl (default: \"0\")\n- `-v, --verbose`: Enable verbose logging (default: false)\n- `--browser-path`: Path to browser executable. Will use playwright default if not provided\n- `-o, --output`: When used with --single-file, specifies the output filename. Otherwise specifies the output directory (default: \"output\")\n- `--single-file`: Output all results to a single markdown file instead of a directory structure (default: false)\n- `-c, --concurrency`: Number of concurrent pages to use (default: \"4\")\n- `--scope-url`: URL that defines the crawling scope. Links outside this scope will be ignored\n- `-f, --force`: When enabled, if page load is taking too long, captures page content 1 second before the timeout would occur. This ensures content is retrieved even if the page never fully loads (default: false)\n- `-t, --timeout`: Navigation timeout in milliseconds (default: 10000)\n\n## Programmatic Usage\n\nYou can also use the package programmatically in your Node.js applications:\n\n```typescript\nimport { crawl } from \"crawldown\";\n\nasync function main() {\n  const results = await crawl({\n    url: \"https://example.com\", // Required: URL to crawl\n    depth: 1, // Optional: how deep to crawl links (default: 0)\n    browserPath: \"/path/to/chrome\", // Optional: custom browser executable path\n    concurrency: 4, // Optional: number of concurrent pages (default: 4)\n    scopeUrl: \"https://example.com/docs\", // Optional: URL that defines crawling scope\n    force: false, // Optional: force scraping before timeout (default: false)\n    timeout: 10000, // Optional: navigation timeout in ms (default: 10000)\n  });\n\n  // Each result contains:\n  // - url: string\n  // - title: string\n  // - markdown: string\n  for (const result of results) {\n    console.log(`Title: ${result.title}`);\n    console.log(`URL: ${result.url}`);\n    console.log(`Content:\\n${result.markdown}`);\n  }\n}\n```\n\n## Credits\n\nThis project was inspired by:\n\n- [repomix](https://github.com/yamadashy/repomix) - An awesome CLI tool to combine whole repositories into a single markdown file\n- [crawl4ai](https://github.com/unclecode/crawl4ai) - A feature-rich Python web crawler\n\n## License\n\n[Mozilla Public License 2.0](./LICENSE.md) License\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fericc-ch%2Fcrawldown","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fericc-ch%2Fcrawldown","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fericc-ch%2Fcrawldown/lists"}