https://github.com/jaredkrinke/link_checker

Link checker and web crawler for Deno
https://github.com/jaredkrinke/link_checker

deno link-checker web-crawler

Last synced: 3 months ago
JSON representation

Link checker and web crawler for Deno

• Host: GitHub
• URL: https://github.com/jaredkrinke/link_checker
• Owner: jaredkrinke
• License: mit-0
• Created: 2021-12-15T22:04:42.000Z (about 3 years ago)
• Default Branch: main
• Last Pushed: 2023-03-12T23:32:37.000Z (almost 2 years ago)
• Last Synced: 2024-10-19T04:09:23.600Z (4 months ago)
• Topics: deno, link-checker, web-crawler
• Language: TypeScript
• Homepage:
• Size: 63.5 KB
• Stars: 5
• Watchers: 3
• Forks: 1
• Open Issues: 2
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# link_checker

Link checker and web crawler for Deno.

Note: this module is currently functional but mostly undocumented.

# Link checker

## Usage

```text

$ deno run --allow-read --allow-net https://deno.land/x/link_checker/check.ts --help

Usage: deno run [--allow-read] [--allow-net] check.ts  [options]

Options:

  -x, --check-external-links  Check external links; note: consider using "-c N" to speed this up (default: false)

  -f, --check-fragments       Check URL fragment/hash for internal links (default: true)

  -c, --concurrency      Maximum concurrency (default: 1)

  -b, --base-url         Base URL for the site (default: entry point parent)

  --index-name          Index name for file system directories (default: "index.html")

  -v, --verbose               Enable verbose logging

  --version                   Display module version

  -h, -?, --help              Display usage information

```

# Web crawler

## Usage

```text

$ deno run --allow-read --allow-net https://deno.land/x/link_checker/crawl.ts --help 

Usage: deno run [--allow-read] [--allow-net] crawl.ts  [options]

Options:

  -x, --external-links   Strategy for external links: ignore, check, follow (default: "ignore")

  -c, --concurrency           Maximum concurrency (default: 1)

  -d, --depth                 Maximum crawl depth (default: Infinity)

  -b, --base-url              Base URL for the site (default: entry point parent)

  --index-name               Index name for file system directories (default: "index.html")

  --version                        Display module version

  -h, -?, --help                   Display usage information

```