{"id":13404798,"url":"https://github.com/janreges/siteone-crawler","last_synced_at":"2026-03-18T01:25:36.222Z","repository":{"id":198317107,"uuid":"700333533","full_name":"janreges/siteone-crawler","owner":"janreges","description":"SiteOne Crawler is a website analyzer and exporter you'll ♥ as a Dev/DevOps, QA engineer, website owner or consultant. Works on all popular platforms - Windows, macOS and Linux (x64 and arm64 too).","archived":false,"fork":false,"pushed_at":"2024-08-28T21:50:55.000Z","size":24070,"stargazers_count":142,"open_issues_count":3,"forks_count":8,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-08-29T10:28:25.578Z","etag":null,"topics":["analyzer","crawler","crawling","performance","qa","quality-assessment","security","seo","seotools","stress-testing","swoole","testing","website"],"latest_commit_sha":null,"homepage":"https://crawler.siteone.io/","language":"PHP","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/janreges.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-10-04T11:58:19.000Z","updated_at":"2024-08-28T21:50:59.000Z","dependencies_parsed_at":null,"dependency_job_id":"9a06b128-f266-4a76-b8e0-b4105d26a51f","html_url":"https://github.com/janreges/siteone-crawler","commit_stats":null,"previous_names":["janreges/fast-website-crawler","janreges/siteone-website-crawler","janreges/siteone-crawler"],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/janreges%2Fsiteone-crawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/janreges%2Fsiteone-crawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/janreges%2Fsiteone-crawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/janreges%2Fsiteone-crawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/janreges","download_url":"https://codeload.github.com/janreges/siteone-crawler/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243554070,"owners_count":20309862,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["analyzer","crawler","crawling","performance","qa","quality-assessment","security","seo","seotools","stress-testing","swoole","testing","website"],"created_at":"2024-07-30T19:01:51.566Z","updated_at":"2026-03-18T01:25:36.206Z","avatar_url":"https://github.com/janreges.png","language":"PHP","funding_links":[],"categories":["Applications","PHP","Miscellaneous",":hammer_and_wrench: Standalone Open Source SEO Tools","⚙️ Backend \u0026 APIs","Data Processing"],"sub_categories":["Web",":gear: OpenClaw SEO Skills"],"readme":"# SiteOne Crawler\r\n\r\nSiteOne Crawler is a powerful and easy-to-use **website analyzer, cloner, and converter** designed for developers seeking security and performance insights, SEO specialists identifying optimization opportunities, and website owners needing reliable backups and offline versions.\r\n\r\n**Now rewritten in Rust** for maximum performance, minimal resource usage, and zero runtime dependencies. The transition from PHP+Swoole to Rust resulted in **25% faster execution** and **30% lower memory consumption** while producing identical output.\r\n\r\n**Discover the SiteOne Crawler advantage:**\r\n\r\n*   **Run Anywhere:** Single native binary for **🪟 Windows**, **🍎 macOS**, and **🐧 Linux** (x64 \u0026 arm64). No runtime dependencies.\r\n*   **Work Your Way:** Launch the binary without arguments for an **interactive wizard** 🧙 with 10 preset modes, use the extensive **command-line interface** 📟 ([releases](https://github.com/janreges/siteone-crawler/releases), [▶️ video](https://www.youtube.com/watch?v=25T_yx13naA\u0026list=PL9mElgTe-s1Csfg0jXWmDS0MHFN7Cpjwp)) for automation and power, or enjoy the intuitive **desktop GUI application** 💻 ([GUI app](https://github.com/janreges/siteone-crawler-gui), [▶️ video](https://www.youtube.com/watch?v=rFW8LNEVNdw)) for visual control.\r\n*   **Rich Output Formats:** Interactive **HTML audit report** 📊 with sortable tables and quality scoring (0.0-10.0) (see [nextjs.org sample](https://crawler.siteone.io/html/2024-08-23/forever/cl8xw4r-fdag8wg-44dd.html)), detailed **JSON** for programmatic consumption, and human-readable **text** for terminal. Send HTML reports directly to your inbox via **built-in SMTP mailer** 📧.\r\n*   **CI/CD Integration:** Built-in **quality gate** (`--ci`) with configurable thresholds — exit code 10 on failure enables automated deployment blocking. Also useful for **cache warming** — crawling the entire site after deployment populates your reverse proxy/CDN cache.\r\n*   **Offline \u0026 Markdown Power:** Create complete **offline clones** 💾 for browsing without a server ([nextjs.org clone](https://crawler.siteone.io/examples-exports/nextjs.org/)) or convert entire websites into clean **Markdown** 📝 — perfect for backups, documentation, or feeding content to AI models ([examples](https://github.com/janreges/siteone-crawler-markdown-examples/)).\r\n*   **Deep Crawling \u0026 Analysis:** Thoroughly crawl every page and asset, identify errors (404s, redirects), generate **sitemaps** 🗺️, and even get **email summaries** 📧 (watch [▶️ video example](https://www.youtube.com/watch?v=PHIFSOmk0gk)).\r\n*   **Learn More:** Dive into the 🌐 [Project Website](https://crawler.siteone.io/), explore the detailed [Documentation](https://crawler.siteone.io/configuration/command-line-options/), or check the [JSON](docs/JSON-OUTPUT.md)/[Text](docs/TEXT-OUTPUT.md) output specs.\r\n\r\nGIF animation of the crawler in action (also available as a [▶️ video](https://www.youtube.com/watch?v=25T_yx13naA\u0026list=PL9mElgTe-s1Csfg0jXWmDS0MHFN7Cpjwp)):\r\n\r\n![SiteOne Crawler](docs/siteone-crawler-command-line.gif)\r\n\r\n## Table of contents\r\n\r\n- [✨ Features](#-features)\r\n    * [🕷️ Crawler](#️-crawler)\r\n    * [🛠️ Dev/DevOps assistant](#️-devdevops-assistant)\r\n    * [📊 Analyzer](#-analyzer)\r\n    * [📧 Reporter](#-reporter)\r\n    * [💾 Offline website generator](#-offline-website-generator)\r\n    * [📝 Website to markdown converter](#-website-to-markdown-converter)\r\n    * [🗺️ Sitemap generator](#️-sitemap-generator)\r\n- [🚀 Installation](#-installation)\r\n    * [📦 Pre-built binaries](#-pre-built-binaries)\r\n    * [🍺 Homebrew (macOS / Linux)](#-homebrew-macos--linux)\r\n    * [🐧 Debian / Ubuntu (apt)](#-debian--ubuntu-apt)\r\n    * [🎩 Fedora / RHEL (dnf)](#-fedora--rhel-dnf)\r\n    * [🦎 openSUSE / SLES (zypper)](#-opensuse--sles-zypper)\r\n    * [🏔️ Alpine Linux (apk)](#️-alpine-linux-apk)\r\n    * [🔨 Build from source](#-build-from-source)\r\n- [▶️ Usage](#️-usage)\r\n    * [Interactive wizard](#interactive-wizard)\r\n    * [Basic example](#basic-example)\r\n    * [CI/CD example](#cicd-example)\r\n    * [Fully-featured example](#fully-featured-example)\r\n    * [⚙️ Arguments](#️-arguments)\r\n        + [Basic settings](#basic-settings)\r\n        + [Output settings](#output-settings)\r\n        + [Resource filtering](#resource-filtering)\r\n        + [Advanced crawler settings](#advanced-crawler-settings)\r\n        + [File export settings](#file-export-settings)\r\n        + [Mailer options](#mailer-options)\r\n        + [Upload options](#upload-options)\r\n        + [Offline exporter options](#offline-exporter-options)\r\n        + [Markdown exporter options](#markdown-exporter-options)\r\n        + [Sitemap options](#sitemap-options)\r\n        + [Expert options](#expert-options)\r\n        + [Fastest URL analyzer](#fastest-url-analyzer)\r\n        + [SEO and OpenGraph analyzer](#seo-and-opengraph-analyzer)\r\n        + [Slowest URL analyzer](#slowest-url-analyzer)\r\n        + [Built-in HTTP server](#built-in-http-server)\r\n        + [CI/CD settings](#cicd-settings)\r\n- [🏆 Quality Scoring](#-quality-scoring)\r\n- [🔄 CI/CD Integration](#-cicd-integration)\r\n- [📄 Output Examples](#-output-examples)\r\n- [🧪 Testing](#-testing)\r\n- [⚠️ Disclaimer](#️-disclaimer)\r\n- [📜 License](#-license)\r\n\r\n## ✨ Features\r\n\r\nIn short, the main benefits can be summarized in these points:\r\n\r\n- **🕷️ Crawler** - very powerful crawler of the entire website reporting useful information about each URL (status code,\r\n  response time, size, custom headers, titles, etc.)\r\n- **🛠️ Dev/DevOps assistant** - offers stress/load testing with configurable concurrent workers (`--workers`) and request\r\n  rate (`--max-reqs-per-sec`), cache warming, localhost testing, and rich URL/content-type filtering\r\n- **📊 Analyzer** - analyzes all webpages and reports strange or error behaviour and useful statistics (404, redirects, bad\r\n  practices, SEO and security issues, heading structures, etc.)\r\n- **📧 Reporter** - interactive **HTML audit report**, structured **JSON**, and colored **text** output; built-in\r\n  **SMTP mailer** sends HTML reports directly to your inbox\r\n- **💾 Offline website generator** - clone entire websites to browsable local HTML files (no server needed) including all\r\n  assets. Supports **multi-domain clones** — include subdomains or external domains with intelligent cross-linking.\r\n- **📝 Website to markdown converter** - export the entire website to browsable text markdown (viewable on GitHub or any\r\n  text editor), or generate a **single-file markdown** with smart header/footer deduplication — ideal for **feeding to AI\r\n  tools**. Includes a **built-in web server** that renders markdown exports as styled HTML pages.\r\n  See [markdown examples](https://github.com/janreges/siteone-crawler-markdown-examples/).\r\n- **🗺️ Sitemap generator** - allows you to generate `sitemap.xml` and `sitemap.txt` files with a list of all pages on your\r\n  website\r\n- **🏆 Quality scoring** - automatic quality scoring (0.0-10.0) across 5 categories: Performance, SEO, Security, Accessibility, Best Practices\r\n- **🔄 CI/CD quality gate** - configurable thresholds with exit code 10 on failure for automated pipelines; also\r\n  useful as a **post-deployment cache warmer** for reverse proxies and CDNs\r\n\r\nThe following features are summarized in greater detail:\r\n\r\n### 🕷️ Crawler\r\n\r\n- **all major platforms** supported without dependencies (🐧 Linux, 🪟 Windows, 🍎 macOS, arm64) — single native binary\r\n- has incredible **🚀 native Rust performance** with async I/O and multi-threaded crawling\r\n- provides simulation of **different device types** (desktop/mobile/tablet) thanks to predefined User-Agents\r\n- will crawl **all files**, styles, scripts, fonts, images, documents, etc. on your website\r\n- will respect the `robots.txt` file and will not crawl the pages that are not allowed\r\n- has a **beautiful interactive** and **🎨 colourful output**\r\n- it will **clearly warn you** ⚠️ of any wrong use of the tool (e.g. input parameters validation or wrong permissions)\r\n- as `--url` parameter, you can specify also a `sitemap.xml` file (or [sitemap index](https://www.sitemaps.org/protocol.html#index)),\r\n  which will be processed as a list of URLs. In sitemap-only mode, the crawler follows only URLs from\r\n  the sitemap — it does not discover additional links from HTML pages. Gzip-compressed sitemaps (`*.xml.gz`)\r\n  are fully supported, both as direct URLs and when referenced from sitemap index files.\r\n- respects the HTML `\u003cbase href\u003e` tag when resolving relative URLs on pages that use it.\r\n\r\n### 🛠️ Dev/DevOps assistant\r\n\r\n- allows testing **public** and **local projects on specific ports** (e.g. `http://localhost:3000/`)\r\n- works as a **stress/load tester** — configure the number of **concurrent workers** (`--workers`) and the **maximum\r\n  requests per second** (`--max-reqs-per-sec`) to simulate various traffic levels and test your infrastructure's\r\n  resilience against high load or DoS scenarios\r\n- combine with **rich filtering options** — include/ignore URLs by regex (`--include-regex`, `--ignore-regex`), disable\r\n  specific asset types (`--disable-javascript`, `--disable-images`, etc.), or limit crawl depth (`--max-depth`) to focus\r\n  the load on specific parts of your website\r\n- will help you **warm up the application cache** or the **cache on the reverse proxy** of the entire website\r\n\r\n### 📊 Analyzer\r\n\r\n- will **find the weak points** or **strange behavior** of your website\r\n- built-in analyzers cover SEO, security headers, accessibility, best practices, performance, SSL/TLS, caching, and more\r\n\r\n### 📧 Reporter\r\n\r\nThree output formats:\r\n\r\n- **Interactive HTML report** — a self-contained `.html` file with sortable tables, quality scores, color-coded\r\n  findings, and sections for SEO, security, accessibility, performance, headers, redirects, 404s, and more. Open it\r\n  in any browser — no server needed.\r\n- **JSON output** — structured data with all crawled URLs, response details, analysis findings, scores, and CI/CD gate\r\n  results. Ideal for programmatic consumption, dashboards, and integrations.\r\n- **Text output** — human-readable colored terminal output with tables, progress bars, and summaries.\r\n\r\nAdditional reporting features:\r\n\r\n- **Built-in SMTP mailer** — send the HTML audit report directly to one or more email addresses via your own SMTP\r\n  server. Configure sender, recipients, subject template, and SMTP credentials via CLI options.\r\n- will provide you with data for **SEO analysis**, just add the `Title`, `Keywords` and `Description` extra columns\r\n- will provide useful **summaries and statistics** at the end of the processing\r\n\r\n### 💾 Offline website generator\r\n\r\n- will help you **export the entire website** to offline form, where it is possible to browse the site through local\r\n  HTML files (without HTTP server) including all documents, images, styles, scripts, fonts, etc.\r\n- supports **multi-domain clones** — include subdomains (`*.mysite.tld`) or entirely different domains in a single\r\n  offline export. All URLs across included domains are **intelligently rewritten to relative paths**, so the resulting\r\n  offline version cross-links pages between domains seamlessly — you get one unified browsable clone.\r\n- you can **limit what assets** you want to download and export (see `--disable-*` directives) .. for some types of\r\n  websites the best result is with the `--disable-javascript` option.\r\n- you can specify by `--allowed-domain-for-external-files` (short `-adf`) from which **external domains** it is possible\r\n  to **download** assets (JS, CSS, fonts, images, documents) including `*` option for all domains.\r\n- you can specify by `--allowed-domain-for-crawling` (short `-adc`) which **other domains** should be included in the\r\n  **crawling** if there are any links pointing to them. You can enable e.g. `mysite.*` to export all language mutations\r\n  that have a different TLD or `*.mysite.tld` to export all subdomains.\r\n- you can use `--single-page` to **export only one page** to which the URL is given (and its assets), but do not follow\r\n  other pages.\r\n- you can use `--single-foreign-page` to **export only one page** from another domain (if allowed by `--allowed-domain-for-crawling`),\r\n  but do not follow other pages.\r\n- you can use `--replace-content` to **replace content** in HTML/JS/CSS with `foo -\u003e bar` or regexp in PCRE format, e.g.\r\n  `/card[0-9]/i -\u003e card`. Can be specified multiple times.\r\n- you can use `--replace-query-string` to **replace chars in query string** in the filename.\r\n- you can use `--max-depth` to set the **maximum crawling depth** (for pages, not assets). `1` means `/about` or `/about/`,\r\n  `2` means `/about/contacts` etc.\r\n- you can use it to **export your website to a static form** and host it on GitHub Pages, Netlify, Vercel, etc. as a\r\n  static backup and part of your **disaster recovery plan** or **archival/legal needs**\r\n- works great with **older conventional websites** but also **modern ones**, built on frameworks like Next.js, Nuxt.js,\r\n  SvelteKit, Astro, Gatsby, etc. When a JS framework is detected, the export also performs some framework-specific code\r\n  modifications for optimal results.\r\n- **try it** for your website, and you will be very pleasantly surprised :-)\r\n\r\n### 📝 Website to markdown converter\r\n\r\nTwo export modes:\r\n\r\n- **Multi-file markdown** — exports the entire website with all subpages to a directory of **browsable `.md` files**.\r\n  The markdown renders nicely when uploaded to GitHub, viewed in VS Code, or any text editor. Links between pages are\r\n  converted to relative `.md` links so you can navigate between files. Optionally includes images and other files\r\n  (PDF, etc.).\r\n- **Single-file markdown** — combines all pages into **one large markdown file** with smart removal of duplicate website\r\n  headers and footers across pages. Ideal for **feeding entire website content to AI tools** (ChatGPT, Claude, etc.)\r\n  that process markdown more effectively than raw HTML.\r\n\r\nSmart conversion features:\r\n\r\n- **collapsible accordions** — large link lists (menus, navigation, footer links with 8+ items) are automatically\r\n  collapsed into `\u003cdetails\u003e` accordions with contextual labels (\"Menu\", \"Links\") for better readability\r\n- content before the main heading (typically h1) — such as the site header and navigation — is moved to the end of the\r\n  page below a `---` separator, so the actual page content comes first\r\n- you can set multiple selectors (CSS-like) to **remove unwanted elements** from the exported markdown\r\n- **code block detection** and **syntax highlighting** for popular programming languages\r\n- HTML tables are converted to proper **markdown tables**\r\n\r\nBuilt-in web server:\r\n\r\n- use `--serve-markdown=\u003cdir\u003e` to start a **built-in HTTP server** that renders your markdown export as styled HTML\r\n  pages with tables, dark/light mode, breadcrumb navigation, and accordion support — perfect for browsing and sharing\r\n  the export locally or on a network\r\n\r\n💡 Tip: you can push the exported markdown folder to your GitHub repository, where it will be automatically rendered as a browsable\r\ndocumentation. You can look at the [examples](https://github.com/janreges/siteone-crawler-markdown-examples/) of converted websites to markdown.\r\n\r\nSee all available [markdown exporter options](#markdown-exporter-options).\r\n\r\n### 🗺️ Sitemap generator\r\n\r\n- will help you create a `sitemap.xml` and `sitemap.txt` for your website\r\n- you can set the priority of individual pages based on the number of slashes in the URL\r\n\r\nDon't hesitate and try it. You will love it as we do! ❤️\r\n\r\n## 🚀 Installation\r\n\r\n### 📦 Pre-built binaries\r\n\r\nDownload pre-built binaries from [🐙 GitHub releases](https://github.com/janreges/siteone-crawler/releases) for all major platforms (🐧 Linux, 🪟 Windows, 🍎 macOS, x64 \u0026 arm64).\r\n\r\nThe binary is self-contained — no runtime dependencies required.\r\n\r\n```bash\r\n# Linux / macOS — download, extract, run\r\n./siteone-crawler --url=https://my.domain.tld\r\n```\r\n\r\n**Note for macOS users**: In case that Mac refuses to start the crawler from your Download folder, move the entire folder with the Crawler **via the terminal** to another location, for example to the homefolder `~`.\r\n\r\n### 🍺 Homebrew (macOS / Linux)\r\n\r\n```bash\r\nbrew install janreges/tap/siteone-crawler\r\nsiteone-crawler --url=https://my.domain.tld\r\n```\r\n\r\n### 🐧 Debian / Ubuntu (apt)\r\n\r\n```bash\r\ncurl -1sLf 'https://dl.cloudsmith.io/public/janreges/siteone-crawler/setup.deb.sh' | sudo -E bash\r\nsudo apt-get install siteone-crawler\r\n```\r\n\r\n### 🎩 Fedora / RHEL (dnf)\r\n\r\n```bash\r\ncurl -1sLf 'https://dl.cloudsmith.io/public/janreges/siteone-crawler/setup.rpm.sh' | sudo -E bash\r\nsudo dnf install siteone-crawler\r\n```\r\n\r\n### 🦎 openSUSE / SLES (zypper)\r\n\r\n```bash\r\ncurl -1sLf 'https://dl.cloudsmith.io/public/janreges/siteone-crawler/setup.rpm.sh' | sudo -E bash\r\nsudo zypper install siteone-crawler\r\n```\r\n\r\n### 🏔️ Alpine Linux (apk)\r\n\r\n```bash\r\ncurl -1sLf 'https://dl.cloudsmith.io/public/janreges/siteone-crawler/setup.alpine.sh' | sudo -E bash\r\nsudo apk add siteone-crawler\r\n```\r\n\r\n### 🔨 Build from source\r\n\r\nRequires [Rust](https://www.rust-lang.org/tools/install) 1.85 or later.\r\n\r\n```bash\r\ngit clone https://github.com/janreges/siteone-crawler.git\r\ncd siteone-crawler\r\n\r\n# Build optimized release binary\r\ncargo build --release\r\n\r\n# Run\r\n./target/release/siteone-crawler --url=https://my.domain.tld\r\n```\r\n\r\n## ▶️ Usage\r\n\r\n### Interactive wizard\r\n\r\nRun the binary **without any arguments** and an interactive wizard will guide you through the\r\nconfiguration. Choose from 10 preset modes, enter the target URL, fine-tune settings with\r\narrow keys, and the crawler starts immediately — no need to remember CLI flags.\r\n\r\n```\r\n? Choose a crawl mode:\r\n❯ Quick Audit               Fast site health overview — crawls all pages and assets\r\n  SEO Analysis               Extract titles, descriptions, keywords, and OpenGraph tags\r\n  Performance Test           Measure response times with cache disabled — find bottlenecks\r\n  Security Check             Check SSL/TLS, security headers, and redirects site-wide\r\n  Offline Clone              Download entire website with all assets for offline browsing\r\n  Markdown Export            Convert pages to Markdown for AI models or documentation\r\n  Stress Test                High-concurrency load test with cache-busting random params\r\n  Single Page                Deep analysis of a single URL — SEO, security, performance\r\n  Large Site Crawl           High-throughput HTML-only crawl for large sites (100k+ pages)\r\n  Custom                     Start from defaults and configure every option manually\r\n  ──────────────────────────────────────\r\n  Browse offline export      Serve a previously exported offline site via HTTP\r\n  Browse markdown export     Serve a previously exported markdown site via HTTP\r\n[↑↓ to move, enter to select, type to filter]\r\n```\r\n\r\nAfter selecting a preset and entering the URL, the wizard shows a settings form where you can\r\nadjust workers, timeout, content types, export options, and more. A configuration summary with the\r\nequivalent CLI command is displayed before the crawl starts — copy it for future use without the\r\nwizard.\r\n\r\nIf existing offline or markdown exports are detected in `./tmp/`, the wizard also offers to\r\n**serve them via the built-in HTTP server** directly from the menu.\r\n\r\n### Basic example\r\n\r\nTo run the crawler from the command line, provide the required arguments:\r\n\r\n```bash\r\n./siteone-crawler --url=https://mydomain.tld/ --device=mobile\r\n```\r\n\r\n### CI/CD example\r\n\r\n```bash\r\n# Fail deployment if quality score \u003c 7.0 or any 5xx errors\r\n./siteone-crawler --url=https://mydomain.tld/ --ci --ci-min-score=7.0 --ci-max-5xx=0\r\necho $?  # 0 = pass, 10 = fail\r\n```\r\n\r\n### Fully-featured example\r\n\r\n```bash\r\n./siteone-crawler --url=https://mydomain.tld/ \\\r\n  --output=text \\\r\n  --workers=2 \\\r\n  --max-reqs-per-sec=10 \\\r\n  --memory-limit=2048M \\\r\n  --resolve='mydomain.tld:443:127.0.0.1' \\\r\n  --timeout=5 \\\r\n  --proxy=proxy.mydomain.tld:8080 \\\r\n  --http-auth=myuser:secretPassword123 \\\r\n  --user-agent=\"My User-Agent String\" \\\r\n  --extra-columns=\"DOM,X-Cache(10),Title(40),Keywords(50),Description(50\u003e),Heading1=xpath://h1/text()(20\u003e),ProductPrice=regexp:/Price:\\s*\\$?(\\d+(?:\\.\\d{2})?)/i#1(10)\" \\\r\n  --accept-encoding=\"gzip, deflate\" \\\r\n  --url-column-size=100 \\\r\n  --max-queue-length=3000 \\\r\n  --max-visited-urls=10000 \\\r\n  --max-url-length=5000 \\\r\n  --max-non200-responses-per-basename=10 \\\r\n  --include-regex=\"/^.*\\/technologies.*/\" \\\r\n  --include-regex=\"/^.*\\/fashion.*/\" \\\r\n  --ignore-regex=\"/^.*\\/downloads\\/.*\\.pdf$/i\" \\\r\n  --analyzer-filter-regex=\"/^.*$/i\" \\\r\n  --remove-query-params \\\r\n  --add-random-query-params \\\r\n  --transform-url=\"live-site.com -\u003e local-site.local\" \\\r\n  --transform-url=\"/cdn\\.live-site\\.com/ -\u003e local-site.local/cdn\" \\\r\n  --show-scheme-and-host \\\r\n  --do-not-truncate-url \\\r\n  --output-html-report=tmp/myreport.html \\\r\n  --html-report-options=\"summary,seo-opengraph,visited-urls,security,redirects\" \\\r\n  --output-json-file=/dir/report.json \\\r\n  --output-text-file=/dir/report.txt \\\r\n  --add-timestamp-to-output-file \\\r\n  --add-host-to-output-file \\\r\n  --offline-export-dir=tmp/mydomain.tld \\\r\n  --replace-content='/\u003cfoo[^\u003e]+\u003e/ -\u003e \u003cbar\u003e' \\\r\n  --ignore-store-file-error \\\r\n  --sitemap-xml-file=/dir/sitemap.xml \\\r\n  --sitemap-txt-file=/dir/sitemap.txt \\\r\n  --sitemap-base-priority=0.5 \\\r\n  --sitemap-priority-increase=0.1 \\\r\n  --markdown-export-dir=tmp/mydomain.tld.md \\\r\n  --markdown-export-single-file=tmp/mydomain.tld.combined.md \\\r\n  --markdown-move-content-before-h1-to-end \\\r\n  --markdown-disable-images \\\r\n  --markdown-disable-files \\\r\n  --markdown-remove-links-and-images-from-single-file \\\r\n  --markdown-exclude-selector='.exclude-me' \\\r\n  --markdown-replace-content='/\u003cfoo[^\u003e]+\u003e/ -\u003e \u003cbar\u003e' \\\r\n  --markdown-replace-query-string='/[a-z]+=[^\u0026]*(\u0026|$)/i -\u003e $1__$2' \\\r\n  --mail-to=your.name@my-mail.tld \\\r\n  --mail-to=your.friend.name@my-mail.tld \\\r\n  --mail-from=crawler@my-mail.tld \\\r\n  --mail-from-name=\"SiteOne Crawler\" \\\r\n  --mail-subject-template=\"Crawler Report for %domain% (%date%)\" \\\r\n  --mail-smtp-host=smtp.my-mail.tld \\\r\n  --mail-smtp-port=25 \\\r\n  --mail-smtp-user=smtp.user \\\r\n  --mail-smtp-pass=secretPassword123 \\\r\n  --ci --ci-min-score=7.0 --ci-min-security=8.0\r\n```\r\n\r\n## ⚙️ Arguments\r\n\r\nFor a clearer list, I recommend going to the documentation: 🌐 https://crawler.siteone.io/configuration/command-line-options/\r\n\r\n### Basic settings\r\n\r\n| Parameter | Description |\r\n|-----------|-------------|\r\n| `--url=\u003curl\u003e` | Required. HTTP or HTTPS URL address of the website or sitemap xml to be crawled.\u003cbr\u003eUse quotation marks `''` if the URL contains query parameters. |\r\n| `--single-page` | Load only one page to which the URL is given (and its assets), but do not follow other pages. |\r\n| `--max-depth=\u003cint\u003e` | Maximum crawling depth (for pages, not assets). Default is `0` (no limit). `1` means `/about`\u003cbr\u003eor `/about/`, `2` means `/about/contacts` etc. |\r\n| `--device=\u003cval\u003e` | Device type for choosing a predefined User-Agent. Ignored when `--user-agent` is defined.\u003cbr\u003eSupported values: `desktop`, `mobile`, `tablet`. Default is `desktop`. |\r\n| `--user-agent=\u003cval\u003e` | Custom User-Agent header. Use quotation marks. If specified, it takes precedence over\u003cbr\u003ethe device parameter. If you add `!` at the end, the siteone-crawler/version will not be\u003cbr\u003eadded as a signature at the end of the final user-agent. |\r\n| `--timeout=\u003cint\u003e` | Request timeout in seconds. Default is `5`. |\r\n| `--proxy=\u003chost:port\u003e` | HTTP proxy to use in `host:port` format. Host can be hostname, IPv4 or IPv6. |\r\n| `--http-auth=\u003cuser:pass\u003e` | Basic HTTP authentication in `username:password` format. |\r\n| `--config-file=\u003cfile\u003e` | Load CLI options from a config file. One option per line, `#` comments allowed.\u003cbr\u003eWithout this flag, auto-discovers `~/.siteone-crawler.conf` or `/etc/siteone-crawler.conf`.\u003cbr\u003eCLI arguments override config file values. |\r\n\r\n### Output settings\r\n\r\n| Parameter | Description |\r\n|-----------|-------------|\r\n| `--output=\u003cval\u003e` | Output type. Supported values: `text`, `json`. Default is `text`. |\r\n| `--extra-columns=\u003cvalues\u003e` | Comma delimited list of extra columns added to output table. You can specify HTTP headers\u003cbr\u003e(e.g. `X-Cache`), predefined values (`Title`, `Keywords`, `Description`, `DOM`), or custom\u003cbr\u003eextraction from text files (HTML, JS, CSS, TXT, JSON, XML, etc.) using XPath or regexp.\u003cbr\u003eFor custom extraction, use the format `Custom_column_name=method:pattern#group(length)`, where\u003cbr\u003e`method` is `xpath` or `regexp`, `pattern` is the extraction pattern, an optional `#group` specifies the\u003cbr\u003ecapturing group (or node index for XPath) to return (defaulting to the entire match or first node), and an\u003cbr\u003eoptional `(length)` sets the maximum output length (append `\u003e` to disable truncation).\u003cbr\u003eFor example, use `Heading1=xpath://h1/text()(20\u003e)` to extract the text of the first H1 element\u003cbr\u003efrom the HTML document, and `ProductPrice=regexp:/Price:\\s*\\$?(\\d+(?:\\.\\d{2})?)/i#1(10)`\u003cbr\u003eto extract a numeric price (e.g., \"29.99\") from a string like \"Price: $29.99\". |\r\n| `--url-column-size=\u003cnum\u003e` | Basic URL column width. By default, it is calculated from the size of your terminal window. |\r\n| `--rows-limit=\u003cnum\u003e` | Max. number of rows to display in tables with analysis results.\u003cbr\u003eDefault is `200`. |\r\n| `--timezone=\u003cval\u003e` | Timezone for datetimes in HTML reports and timestamps in output folders/files, e.g. `Europe/Prague`.\u003cbr\u003eDefault is `UTC`. |\r\n| `--do-not-truncate-url` | In the text output, long URLs are truncated by default to `--url-column-size` so the table does not\u003cbr\u003ewrap due to long URLs. With this option, you can turn off the truncation. |\r\n| `--show-scheme-and-host` | On text output, show scheme and host also for origin domain URLs. |\r\n| `--hide-progress-bar` | Hide progress bar visible in text and JSON output for more compact view. |\r\n| `--hide-columns=\u003clist\u003e` | Hide specified columns from the progress table. Comma-separated list of column names:\u003cbr\u003e`type`, `time`, `size`, `cache`. Example: `--hide-columns=cache` or `--hide-columns=cache,type`. |\r\n| `--no-color` | Disable colored output. |\r\n| `--force-color` | Force colored output regardless of support detection. |\r\n| `--show-inline-criticals` | Show criticals from the analyzer directly in the URL table. |\r\n| `--show-inline-warnings` | Show warnings from the analyzer directly in the URL table. |\r\n\r\n### Resource filtering\r\n\r\n| Parameter | Description |\r\n|-----------|-------------|\r\n| `--disable-all-assets` | Disables crawling of all assets and files and only crawls pages in href attributes.\u003cbr\u003eShortcut for calling all other `--disable-*` flags. |\r\n| `--disable-javascript` | Disables JavaScript downloading and removes all JavaScript code from HTML,\u003cbr\u003eincluding `onclick` and other `on*` handlers. |\r\n| `--disable-styles` | Disables CSS file downloading and at the same time removes all style definitions\u003cbr\u003eby `\u003cstyle\u003e` tag or inline by style attributes. |\r\n| `--disable-fonts` | Disables font downloading and also removes all font/font-face definitions from CSS. |\r\n| `--disable-images` | Disables downloading of all images and replaces found images in HTML with placeholder image only. |\r\n| `--disable-files` | Disables downloading of any files (typically downloadable documents) to which various links point. |\r\n| `--remove-all-anchor-listeners` | On all links on the page remove any event listeners. Useful on some types of sites with modern\u003cbr\u003eJS frameworks that would like to compose content dynamically (React, Svelte, Vue, Angular, etc.). |\r\n\r\n### Advanced crawler settings\r\n\r\n| Parameter | Description |\r\n|-----------|-------------|\r\n| `--workers=\u003cint\u003e` | Maximum number of concurrent workers (threads).\u003cbr\u003eCrawler will not make more simultaneous requests to the server than this number.\u003cbr\u003eUse carefully! A high number of workers can cause a DoS attack. Default is `3`. |\r\n| `--max-reqs-per-sec=\u003cval\u003e` | Max requests/s for whole crawler. Be careful not to cause a DoS attack. Default value is `10`. |\r\n| `--memory-limit=\u003csize\u003e` | Memory limit in units `M` (Megabytes) or `G` (Gigabytes). Default is `2048M`. |\r\n| `--resolve=\u003chost:port:ip\u003e` | Custom DNS resolution in `domain:port:ip` format. Same as [curl --resolve](https://everything.curl.dev/usingcurl/connections/name.html?highlight=resolve#provide-a-custom-ip-address-for-a-name).\u003cbr\u003eCan be specified multiple times. |\r\n| `--allowed-domain-for-external-files=\u003cdomain\u003e` | Enable loading of file content from another domain (e.g. CDN).\u003cbr\u003eCan be specified multiple times. Use `*` for all domains. |\r\n| `--allowed-domain-for-crawling=\u003cdomain\u003e` | Allow crawling of other listed domains — typically language mutations on other domains.\u003cbr\u003eCan be specified multiple times. Use wildcards like `*.mysite.tld`. |\r\n| `--single-foreign-page` | When crawling of other domains is allowed, ensures that only the linked page\u003cbr\u003eand its assets are crawled from foreign domains. |\r\n| `--include-regex=\u003cregex\u003e` | PCRE-compatible regular expression for URLs that should be included.\u003cbr\u003eCan be specified multiple times. Example: `--include-regex='/^\\/public\\//'` |\r\n| `--ignore-regex=\u003cregex\u003e` | PCRE-compatible regular expression for URLs that should be ignored.\u003cbr\u003eCan be specified multiple times. |\r\n| `--regex-filtering-only-for-pages` | Apply `*-regex` rules only to page URLs, not static assets. |\r\n| `--analyzer-filter-regex` | PCRE-compatible regular expression for filtering analyzers by name. |\r\n| `--accept-encoding=\u003cval\u003e` | Custom `Accept-Encoding` request header. Default is `gzip, deflate, br`. |\r\n| `--remove-query-params` | Remove query parameters from found URLs. |\r\n| `--add-random-query-params` | Add random query parameters to each URL to bypass caches. |\r\n| `--transform-url=\u003cfrom-\u003eto\u003e` | Transform URLs before crawling. Use `from -\u003e to` for simple replacement or `/regex/ -\u003e replacement`.\u003cbr\u003eCan be specified multiple times. |\r\n| `--force-relative-urls` | Normalize all discovered URLs matching the initial domain (incl. www variant and protocol\u003cbr\u003edifferences) to canonical form. Prevents duplicate files in offline export when the site\u003cbr\u003euses inconsistent URL formats (http/https, www/non-www). |\r\n| `--ignore-robots-txt` | Ignore robots.txt content. |\r\n| `--http-cache-dir=\u003cdir\u003e` | Cache dir for HTTP responses. Disable with `--http-cache-dir='off'` or `--no-cache`.\u003cbr\u003eDefault is `~/.cache/siteone-crawler/http-cache` (XDG-compliant, respects `$XDG_CACHE_HOME`). |\r\n| `--http-cache-compression` | Enable compression for HTTP cache storage. |\r\n| `--http-cache-ttl=\u003cval\u003e` | TTL for HTTP cache entries (e.g. `1h`, `7d`, `30m`). Use `0` for infinite. Default is `24h`. |\r\n| `--no-cache` | Disable HTTP cache completely. Shortcut for `--http-cache-dir='off'`. |\r\n| `--max-queue-length=\u003cnum\u003e` | Maximum length of the waiting URL queue. Default is `9000`. |\r\n| `--max-visited-urls=\u003cnum\u003e` | Maximum number of visited URLs. Default is `10000`. |\r\n| `--max-skipped-urls=\u003cnum\u003e` | Maximum number of skipped URLs. Default is `10000`. |\r\n| `--max-url-length=\u003cnum\u003e` | Maximum supported URL length in chars. Default is `2083`. |\r\n| `--max-non200-responses-per-basename=\u003cnum\u003e` | Protection against looping with dynamic non-200 URLs. Default is `5`. |\r\n\r\n### File export settings\r\n\r\n| Parameter | Description |\r\n|-----------|-------------|\r\n| `--output-html-report=\u003cfile\u003e` | Save HTML report into that file. Set to empty `''` to disable HTML report.\u003cbr\u003eBy default saved into `tmp/%domain%.report.%datetime%.html`. |\r\n| `--html-report-options=\u003csections\u003e` | Comma-separated list of sections to include in HTML report.\u003cbr\u003eAvailable sections: `summary`, `seo-opengraph`, `image-gallery`, `video-gallery`, `visited-urls`, `dns-ssl`, `crawler-stats`, `crawler-info`, `headers`, `content-types`, `skipped-urls`, `external-links`, `caching`, `best-practices`, `accessibility`, `security`, `redirects`, `404-pages`, `slowest-urls`, `fastest-urls`, `source-domains`.\u003cbr\u003eDefault: all sections. |\r\n| `--output-json-file=\u003cfile\u003e` | File path for JSON output. Set to empty `''` to disable JSON file.\u003cbr\u003eBy default saved into `tmp/%domain%.output.%datetime%.json`.\u003cbr\u003eSee [JSON Output Documentation](docs/JSON-OUTPUT.md) for format details. |\r\n| `--output-text-file=\u003cfile\u003e` | File path for TXT output. Set to empty `''` to disable TXT file.\u003cbr\u003eBy default saved into `tmp/%domain%.output.%datetime%.txt`.\u003cbr\u003eSee [Text Output Documentation](docs/TEXT-OUTPUT.md) for format details. |\r\n| `--add-timestamp-to-output-file` | Append timestamp to output filenames (HTML report, JSON, TXT) except sitemaps. |\r\n| `--add-host-to-output-file` | Append initial URL host to output filenames (HTML report, JSON, TXT) except sitemaps. |\r\n\r\n**Default output directory:** Report files are saved into `./tmp/` in the current working directory. If `./tmp/` cannot be created (e.g. read-only filesystem), the crawler falls back to the platform's XDG data directory (`~/.local/share/siteone-crawler/` on Linux, `~/Library/Application Support/siteone-crawler/` on macOS, `%APPDATA%\\siteone-crawler\\` on Windows) and prints a notice to stderr.\r\n\r\n### Mailer options\r\n\r\n| Parameter | Description |\r\n|-----------|-------------|\r\n| `--mail-to=\u003cemail\u003e` | Recipients of HTML e-mail reports. Required for mailer activation.\u003cbr\u003eYou can specify multiple emails separated by comma. |\r\n| `--mail-from=\u003cemail\u003e` | E-mail sender address. Default is `siteone-crawler@your-hostname.com`. |\r\n| `--mail-from-name=\u003cval\u003e` | E-mail sender name. Default is `SiteOne Crawler`. |\r\n| `--mail-subject-template=\u003cval\u003e` | E-mail subject template. You can use `%domain%`, `%date%` and `%datetime%`.\u003cbr\u003eDefault is `Crawler Report for %domain% (%date%)`. |\r\n| `--mail-smtp-host=\u003chost\u003e` | SMTP host for sending emails. Default is `localhost`. |\r\n| `--mail-smtp-port=\u003cport\u003e` | SMTP port for sending emails. Default is `25`. |\r\n| `--mail-smtp-user=\u003cuser\u003e` | SMTP user, if your SMTP server requires authentication. |\r\n| `--mail-smtp-pass=\u003cpass\u003e` | SMTP password, if your SMTP server requires authentication. |\r\n\r\n### Upload options\r\n\r\n| Parameter | Description |\r\n|-----------|-------------|\r\n| `--upload` | Enable HTML report upload to `--upload-to`. |\r\n| `--upload-to=\u003curl\u003e` | URL of the endpoint where to send the HTML report. Default is `https://crawler.siteone.io/up`. |\r\n| `--upload-retention=\u003cval\u003e` | How long should the HTML report be kept in the online version?\u003cbr\u003eValues: 1h / 4h / 12h / 24h / 3d / 7d / 30d / 365d / forever.\u003cbr\u003eDefault is `30d`. |\r\n| `--upload-password=\u003cval\u003e` | Optional password (user will be 'crawler') to display the online HTML report. |\r\n| `--upload-timeout=\u003cint\u003e` | Upload timeout in seconds. Default is `3600`. |\r\n\r\n### Offline exporter options\r\n\r\n| Parameter | Description |\r\n|-----------|-------------|\r\n| `--offline-export-dir=\u003cdir\u003e` | Path to directory where to save the offline version of the website. |\r\n| `--offline-export-store-only-url-regex=\u003cregex\u003e` | Debug: store only URLs matching these PCRE regexes. Can be specified multiple times. |\r\n| `--offline-export-remove-unwanted-code=\u003c1/0\u003e` | Remove unwanted code for offline mode (analytics, social networks, etc.). Default is `1`. |\r\n| `--offline-export-no-auto-redirect-html` | Disable automatic creation of redirect HTML files for subfolders containing `index.html`. |\r\n| `--offline-export-preserve-url-structure` | Preserve the original URL path structure. E.g. `/about` is stored as `about/index.html`\u003cbr\u003einstead of `about.html`. Useful for web server deployment where the clone should maintain\u003cbr\u003ethe same URL hierarchy as the original site. |\r\n| `--replace-content=\u003cval\u003e` | Replace content in HTML/JS/CSS with `foo -\u003e bar` or PCRE regexp.\u003cbr\u003eCan be specified multiple times. |\r\n| `--replace-query-string=\u003cval\u003e` | Replace characters in query string filenames.\u003cbr\u003eCan be specified multiple times. |\r\n| `--offline-export-lowercase` | Convert all filenames to lowercase for offline export. Useful for case-insensitive filesystems. |\r\n| `--ignore-store-file-error` | Ignore any file storing errors and continue. |\r\n| `--disable-astro-inline-modules` | Disable inlining of Astro module scripts for offline export.\u003cbr\u003eScripts will remain as external files with corrected relative paths. |\r\n\r\n### Markdown exporter options\r\n\r\n| Parameter | Description |\r\n|-----------|-------------|\r\n| `--markdown-export-dir=\u003cdir\u003e` | Path to directory where to save the markdown version of the website. |\r\n| `--markdown-export-single-file=\u003cfile\u003e` | Path to a file for combined markdown. Requires `--markdown-export-dir`. |\r\n| `--markdown-move-content-before-h1-to-end` | Move content before main H1 heading to the end of the markdown. |\r\n| `--markdown-disable-images` | Do not export and show images in markdown files. |\r\n| `--markdown-disable-files` | Do not export files other than HTML/CSS/JS/fonts/images (e.g. PDF, ZIP). |\r\n| `--markdown-remove-links-and-images-from-single-file` | Remove links and images from combined single file. |\r\n| `--markdown-exclude-selector=\u003cval\u003e` | Exclude DOM elements by CSS selector from markdown export.\u003cbr\u003eCan be specified multiple times. |\r\n| `--markdown-replace-content=\u003cval\u003e` | Replace text content with `foo -\u003e bar` or PCRE regexp.\u003cbr\u003eCan be specified multiple times. |\r\n| `--markdown-replace-query-string=\u003cval\u003e` | Replace characters in query string filenames.\u003cbr\u003eCan be specified multiple times. |\r\n| `--markdown-export-store-only-url-regex=\u003cregex\u003e` | Debug: store only URLs matching these PCRE regexes. Can be specified multiple times. |\r\n| `--markdown-ignore-store-file-error` | Ignore any file storing errors and continue. |\r\n\r\n### Sitemap options\r\n\r\n| Parameter | Description |\r\n|-----------|-------------|\r\n| `--sitemap-xml-file=\u003cfile\u003e` | File path for generated XML Sitemap. Extension `.xml` added if not specified. |\r\n| `--sitemap-txt-file=\u003cfile\u003e` | File path for generated TXT Sitemap. Extension `.txt` added if not specified. |\r\n| `--sitemap-base-priority=\u003cnum\u003e` | Base priority for XML sitemap. Default is `0.5`. |\r\n| `--sitemap-priority-increase=\u003cnum\u003e` | Priority increase based on slashes in URL. Default is `0.1`. |\r\n\r\n### Expert options\r\n\r\n| Parameter | Description |\r\n|-----------|-------------|\r\n| `--debug` | Activate debug mode. |\r\n| `--debug-log-file=\u003cfile\u003e` | Log file for debug messages. When set without `--debug`, logging is active without visible output. |\r\n| `--debug-url-regex=\u003cregex\u003e` | Regex for URL(s) to debug. Can be specified multiple times. |\r\n| `--result-storage=\u003cval\u003e` | Result storage type. Values: `memory` or `file`. Use `file` for large websites. Default is `memory`. |\r\n| `--result-storage-dir=\u003cdir\u003e` | Directory for `--result-storage=file`. Default is `tmp/result-storage`. |\r\n| `--result-storage-compression` | Enable compression for results storage. |\r\n| `--http-cache-dir=\u003cdir\u003e` | Cache dir for HTTP responses. Disable with `--http-cache-dir='off'` or `--no-cache`.\u003cbr\u003eDefault is `~/.cache/siteone-crawler/http-cache` (XDG-compliant, respects `$XDG_CACHE_HOME`). |\r\n| `--http-cache-compression` | Enable compression for HTTP cache storage. |\r\n| `--http-cache-ttl=\u003cval\u003e` | TTL for HTTP cache entries (e.g. `1h`, `7d`, `30m`). Use `0` for infinite. Default is `24h`. |\r\n| `--websocket-server=\u003chost:port\u003e` | Start crawler with websocket server on given host:port. |\r\n| `--console-width=\u003cint\u003e` | Enforce a fixed console width, disabling automatic detection. |\r\n\r\n### Fastest URL analyzer\r\n\r\n| Parameter | Description |\r\n|-----------|-------------|\r\n| `--fastest-urls-top-limit=\u003cint\u003e` | Number of URLs in TOP fastest list. Default is `20`. |\r\n| `--fastest-urls-max-time=\u003cval\u003e` | Maximum response time for an URL to be considered fast. Default is `1`. |\r\n\r\n### SEO and OpenGraph analyzer\r\n\r\n| Parameter | Description |\r\n|-----------|-------------|\r\n| `--max-heading-level=\u003cint\u003e` | Max heading level from 1 to 6 for analysis. Default is `3`. |\r\n\r\n### Slowest URL analyzer\r\n\r\n| Parameter | Description |\r\n|-----------|-------------|\r\n| `--slowest-urls-top-limit=\u003cint\u003e` | Number of URLs in TOP slowest list. Default is `20`. |\r\n| `--slowest-urls-min-time=\u003cval\u003e` | Minimum response time threshold for slow URLs. Default is `0.01`. |\r\n| `--slowest-urls-max-time=\u003cval\u003e` | Maximum response time for very slow evaluation. Default is `3`. |\r\n\r\n### Built-in HTTP server\r\n\r\nBrowse exported markdown or offline HTML files through a local web server with a built-in viewer.\r\n\r\n| Parameter | Description |\r\n|-----------|-------------|\r\n| `--serve-markdown=\u003cdir\u003e` | Start built-in HTTP server for browsing a markdown export directory.\u003cbr\u003eRenders `.md` files as styled HTML with tables, accordions, dark/light mode, and breadcrumb navigation. |\r\n| `--serve-offline=\u003cdir\u003e` | Start built-in HTTP server for browsing an offline HTML export directory.\u003cbr\u003eServes static files with Content-Security-Policy restricting assets to the same origin. |\r\n| `--serve-port=\u003cint\u003e` | Port for the built-in HTTP server. Default is `8321`. |\r\n| `--serve-bind-address=\u003caddr\u003e` | Bind address for the built-in HTTP server. Default is `127.0.0.1` (localhost only).\u003cbr\u003eUse `0.0.0.0` to listen on all network interfaces and their IP addresses. |\r\n\r\n**Example:**\r\n\r\n```bash\r\n# Browse markdown export\r\n./siteone-crawler --serve-markdown=./exports/markdown\r\n\r\n# Browse offline export on custom port, accessible from network\r\n./siteone-crawler --serve-offline=./exports/offline --serve-port=9000 --serve-bind-address=0.0.0.0\r\n```\r\n\r\n### CI/CD settings\r\n\r\n| Parameter | Description |\r\n|-----------|-------------|\r\n| `--ci` | Enable CI/CD quality gate. Crawler exits with code 10 if thresholds are not met. Default file outputs (HTML, JSON, TXT reports) are suppressed unless explicitly requested via `--output-*` options. |\r\n| `--ci-min-score=\u003cval\u003e` | Minimum overall quality score (0.0-10.0). Default is `5.0`. |\r\n| `--ci-min-performance=\u003cval\u003e` | Minimum Performance category score (0.0-10.0). Default is `5.0`. |\r\n| `--ci-min-seo=\u003cval\u003e` | Minimum SEO category score (0.0-10.0). Default is `5.0`. |\r\n| `--ci-min-security=\u003cval\u003e` | Minimum Security category score (0.0-10.0). Default is `5.0`. |\r\n| `--ci-min-accessibility=\u003cval\u003e` | Minimum Accessibility category score (0.0-10.0). Default is `3.0`. |\r\n| `--ci-min-best-practices=\u003cval\u003e` | Minimum Best Practices category score (0.0-10.0). Default is `5.0`. |\r\n| `--ci-max-404=\u003cint\u003e` | Maximum number of 404 responses allowed. Default is `0`. |\r\n| `--ci-max-5xx=\u003cint\u003e` | Maximum number of 5xx server error responses allowed. Default is `0`. |\r\n| `--ci-max-criticals=\u003cint\u003e` | Maximum number of critical analysis findings allowed. Default is `0`. |\r\n| `--ci-max-warnings=\u003cint\u003e` | Maximum number of warning analysis findings allowed. Not checked by default. |\r\n| `--ci-max-avg-response=\u003cval\u003e` | Maximum average response time in seconds. Not checked by default. |\r\n| `--ci-min-pages=\u003cint\u003e` | Minimum number of HTML pages that must be found. Default is `10`. |\r\n| `--ci-min-assets=\u003cint\u003e` | Minimum number of assets (JS, CSS, images, fonts) that must be found. Default is `10`. |\r\n| `--ci-min-documents=\u003cint\u003e` | Minimum number of documents (PDF, etc.) that must be found. Default is `0` (not checked). |\r\n\r\n**Default behavior with `--ci` alone:** overall score \u003e= 5.0, each category score \u003e= 5.0 (Performance, SEO, Security, Best Practices) and Accessibility \u003e= 3.0, 404 errors \u003c= 0, 5xx errors \u003c= 0, critical findings \u003c= 0, HTML pages \u003e= 10, assets \u003e= 10. File outputs (HTML, JSON, TXT reports) are not generated. To save reports in CI mode, specify the desired output explicitly, e.g. `--ci --output-html-report=report.html`.\r\n\r\n## 🏆 Quality Scoring\r\n\r\nThe crawler automatically calculates a quality score (0.0-10.0) across 5 weighted categories:\r\n\r\n| Category | Weight | What it measures |\r\n|----------|--------|------------------|\r\n| **Performance** | 20% | Response times, slow URLs |\r\n| **SEO** | 20% | Missing H1, title uniqueness, meta descriptions, 404s, redirects |\r\n| **Security** | 25% | SSL/TLS certificates, security headers, unsafe protocols |\r\n| **Accessibility** | 20% | Lang attribute, image alt text, form labels, ARIA, heading levels |\r\n| **Best Practices** | 15% | Duplicate/large SVGs, deep DOM, Brotli/WebP support |\r\n\r\nThe overall score is a weighted average of all categories. Scores are displayed in a colored box in the console output and included in JSON and HTML report outputs.\r\n\r\nScore labels:\r\n- **9.0-10.0** — Excellent (green)\r\n- **7.0-8.9** — Good (blue)\r\n- **5.0-6.9** — Fair (yellow)\r\n- **3.0-4.9** — Poor (purple)\r\n- **0.0-2.9** — Critical (red)\r\n\r\n## 🔄 CI/CD Integration\r\n\r\nThe `--ci` flag enables a quality gate that evaluates configurable thresholds after crawling completes. When any threshold is not met, the crawler exits with **code 10** (distinct from exit code 1 for runtime errors). In CI mode, default file outputs (HTML, JSON, TXT reports) are automatically suppressed — only the console output and exit code matter. If you need report files in CI, specify them explicitly (e.g. `--output-html-report=report.html`).\r\n\r\n**Bonus: Cache warming** — running the crawler as a post-deployment step in your CI/CD pipeline crawls every page and asset on your site, which populates the HTML/asset cache on your **reverse proxy** (Varnish, Nginx) or **CDN** (Cloudflare, CloudFront). This way, the first real visitors always hit a warm cache instead of cold origin requests.\r\n\r\n### Exit codes\r\n\r\n| Code | Meaning |\r\n|------|---------|\r\n| `0` | Success (with `--ci` this also means all quality thresholds passed) |\r\n| `1` | Runtime error |\r\n| `2` | Help/version displayed |\r\n| `3` | No pages crawled (e.g. DNS failure, timeout, connection refused) |\r\n| `10` | CI/CD quality gate failed |\r\n| `101` | Configuration error |\r\n\r\n### Example: GitHub Actions\r\n\r\n```yaml\r\n- name: Check website quality\r\n  run: |\r\n    ./siteone-crawler \\\r\n      --url=https://staging.example.com \\\r\n      --ci \\\r\n      --ci-min-score=7.0 \\\r\n      --ci-min-security=8.0 \\\r\n      --ci-max-404=0 \\\r\n      --ci-max-5xx=0\r\n```\r\n\r\n### Example: GitLab CI\r\n\r\n```yaml\r\nquality_check:\r\n  script:\r\n    - ./siteone-crawler --url=$STAGING_URL --ci --ci-min-score=6.0\r\n  allow_failure: false\r\n```\r\n\r\n### Console output\r\n\r\nWhen `--ci` is enabled, a quality gate box is displayed after the quality scores:\r\n\r\n```\r\n╔══════════════════════════════════════════════════════════════╗\r\n║                      CI/CD QUALITY GATE                      ║\r\n╠══════════════════════════════════════════════════════════════╣\r\n║  [PASS] Overall score: 7.2 \u003e= 5                              ║\r\n║  [PASS] 404 errors: 0 \u003c= 0                                   ║\r\n║  [PASS] 5xx errors: 0 \u003c= 0                                   ║\r\n║  [FAIL] Critical findings: 2 \u003e 0 (max: 0)                    ║\r\n╠══════════════════════════════════════════════════════════════╣\r\n║  RESULT: FAIL (1 of 4 checks failed) — exit code 10          ║\r\n╚══════════════════════════════════════════════════════════════╝\r\n```\r\n\r\n### JSON output\r\n\r\nWhen using `--output=json --ci`, the JSON includes a `ciGate` object:\r\n\r\n```json\r\n{\r\n  \"ciGate\": {\r\n    \"passed\": false,\r\n    \"exitCode\": 10,\r\n    \"checks\": [\r\n      {\"metric\": \"Overall score\", \"operator\": \"\u003e=\", \"threshold\": 5.0, \"actual\": 7.2, \"passed\": true},\r\n      {\"metric\": \"404 errors\", \"operator\": \"\u003c=\", \"threshold\": 0.0, \"actual\": 0.0, \"passed\": true},\r\n      {\"metric\": \"Critical findings\", \"operator\": \"\u003c=\", \"threshold\": 0.0, \"actual\": 2.0, \"passed\": false}\r\n    ]\r\n  }\r\n}\r\n```\r\n\r\n## 📄 Output Examples\r\n\r\nTo understand the richness of the data provided by the crawler, you can examine real output examples generated from crawling `crawler.siteone.io`:\r\n\r\n*   **Text Output Example:** [`docs/OUTPUT-crawler.siteone.io.txt`](docs/OUTPUT-crawler.siteone.io.txt)\r\n    *   Provides a human-readable summary suitable for quick review.\r\n    *   See the detailed [Text Output Documentation](docs/TEXT-OUTPUT.md).\r\n*   **JSON Output Example:** [`docs/OUTPUT-crawler.siteone.io.json`](docs/OUTPUT-crawler.siteone.io.json)\r\n    *   Provides structured data ideal for programmatic consumption and detailed analysis.\r\n    *   See the detailed [JSON Output Documentation](docs/JSON-OUTPUT.md).\r\n\r\nThese examples showcase the various tables and metrics generated, demonstrating the tool's capabilities in analyzing website structure, performance, SEO, security, and more.\r\n\r\n## 🧪 Testing\r\n\r\n```bash\r\ncargo test                                       # unit tests + offline integration tests\r\ncargo test --test integration_crawl -- --ignored --test-threads=1  # network integration tests (crawls crawler.siteone.io)\r\n```\r\n\r\nUnit tests live in each source file (`#[cfg(test)] mod tests`). Integration tests are in `tests/integration_crawl.rs` — network-dependent tests are `#[ignore]` by default so that `cargo test` stays fast and offline.\r\n\r\n## ⚠️ Disclaimer\r\n\r\nPlease use responsibly and ensure that you have the necessary permissions when crawling websites. Some sites may have\r\nrules against automated access detailed in their robots.txt.\r\n\r\n**The author is not responsible for any consequences caused by inappropriate use or deliberate misuse of this tool.**\r\n\r\n## 📜 License\r\n\r\nThis work is licensed under a [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) license.\r\n\r\n## Powered by\r\n\r\n[![Hosted By: Cloudsmith](https://img.shields.io/badge/OSS%20hosting%20by-cloudsmith-blue?logo=cloudsmith\u0026style=for-the-badge)](https://cloudsmith.com)\r\n\r\nPackage repository hosting is graciously provided by  [Cloudsmith](https://cloudsmith.com).\r\nCloudsmith is the only fully hosted, cloud-native, universal package management solution, that\r\nenables your organization to create, store and share packages in any format, to any place, with total\r\nconfidence.\r\n\r\n[![PhpStorm logo.](https://resources.jetbrains.com/storage/products/company/brand/logos/PhpStorm.svg)](https://jb.gg/OpenSourceSupport)\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjanreges%2Fsiteone-crawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjanreges%2Fsiteone-crawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjanreges%2Fsiteone-crawler/lists"}