https://github.com/willf/lychee-sitemap
Create a sitemap that lychee can use (or any website really)
https://github.com/willf/lychee-sitemap
Last synced: about 2 months ago
JSON representation
Create a sitemap that lychee can use (or any website really)
- Host: GitHub
- URL: https://github.com/willf/lychee-sitemap
- Owner: willf
- Created: 2026-06-22T17:15:31.000Z (about 2 months ago)
- Default Branch: main
- Last Pushed: 2026-06-22T17:27:49.000Z (about 2 months ago)
- Last Synced: 2026-06-22T19:18:52.759Z (about 2 months ago)
- Language: Rust
- Size: 23.4 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# lychee-sitemap
[](https://github.com/willf/lychee-sitemap/actions/workflows/ci.yml)
[](https://github.com/willf/lychee-sitemap/actions/workflows/release.yml)
[](https://github.com/willf/lychee-sitemap/releases)
[](https://www.rust-lang.org/)
A Rust CLI that crawls a website recursively and emits a sitemap XML document.
## Features
- Recursive crawling from a starting URL
- Same-site restriction (won't crawl external domains)
- URL deduplication (including fragment removal like `#section`)
- Configurable crawl limits (`max-pages`, `max-depth`)
- Bounded concurrent crawling (`--concurrency`)
- Structured JSON logs to stderr with timestamps
- Verbosity controls (`-v`, `-vv`, `-vvv` or repeated `--verbose`)
- Output to stdout by default
- Optional file output (`--output` or `--write-file` for `sitemap.xml`)
## Requirements
- Rust toolchain (stable)
## Build
```bash
cargo build --release
```
## Usage
Print sitemap to stdout:
```bash
cargo run -- https://example.com
```
Write sitemap to `sitemap.xml`:
```bash
cargo run -- https://example.com --write-file
```
Write sitemap to a custom file:
```bash
cargo run -- https://example.com --output my-sitemap.xml
```
Tune crawl limits:
```bash
cargo run -- https://example.com --max-pages 1000 --max-depth 10
```
Tune concurrency:
```bash
cargo run -- https://example.com --concurrency 32
```
Enable logs:
```bash
# info-level logs
cargo run -- https://example.com -v
# debug-level logs
cargo run -- https://example.com -vv
# trace-level logs
cargo run -- https://example.com -vvv
```
With logging enabled, sitemap XML still goes to stdout while structured logs go to stderr.
That means shell redirection like `> sitemap.xml` captures XML only.
## Example output
```xml
https://example.com/
```
## Test
```bash
cargo test
```
The integration test spins up a local HTTP server and verifies recursive crawling, deduplication, and same-site filtering behavior.
## GitHub Actions
This repository includes two workflows:
- CI: [.github/workflows/ci.yml](.github/workflows/ci.yml)
- Runs on pushes to `main` and pull requests
- Checks formatting (`cargo fmt --check`)
- Runs clippy with warnings denied
- Runs tests on Linux, macOS, and Windows
- Ensures release builds compile on Linux, macOS, and Windows
- Release: [.github/workflows/release.yml](.github/workflows/release.yml)
- Runs when you push a tag matching `v*` (for example `v0.1.0`)
- Builds release binaries on Linux, macOS, and Windows
- Packages artifacts (`.tar.gz` for Unix, `.zip` for Windows)
- Publishes a GitHub Release with generated notes and attached binaries
Create a release by tagging and pushing:
```bash
git tag v0.1.0
git push origin v0.1.0
```