Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/s0rg/crawley
The unix-way web crawler
https://github.com/s0rg/crawley
cli crawler go golang golang-application pentest pentest-tool pentesting unix-way web-crawler web-scraping web-spider
Last synced: 6 days ago
JSON representation
The unix-way web crawler
- Host: GitHub
- URL: https://github.com/s0rg/crawley
- Owner: s0rg
- License: mit
- Created: 2021-10-27T18:48:51.000Z (about 3 years ago)
- Default Branch: master
- Last Pushed: 2024-09-08T22:53:31.000Z (2 months ago)
- Last Synced: 2024-09-08T23:46:36.172Z (2 months ago)
- Topics: cli, crawler, go, golang, golang-application, pentest, pentest-tool, pentesting, unix-way, web-crawler, web-scraping, web-spider
- Language: Go
- Homepage:
- Size: 203 KB
- Stars: 250
- Watchers: 2
- Forks: 14
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Security: SECURITY.md
Awesome Lists containing this project
- awesome-cli-apps - crawley - Unix-way web crawler. (Utilities / Calendars)
- awesome-bugbounty-tools - crawley - fast, feature-rich unix-way web scraper/crawler written in Golang. (Recon / Content Discovery)
- fucking-awesome-cli-apps - crawley - Unix-way web crawler. (Utilities / Calendars)
- awesome-go-extra - crawley - way web crawler|64|2|0|2021-10-27T18:48:51Z|2022-08-04T20:23:59Z| (Go Tools / Other Software)
README
[](https://github.com/s0rg/crawley/blob/main/LICENSE)
[](https://app.fossa.com/projects/git%2Bgithub.com%2Fs0rg%2Fcrawley?ref=badge_shield)
[](go.mod)
[](https://github.com/s0rg/crawley/releases/latest)
[](https://github.com/avelino/awesome-go)
[](https://github.com/s0rg/crawley/actions?query=workflow%3Aci)
[](https://goreportcard.com/report/github.com/s0rg/crawley)
[](https://codeclimate.com/github/s0rg/crawley/maintainability)
[](https://codeclimate.com/github/s0rg/crawley/test_coverage)
[](https://libraries.io/github/s0rg/crawley)
# crawley
Crawls web pages and prints any link it can find.
# features
- fast html SAX-parser (powered by [x/net/html](https://golang.org/x/net/html))
- js/css lexical parsers (powered by [tdewolff/parse](https://github.com/tdewolff/parse)) - extract api endpoints from js code and `url()` properties
- small (below 1500 SLOC), idiomatic, 100% test covered codebase
- grabs most of useful resources urls (pics, videos, audios, forms, etc...)
- found urls are streamed to stdout and guranteed to be unique (with fragments omitted)
- scan depth (limited by starting host and path, by default - 0) can be configured
- can be polite - crawl rules and sitemaps from `robots.txt`
- `brute` mode - scan html comments for urls (this can lead to bogus results)
- make use of `HTTP_PROXY` / `HTTPS_PROXY` environment values + handles proxy auth (use `HTTP_PROXY="socks5://127.0.0.1:1080/" crawley` for socks5)
- directory-only scan mode (aka `fast-scan`)
- user-defined cookies, in curl-compatible format (i.e. `-cookie "ONE=1; TWO=2" -cookie "ITS=ME" -cookie @cookie-file`)
- user-defined headers, same as curl: `-header "ONE: 1" -header "TWO: 2" -header @headers-file`
- tag filter - allow to specify tags to crawl for (single: `-tag a -tag form`, multiple: `-tag a,form`, or mixed)
- url ignore - allow to ignore urls with matched substrings from crawling (i.e.: `-ignore logout`)
- subdomains support - allow depth crawling for subdomains as well (e.g. `crawley http://some-test.site` will be able to crawl `http://www.some-test.site`)
# examples
```sh
# print all links from first page:
crawley http://some-test.site
# print all js files and api endpoints:
crawley -depth -1 -tag script -js http://some-test.site
# print all endpoints from js:
crawley -js http://some-test.site/app.js
# download all png images from site:
crawley -depth -1 -tag img http://some-test.site | grep '\.png$' | wget -i -
# fast directory traversal:
crawley -headless -delay 0 -depth -1 -dirs only http://some-test.site
```
# installation
- [binaries / deb / rpm](https://github.com/s0rg/crawley/releases) for Linux, FreeBSD, macOS and Windows.
- [archlinux](https://aur.archlinux.org/packages/crawley-bin/) you can use your favourite AUR helper to install it, e. g. `paru -S crawley-bin`.
# usage
```
crawley [flags] url
possible flags with default values:
-all
scan all known sources (js/css/...)
-brute
scan html comments
-cookie value
extra cookies for request, can be used multiple times, accept files with '@'-prefix
-css
scan css for urls
-delay duration
per-request delay (0 - disable) (default 150ms)
-depth int
scan depth (set -1 for unlimited)
-dirs string
policy for non-resource urls: show / hide / only (default "show")
-header value
extra headers for request, can be used multiple times, accept files with '@'-prefix
-headless
disable pre-flight HEAD requests
-ignore value
patterns (in urls) to be ignored in crawl process
-js
scan js code for endpoints
-proxy-auth string
credentials for proxy: user:password
-robots string
policy for robots.txt: ignore / crawl / respect (default "ignore")
-silent
suppress info and error messages in stderr
-skip-ssl
skip ssl verification
-subdomains
support subdomains (e.g. if www.domain.com found, recurse over it)
-tag value
tags filter, single or comma-separated tag names
-timeout duration
request timeout (min: 1 second, max: 10 minutes) (default 5s)
-user-agent string
user-agent string
-version
show version
-workers int
number of workers (default - number of CPU cores)
```
# flags autocompletion
Crawley can handle flags autocompletion in bash and zsh via `complete`:
```bash
complete -C "/full-path-to/bin/crawley" crawley
```
# license
[](https://app.fossa.com/projects/git%2Bgithub.com%2Fs0rg%2Fcrawley?ref=badge_large)