https://github.com/alphasights/crawl

Crawl pages within a domain, reporting any page that returns a bad response code
https://github.com/alphasights/crawl

Last synced: 5 days ago
JSON representation

Crawl pages within a domain, reporting any page that returns a bad response code

• Host: GitHub
• URL: https://github.com/alphasights/crawl
• Owner: alphasights
• License: mit
• Created: 2011-11-04T14:06:36.000Z (about 13 years ago)
• Default Branch: master
• Last Pushed: 2018-03-26T16:13:26.000Z (over 6 years ago)
• Last Synced: 2024-04-25T15:22:26.764Z (7 months ago)
• Language: Ruby
• Homepage:
• Size: 50.8 KB
• Stars: 5
• Watchers: 55
• Forks: 6
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE.txt

Awesome Lists containing this project

README

        
# Crawl

[![Circle CI](https://circleci.com/gh/alphasights/crawl/tree/master.svg?style=shield&circle-token=188c328ee0669c8eba8dbda7f7f9943445a50c70)](https://circleci.com/gh/alphasights/crawl/tree/master)

Crawl pages within a domain, reporting any page that returns a bad response code

Usage:

    > crawl [options] domain

    Usage: crawl [options] domain

    -s, --start /home,/about         Starting path(s), defaults to /

    -u, --username username          Basic auth username

    -p, --password password          Basic auth password

    -c, --connections count          Max mumber of parallel connections to use. The default is 5.

    -v, --verbose                    Give details when crawling

    -h, --help                       Show this message

        --version                    Print version

Example:

    > crawl https://engineering.alphasights.com --connections=5 --start=/ --verbose

      Adding /

    Fetching / ...

      Adding /positions/ruby-developer

      Adding /positions/js-ember-developer

      Adding /positions/ux-ui-designer

      Adding /positions/support-specialist

    Fetching /positions/ruby-developer

    Fetching /positions/js-ember-developer ...

    Fetching /positions/ux-ui-designer ...

    Fetching /positions/support-specialist ...

    5 pages crawled without errors.

### Copyright and License

Copyright AlphaSights and Contributors, 2015

[MIT Licence](LICENSE.txt)