https://github.com/vindecodex/automated-crawler-wget

Using wget to crawl site
https://github.com/vindecodex/automated-crawler-wget

crawler shell-script

Last synced: 23 days ago
JSON representation

Using wget to crawl site

• Host: GitHub
• URL: https://github.com/vindecodex/automated-crawler-wget
• Owner: vindecodex
• License: mit
• Created: 2019-12-04T09:02:59.000Z (about 5 years ago)
• Default Branch: master
• Last Pushed: 2020-08-12T02:29:54.000Z (over 4 years ago)
• Last Synced: 2024-12-30T07:52:21.549Z (25 days ago)
• Topics: crawler, shell-script
• Language: Shell
• Homepage:
• Size: 4.88 KB
• Stars: 1
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: readme.md
  • License: LICENSE

Awesome Lists containing this project

README

        
##### Simple script for crawling(Download)

It will download specific html file of the specified url with page number (automated). Added sleep to avoid blocked by the website you are trying to crawl.