https://github.com/opensauce04/linkshovel

A simple webcrawler written in ruby
https://github.com/opensauce04/linkshovel

nokogiri ruby web-crawler webcrawler

Last synced: 20 days ago
JSON representation

A simple webcrawler written in ruby

• Host: GitHub
• URL: https://github.com/opensauce04/linkshovel
• Owner: OpenSauce04
• License: unlicense
• Created: 2019-10-23T12:45:30.000Z (about 5 years ago)
• Default Branch: master
• Last Pushed: 2022-10-21T03:34:15.000Z (over 2 years ago)
• Last Synced: 2023-03-05T21:01:40.324Z (almost 2 years ago)
• Topics: nokogiri, ruby, web-crawler, webcrawler
• Language: Ruby
• Homepage:
• Size: 33.2 KB
• Stars: 1
• Watchers: 1
• Forks: 0
• Open Issues: 1
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# LinkShovel

A simple webcrawler that will scrape every URL linked to on a given site, and will put it into a `links.txt` file. It will then randomly pick a link from links.txt and will search that page for links, and will then add _those_ to links.txt. This will be repeated indefinitely until the program is turned off