https://github.com/opensauce04/linkshovel

A simple webcrawler written in ruby
https://github.com/opensauce04/linkshovel

nokogiri ruby web-crawler webcrawler

Last synced: 3 months ago
JSON representation

A simple webcrawler written in ruby

• Host: GitHub
• URL: https://github.com/opensauce04/linkshovel
• Owner: OpenSauce04
• License: unlicense
• Created: 2019-10-23T12:45:30.000Z (over 5 years ago)
• Default Branch: master
• Last Pushed: 2023-04-12T06:09:41.000Z (about 2 years ago)
• Last Synced: 2025-01-01T22:33:39.149Z (5 months ago)
• Topics: nokogiri, ruby, web-crawler, webcrawler
• Language: Ruby
• Homepage:
• Size: 36.1 KB
• Stars: 1
• Watchers: 1
• Forks: 0
• Open Issues: 3
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# LinkShovel

A simple webcrawler that will scrape every URL linked to on a given site, and will put it into a `links.txt` file. It will then randomly pick a link from links.txt and will search that page for links, and will then add _those_ to links.txt. This will be repeated indefinitely until the program is turned off