https://github.com/kvpmx/web-crawler

A simple web crawler written in Ruby
https://github.com/kvpmx/web-crawler

Last synced: 10 months ago
JSON representation

A simple web crawler written in Ruby

• Host: GitHub
• URL: https://github.com/kvpmx/web-crawler
• Owner: kvpmx
• Created: 2025-09-21T13:57:34.000Z (10 months ago)
• Default Branch: main
• Last Pushed: 2025-09-21T14:41:13.000Z (10 months ago)
• Last Synced: 2025-09-21T16:28:38.059Z (10 months ago)
• Language: Ruby
• Size: 3.91 KB
• Stars: 0
• Watchers: 0
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

          
# 🕷️ Web Crawler

The project focuses on crawling the website [Books to Scrape](http://books.toscrape.com/),

a sandbox site created specifically for practicing web scraping.

This makes it a safe and legal choice for experimentation.

The purpose of the crawler is to collect structured data about books

and store it in a convenient format for later analysis.

The crawler begins at the homepage and retrieves all the data about available books.

From every catalogue page, links to individual book detail pages are collected.

On each detail page, the crawler extracts comprehensive information

such as the book title, description, price, availability, rating etc. 

Additionally, the URL of the book’s page and the cover image link are gathered.

An optional step is downloading the images themselves into a local folder.

All collected data is stored in a CSV file. Each row corresponds to a single book.

This ensures that the dataset is complete and ready for analysis or integration into other tools.