https://github.com/ravenxd0/crawler

Web Crawler in Rust
https://github.com/ravenxd0/crawler

Last synced: about 1 month ago
JSON representation

Web Crawler in Rust

• Host: GitHub
• URL: https://github.com/ravenxd0/crawler
• Owner: ravenxd0
• Created: 2023-02-16T13:54:02.000Z (about 2 years ago)
• Default Branch: main
• Last Pushed: 2023-02-16T19:23:09.000Z (about 2 years ago)
• Last Synced: 2024-09-24T16:10:04.050Z (5 months ago)
• Language: Rust
• Size: 2.93 KB
• Stars: 1
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

• awesome-rust-list - ravenxd0/crawler
• awesome-rust-list - ravenxd0/crawler

README

        
# crawler

It is a Simple Web Crawler Implementation.

A web crawler, also known as a spider, is an automated program or script that systematically browses the internet and indexes or downloads content from web pages.

Dependencies:

```toml

[dependencies]

rayon = "1.6.1"

reqwest = { version =  "0.11.14" , features = ["json","blocking"] }

select = "0.6.0"

```