https://github.com/tawilkinson/rust_web_crawler

A simple web crawler in rust
https://github.com/tawilkinson/rust_web_crawler

Last synced: about 1 month ago
JSON representation

A simple web crawler in rust

• Host: GitHub
• URL: https://github.com/tawilkinson/rust_web_crawler
• Owner: tawilkinson
• Created: 2019-07-27T08:58:39.000Z (over 5 years ago)
• Default Branch: master
• Last Pushed: 2023-06-21T22:55:01.000Z (over 1 year ago)
• Last Synced: 2024-09-24T16:10:03.826Z (5 months ago)
• Language: Rust
• Size: 168 KB
• Stars: 1
• Watchers: 2
• Forks: 0
• Open Issues: 1
• Metadata Files:
  • Readme: readme.md

Awesome Lists containing this project

• awesome-rust-list - tawilkinson/rust_web_crawler
• awesome-rust-list - tawilkinson/rust_web_crawler

README

        
# Rust Web Crawler

A very simple web crawler in [rust](https://www.rust-lang.org/).

## Usage

`cargo run `

The crawler will then check the `robots.txt` and crawl the site if allowed.