https://github.com/wcygan/crawler
web crawler
https://github.com/wcygan/crawler
crawler crawling tokio tokio-rs web-crawler
Last synced: about 2 months ago
JSON representation
web crawler
- Host: GitHub
- URL: https://github.com/wcygan/crawler
- Owner: wcygan
- Created: 2023-02-13T01:57:52.000Z (over 3 years ago)
- Default Branch: master
- Last Pushed: 2023-09-15T04:44:43.000Z (almost 3 years ago)
- Last Synced: 2025-01-12T12:20:27.048Z (over 1 year ago)
- Topics: crawler, crawling, tokio, tokio-rs, web-crawler
- Language: Rust
- Homepage:
- Size: 124 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: readme.md
Awesome Lists containing this project
README
# Crawler
A web crawler written in Rust.
This crawler creates a web graph by exploring all URLs that it finds.
## Design
The crawler is split into two parts:
1. The connection pool
2. The parser pool
The crawler will spin up as many connections & parsers as you specify.
The connection pool will handle all HTTP requests, while the parser pool will handle all HTML parsing.
Requests to the same domain are rate limited to avoid being blocked by the server.
The URL mapping is written to an index which can be written to disk during shutdown.
## Resources
- [Tokio](https://crates.io/crates/tokio) - asynchronous runtime
- [Tokio-utils](https://crates.io/crates/tokio-utils) - rate limiter, graceful shutdown
- [Reqwest](https://crates.io/crates/reqwest/) - HTTP client
- [Dashmap](https://crates.io/crates/dashmap/) - concurrent hash map