https://github.com/messense/robotparser-rs

robots.txt parser for Rust.
https://github.com/messense/robotparser-rs

http robots-parser rust

Last synced: about 2 months ago
JSON representation

robots.txt parser for Rust.

• Host: GitHub
• URL: https://github.com/messense/robotparser-rs
• Owner: messense
• License: mit
• Created: 2015-06-27T17:28:41.000Z (almost 10 years ago)
• Default Branch: master
• Last Pushed: 2022-03-30T02:24:19.000Z (about 3 years ago)
• Last Synced: 2025-03-28T04:46:26.280Z (2 months ago)
• Topics: http, robots-parser, rust
• Language: Rust
• Homepage: https://crates.io/crates/robotparser
• Size: 19.6 MB
• Stars: 18
• Watchers: 4
• Forks: 9
• Open Issues: 2
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# robotparser-rs

[![GitHub Actions](https://github.com/messense/robotparser-rs/workflows/CI/badge.svg)](https://github.com/messense/robotparser-rs/actions?query=workflow%3ACI)

[![Coverage Status](https://coveralls.io/repos/messense/robotparser-rs/badge.svg)](https://coveralls.io/r/messense/robotparser-rs)

[![codecov](https://codecov.io/gh/messense/robotparser-rs/branch/master/graph/badge.svg)](https://codecov.io/gh/messense/robotparser-rs)

[![Crates.io](https://img.shields.io/crates/v/robotparser.svg)](https://crates.io/crates/robotparser)

[![Dependency status](https://deps.rs/repo/github/messense/robotparser-rs/status.svg)](https://deps.rs/repo/github/messense/robotparser-rs)

robots.txt parser for Rust.

## Installation

Add it to your ``Cargo.toml``:

```toml

[dependencies]

robotparser = "0.11"

```

## Examples

```rust

use robotparser::http::RobotsTxtClient;

use robotparser::service::RobotsTxtService;

use reqwest::Client;

use url::Url;

fn main() {

    let client = Client::new();

    let robots_txt_url = Url::parse("https://www.python.org/robots.txt").unwrap();

    let robots_txt = client.fetch_robots_txt(robots_txt_url.origin()).unwrap().get_result();

    let fetch_url = Url::parse("https://www.python.org/robots.txt").unwrap();

    assert!(robots_txt.can_fetch("*", &fetch_url));

}

```

## License

This work is released under the MIT license. A copy of the license is provided in the [LICENSE](./LICENSE) file.