https://github.com/jethronap/rust-webscraper
A rudimentary web scraper written in Rust
https://github.com/jethronap/rust-webscraper
cli rust-lang webscraper
Last synced: 8 months ago
JSON representation
A rudimentary web scraper written in Rust
- Host: GitHub
- URL: https://github.com/jethronap/rust-webscraper
- Owner: jethronap
- License: mit
- Created: 2024-12-27T15:37:58.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2025-01-08T21:57:08.000Z (over 1 year ago)
- Last Synced: 2025-01-08T22:36:22.004Z (over 1 year ago)
- Topics: cli, rust-lang, webscraper
- Language: Rust
- Homepage:
- Size: 29.3 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Rust Webscraper
This is a simple webscraper written in Rust. It allows you to fetch and parse HTML content from web pages.
## Features
- Fetch HTML content from a given URL
- Parse and extract data from HTML
## Requirements
- Rust (latest stable version)
## Installation
1. Clone the repository:
```sh
git clone https://github.com/yourusername/rust-webscraper.git
```
2. Navigate to the project directory:
```sh
cd rust-webscraper
```
3. Build the project:
```sh
cargo build
```
## Usage
Run the webscraper with a target URL a designated timeout and css selector:
```sh
cargo run -- --url https://example.com --timeout 15 --selector a
```
Run the websraper without cli arguments:
```sh
cargo run
```
The scraper will use the default values provided in the [configuration file](/config.toml).
Currently, the scraper saves extracted data to a `.json` file inside a `backup` folder at the root of the project.
Below is a excerpt of the output when run without any args.
```json
[
...
{
"tag": "a",
"content": "\n Read Contribution Guide\n ",
"attributes": {
"class": "button button-secondary",
"href": "https://rustc-dev-guide.rust-lang.org/getting-started.html"
}
},
{
"tag": "a",
"content": "See individual contributors",
"attributes": {
"class": "button button-secondary",
"href": "https://thanks.rust-lang.org/"
}
},
{
"tag": "a",
"content": "See Foundation members",
"attributes": {
"class": "button button-secondary",
"href": "https://foundation.rust-lang.org/members"
}
},
{
"tag": "a",
"content": "Documentation",
"attributes": {
"href": "/learn"
}
},
{
"tag": "a",
"content": "Rust Forge (Contributor Documentation)",
"attributes": {
"href": "http://forge.rust-lang.org"
}
},
{
"tag": "a",
"content": "Ask a Question on the Users Forum",
"attributes": {
"href": "https://users.rust-lang.org"
}
},
{
"tag": "a",
"content": "Code of Conduct",
"attributes": {
"href": "/policies/code-of-conduct"
}
},
{
"tag": "a",
"content": "Licenses",
"attributes": {
"href": "/policies/licenses"
}
},
{
"tag": "a",
"content": "Logo Policy and Media Guide",
"attributes": {
"href": "https://foundation.rust-lang.org/policies/logo-policy-and-media-guide/"
}
},
...
]
```
## Testing
Use the following command to run the unit tests
```sh
cargo test
```
## License
This project is licensed under the MIT License.