https://github.com/goto-eof/octoscraper
A web scraper tool implemented in Rust.
https://github.com/goto-eof/octoscraper
imagedownloader scraper webscraper
Last synced: about 1 month ago
JSON representation
A web scraper tool implemented in Rust.
- Host: GitHub
- URL: https://github.com/goto-eof/octoscraper
- Owner: goto-eof
- License: gpl-3.0
- Created: 2023-03-23T22:25:58.000Z (over 3 years ago)
- Default Branch: master
- Last Pushed: 2023-09-02T01:01:15.000Z (almost 3 years ago)
- Last Synced: 2025-05-22T10:13:36.641Z (about 1 year ago)
- Topics: imagedownloader, scraper, webscraper
- Language: Rust
- Homepage:
- Size: 333 KB
- Stars: 1
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
```
___ _ __
/___\___| |_ ___ / _\ ___ _ __ __ _ _ __ ___ _ __
// // __| __/ _ \\ \ / __| '__/ _` | '_ \ / _ \ '__|
/ \_// (__| || (_) |\ \ (__| | | (_| | |_) | __/ |
\___/ \___|\__\___/\__/\___|_| \__,_| .__/ \___|_|
|_|
```
### Description
OctoScraper is a multithread web scraper tool implemented in Rust.
### Execute it
Download the executable from [here](https://github.com/goto-eof/octoscraper/releases) and run it.
```
./octoscraper -w http://dodu.it -e .png,.PNG -d DIRECTORY_NAME -s 100 -t 90000 -i true -l 3 -a OctoScraper
```
### Examples
Download midi and mp3 files, no same domain, scan all website
```
./octoscraper -w http://audiomidimania.com -oa true -sd false -r false
```
Download midi and mp3 files, same domain, only from the page passed as parameter
```
./octoscraper -w http://ininternet.org/midi_file.htm -oa true -sd true -r true
```
Download image files, no same domain
```
./octoscraper -w https://wallpaper.mob.org/ -oi true -sd false -si 1000000
```
Download video, process only the page passed as parameter
```
./octoscraper -w http://www.w3schools.com/html/html5_video.asp -ov true -r true
```
where
| argument | meaning | value example |
-----------|--------------------------------------------------------------------|--------------------------|
| -h |Help |
| -w |website - with http/https | http://dodu.it or http://audiomidimania.com or http://wallpaper.mob.org or http://www.w3schools.com/html/html5_video.asp|
| -sd |same domain | true|
| -oi | enable image extractor | true |
| -ov | enable video extractor | true |
| -oa | enable audio extractor | true |
| -oo | enable other file extractor | true |
| -si | minimum image file size (in bytes) |1000000 |
| -sv | minimum video file size (in bytes) |1000000 |
| -sa | minimum audio file size (in bytes) |1000000 |
| -so | minimum other file size (in bytes) |1000000 |
| -ei |list of image extensions separated by comma | .jpg,.JPG,.png,.PNG |
| -ev | list of video extensions separated by comma | .ogg,.OGG,.MP4,.mp4 |
| -ea | list of audio extensions separated by comma | .mp3,.MP3,.midi,.MIDI |
| -eo | list of other file extensions separated by comma | .zip,.ZIP,.exe,.EXE,.pdf,.PDF |
| -d |directory where files will be saved | Images|
| -s |sleep time in millis before making the request | 1000|
| -t |download timeout | 90000|
| -i |insistent mode (it retries until download succeed) | true|
| -l |download limit (by default it makes as much requests as possibile) | 3|
| -a |user agent | OctoScraper|
| -c |enables downloaded file hash check for avoiding duplicate downloads | true|
|-r | process only the root link (process only one page) | false|
| -u | consider unique resources by filename (1) or by link (2). Allowed values: 1 or 2| 1 |
### For developers
Allow reqwest crate to work properly:
```
sudo apt install libssl-dev
```
Run application with your configuration:
```
cargo run -- -w http://dodu.it -e .png,.PNG -d DIRECTORY_NAME -s 100 -t 90000 -i true -l 3 -a OctoScraper
```
### Tests
```
cargo test
```
### Screenshot
Tested on Linux, MacOS and Windows.
if any problems arise, feel free to [contact me](http://andre-i.eu/#contactme).