{"id":18821690,"url":"https://github.com/goto-eof/octoscraper","last_synced_at":"2026-06-23T23:31:49.673Z","repository":{"id":147782566,"uuid":"618168200","full_name":"goto-eof/octoscraper","owner":"goto-eof","description":"A web scraper tool implemented in Rust.","archived":false,"fork":false,"pushed_at":"2023-09-02T01:01:15.000Z","size":341,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-05-22T10:13:36.641Z","etag":null,"topics":["imagedownloader","scraper","webscraper"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/goto-eof.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-03-23T22:25:58.000Z","updated_at":"2023-04-19T17:36:47.000Z","dependencies_parsed_at":null,"dependency_job_id":"d6e7bc9f-04cf-4f66-acc9-173f2feebf30","html_url":"https://github.com/goto-eof/octoscraper","commit_stats":null,"previous_names":[],"tags_count":19,"template":false,"template_full_name":null,"purl":"pkg:github/goto-eof/octoscraper","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/goto-eof%2Foctoscraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/goto-eof%2Foctoscraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/goto-eof%2Foctoscraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/goto-eof%2Foctoscraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/goto-eof","download_url":"https://codeload.github.com/goto-eof/octoscraper/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/goto-eof%2Foctoscraper/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34711176,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-23T02:00:07.161Z","response_time":65,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["imagedownloader","scraper","webscraper"],"created_at":"2024-11-08T00:45:16.974Z","updated_at":"2026-06-23T23:31:49.668Z","avatar_url":"https://github.com/goto-eof.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"```\n        ___     _        __                                \n        /___\\___| |_ ___ / _\\ ___ _ __ __ _ _ __   ___ _ __ \n       //  // __| __/ _ \\\\ \\ / __| '__/ _` | '_ \\ / _ \\ '__|\n      / \\_// (__| || (_) |\\ \\ (__| | | (_| | |_) |  __/ |   \n      \\___/ \\___|\\__\\___/\\__/\\___|_|  \\__,_| .__/ \\___|_|   \n                                           |_|              \n```    \n### Description\nOctoScraper is a multithread web scraper tool implemented in Rust.\n\n### Execute it\nDownload the executable from [here](https://github.com/goto-eof/octoscraper/releases) and run it.\n```\n./octoscraper -w http://dodu.it -e .png,.PNG -d DIRECTORY_NAME -s 100 -t 90000 -i true -l 3 -a OctoScraper\n```\n\n### Examples\nDownload midi and mp3 files, no same domain, scan all website\n```\n./octoscraper -w http://audiomidimania.com  -oa true -sd false -r false\n```\nDownload midi and mp3 files, same domain, only from the page passed as parameter \n```\n./octoscraper -w http://ininternet.org/midi_file.htm -oa true -sd true -r true\n```\nDownload image files, no same domain\n```\n./octoscraper -w https://wallpaper.mob.org/ -oi true -sd false -si 1000000\n```\nDownload video, process only the page passed as parameter\n```\n./octoscraper -w http://www.w3schools.com/html/html5_video.asp -ov true -r true\n```\n\nwhere \n\n | argument | meaning                                                            | value example                  |\n -----------|--------------------------------------------------------------------|--------------------------|\n | -h       |Help                                                                |\n | -w       |website - with http/https                     |  http://dodu.it or http://audiomidimania.com or http://wallpaper.mob.org or http://www.w3schools.com/html/html5_video.asp|\n | -sd      |same domain                                                         |  true|\n |   -oi    | enable image extractor                                            | true |\n |   -ov    | enable video extractor                                            | true |\n |   -oa    |  enable audio extractor                                            | true |\n |   -oo    |  enable other file extractor                                            | true |\n |  -si     |   minimum image file size (in bytes)                                    |1000000                   |\n |  -sv     |   minimum video file size (in bytes)                                    |1000000                   |\n |  -sa     |   minimum audio file size (in bytes)                                    |1000000                   |\n |  -so     |   minimum other file size (in bytes)                                    |1000000                   |\n |   -ei\t|list of image extensions separated by comma                        | .jpg,.JPG,.png,.PNG |\n |   -ev\t| list of video extensions separated by comma                       | .ogg,.OGG,.MP4,.mp4 |\n |   -ea\t| list of audio extensions separated by comma                      | .mp3,.MP3,.midi,.MIDI |\n |   -eo\t| list of other file extensions separated by comma                      | .zip,.ZIP,.exe,.EXE,.pdf,.PDF |\n | -d       |directory where files will be saved                                 | Images|\n | -s       |sleep time in millis before making the request                      | 1000|\n | -t       |download timeout                                                    | 90000|\n | -i       |insistent mode (it retries until download succeed)                  | true|\n | -l       |download limit (by default it makes as much requests as possibile)  | 3|\n | -a       |user agent                                                          | OctoScraper|\n | -c       |enables downloaded file hash check for avoiding duplicate downloads | true|\n |-r        | process only the root link (process only one page)                                         | false|\n | -u       | consider unique resources by filename (1) or by link (2). Allowed values: 1 or 2| 1 |\n\n\n### For developers\n\nAllow reqwest crate to work properly:\n```\nsudo apt install libssl-dev\n```\nRun application with your configuration:\n```\ncargo run -- -w http://dodu.it -e .png,.PNG -d DIRECTORY_NAME -s 100 -t 90000 -i true -l 3 -a OctoScraper\n```\n\n### Tests\n```\ncargo test\n```\n\n### Screenshot\n![image](https://user-images.githubusercontent.com/6343630/229301705-de5ac86a-d44e-4e8b-99eb-99e30bc17296.png)\n\n\n\n\n\nTested on Linux, MacOS and Windows.\n\nif any problems arise, feel free to [contact me](http://andre-i.eu/#contactme).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoto-eof%2Foctoscraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgoto-eof%2Foctoscraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoto-eof%2Foctoscraper/lists"}