{"id":17808190,"url":"https://github.com/holmofy/reqwest-scraper","last_synced_at":"2025-03-17T15:30:36.578Z","repository":{"id":247384906,"uuid":"825705731","full_name":"holmofy/reqwest-scraper","owner":"holmofy","description":"web scraping integration with reqwest","archived":false,"fork":false,"pushed_at":"2024-10-27T14:21:19.000Z","size":138,"stargazers_count":4,"open_issues_count":0,"forks_count":3,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-28T00:51:17.739Z","etag":null,"topics":["proc-macro","proc-macro-derive","reqwest","rust","scraper"],"latest_commit_sha":null,"homepage":"https://docs.rs/reqwest-scraper/","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/holmofy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-08T10:48:28.000Z","updated_at":"2025-01-12T18:14:19.000Z","dependencies_parsed_at":"2024-08-03T13:55:06.468Z","dependency_job_id":null,"html_url":"https://github.com/holmofy/reqwest-scraper","commit_stats":null,"previous_names":["holmofy/reqwest-scraper"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/holmofy%2Freqwest-scraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/holmofy%2Freqwest-scraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/holmofy%2Freqwest-scraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/holmofy%2Freqwest-scraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/holmofy","download_url":"https://codeload.github.com/holmofy/reqwest-scraper/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243865503,"owners_count":20360452,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["proc-macro","proc-macro-derive","reqwest","rust","scraper"],"created_at":"2024-10-27T15:09:02.223Z","updated_at":"2025-03-17T15:30:36.225Z","avatar_url":"https://github.com/holmofy.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"## reqwest-scraper - Web scraping integration with reqwest\n\n[![crates.io](https://img.shields.io/crates/v/reqwest-scraper.svg)](https://crates.io/crates/reqwest-scraper)\n[![Documentation](https://docs.rs/reqwest-scraper/badge.svg)](https://docs.rs/reqwest-scraper)\n[![CI](https://github.com/holmofy/reqwest-scraper/workflows/Publish/badge.svg)](https://github.com/holmofy/reqwest-scraper/actions?query=workflow%3APublish)\n\nExtends [reqwest](https://github.com/seanmonstar/reqwest) to support multiple web scraping methods.\n\n### Features\n\n* [x] Use [JsonPath](#jsonpath) to select fields in json response\n* [x] Select elements in HTML response using [CSS selector](#css-selector)\n* [x] Evalute the value in HTML response using [xpath expression](#xpath)\n* [x] [Derive macro extract](#macros)\n\n### Start Guide\n\n* add dependency\n    ```toml\n    reqwest = { version = \"0.12\", features = [\"json\"] }\n    reqwest-scraper=\"0.3.2\"\n    ```\n* use ScraperResponse\n    ```rust\n    use reqwest_scraper::ScraperResponse;\n    ```\n\n\n\u003ch3 id=\"jsonpath\"\u003eJsonPath\u003c/h3\u003e\n\n* `Json::select\u003cT: DeserializeOwned\u003e(path: \u0026str) -\u003e Result\u003cVec\u003cT\u003e\u003e`\n* `Json::select_one\u003cT: DeserializeOwned\u003e(path: \u0026str) -\u003e Result\u003cT\u003e`\n* `Json::select_as_str(path: \u0026str) -\u003e Result\u003cString\u003e`\n\n[**example**](./examples/json.rs):\n\n```rust\nuse reqwest_scraper::ScraperResponse;\n\npub async fn request() -\u003e Result\u003c()\u003e {\n    let json = reqwest::Client::builder()\n        .build()?\n        .get(\"https://api.github.com/search/repositories?q=rust\")\n        .header(\"User-Agent\", \"Rust Reqwest\")\n        .send()\n        .await?\n        .jsonpath()\n        .await?;\n\n    let total_count = json.select_as_str(\"$.total_count\")?;\n    let names: Vec\u003cString\u003e = json.select(\"$.items[*].full_name\")?;\n\n    println!(\"{}\", total_count);\n    println!(\"{}\", names.join(\"\\t\"));\n\n    Ok(())\n}\n```\n\n\u003ch3 id=\"css-selector\"\u003eCSS selector\u003c/h3\u003e\n\n* `Html::select(selector: \u0026str) -\u003e Result\u003cSelectable\u003e`\n* `Selectable::iter() -\u003e impl Iterator\u003cSelectItem\u003e`\n* `Selectable::first() -\u003e Option\u003cSelectItem\u003e`\n* `SelectItem::name() -\u003e \u0026str`\n* `SelectItem::id() -\u003e Option\u003c\u0026str\u003e`\n* `SelectItem::has_class(class: \u0026str, case_sensitive: CaseSensitivity) -\u003e bool`\n* `SelectItem::classes() -\u003e Classes`\n* `SelectItem::attrs() -\u003e Attrs`\n* `SelectItem::attr(attr: \u0026str) -\u003e Option\u003c\u0026str\u003e`\n* `SelectItem::text() -\u003e String`\n* `SelectItem::html() -\u003e String`\n* `SelectItem::inner_html() -\u003e String`\n* `SelectItem::children() -\u003e impl Iterator\u003cSelectItem\u003e`\n* `SelectItem::find(selector: \u0026str) -\u003e Result\u003cSelectable\u003e`\n\n[**example**](./examples/html.rs):\n\n```rust\nuse reqwest_scraper::ScraperResponse;\n\nasync fn request() -\u003e Result\u003c()\u003e {\n    let html = reqwest::get(\"https://github.com/holmofy\")\n        .await?\n        .css_selector()\n        .await?;\n\n    assert_eq!(\n        html.select(\".p-name\")?.iter().nth(0).unwrap().text().trim(),\n        \"holmofy\"\n    );\n\n    let select_result = html.select(\".vcard-details \u003e li.vcard-detail\")?;\n\n    for detail_item in select_result.iter() {\n        println!(\"{}\", detail_item.attr(\"aria-label\").unwrap())\n    }\n\n    Ok(())\n}\n```\n\n\u003ch3 id=\"xpath\"\u003eXPath\u003c/h3\u003e\n\n* `XHtml::select(xpath: \u0026str) -\u003e Result\u003cXPathResult\u003e`\n* `XPathResult::as_nodes() -\u003e Vec\u003cNode\u003e`\n* `XPathResult::as_strs() -\u003e Vec\u003cString\u003e`\n* `XPathResult::as_node() -\u003e Option\u003cNode\u003e`\n* `XPathResult::as_str() -\u003e Option\u003cString\u003e`\n* `Node::name() -\u003e String`\n* `Node::id() -\u003e Option\u003cString\u003e`\n* `Node::classes() -\u003e HashSet\u003cString\u003e`\n* `Node::attr(attr: \u0026str) -\u003e Option\u003cString\u003e`\n* `Node::has_attr(attr: \u0026str) -\u003e bool`\n* `Node::text() -\u003e String`\n* TODO: `Node::html() -\u003e String`\n* TODO: `Node::inner_html() -\u003e String`\n* `Node::children() -\u003e Vec\u003cNode\u003e`\n* `Node::findnodes(relative_xpath: \u0026str) -\u003e Result\u003cVec\u003cNode\u003e\u003e`\n* `Node::findvalues(relative_xpath: \u0026str) -\u003e Result\u003cVec\u003cString\u003e\u003e`\n* `Node::findnode(relative_xpath: \u0026str) -\u003e Result\u003cOption\u003cNode\u003e\u003e`\n* `Node::findvalue(relative_xpath: \u0026str) -\u003e Result\u003cOption\u003cString\u003e\u003e`\n\n[**example**](./examples/xpath.rs):\n\n```rust\nasync fn request() -\u003e Result\u003c()\u003e {\n    let html = reqwest::get(\"https://github.com/holmofy\")\n        .await?\n        .xpath()\n        .await?;\n\n    // simple extract element\n    let name = html\n        .select(\"//span[contains(@class,'p-name')]\")?\n        .as_node()\n        .unwrap()\n        .text();\n    println!(\"{}\", name);\n    assert_eq!(name.trim(), \"holmofy\");\n\n    // iterate elements\n    let select_result = html\n        .select(\"//ul[contains(@class,'vcard-details')]/li[contains(@class,'vcard-detail')]\")?\n        .as_nodes();\n\n    println!(\"{}\", select_result.len());\n\n    for item in select_result.into_iter() {\n        let attr = item.attr(\"aria-label\").unwrap_or_else(|| \"\".into());\n        println!(\"{}\", attr);\n        println!(\"{}\", item.text());\n    }\n\n    // attribute extract\n    let select_result = html\n        .select(\"//ul[contains(@class,'vcard-details')]/li[contains(@class,'vcard-detail')]/@aria-label\")?\n        .as_strs();\n\n    println!(\"{}\", select_result.len());\n    select_result.into_iter().for_each(|s| println!(\"{}\", s));\n\n    Ok(())\n}\n```\n\n\u003ch3 id=\"macros\"\u003eDerive macro extract\u003c/h3\u003e\n\n**use `FromCssSelector` \u0026 `selector` to extract html element into struct**\n```rust\n// define struct and derive the FromCssSelector trait\n#[derive(Debug, FromCssSelector)]\n#[selector(path = \"#user-repositories-list \u003e ul \u003e li\")]\nstruct Repo {\n    #[selector(path = \"a[itemprop~='name']\", default = \"\u003cunname\u003e\", text)]\n    name: String,\n\n    #[selector(path = \"span[itemprop~='programmingLanguage']\", text)]\n    program_lang: Option\u003cString\u003e,\n\n    #[selector(path = \"div.topics-row-container\u003ea\", text)]\n    topics: Vec\u003cString\u003e,\n}\n\n// request\nlet html = reqwest::get(\"https://github.com/holmofy?tab=repositories\")\n    .await?\n    .css_selector()\n    .await?;\n\n// Use the generated `from_html` method to extract data into the struct\nlet items = Repo::from_html(html)?;\nitems.iter().for_each(|item| println!(\"{:?}\", item));\n```\n\n**use `FromXPath` \u0026 `xpath` to extract html element into struct**\n```rust\n// define struct and derive the FromXPath trait\n#[derive(Debug, FromXPath)]\n#[xpath(path = \"//div[@id='user-repositories-list']/ul/li\")]\nstruct Repo {\n    #[xpath(path = \".//a[contains(@itemprop,'name')]/text()\", default = \"\u003cunname\u003e\")]\n    name: String,\n\n    #[xpath(path = \".//span[contains(@itemprop,'programmingLanguage')]/text()\")]\n    program_lang: Option\u003cString\u003e,\n\n    #[xpath(path = \".//div[contains(@class,'topics-row-container')]/a/text()\")]\n    topics: Vec\u003cString\u003e,\n}\n\nlet html = reqwest::get(\"https://github.com/holmofy?tab=repositories\")\n    .await?\n    .xpath()\n    .await?;\n\n// Use the generated `from_xhtml` method to extract data into the struct\nlet items = Repo::from_xhtml(html)?;\nitems.iter().for_each(|item| println!(\"{:?}\", item));\n```\n\n\n## Related Projects\n\n* [reqwest](https://github.com/seanmonstar/reqwest)\n* [scraper](https://github.com/causal-agent/scraper)\n* [nipper](https://github.com/importcjj/nipper)\n* [jsonpath_lib](https://github.com/freestrings/jsonpath)\n* [unhtml.rs](https://github.com/Hexilee/unhtml.rs)\n* [xpath-scraper](https://github.com/Its-its/xpath-scraper)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fholmofy%2Freqwest-scraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fholmofy%2Freqwest-scraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fholmofy%2Freqwest-scraper/lists"}