{"id":13464685,"url":"https://github.com/miyagawa/web-scraper","last_synced_at":"2025-07-02T15:36:08.191Z","repository":{"id":479894,"uuid":"105434","full_name":"miyagawa/web-scraper","owner":"miyagawa","description":"Perl web scraping toolkit","archived":false,"fork":false,"pushed_at":"2017-04-26T01:43:06.000Z","size":312,"stargazers_count":102,"open_issues_count":9,"forks_count":31,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-04-01T22:03:16.418Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://search.cpan.org/dist/Web-Scraper","language":"Perl","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/miyagawa.png","metadata":{"files":{"readme":"README.md","changelog":"Changes","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2009-01-11T21:05:25.000Z","updated_at":"2025-03-01T16:39:51.000Z","dependencies_parsed_at":"2022-07-04T19:00:56.204Z","dependency_job_id":null,"html_url":"https://github.com/miyagawa/web-scraper","commit_stats":null,"previous_names":[],"tags_count":14,"template":false,"template_full_name":null,"purl":"pkg:github/miyagawa/web-scraper","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/miyagawa%2Fweb-scraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/miyagawa%2Fweb-scraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/miyagawa%2Fweb-scraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/miyagawa%2Fweb-scraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/miyagawa","download_url":"https://codeload.github.com/miyagawa/web-scraper/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/miyagawa%2Fweb-scraper/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263166714,"owners_count":23424224,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T14:00:48.637Z","updated_at":"2025-07-02T15:36:08.083Z","avatar_url":"https://github.com/miyagawa.png","language":"Perl","funding_links":[],"categories":["All","Perl"],"sub_categories":[],"readme":"# NAME\n\nWeb::Scraper - Web Scraping Toolkit using HTML and CSS Selectors or XPath expressions\n\n# SYNOPSIS\n\n    use URI;\n    use Web::Scraper;\n    use Encode;\n\n    # First, create your scraper block\n    my $authors = scraper {\n        # Parse all TDs inside 'table[width=\"100%]\"', store them into\n        # an array 'authors'.  We embed other scrapers for each TD.\n        process 'table[width=\"100%\"] td', \"authors[]\" =\u003e scraper {\n          # And, in each TD,\n          # get the URI of \"a\" element\n          process \"a\", uri =\u003e '@href';\n          # get text inside \"small\" element\n          process \"small\", fullname =\u003e 'TEXT';\n        };\n    };\n\n    my $res = $authors-\u003escrape( URI-\u003enew(\"http://search.cpan.org/author/?A\") );\n\n    # iterate the array 'authors'\n    for my $author (@{$res-\u003e{authors}}) {\n        # output is like:\n        # Andy Adler      http://search.cpan.org/~aadler/\n        # Aaron K Dancygier       http://search.cpan.org/~aakd/\n        # Aamer Akhter    http://search.cpan.org/~aakhter/\n        print Encode::encode(\"utf8\", \"$author-\u003e{fullname}\\t$author-\u003e{uri}\\n\");\n    }\n\nThe structure would resemble this (visually)\n  {\n    authors =\u003e \\[\n      { fullname =\u003e $fullname, link =\u003e $uri },\n      { fullname =\u003e $fullname, link =\u003e $uri },\n    \\]\n  }\n\n# DESCRIPTION\n\nWeb::Scraper is a web scraper toolkit, inspired by Ruby's equivalent\nScrapi. It provides a DSL-ish interface for traversing HTML documents and\nreturning a neatly arranged Perl data structure.\n\nThe _scraper_ and _process_ blocks provide a method to define what segments\nof a document to extract.  It understands HTML and CSS Selectors as well as\nXPath expressions.\n\n# METHODS\n\n## scraper\n\n    $scraper = scraper { ... };\n\nCreates a new Web::Scraper object by wrapping the DSL code that will be fired when _scrape_ method is called.\n\n## scrape\n\n    $res = $scraper-\u003escrape(URI-\u003enew($uri));\n    $res = $scraper-\u003escrape($html_content);\n    $res = $scraper-\u003escrape(\\$html_content);\n    $res = $scraper-\u003escrape($http_response);\n    $res = $scraper-\u003escrape($html_element);\n\nRetrieves the HTML from URI, HTTP::Response, HTML::Tree or text\nstrings and creates a DOM object, then fires the callback scraper code\nto retrieve the data structure.\n\nIf you pass URI or HTTP::Response object, Web::Scraper will\nautomatically guesses the encoding of the content by looking at\nContent-Type headers and META tags. Otherwise you need to decode the\nHTML to Unicode before passing it to _scrape_ method.\n\nYou can optionally pass the base URL when you pass the HTML content as\na string instead of URI or HTTP::Response.\n\n    $res = $scraper-\u003escrape($html_content, \"http://example.com/foo\");\n\nThis way Web::Scraper can resolve the relative links found in the document.\n\n## process\n\n    scraper {\n        process \"tag.class\", key =\u003e 'TEXT';\n        process '//tag[contains(@foo, \"bar\")]', key2 =\u003e '@attr';\n        process '//comment()', 'comments[]' =\u003e 'TEXT';\n    };\n\n_process_ is the method to find matching elements from HTML with CSS\nselector or XPath expression, then extract text or attributes into the\nresult stash.\n\nIf the first argument begins with \"//\" or \"id(\" it's treated as an\nXPath expression and otherwise CSS selector.\n\n    # \u003cspan class=\"date\"\u003e2008/12/21\u003c/span\u003e\n    # date =\u003e \"2008/12/21\"\n    process \".date\", date =\u003e 'TEXT';\n\n    # \u003cdiv class=\"body\"\u003e\u003ca href=\"http://example.com/\"\u003efoo\u003c/a\u003e\u003c/div\u003e\n    # link =\u003e URI-\u003enew(\"http://example.com/\")\n    process \".body \u003e a\", link =\u003e '@href';\n\n    # \u003cdiv class=\"body\"\u003e\u003c!-- HTML Comment here --\u003e\u003ca href=\"http://example.com/\"\u003efoo\u003c/a\u003e\u003c/div\u003e\n    # comment =\u003e \" HTML Comment here \"\n    #\n    # NOTES: A comment nodes are accessed when installed\n    # the HTML::TreeBuilder::XPath (version \u003e= 0.14) and/or\n    # the HTML::TreeBuilder::LibXML (version \u003e= 0.13)\n    process \"//div[contains(@class, 'body')]/comment()\", comment =\u003e 'TEXT';\n\n    # \u003cdiv class=\"body\"\u003e\u003ca href=\"http://example.com/\"\u003efoo\u003c/a\u003e\u003c/div\u003e\n    # link =\u003e URI-\u003enew(\"http://example.com/\"), text =\u003e \"foo\"\n    process \".body \u003e a\", link =\u003e '@href', text =\u003e 'TEXT';\n\n    # \u003cul\u003e\u003cli\u003efoo\u003c/li\u003e\u003cli\u003ebar\u003c/li\u003e\u003c/ul\u003e\n    # list =\u003e [ \"foo\", \"bar\" ]\n    process \"li\", \"list[]\" =\u003e \"TEXT\";\n\n    # \u003cul\u003e\u003cli id=\"1\"\u003efoo\u003c/li\u003e\u003cli id=\"2\"\u003ebar\u003c/li\u003e\u003c/ul\u003e\n    # list =\u003e [ { id =\u003e \"1\", text =\u003e \"foo\" }, { id =\u003e \"2\", text =\u003e \"bar\" } ];\n    process \"li\", \"list[]\" =\u003e { id =\u003e '@id', text =\u003e \"TEXT\" };\n\n## process\\_first\n\n`process_first` is the same as `process` but stops when the first matching\nresult is found.\n\n    # \u003cspan class=\"date\"\u003e2008/12/21\u003c/span\u003e\n    # \u003cspan class=\"date\"\u003e2008/12/22\u003c/span\u003e\n    # date =\u003e \"2008/12/21\"\n    process_first \".date\", date =\u003e 'TEXT';\n\n## result\n\n`result` allows to return not the default value after processing but a single\nvalue specified by a key or a hash reference built from several keys.\n\n    process 'a', 'want[]' =\u003e 'TEXT';\n    result 'want';\n\n# EXAMPLES\n\nThere are many examples in the `eg/` dir packaged in this distribution.\nIt is recommended to look through these.\n\n# NESTED SCRAPERS\n\nScrapers can be nested thus allowing to scrape already captured data.\n\n    # \u003cul\u003e\n    # \u003cli class=\"foo\"\u003e\u003ca href=\"foo1\"\u003ebar1\u003c/a\u003e\u003c/li\u003e\n    # \u003cli class=\"bar\"\u003e\u003ca href=\"foo2\"\u003ebar2\u003c/a\u003e\u003c/li\u003e\n    # \u003cli class=\"foo\"\u003e\u003ca href=\"foo3\"\u003ebar3\u003c/a\u003e\u003c/li\u003e\n    # \u003c/ul\u003e\n    # friends =\u003e [ {href =\u003e 'foo1'}, {href =\u003e 'foo2'} ];\n    process 'li', 'friends[]' =\u003e scraper {\n      process 'a', href =\u003e '@href',\n    };\n\n# FILTERS\n\nFilters are applied to the result after processing. They can be declared as\nanonymous subroutines or as class names.\n\n    process $exp, $key =\u003e [ 'TEXT', sub { s/foo/bar/ } ];\n    process $exp, $key =\u003e [ 'TEXT', 'Something' ];\n    process $exp, $key =\u003e [ 'TEXT', '+MyApp::Filter::Foo' ];\n\nFilters can be stacked\n\n    process $exp, $key =\u003e [ '@href', 'Foo', '+MyApp::Filter::Bar', \\\u0026baz ];\n\nMore about filters you can find in [Web::Scraper::Filter](https://metacpan.org/pod/Web::Scraper::Filter) documentation.\n\n# XML backends\n\nBy default [HTML::TreeBuilder::XPath](https://metacpan.org/pod/HTML::TreeBuilder::XPath) is used, this can be replaces by\na [XML::LibXML](https://metacpan.org/pod/XML::LibXML) backend using [Web::Scraper::LibXML](https://metacpan.org/pod/Web::Scraper::LibXML) module.\n\n    use Web::Scraper::LibXML;\n\n    # same as Web::Scraper\n    my $scraper = scraper { ... };\n\n# AUTHOR\n\nTatsuhiko Miyagawa \u003cmiyagawa@bulknews.net\u003e\n\n# LICENSE\n\nThis library is free software; you can redistribute it and/or modify\nit under the same terms as Perl itself.\n\n# SEE ALSO\n\n[http://blog.labnotes.org/category/scrapi/](http://blog.labnotes.org/category/scrapi/)\n\n[HTML::TreeBuilder::XPath](https://metacpan.org/pod/HTML::TreeBuilder::XPath)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmiyagawa%2Fweb-scraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmiyagawa%2Fweb-scraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmiyagawa%2Fweb-scraper/lists"}