{"id":54719,"url":"https://github.com/duyet/awesome-web-scraper","name":"awesome-web-scraper","description":"A collection of awesome web scaper, crawler.","projects_count":36,"last_synced_at":"2026-07-29T16:00:28.392Z","repository":{"id":54154376,"uuid":"51383372","full_name":"duyet/awesome-web-scraper","owner":"duyet","description":"A collection of awesome web scaper, crawler.","archived":false,"fork":false,"pushed_at":"2024-04-04T13:12:43.000Z","size":50,"stargazers_count":291,"open_issues_count":40,"forks_count":49,"subscribers_count":10,"default_branch":"master","last_synced_at":"2026-07-07T17:41:56.277Z","etag":null,"topics":["awesome","awesome-list","goutte","phantomjs","php","scrapy","slimerjs","spider","storage","web-crawler","web-scraper"],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/duyet.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null},"funding":{"github":"duyet","ko_fi":"duyet"}},"created_at":"2016-02-09T17:18:04.000Z","updated_at":"2026-06-13T17:22:16.000Z","dependencies_parsed_at":"2024-01-18T22:33:24.331Z","dependency_job_id":"a16b2a7b-456f-46b4-80a7-192b4737f391","html_url":"https://github.com/duyet/awesome-web-scraper","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/duyet/awesome-web-scraper","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/duyet%2Fawesome-web-scraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/duyet%2Fawesome-web-scraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/duyet%2Fawesome-web-scraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/duyet%2Fawesome-web-scraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/duyet","download_url":"https://codeload.github.com/duyet/awesome-web-scraper/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/duyet%2Fawesome-web-scraper/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":36038877,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-07-20T02:08:10.276Z","status":"online","status_checked_at":"2026-07-29T02:00:04.910Z","response_time":95,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"created_at":"2024-01-18T21:51:51.711Z","updated_at":"2026-07-29T16:00:28.393Z","primary_language":null,"list_of_lists":false,"displayable":true,"categories":["C/C++","Java","PHP","C#","Contributing","Ruby","Nodejs","Python","Rust","Go","Erlang"],"sub_categories":[],"readme":"# Awesome Web Scraper [![Awesome](https://cdn.rawgit.com/sindresorhus/awesome/d7305f38d29fed78fa85652e3a63e154dd8e8829/media/badge.svg)](https://github.com/sindresorhus/awesome)\n\nA collection of awesome web scaper, crawler.\n\n## Java\n* [Apache Nutch](http://nutch.apache.org/) - Highly extensible, highly scalable Web crawler. Pluggable parsing, protocols, storage and indexing.\n* [websphinx](http://www.cs.cmu.edu/~rcm/websphinx/) - Website-Specific Processors for HTML INformation eXtraction.\n* [Open Search Server](http://www.opensearchserver.com/) - A full set of search functions. Build your own indexing strategy. Parsers extract full-text data. The crawlers can index everything.\n* [crawler4j](https://github.com/yasserg/crawler4j) - open source web crawler for Java which provides a simple interface for crawling the Web. Using it, you can setup a multi-threaded web crawler in few minutes.\n\n## C/C++\n* [HTTrack](http://www.httrack.com/) - Highly extensible, highly scalable Web crawler. Pluggable parsing, protocols, storage and indexing.\n\n## C#\n* [ccrawler](https://code.google.com/archive/p/ccrawler/) - Built in C# 3.5 version. it contains a simple extention of web content categorizer, which can saparate between the web page depending on their content.\n\n## Erlang\n* [ebot](https://github.com/matteoredaelli/ebot) - Opensource Web Crawler built on top of a nosql database (apache couchdb, riak), AMQP database (rabbitmq), webmachine and mochiweb.\n\n## Python\n* [scrapy](https://github.com/scrapy/scrapy) - Scrapy, a fast high-level web crawling \u0026 scraping framework for Python.\n* [gdom](https://github.com/syrusakbary/gdom) - gdom, DOM Traversing and Scraping using GraphQL.\n* [trafilatura](https://github.com/adbar/trafilatura) - Library and command-line tool to extract metadata, main text, and comments.\n* [extractnet](https://github.com/currentsapi/extractnet) - machine learning based content \u0026 metadata extraction framework for Python\n* [Scrapegraph-ai](https://github.com/VinciGit00/Scrapegraph-ai) - An open source library for making scraping with the use of the AI\n\n## PHP \n* [Goutte](https://github.com/FriendsOfPHP/Goutte) - Goutte, a simple PHP Web Scraper.\n* [DiDOM](https://github.com/Imangazaliev/DiDOM) - Simple and fast HTML parser.\n* [simple_html_dom](https://github.com/samacs/simple_html_dom) - Just a Simple HTML DOM library fork.\n* [PHPCrawl](http://phpcrawl.cuab.de/) - PHPCrawl is a framework for crawling/spidering websites written in PHP.\n* [Crawler](https://www.crwlr.software/packages/crawler) - A library for Rapid Web Crawler and Scraper Development.\n\n## Nodejs\n\n* [puppeteer](https://github.com/GoogleChrome/puppeteer) - Headless Chrome Node API https://pptr.dev.\n* [Phantomjs](https://github.com/ariya/phantomjs) - Scriptable Headless WebKit.\n* [node-crawler](https://github.com/bda-research/node-crawler) - Web Crawler/Spider for NodeJS + server-side jQuery.\n* [node-simplecrawler](https://github.com/simplecrawler/simplecrawler) - Flexible event driven crawler for node.\n* [spider](https://github.com/mikeal/spider) - Programmable spidering of web sites with node.js and jQuery.\n* [slimerjs](https://github.com/laurentj/slimerjs) - A PhantomJS-like tool running Gecko.\n* [casperjs](https://github.com/casperjs/casperjs) - Navigation scripting \u0026 testing utility for PhantomJS and SlimerJS.\n* [zombie](https://github.com/assaf/zombie) - Insanely fast, full-stack, headless browser testing using node.js.\n* [nightmare](https://github.com/segmentio/nightmare) - Nightmare is a high level wrapper for PhantomJS that lets you automate browser tasks\n* [jsdom](https://github.com/jsdom/jsdom) - A JavaScript implementation of the WHATWG DOM and HTML standards, for use with node.js\n* [xray](https://github.com/matthewmueller/x-ray) - The next web scraper. See through the `\u003chtml\u003e` noise.\n* [lightcrawler](https://github.com/github/lightcrawler) - Crawl a website and run it through Google lighthouse.\n\n## Ruby\n* [wombat](https://github.com/felipecsl/wombat) - Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.\n\n## Go\n* [gocrawl](https://github.com/PuerkitoBio/gocrawl) - Polite, slim and concurrent web crawler.\n* [fetchbot](https://github.com/PuerkitoBio/fetchbot) - A simple and flexible web crawler that follows the robots.txt policies and crawl delays.\n\n## Rust\n* [scraper](https://github.com/causal-agent/scraper) - HTML parsing and querying with CSS selectors.\n* [reqwest](https://github.com/seanmonstar/reqwest) - An ergonomic, batteries-included HTTP Client for Rust.\n\n\n---------------------\n\n## License\n[MIT](LICENSE)\n\n## Contributing\n\nPlease, read the [Contribution Guidelines](https://github.com/duyetdev/awesome-web-scraper/blob/master/CONTRIBUTING.md) before submitting your suggestion.\n\nFeel free to [open an issue](https://github.com/duyetdev/awesome-web-scraper/issues) or [create a pull request](https://github.com/duyetdev/awesome-web-scraper/pulls) with your additions.\n","projects_url":"https://awesome.ecosyste.ms/api/v1/lists/duyet%2Fawesome-web-scraper/projects"}