https://github.com/lorien/awesome-web-scraping

List of libraries, tools and APIs for web scraping and data processing.
https://github.com/lorien/awesome-web-scraping

List: awesome-web-scraping

captcha-bypass captcha-recaptcha crawler crawling crawling-framework crawling-python crawling-tool scraping scraping-framework scraping-python scraping-tool spider web-scraping webscraping

Last synced: 24 days ago
JSON representation

List of libraries, tools and APIs for web scraping and data processing.

• Host: GitHub
• URL: https://github.com/lorien/awesome-web-scraping
• Owner: lorien
• License: other
• Created: 2015-08-12T19:55:27.000Z (about 9 years ago)
• Default Branch: master
• Last Pushed: 2024-05-09T05:22:56.000Z (6 months ago)
• Last Synced: 2024-05-19T03:34:19.351Z (6 months ago)
• Topics: captcha-bypass, captcha-recaptcha, crawler, crawling, crawling-framework, crawling-python, crawling-tool, scraping, scraping-framework, scraping-python, scraping-tool, spider, web-scraping, webscraping
• Language: Makefile
• Homepage:
• Size: 449 KB
• Stars: 6,365
• Watchers: 230
• Forks: 772
• Open Issues: 2
• Metadata Files:
  • Readme: README.md
  • Contributing: CONTRIBUTING.md
  • License: LICENSE

Awesome Lists containing this project

• my-awesome-starred - awesome-web-scraping - List of libraries, tools and APIs for web scraping and data processing. (Makefile)
• awesome - lorien/awesome-web-scraping - List of libraries, tools and APIs for web scraping and data processing. (Makefile)
• awesome-awesome - awesome-web-scraping
• awesome-open-source-marketing - lorien/awesome-web-scraping
• fucking-lists - awesome-web-scraping
• awesomelist - awesome-web-scraping
• AwesomeGenomics - scrap the web
• awesome-rainmana - lorien/awesome-web-scraping - List of libraries, tools and APIs for web scraping and data processing. (Makefile)
• awesome-golang-repositories - awesome-web-scraping
• collection - awesome-web-scraping
• lists - awesome-web-scraping
• awesome-browser-automation - Awesome Web Scraping - Comprehensive list of tools, programming libraries and web services used in web scraping. (Resources / Related tools)
• jimsghstars - lorien/awesome-web-scraping - List of libraries, tools and APIs for web scraping and data processing. (Makefile)
• awesome-security-collection - **3198**星

README

        
# Awesome Web Scraping

Lists of packages, services and manuals related to web scraping.

## Topics

* [Python](https://github.com/lorien/web-scraping/blob/master/python.md) - Python packages

* [PHP](https://github.com/lorien/web-scraping/blob/master/php.md) - PHP packages

* [Ruby](https://github.com/lorien/web-scraping/blob/master/ruby.md) - Ruby packages

* [JavaScript](https://github.com/lorien/web-scraping/blob/master/javascript.md) - JavaScript packages

* [Go](https://github.com/lorien/web-scraping/blob/master/golang.md) - Go packages

* [Web Scraping Manuals](https://github.com/lorien/awesome-web-scraping/blob/master/manuals.md) - list of articles and books teaching web scraping

* [dhamaniasad / HeadlessBrowsers](https://github.com/dhamaniasad/HeadlessBrowsers) - list of (almost) all headless web browsers in existence

* [DNS over HTTPS providers](https://github.com/curl/curl/wiki/DNS-over-HTTPS) - list of DNS over HTTPs providers

* [Awesome Pastebins](https://github.com/lorien/awesome-pastebins) - list of pastebin sites

## Captcha Solving Services

* [https://2captcha.com](https://2captcha.com/?from=3019071)

## Proxy Server Marketplaces

* https://www.blackhatworld.com/forums/proxies-for-sale.112/

* https://forum.antichat.com/forums/147/

## Telegram Discussion Groups

* [@grablab](https://t.me/grablab) - talks in English

* [@grablab_ru](https://t.me/grablab_ru) - talks in Russian

## How to Contribute to This List

See [Contributing](https://github.com/lorien/web-scraping/blob/master/CONTRIBUTING.md) guide.

## Credits

The list is based initially on some data from these sources [awesome-python](https://github.com/vinta/awesome-python), [awesome-php](https://github.com/ziadoz/awesome-php), [awesome-ruby](https://github.com/markets/awesome-ruby), [ruby-nlp](https://github.com/diasks2/ruby-nlp), [awesome-javascript](https://github.com/sorrycc/awesome-javascript)