awesome-web-scraping
Best scraping tools collection in town. Find everything you need for scraping, crawling, and processing data from the web
https://github.com/lukas-bear/awesome-web-scraping
Last synced: 9 days ago
JSON representation
Anti-Bot Solutions
Browser Fingerprinting
CAPTCHA Solvers
- 2captcha - Human captcha solving
- Anti-Captcha - Automated solving
- DeathByCaptcha - API-based solving
Proxy Services
- anyIP.io - Reliable proxy solutions, solid mobile proxies
- Bright Data - Enterprise proxy network
- Oxylabs - Proxy and scraping solutions
- ScraperAPI - Proxy API service
- IPRotate - IP rotation service
- SOAX
- ProxyEmpire
- NetNut
- Smartproxy
Browser Automation
Anti-detect Browsers
- Ghost Browser
- Dolphin Anty
- GoLogin
- Incogniton
- MoreLogin
- Lalicat
- HideMyAcc
- BitBrowser
- Multilogin
- AdsPower
Headless Browsers
- PhantomJS - Scriptable headless WebKit
Testing Frameworks
- Selenium - Browser automation standard
- Playwright - Modern web testing
- Cypress - JavaScript testing framework
Data Processing
Data Storage
- MongoDB - Document database
- Elasticsearch - Search and analytics
- PostgreSQL - Relational database
- Redis - In-memory data store
Natural Language Processing
- NLTK - Natural Language Toolkit
- spaCy - Industrial-strength NLP
- langdetect - Language detection
Resources
Community
Documentation
Tutorials
Specialized Tools
HTML/XML Processing
- XPath - XML path language
- CSS Selectors - Pattern matching syntax
- xmltodict - XML to Python dict converter
- html5lib - HTML parser and serializer
Network Utilities
- Charles Proxy - Web debugging proxy
- Fiddler - Web debugging proxy
- Proxychains - Proxy chaining tool
- mitmproxy - Interactive HTTPS proxy
Text Processing
- Dateparser - Date parsing library
- Price-parser - Price extraction
- Phonenumbers - Phone number parsing
- Ftfy - Text encoding fixer