awesome-web-scraping
A curated list of web scraping tools, libraries, and resources.
https://github.com/crawlee-cloud/awesome-web-scraping
Last synced: 24 days ago
JSON representation
<a name="ai--llm-scraping"></a>🤖 AI & LLM Scraping
- Crawl4AI - source LLM-friendly web crawler and scraper |
- ScrapeGraphAI - based AI scraping |
<a name="anti-detection"></a>🛡️ Anti-Detection
- curl-impersonate
- Rebrowser Patches - detection |
- undetected-chromedriver - detection |
- Camoufox
<a name="browser-automation"></a>🎭 Browser Automation
- rod - level Chrome DevTools controller |
- Steel - | Browser API for AI agents |
- Playwright - browser automation by Microsoft |
- Puppeteer
<a name="captcha-solving"></a>🧩 CAPTCHA Solving
- Anti-Captcha
- nocaptchaai
- 2Captcha - powered CAPTCHA solving service |
<a name="data-extraction"></a>⛏️ Data Extraction
- Beautiful Soup
- jsdom
- Parsel
- Selectolax - fast HTML5 parser using Modest engine |
- Cheerio - like HTML parsing |
- lxml - performance XML/HTML processing |
<a name="scheduling"></a>⏰ Scheduling
<a name="scraping-frameworks"></a>🕷️ Scraping Frameworks
<a name="tutorials"></a>📚 Tutorials
<a name="utils--user-agents"></a>🧰 Utils & User Agents
- fake-useragent - Agent generator |
- protego - Python robots.txt parser |
- robots-parser
- user-agents - Agent generator |
📝 License
🚀 Self-Hosted Platforms
- Crawlee Cloud - source, self-hosted Actor platform |