awesome-web-scraping

A curated list of web scraping tools, libraries, and resources.
https://github.com/crawlee-cloud/awesome-web-scraping

Last synced: 4 days ago
JSON representation

<a name="ai--llm-scraping"></a>🤖 AI & LLM Scraping
- Stagehand - powered programmable browser |
- Crawl4AI - source LLM-friendly web crawler and scraper |
- Firecrawl - ready markdown |
<a name="anti-detection"></a>🛡️ Anti-Detection
- Camoufox
- curl-impersonate
- Rebrowser Patches - detection |
- undetected-chromedriver - detection |
<a name="browser-automation"></a>🎭 Browser Automation
- rod - level Chrome DevTools controller |
- Steel - | Browser API for AI agents |
- Browserbase - | Serverless headless browser platform |
<a name="captcha-solving"></a>🧩 CAPTCHA Solving
- CapMonster Cloud - powered cloud CAPTCHA solving service |
- nocaptchaai
- Anti-Captcha
<a name="data-extraction"></a>⛏️ Data Extraction
- Beautiful Soup
- jsdom
- Parsel
- Selectolax - fast HTML5 parser using Modest engine |
<a name="scheduling"></a>⏰ Scheduling
- BullMQ - based job queue |
- node-cron - like scheduler |
- Celery
<a name="scraping-frameworks"></a>🕷️ Scraping Frameworks
- Ferret
- Crawlee
<a name="tutorials"></a>📚 Tutorials
<a name="utils--user-agents"></a>🧰 Utils & User Agents
- fake-useragent - Agent generator |
- protego - Python robots.txt parser |
- robots-parser
- user-agents - Agent generator |
📝 License
- ![CC0
🚀 Self-Hosted Platforms
- Crawlee Cloud - source, self-hosted Actor platform |

Programming Languages

Categories

Sub Categories

Keywords

ecosyste.ms