Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

Projects in Awesome Lists tagged with web-scraping

A curated list of projects in awesome lists tagged with web-scraping .

https://github.com/d4vinci/scrapling

🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!

ai ai-scraping automation crawler crawling crawling-python data data-extraction mcp mcp-server playwright python scraping selectors stealth web-scraper web-scraping web-scraping-python webscraping xpath

Last synced: 27 Aug 2026

https://github.com/scrapy/scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.

crawler crawling framework hacktoberfest python scraping web-scraping web-scraping-python

Last synced: 11 Jul 2026

https://github.com/mintplex-labs/anything-llm

The all-in-one Desktop & Docker AI application with built-in RAG, AI agents, No-code agent builder, MCP compatibility, and more.

ai-agents custom-ai-agents deepseek kimi llama3 llm lmstudio local-llm localai mcp mcp-servers moonshot multimodal no-code ollama qwen3 rag vector-database web-scraping

Last synced: 18 Feb 2026

https://github.com/dgtlmoon/changedetection.io

The best and simplest free open source web page change detection, website watcher, restock monitor and notification service. Restock Monitor, change detection. Designed for simplicity - Simply monitor which websites had a text change for free. Free Open source web page change detection, Website defacement monitoring, Price change notification

back-in-stock change-alert change-detection change-monitoring changedetection monitoring notifications restock-monitor self-hosted url-monitor web-scraping website-change-detection website-change-detector website-change-monitor website-change-notification website-change-tracker website-defacement-monitoring website-monitor website-monitoring website-watcher

Last synced: 28 Apr 2026

https://github.com/apifytech/apify-js

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

apify automation crawler crawling headless headless-chrome javascript nodejs npm playwright puppeteer scraper scraping typescript web-crawler web-crawling web-scraping

Last synced: 06 Jul 2025

https://github.com/apify/crawlee

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

apify automation crawler crawling headless headless-chrome javascript nodejs npm playwright puppeteer scraper scraping typescript web-crawler web-crawling web-scraping

Last synced: 03 Nov 2025

https://github.com/yusufkaraaslan/skill_seekers

Convert documentation websites, GitHub repositories, and PDFs into Claude AI skills with automatic conflict detection

ai-tools ast-parser automation claude-ai claude-skills code-analysis conflict-detection documentation documentation-generator github github-scraper mcp mcp-server multi-source ocr pdf python web-scraping

Last synced: 28 Aug 2026

https://github.com/getmaxun/maxun

🔥 Open Source No Code Web Data Extraction Platform • Turn Websites To APIs & Spreadsheets With No-Code Robots In Minutes 🔥

agents api automation browser browser-automation data-extraction no-code no-code-web-scraper playwright robotic-process-automation rpa scraper self-hosted web-agent web-automation web-scraper web-scraping web-scraping-agent webscraping website-to-api

Last synced: 23 Jan 2026

https://github.com/evil0ctal/douyin_tiktok_download_api

🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、TikTok、Bilibili数据爬取工具，支持API调用，在线批量解析及下载。

api async crawler douyin douyin-api douyin-scraper douyin-tiktok-api douyin-tiktok-download fastapi no-watermark online-parsing python pywebio scraper spider tiktok tiktok-api tiktok-scraper tiktok-signature web-scraping

Last synced: 12 May 2025

https://github.com/Evil0ctal/Douyin_TikTok_Download_API

🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、TikTok、Bilibili数据爬取工具，支持API调用，在线批量解析及下载。

api async crawler douyin douyin-api douyin-scraper douyin-tiktok-api douyin-tiktok-download fastapi no-watermark online-parsing python pywebio scraper spider tiktok tiktok-api tiktok-scraper tiktok-signature web-scraping

Last synced: 26 Mar 2025

https://github.com/cporter202/API-mega-list

This GitHub repo is a powerhouse collection of APIs you can start using immediately to build everything from simple automations to full-scale applications. One of the most valuable API lists on GitHub—period. 💪

ai-tools api api-collection api-directory api-list apify artificial-intelligence automation awesome-list data-collection developer-resources developer-tools ecommerce lead-generation mcp open-source public-apis real-estate social-media web-scraping

Last synced: 11 Aug 2026

https://github.com/firecrawl/firecrawl-mcp-server

🔥 Official Firecrawl MCP Server - Adds powerful web scraping and search to Cursor, Claude and any other LLM clients.

batch-processing claude content-extraction data-collection firecrawl firecrawl-ai javascript-rendering llm-tools mcp mcp-server model-context-protocol search-api web-crawler web-scraping

Last synced: 27 Aug 2026

https://github.com/apify/crawlee-python

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

apify automation beautifulsoup crawler crawling hacktoberfest headless headless-chrome pip playwright python scraper scraping web-crawler web-crawling web-scraping

Last synced: 06 Mar 2026

https://github.com/D4Vinci/Scrapling

🕷️ An undetectable, powerful, flexible, high-performance Python library to make Web Scraping Easy and Effortless as it should be!

ai ai-scraping automation crawler crawling crawling-python data data-extraction hacktoberfest playwright python python3 scraping selectors stealth web-scraper web-scraping web-scraping-python webscraping xpath

Last synced: 13 May 2025

https://github.com/jo-inc/camofox-browser

Stealth headless browser for AI agents — bypass Cloudflare, bot detection, and anti-scraping. Drop-in Puppeteer/Playwright replacement.

ai-agent anti-bot antidetect-browser automation bot-detection browser-automation cloudflare-bypass headless-browser javascript nodejs playwright puppeteer scraping stealth-browser web-scraping

Last synced: 24 May 2026

https://github.com/knockoutez/wigolo

The go-to web for your AI coding agent — local-first search, fetch, crawl & research over MCP. No API keys, no cloud, $0/query. Public beta.

agent ai ai-agent claude cli developer-tools local-first mcp mcp-server metasearch model-context-protocol nodejs privacy rag search search-engine typescript web-crawler web-scraping web-search

Last synced: 27 Aug 2026

https://github.com/KnockOutEZ/wigolo

The go-to web for your AI coding agent — local-first search, fetch, crawl & research over MCP. No API keys, no cloud, $0/query. Public beta.

agent ai ai-agent claude cli developer-tools local-first mcp mcp-server metasearch model-context-protocol nodejs privacy rag search search-engine typescript web-crawler web-scraping web-search

Last synced: 04 Aug 2026

https://github.com/lexiforest/curl_cffi

Python binding for curl-impersonate fork via cffi. A http client that can impersonate browser tls/ja3/http2 fingerprints.

akamai-fingerprint curl curl-impersonate fingerprinting http http-client http2-fingerprint https ja3 ja3-fingerprint tls-fingerprint web-scraping

Last synced: 14 May 2025

https://github.com/snooppr/snoop

Snoop — инструмент разведки на основе открытых данных (OSINT world)

blueteam ctf geo geocoder infosec ip nickname osint parser pentest police redteam scanner scraping security termux username username-checker username-search web-scraping

Last synced: 14 May 2025

https://github.com/browser-act/skills

Browser automation CLI built for AI agents. Break through anti-bot walls, hand off to humans across platforms when stuck. Parallel multi-task execution, independent multi-session operation, isolated multi-account browsing.

ai-agents automation claude-cli claude-code claude-code-skills claude-skills codex codex-cli codex-skill cursor data-extraction no-code openclaw openclaw-cli openclaw-skill openclaw-skills web-data-extraction web-scraping web-scraping-api

Last synced: 28 Jun 2026

https://github.com/microlinkhq/browserless

The headless Chrome/Chromium driver on top of Puppeteer. Take screenshots, generate PDFs, extract text and HTML with a production-ready API.

automation browser-automation chromium lighthouse pdf-generation screenshot web-crawler web-scraping

Last synced: 14 Jul 2026

https://github.com/Altimis/Scweet

Scrape tweets, profiles, followers and following from Twitter/X, no API key needed. Python library with smart multi-account pooling, proxy support and async.

apify data-collection graphql proxy python python-library scrape-tweets scraper social-media tweet-scraper tweets twitter twitter-data twitter-followers twitter-following twitter-scraper twitter-search twitter-x web-scraping x

Last synced: 14 Aug 2026

https://github.com/a9t9/rpa

Ui.Vision Open-Source RPA Software with Computer Vision, OCR, Anthropic Computer Use/LLM. Selenium IDE import/export.

anthropic anthropic-claude browser-automation browser-extension computer-use data-driven-tests imacros selenium-ide web-automation web-scraping

Last synced: 16 May 2025

https://github.com/tidyverse/rvest

Simple web scraping for R

html r web-scraping

Last synced: 16 Dec 2025

https://github.com/roach-php/core

The complete web scraping toolkit for PHP.

crawling php web-scraping

Last synced: 13 May 2025

https://github.com/rushter/selectolax

Python binding to Modest and Lexbor engines (fast HTML5 parser with CSS selectors).

css html5 modest-engine parser python web-scraping

Last synced: 13 May 2025

https://github.com/gosom/google-maps-scraper

scrape data data from Google Maps. Extracts data such as the name, address, phone number, website URL, rating, reviews number, latitude and longitude, reviews,email and more for each place

distributed-scraper distributed-scraping golang google-maps google-maps-scraping web-scraper web-scraping

Last synced: 28 Dec 2025

https://github.com/firecrawl/open-scouts

AI-powered web monitoring platform. Create automated scouts that search the web and send email alerts when they find what you're looking for.

ai-agents alerts automation email-notifications firecrawl nextjs open-source openai posthog react resend supabase tailwindcss typescript vercel web-monitoring web-scraping

Last synced: 11 Feb 2026

https://github.com/A9T9/RPA

UI.Vision: Open-Source RPA Software (formerly Kantu) - Modern Robotic Process Automation with Selenium IDE++

autohotkey automation browser-automation browser-extension data-driven-tests imacros opencv selenium-ide sikulix ui-tests uipath visual-recognition web-automation web-scraping webassembly

Last synced: 22 Mar 2025

https://github.com/orangecoding/fredy

❤️ Fredy - [F]ind [R]eal [E]state [D]amn Eas[y] - Fredy keeps searching for new apartments, houses, and flats in Germany on platforms like ImmoScout24, Immowelt, Immonet, eBay Kleinanzeigen, and WG-Gesucht and instantly delivers the results to you via Slack, Telegram, Email, Discord or ntfy, so you can focus on the more important things in life ;)

appartment appartments crawler ebay-kleinanzeigen fredy immobilien immobilienscout24 immonet immoscout24 immowelt nodejs puppeteer real-estate real-estate-search self-hosted telegrambots web-scraping wg-gesucht wohnungssuche

Last synced: 11 Jun 2026

https://github.com/manavarya09/design-extract

Extract any website's complete design system with one command. DTCG tokens, semantic+primitive+composite, MCP server for Claude Code/Cursor/Windsurf, multi-platform emitters (iOS SwiftUI, Android Compose, Flutter, WordPress), Tailwind v4, Figma variables, shadcn/ui, CSS health audit, WCAG remediation, Chrome extension. MIT, Playwright, Node 20+.

accessibility agent-skill ai chrome-extension claude-code-plugin cli css cursor design-system design-to-code design-tokens dtcg figma mcp-server npx playwright shadcn-ui skills-sh tailwind web-scraping

Last synced: 24 Apr 2026

https://github.com/austin-weeks/miasma

Trap AI web scrapers in an endless poison pit.

ai anti-ai anti-spam free-software web-scraping

Last synced: 11 Jun 2026

https://github.com/intoli/user-agents

A JavaScript library for generating random user agents with data that's updated daily.

browser-automation browsers javascript navigator random randomization user-agent user-agent-spoofer web-scraping

Last synced: 13 May 2025

https://github.com/platonai/pulsarRPA

Automate webpages at scale, scrape web data completely and accurately with high performance, distributed AI-RPA.

ai-agents ai-crawler ai-rpa ai-scrarper crawler rpa scraper scraping web-crawler web-scraping

Last synced: 01 Apr 2025

https://github.com/postmodern/spidr

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

crawler ruby scraper spider spider-links web web-crawler web-scraper web-scraping web-spider

Last synced: 13 May 2025

https://github.com/DataHenHQ/till

DataHen Till is a companion tool to your existing web scraper that instantly makes it scalable, maintainable, and more unblockable, with minimal code changes on your scraper. Integrates with any scraper in 5 minutes.

crawler man-in-the-middle mitm proxy-server scraper scraping web-scraping

Last synced: 15 Mar 2025

https://github.com/je-suis-tm/web-scraping

Detailed web scraping tutorials for dummies with financial data crawlers on Reddit WallStreetBets, CME (both options and futures), US Treasury, CFTC, LME, MacroTrends, SHFE and alternative data crawlers on Tomtom, BBC, Wall Street Journal, Al Jazeera, Reuters, Financial Times, Bloomberg, CNN, Fortune, The Economist

bloomberg data-scraper data-scraping financial-data financial-times futures futures-historical-data news-scraper news-websites newsletter options-data python-web-scraper reuters scrapper sraping wall-street-journal wallstreetbets web-scraper web-scrapers web-scraping

Last synced: 04 Apr 2025

https://github.com/gildas-lormeau/single-file-cli

CLI tool for saving a faithful copy of a complete web page in a single HTML file (based on SingleFile)

archiving cli crawler deno dockerfile nodejs scraping-websites single-file web-archiving web-crawler web-scraper web-scraping

Last synced: 15 May 2025

https://github.com/rebrowser/rebrowser-patches

Collection of patches for puppeteer and playwright to avoid automation detection and leaks. Helps to avoid Cloudflare and DataDome CAPTCHA pages. Easy to patch/unpatch, can be enabled/disabled on demand.

automation bot bot-detection chrome chromedriver cloudflare crawler crawling datadome headless headless-chrome playwright puppeteer puppeteer-extra rebrowser scraping selenium stealth web-scraping webdriver

Last synced: 14 May 2025

https://github.com/sardanioss/httpcloak

Go HTTP client with browser-identical TLS/HTTP2 fingerprinting. Bypass bot detection by perfectly mimicking Chrome, Firefox, and Safari at the cryptographic level (JA3/JA4, Akamai fingerprint, header order). Supports HTTP/1.1, HTTP/2, HTTP/3, sessions, cookies, and proxies.

anti-bot bot-detection browser-fingerprint browser-fingerprinting cloudflare go golang http-client http2 http3 ja3-fingerprint ja4-fingerprint js nodejs python python3 quic tls-fingerprint tls-fingerprinting web-scraping

Last synced: 20 Feb 2026

https://github.com/tinyfish-io/agentql

AgentQL is a suite of tools for connecting your AI to the web. Featuring a query language and Playwright integrations for interacting with elements and extracting data quickly, precisely, and at scale. Includes REST API, Python and JavaScript SDKs, browser debugger.

agent ai aiagent automation javascript playwright python rpa scraping web web-scraping web-scraping-colabs web-scraping-javascript web-scraping-python web-scrapping webagent

Last synced: 15 May 2025

https://github.com/alecxe/scrapy-fake-useragent

Random User-Agent middleware based on fake-useragent

python scrapy web-scraping

Last synced: 15 May 2025

https://github.com/dinubs/coolqlcool

Nextjs server to query websites with GraphQL

graphql javascript nextjs schema web-scraping

Last synced: 04 Apr 2025

https://github.com/z0m31en7/uscrapper

Uscrapper Vanta: Dive deeper into the web with this powerful open-source tool. Extract valuable insights with ease and efficiency, from both surface and deep web sources. Empower your data mining and analysis with Vanta's advanced capabilities. Fast, reliable, and user-friendly, Uscrapper Vanta is the ultimate choice for researchers and analysts.

darkweb darkweb-crawler information-extraction information-gathering osint osint-python osint-tool python reconnaissance selenium selenium-webscraper tor web-scraping webcra webcrawler webscraping website-scraper websites

Last synced: 15 May 2025

https://github.com/z0m31en7/Uscrapper

Uscrapper Vanta: Dive deeper into the web with this powerful open-source tool. Extract valuable insights with ease and efficiency, from both surface and deep web sources. Empower your data mining and analysis with Vanta's advanced capabilities. Fast, reliable, and user-friendly, Uscrapper Vanta is the ultimate choice for researchers and analysts.

darkweb darkweb-crawler information-extraction information-gathering osint osint-python osint-tool python reconnaissance selenium selenium-webscraper tor web-scraping webcra webcrawler webscraping website-scraper websites

Last synced: 05 May 2025

https://github.com/oxylabs/how-to-scrape-google-scholar

A guide for extracting titles, authors, and citations from Google Scholar using Python and Oxylabs SERP Scraper API.

google-scholar google-scholar-scraper google-scholar-scrapper google-search-scraper python python-scraper scraper-api web-scraper web-scraping

Last synced: 15 May 2025

https://github.com/oxylabs/quick-start-guide

Python quick start guides to get the most out of Oxylabs' Web Scraper API free trial.

oxylabs scraper scraper-api scraper-python scrapers scraping scraping-websites web-scraper web-scraping

Last synced: 06 Jul 2025

https://github.com/austinoboyle/scrape-linkedin-selenium

`scrape_linkedin` is a python package that allows you to scrape personal LinkedIn profiles & company pages - turning the data into structured json.

linkedin python scrape scraper scraping selenium selenium-webdriver web-scraper web-scraping

Last synced: 04 Apr 2025

https://github.com/oxylabs/how-to-scrape-amazon-prices

A code for extracting best-selling items, search results, and currently available deals from Amazon using Python and Oxylabs E-Commerce Scraper API.

amazon amazon-scraper api python python-scraper scraper-api web-scraper web-scraping

Last synced: 16 May 2025

https://github.com/sangaline/wayback-machine-scraper

A command-line utility and Scrapy middleware for scraping time series data from Archive.org's Wayback Machine.

archive-dot-org command-line-tool python wayback-archiver wayback-machine web-scraping

Last synced: 13 Apr 2025

https://github.com/roniemartinez/dude

dude uncomplicated data extraction: A simple framework for writing web scrapers using Python decorators

async beautifulsoup4 crawler css framework lxml parsel playwright python scraper scraping selenium sync web-scraping webscraping xpath

Last synced: 16 Mar 2025

https://github.com/0xMassi/webclaw

Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.

ai ai-agents ai-scraping cli crawler data-extraction html-to-markdown llm markdown mcp mcp-server rust scraper self-hosted tls-fingerprinting web-crawler web-extraction web-scraper web-scraping webscraping

Last synced: 04 Apr 2026

https://github.com/yusuzech/r-web-scraping-cheat-sheet

Guide, reference and cheatsheet on web scraping using rvest, httr and Rselenium.

cheatsheet httr r rselenium rvest scrape-websites web-scraping webscraping

Last synced: 29 Apr 2026

https://github.com/Johell1NS/browser-search

A skill for AI agents: search the web with SearXNG, browse with Camofox, bypass protections with CloakBrowser. Anti-hallucination by design. Self-hosted, free, unlimited.

agent-skills ai-agent ai-tools browser-automation camoufox cloakbrowser deep-research opencode searxng web-scraping web-search

Last synced: 30 Jul 2026