Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

Projects in Awesome Lists tagged with web-scraper

A curated list of projects in awesome lists tagged with web-scraper .

https://github.com/firecrawl/firecrawl

Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥

ai ai-agents ai-crawler ai-scraping ai-search crawler data-extraction html-to-markdown llm markdown scraper scraping web-crawler web-data web-data-extraction web-scraper web-scraping web-search webscraping

Last synced: 09 Nov 2026

https://github.com/panniantong/agent-reach

Give your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.

agent-infrastructure ai-agent ai-search automation bilibili claude-code cli cursor free-api llm-tools mcp python reddit-scraper twitter-scraper web-scraper xiaohongshu youtube-transcript

Last synced: 09 Nov 2026

https://github.com/d4vinci/scrapling

🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

ai ai-scraping automation crawler crawling crawling-python data data-extraction mcp mcp-server playwright python scraping selectors stealth web-scraper web-scraping web-scraping-python webscraping xpath

Last synced: 09 Nov 2026

https://github.com/gosom/google-maps-scraper

scrape data from Google Maps. Extracts data such as the name, address, phone number, website URL, rating, reviews number, latitude and longitude, reviews,email and more for each place

agent-skills claude-skills codex-skills distributed-scraper distributed-scraping golang google-maps google-maps-scraping web-scraper web-scraping

Last synced: 09 Nov 2026

https://github.com/D4Vinci/Scrapling

🕷️ An undetectable, powerful, flexible, high-performance Python library to make Web Scraping Easy and Effortless as it should be!

ai ai-scraping automation crawler crawling crawling-python data data-extraction hacktoberfest playwright python python3 scraping selectors stealth web-scraper web-scraping web-scraping-python webscraping xpath

Last synced: 13 May 2025

https://github.com/ahmadhassan-bted/jinaclip

Professional-grade, local-first browser extension that transforms any webpage into high-fidelity, LLM-friendly Markdown entirely within the browser's runtime. Zero-cloud, privacy-centric, and designed specifically for AI workflows.

ai-workflows browser-extension chrome-extension- clipboard-api dom-parsing es6 jina-ai-jina-reader llm-friendly local-first markdown-converter mv3 privacy-centric productivity-tool shadow-dom web-scraper

Last synced: 09 Nov 2026

https://github.com/itsowen/cyberscraper-2077

A Powerful web scraper powered by LLM | OpenAI, Gemini & Ollama

ai-scraping gemini-api llm llm-scraper openai scraper web-scraper webscraping

Last synced: 09 Nov 2026

https://github.com/oxylabs/how-to-scrape-amazon-product-data

The process of extracting product data from Amazon using Python, including titles, ratings, prices, images, and descriptions.

amazon amazon-scraper python web-scraper web-scraping web-scraping-python

Last synced: 19 Oct 2026

https://github.com/anaskhan96/soup

Web Scraper in Go, similar to BeautifulSoup

beautifulsoup go golang html-node web-scraper webscraper webscraping

Last synced: 19 Oct 2026

https://github.com/oxylabs/how-to-scrape-google-scholar

A guide for extracting titles, authors, and citations from Google Scholar using Python and Oxylabs SERP Scraper API.

google-scholar google-scholar-scraper google-scholar-scrapper google-search-scraper python python-scraper scraper-api web-scraper web-scraping

Last synced: 19 Oct 2026

https://github.com/oxylabs/how-to-scrape-amazon-prices

A code for extracting best-selling items, search results, and currently available deals from Amazon using Python and Oxylabs E-Commerce Scraper API.

amazon amazon-scraper api python python-scraper scraper-api web-scraper web-scraping

Last synced: 19 Oct 2026

https://github.com/gildas-lormeau/single-file-cli

CLI tool for saving a faithful copy of a complete web page in a single HTML file (based on SingleFile)

archiving cli crawler deno dockerfile nodejs scraping-websites self-extracting single-file singlefile web-archiving web-crawler web-scraper web-scraping

Last synced: 19 Oct 2026

https://github.com/fastcrw/crw

Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary. Self-host or use managed cloud.

ai ai-agents crawler data-extraction docker firecrawl-alternative html-to-markdown llm markdown mcp mcp-server rust scraping-api self-hosted tavily-alternative web-crawler web-scraper web-scraping web-search-api

Last synced: 09 Nov 2026

https://github.com/tholian-network/stealth

:rocket: Stealth - Secure, Peer-to-Peer, Private and Automateable Web Browser/Scraper/Proxy

anonymity browser-automation privacy-protection web-browser web-filter web-proxy web-scraper

Last synced: 19 Oct 2026

https://github.com/oxylabs/ai-scraper-py

AI Scraper is a powerful scraping tool and scrape agent built to automate data extraction with unmatched precision. Ideal for scalable AI scraping tasks across diverse web sources, this tool simplifies complex scraping operations into efficient, intelligent workflows.

ai-agent ai-grounding ai-scraper ai-scraping ai-search ai-studio ai-training firecrawl-alternative open-source-ai scraper-api scraper-api-github search-api tavily-alternative web-scraper web-search-api

Last synced: 19 Oct 2026

https://github.com/postmodern/spidr

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

crawler ruby scraper spider spider-links web web-crawler web-scraper web-scraping web-spider

Last synced: 13 May 2025

https://github.com/k0rnh0li0/onlyfans-dl

OnlyFans content downloader

media-downloader onlyfans python web-scraper

Last synced: 26 Mar 2025

https://github.com/je-suis-tm/web-scraping

Detailed web scraping tutorials for dummies with financial data crawlers on Reddit WallStreetBets, CME (both options and futures), US Treasury, CFTC, LME, MacroTrends, SHFE and alternative data crawlers on Tomtom, BBC, Wall Street Journal, Al Jazeera, Reuters, Financial Times, Bloomberg, CNN, Fortune, The Economist

bloomberg data-scraper data-scraping financial-data financial-times futures futures-historical-data news-scraper news-websites newsletter options-data python-web-scraper reuters scrapper sraping wall-street-journal wallstreetbets web-scraper web-scrapers web-scraping

Last synced: 04 Apr 2025

https://github.com/dondai44423/donsetch

Web fetch, search, and crawl for AI agents. Built from scratch in Rust. No keys, no accounts. AGPL v3.

agpl ai-agents anti-bot crawling mcp pdf rust web-scraper web-search

Last synced: 27 Sep 2026

https://github.com/cassidoo/scrapers

A list of scrapers from around the web.

list scrape-websites scraper web-scraper

Last synced: 06 Feb 2026

https://github.com/oxylabs/quick-start-guide

Python quick start guides to get the most out of Oxylabs' Web Scraper API free trial.

oxylabs scraper scraper-api scraper-python scrapers scraping scraping-websites web-scraper web-scraping

Last synced: 06 Jul 2025

https://github.com/austinoboyle/scrape-linkedin-selenium

`scrape_linkedin` is a python package that allows you to scrape personal LinkedIn profiles & company pages - turning the data into structured json.

linkedin python scrape scraper scraping selenium selenium-webdriver web-scraper web-scraping

Last synced: 04 Apr 2025

https://github.com/0xMassi/webclaw

Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.

ai ai-agents ai-scraping cli crawler data-extraction html-to-markdown llm markdown mcp mcp-server rust scraper self-hosted tls-fingerprinting web-crawler web-extraction web-scraper web-scraping webscraping

Last synced: 04 Apr 2026

https://github.com/paulpierre/markdown-crawler

A multithreaded 🕸️ web crawler that recursively crawls a website and creates a 🔽 markdown file for each page, designed for LLM RAG

html-to-markdown html-to-markdown-converter html2md llm llmops markdown markdown-crawler markdown-parser markdown-scraper md-crawler rag web-scraper

Last synced: 13 Apr 2025

https://github.com/phantominsights/summarizer

A Reddit bot that summarizes news articles written in Spanish or English. It uses a custom built algorithm to rank words and sentences.

nlp praw python3 reddit-bot spacy web-scraper wordcloud

Last synced: 11 Jul 2025

https://github.com/PhantomInsights/summarizer

A Reddit bot that summarizes news articles written in Spanish or English. It uses a custom built algorithm to rank words and sentences.

nlp praw python3 reddit-bot spacy web-scraper wordcloud

Last synced: 02 May 2025

https://github.com/epiqueras/getsy

A simple browser/client-side web scraper.

browser client-side scraper web-scraper

Last synced: 13 Apr 2025

https://github.com/wikimedia/html-metadata

MetaData html scraper and parser for Node.js (supports Promises and callback style)

javascript metadata-extraction metadata-extractor node-module nodejs web-scraper web-scraping

Last synced: 15 May 2025

https://github.com/pinkpixel-dev/web-scout-mcp

An MCP server providing web search and content extraction capabilities. Integrates DuckDuckGo search functionality and URL content extraction into your MCP environment, enabling AI assistants to search the web and extract webpage content.

ai-assistant ai-tools cheerio content-extraction crawler duckduckgo duckduckgo-search google-search mcp mcp-server web-content web-crawler web-scraper web-scraping web-search web-search-agent

Last synced: 09 Nov 2026

https://github.com/oxylabs/how-to-scrape-indeed

A tutorial for collecting job postings from Indeed using Python and Oxylabs Web Scraper API.

api job-posting python scraper-api web-scraper web-scraping

Last synced: 05 Apr 2025

https://github.com/areed1192/python-sec

A simple python library that allows for easy access of the SEC website so that someone can parse filings, collect data, and query documents.

finance python sec securities-and-exchange-commission web-scraper

Last synced: 16 May 2025

https://github.com/oxylabs/playwright-web-scraping

A tutorial for web scraping using Playwright headless browser

playwright web-scraper web-scraping

Last synced: 06 Apr 2025

https://github.com/networkdynamics/pytok

A web scraper for TikTok using Playwright

data-collection tiktok tiktok-api tiktok-scraper web-scraper

Last synced: 19 Jan 2026

https://github.com/oxylabs/chatgpt-web-scraping

Learn to create ChatGPT prompts that generate a web scraping code with proper CSS selectors.

chatgpt chatgpt-scraping gpt-scraping web-scraper web-scraping

Last synced: 09 Apr 2025

https://github.com/sriram-pr/doc-scraper

Go web crawler to scrape documentation sites and convert content to clean Markdown for LLM ingestion (RAG, training data)

data-preparation documentation golang-cli llm mcp mcp-server web-crawler web-scraper

Last synced: 09 Nov 2026

https://github.com/passivebot/facebook-marketplace-scraper

This repository contains a script to scrape Facebook Marketplace data using Playwright, BeautifulSoup and Streamlit.

database facebook facebook-marketing-automation facebook-marketplace playwright playwright-python python sqlite3 web-automation web-scraper web-scraping

Last synced: 15 May 2025

https://github.com/Sriram-PR/doc-scraper

Go web crawler to scrape documentation sites and convert content to clean Markdown for LLM ingestion (RAG, training data).

data-preparation documentation golang-cli llm mcp mcp-server web-crawler web-scraper

Last synced: 19 Sep 2026

https://github.com/khuyentran1401/top-github-scraper

Scape top GitHub repositories and users based on keywords

github github-api python scraping web-scraper web-scraping

Last synced: 01 Aug 2025

https://github.com/scrapehero/yellowpages-scraper

Yellowpages.com Web Scraper written in Python and LXML to extract business details available based on a particular category and location.

business-directory extract html lxml parsing python scraper web-scraper yellow-pages yellow-pages-scraper

Last synced: 04 Apr 2025

https://github.com/janchaloupka/web-scraper-nabidek-pronajmu

Nástroj pro hlídání nových nabídek nemovitostí na populárních realitních serverech. Nabídky jsou vypisovány do Discord roomky.

apartment-finder discord discord-bot docker python renting web-scraper

Last synced: 29 Jun 2025

https://github.com/cobalt-uoft/uoft-scrapers

Public web scraping scripts for the University of Toronto.

open-data toronto uoft web-scraper

Last synced: 02 Apr 2025

https://github.com/nasdin/videorecognition-realtime-autotrainer-alerts

State of the art object detection in real-time using YOLOV3 algorithm. Augmented with a process that allows easy training of the classifier as a plug & play solution . Provides alert if an item in an alert list is detected.

alerts automatic convolutional-neural-networks darknet deep-learning google-image-search image-processing image-recognition machine-learning object-detection real-time tensorflow video-recognition web-scraper webcam webscraping yolo yolo2 yolov2 yolov2-model

Last synced: 09 Oct 2025

https://github.com/mawrkus/jason-the-miner

⛏ A versatile Web scraper for Node.js

crawler crawling javascript scraper scraping web-scraper

Last synced: 08 Apr 2025

https://github.com/jlospinoso/abrade

A fast Web API scraper written in C++ and built on Boost ASIO

boost-asio boost-beast cpp web-scraper

Last synced: 27 Mar 2025

https://github.com/scrapfly/python-scrapfly

Scrapfly Python SDK for headless browsers and proxy rotation

crawler headless-browser python scraper scraping scraping-api sdk web-scraper web-scraping

Last synced: 14 Apr 2025

https://github.com/milahu/opensubtitles-scraper

scrape subtitles from opensubtitles.org

opensubtitles subtitles web-scraper

Last synced: 22 Oct 2025

https://github.com/hexsyro/anti-scraping-playwright

web scraping framework that bypasses common anti-bot measures using advanced evasion techniques.

anti-blocking-scraper anti-bot stealth-scraper stealth-scraping web-scraper web-scraping web-scraping-python

Last synced: 10 Nov 2026

https://github.com/jetkai/proxy-scraper

This is an application that scrapes various Proxy API Endpoints, then compiles the proxies into files within the "/proxies/" directory.

exe gradle httpclient jackson-json jar java jdk11 kotlin launch4j proxies proxy proxy-scrape proxy-scraper scraper scraping selenium-java web-scraper web-scraping

Last synced: 02 Sep 2025

https://github.com/phantominsights/reddit-bots

A collection of Reddit bots that I use to enhance the subreddits I manage.

beautifulsoup praw python3 reddit-bot requests rss web-scraper

Last synced: 29 Apr 2025

https://github.com/luminati-io/linkedin-scraper

Extract LinkedIn data with the #1 LinkedIn Scraper API, including profiles, job postings, company details, connections, and posts. Start your free trial now!

datasets linkedin linkedin-api linkedin-dataset linkedin-profile linkedin-scraper web-scraper

Last synced: 13 Oct 2025

https://github.com/raymelon/tagalog-dictionary-scraper

Builds a Tagalog dictionary by collecting Tagalog words from tagalog.pinoydictionary.com

beautiful-soup database dictionary python scraper tagalog tagalog-dictionary web-scraper web-scraping

Last synced: 14 May 2026

https://github.com/henrylok0/anydownload

CLI & GUI website downloader for offline browsing. Static + Playwright render for SPAs, path discovery, local HTTP preview. npm install -g anydownload

cli download downloads html html-download https nodejs puppeteer web-scraper website website-backup website-downloader webstie

Last synced: 16 May 2026

https://github.com/vedhavyas/scrape

Depth controllable Web scraper and Sitemap Generator in Go

sitemap sitemap-generator web-scraper

Last synced: 14 Oct 2025

https://github.com/knlnks/uber_eats_scraper

An Uber Eats scraper written in python.

python restaurant selenium uber-eats uber-eats-scraper web-scraper

Last synced: 30 Apr 2025

https://github.com/phantominsights/mexican-jobs-2020

Data ETL & Analysis on thousands of job listings from the official Mexican job board (2020 edition).

hacktoberfest job-offers lxml pandas plotly python3 selenium web-scraper

Last synced: 22 Jun 2025

https://github.com/azogue/esiosdata

Web Scraper para datos de demanda, producción y coste de la energía eléctrica en España, y simulador de facturación eléctrica según el PVPC

energy energy-monitor esios python-3 scraper web-scraper

Last synced: 07 May 2025

https://github.com/shobrook/git-pull

Parallelized web scraper for Github

github github-api github-scraper parallel scraper web-scraper

Last synced: 23 Mar 2025

https://github.com/PhantomInsights/tweet-transcriber

A Reddit bot that transcribes tweets from comments and submissions links, mirrors their images and replies back with a formatted Markdown message.

beautifulsoup imgur praw python3 reddit-bot web-scraper

Last synced: 02 May 2025

https://github.com/phantominsights/tweet-transcriber

A Reddit bot that transcribes tweets from comments and submissions links, mirrors their images and replies back with a formatted Markdown message.

beautifulsoup imgur praw python3 reddit-bot web-scraper

Last synced: 29 Apr 2025

https://github.com/henr1ko/pixthief

Stealthy .NET 8 console tool that crawls pages or whole domains and downloads images with optional format conversion.

console-application crawler csharp dotnet http-client image-downloader web-scraper web-scraping webscraper win-x64 windows

Last synced: 15 May 2026

https://github.com/drlordbasil/aurora

AURORA (Artificial Unified Responsive Optimized Reasoning Agent) uses lobes and web research for RAG based memory and learning.

ai chatbot flask groq gui human-computer-interaction llm ml nn ollama ollama-python openai web-scraper

Last synced: 12 May 2025

https://github.com/developerjosh/nekonode-site

Watch high-quality, ad-free anime streaming on NekoNode – your ultimate anime destination!

anime anime-api anime-scraper anime-streaming api nextjs streaming-video web-scraper website

Last synced: 22 Apr 2025

https://github.com/breadrock1/socialnetworkscraper

Web scraping is simply the process of using a social media web scraper to gather data automatically. It saves users time, effort and sometimes money since it’s an automatic process performed by bots. You could take the time to search the web for all mentions of a certain word or find all prices for a certain product, but that would take a lot of time.

facebook facebook-scraping flake8 mailru osint osint-python python python3 scraper scraping site-scraper social-network social-network-analysis twitter vk-api vkontakte web-scraper web-scraping

Last synced: 30 Jul 2025