Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

Projects in Awesome Lists tagged with web-scraping

A curated list of projects in awesome lists tagged with web-scraping .

https://github.com/scrapy/scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.

crawler crawling framework hacktoberfest python scraping web-scraping web-scraping-python

Last synced: 30 Dec 2024

https://github.com/dgtlmoon/changedetection.io

The best and simplest free open source web page change detection, website watcher, restock monitor and notification service. Restock Monitor, change detection. Designed for simplicity - Simply monitor which websites had a text change for free. Free Open source web page change detection, Website defacement monitoring, Price change notification

back-in-stock change-alert change-detection change-monitoring changedetection monitoring notifications restock-monitor self-hosted url-monitor web-scraping website-change-detection website-change-detector website-change-monitor website-change-notification website-change-tracker website-defacement-monitoring website-monitor website-monitoring website-watcher

Last synced: 30 Dec 2024

https://github.com/apify/crawlee

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

apify automation crawler crawling headless headless-chrome javascript nodejs npm playwright puppeteer scraper scraping typescript web-crawler web-crawling web-scraping

Last synced: 30 Dec 2024

https://github.com/evil0ctal/douyin_tiktok_download_api

🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、TikTok、Bilibili数据爬取工具，支持API调用，在线批量解析及下载。

api async crawler douyin douyin-api douyin-scraper douyin-tiktok-api douyin-tiktok-download fastapi no-watermark online-parsing python pywebio scraper spider tiktok tiktok-api tiktok-scraper tiktok-signature web-scraping

Last synced: 30 Dec 2024

https://github.com/Evil0ctal/Douyin_TikTok_Download_API

🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、TikTok、Bilibili数据爬取工具，支持API调用，在线批量解析及下载。

api async crawler douyin douyin-api douyin-scraper douyin-tiktok-api douyin-tiktok-download fastapi no-watermark online-parsing python pywebio scraper spider tiktok tiktok-api tiktok-scraper tiktok-signature web-scraping

Last synced: 29 Oct 2024

https://github.com/getmaxun/maxun

🔥 Open-source no-code web data extraction platform. Turn websites to APIs and spreadsheets with no-code robots in minutes! [In Beta]

agents api automation browser browser-automation data-extraction no-code no-code-web-scraper playwright robotic-process-automation rpa scraper self-hosted web-agent web-automation web-scraper web-scraping web-scraping-agent webscraping website-to-api

Last synced: 02 Jan 2025

https://github.com/seleniumbase/SeleniumBase

📊 Python's all-in-one framework for web crawling, scraping, testing, and reporting. Supports pytest. UC Mode provides stealth. Includes many tools.

anti-detection behave bot-detection chromedriver cloudflare-bypass e2e-testing pytest pytest-plugin python python-scraper selenium selenium-python seleniumbase test-automation unittests web-automation web-scraping web-scraping-python webdriver webkit

Last synced: 29 Oct 2024

https://github.com/seleniumbase/seleniumbase

📊 Python's all-in-one framework for web crawling, scraping, testing, and reporting. Supports pytest. UC Mode provides stealth. Includes many tools.

anti-detection behave bot-detection chromedriver cloudflare-bypass e2e-testing pytest pytest-plugin python python-scraper selenium selenium-python seleniumbase test-automation unittests web-automation web-scraping web-scraping-python webdriver webkit

Last synced: 30 Dec 2024

https://github.com/apify/crawlee-python

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

apify automation beautifulsoup crawler crawling hacktoberfest headless headless-chrome pip playwright python scraper scraping web-crawler web-crawling web-scraping

Last synced: 30 Dec 2024

https://github.com/snooppr/snoop

Snoop — инструмент разведки на основе открытых данных (OSINT world)

blueteam ctf geo geocoder infosec ip nickname osint parser pentest police redteam scanner scraping security termux username username-checker username-search web-scraping

Last synced: 01 Jan 2025

https://github.com/lexiforest/curl_cffi

Python binding for curl-impersonate fork via cffi. A http client that can impersonate browser tls/ja3/http2 fingerprints.

akamai-fingerprint curl curl-impersonate fingerprinting http http-client http2-fingerprint https ja3 ja3-fingerprint tls-fingerprint web-scraping

Last synced: 31 Dec 2024

https://github.com/tidyverse/rvest

Simple web scraping for R

html r web-scraping

Last synced: 31 Dec 2024

https://github.com/roach-php/core

The complete web scraping toolkit for PHP.

crawling php web-scraping

Last synced: 31 Dec 2024

https://github.com/a9t9/rpa

UI.Vision: Open-Source RPA Software (formerly Kantu) - Modern Robotic Process Automation with Selenium IDE++

autohotkey automation browser-automation browser-extension data-driven-tests imacros opencv selenium-ide sikulix ui-tests uipath visual-recognition web-automation web-scraping webassembly

Last synced: 05 Jan 2025

https://github.com/A9T9/RPA

UI.Vision: Open-Source RPA Software (formerly Kantu) - Modern Robotic Process Automation with Selenium IDE++

autohotkey automation browser-automation browser-extension data-driven-tests imacros opencv selenium-ide sikulix ui-tests uipath visual-recognition web-automation web-scraping webassembly

Last synced: 28 Oct 2024

https://github.com/rushter/selectolax

Python binding to Modest and Lexbor engines (fast HTML5 parser with CSS selectors).

css html5 modest-engine parser python web-scraping

Last synced: 01 Jan 2025

https://github.com/intoli/user-agents

A JavaScript library for generating random user agents with data that's updated daily.

browser-automation browsers javascript navigator random randomization user-agent user-agent-spoofer web-scraping

Last synced: 04 Jan 2025

https://github.com/gosom/google-maps-scraper

scrape data data from Google Maps. Extracts data such as the name, address, phone number, website URL, rating, reviews number, latitude and longitude, reviews,email and more for each place

distributed-scraper distributed-scraping golang google-maps google-maps-scraping web-scraper web-scraping

Last synced: 05 Nov 2024

https://github.com/postmodern/spidr

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

crawler ruby scraper spider spider-links web web-crawler web-scraper web-scraping web-spider

Last synced: 02 Jan 2025

https://github.com/DataHenHQ/till

DataHen Till is a companion tool to your existing web scraper that instantly makes it scalable, maintainable, and more unblockable, with minimal code changes on your scraper. Integrates with any scraper in 5 minutes.

crawler man-in-the-middle mitm proxy-server scraper scraping web-scraping

Last synced: 26 Oct 2024

https://github.com/je-suis-tm/web-scraping

Detailed web scraping tutorials for dummies with financial data crawlers on Reddit WallStreetBets, CME (both options and futures), US Treasury, CFTC, LME, MacroTrends, SHFE and alternative data crawlers on Tomtom, BBC, Wall Street Journal, Al Jazeera, Reuters, Financial Times, Bloomberg, CNN, Fortune, The Economist

bloomberg data-scraper data-scraping financial-data financial-times futures futures-historical-data news-scraper news-websites newsletter options-data python-web-scraper reuters scrapper sraping wall-street-journal wallstreetbets web-scraper web-scrapers web-scraping

Last synced: 03 Jan 2025

https://github.com/alecxe/scrapy-fake-useragent

Random User-Agent middleware based on fake-useragent

python scrapy web-scraping

Last synced: 03 Jan 2025

https://github.com/gildas-lormeau/single-file-cli

CLI tool for saving a faithful copy of a complete web page in a single HTML file (based on SingleFile)

archiving cli crawler deno dockerfile nodejs scraping-websites single-file web-archiving web-crawler web-scraper web-scraping

Last synced: 03 Jan 2025

https://github.com/dinubs/coolqlcool

Nextjs server to query websites with GraphQL

graphql javascript nextjs schema web-scraping

Last synced: 04 Jan 2025

https://github.com/oxylabs/quick-start-guide

Python quick start guides to get the most out of Oxylabs' Web Scraper API free trial.

oxylabs scraper scraper-api scraper-python scrapers scraping scraping-websites web-scraper web-scraping

Last synced: 17 Nov 2024

https://github.com/z0m31en7/uscrapper

Uscrapper Vanta: Dive deeper into the web with this powerful open-source tool. Extract valuable insights with ease and efficiency, from both surface and deep web sources. Empower your data mining and analysis with Vanta's advanced capabilities. Fast, reliable, and user-friendly, Uscrapper Vanta is the ultimate choice for researchers and analysts.

darkweb darkweb-crawler information-extraction information-gathering osint osint-python osint-tool python reconnaissance selenium selenium-webscraper tor web-scraping webcra webcrawler webscraping website-scraper websites

Last synced: 04 Jan 2025

https://github.com/z0m31en7/Uscrapper

Uscrapper Vanta: Dive deeper into the web with this powerful open-source tool. Extract valuable insights with ease and efficiency, from both surface and deep web sources. Empower your data mining and analysis with Vanta's advanced capabilities. Fast, reliable, and user-friendly, Uscrapper Vanta is the ultimate choice for researchers and analysts.

darkweb darkweb-crawler information-extraction information-gathering osint osint-python osint-tool python reconnaissance selenium selenium-webscraper tor web-scraping webcra webcrawler webscraping website-scraper websites

Last synced: 13 Nov 2024

https://github.com/austinoboyle/scrape-linkedin-selenium

`scrape_linkedin` is a python package that allows you to scrape personal LinkedIn profiles & company pages - turning the data into structured json.

linkedin python scrape scraper scraping selenium selenium-webdriver web-scraper web-scraping

Last synced: 04 Jan 2025

https://github.com/rebrowser/rebrowser-patches

Collection of patches for puppeteer and playwright to avoid automation detection and leaks. Helps to avoid Cloudflare and DataDome CAPTCHA pages. Easy to patch/unpatch, can be enabled/disabled on demand.

automation bot bot-detection chrome chromedriver cloudflare crawler crawling datadome headless headless-chrome playwright puppeteer puppeteer-extra rebrowser scraping selenium stealth web-scraping webdriver

Last synced: 04 Jan 2025

https://github.com/sangaline/wayback-machine-scraper

A command-line utility and Scrapy middleware for scraping time series data from Archive.org's Wayback Machine.

archive-dot-org command-line-tool python wayback-archiver wayback-machine web-scraping

Last synced: 01 Jan 2025

https://github.com/roniemartinez/dude

dude uncomplicated data extraction: A simple framework for writing web scrapers using Python decorators

async beautifulsoup4 crawler css framework lxml parsel playwright python scraper scraping selenium sync web-scraping webscraping xpath

Last synced: 03 Jan 2025

https://github.com/platonai/PulsarRPA

Automate webpages at scale, scrape web data completely and accurately with high performance, distributed RPA.

crawler data-mining data-science rpa scraper scraping web-automation web-crawler web-mining web-scraping web-sql

Last synced: 05 Nov 2024

https://github.com/yusuzech/r-web-scraping-cheat-sheet

Guide, reference and cheatsheet on web scraping using rvest, httr and Rselenium.

cheatsheet httr r rselenium rvest scrape-websites web-scraping webscraping

Last synced: 09 Nov 2024

https://github.com/tinyfish-io/agentql

AgentQL is an AI-powered query language for web scraping and automation. It uses natural language selectors to find data on any page, including authenticated content. AgentQL queries are self-healing as UI changes and work across similar sites. Users can define structured data output, making AgentQL versatile for developers and data scientists.

automation playwright python scraping web-scraping web-scraping-python web-scrapping

Last synced: 01 Jan 2025

https://github.com/oxylabs/how-to-scrape-google-scholar

A guide for extracting titles, authors, and citations from Google Scholar using Python and Oxylabs SERP Scraper API.

google-scholar google-scholar-scraper google-scholar-scrapper google-search-scraper python python-scraper scraper-api web-scraper web-scraping

Last synced: 30 Dec 2024

https://github.com/City-Bureau/city-scrapers

Scrape, standardize and share public meetings from local government websites

city-scrapers open-data python scrapy web-scraping

Last synced: 06 Nov 2024

https://github.com/serpapi/nokolexbor

High-performance HTML5 parser for Ruby based on Lexbor, with support for both CSS selectors and XPath.

c-extension css html5 parser ruby serpapi web-scraping xpath

Last synced: 31 Dec 2024

https://github.com/infinitbyte/gopa

🕷️ An easy-to-use spider written in Golang. (previous named GOPA.)

crawler crawling elasticsearch lightweight scraping spider web-crawler web-scraping web-spider

Last synced: 14 Dec 2024

https://github.com/infinilabs/crawler

🕷️ An easy-to-use spider written in Golang. (previous named GOPA.)

crawler crawling elasticsearch lightweight scraping spider web-crawler web-scraping web-spider

Last synced: 03 Jan 2025

https://github.com/walissonsilva/web-scraping-python

🌐 Repositório com o conteúdo (slides, exemplos, códigos) da série de vídeos no YouTube sobre Web Scraping com Python.

beautifulsoup python requests selenium web-scraping

Last synced: 31 Oct 2024

https://github.com/oxylabs/python-web-scraping-tutorial

In this Python Web Scraping Tutorial, we will outline everything needed to get started with web scraping. We will begin with simple examples and move on to relatively more complex.

amazon-scraper-python crawler github-python json-database-python python python-projects python-web-crawler python-web-scraper scraper-python scraping web-crawler-python web-scraping web-scraping-api web-scraping-python webscraping

Last synced: 29 Dec 2024

https://github.com/amoudgl/short-jokes-dataset

Python scripts for building 'Short Jokes' dataset, featured on Kaggle

beautiful-soup dataset humor jokes oneliners python scrapers web-scraping

Last synced: 04 Nov 2024

https://github.com/vdutts7/gpt4v-scraper

AI agent that can SEE 👁️, control, navigate, & do stuff for you on your browser.

ai-agents browser-automation gpt-4-vision puppeteer web-scraping

Last synced: 01 Jan 2025

https://github.com/tuhinpal/imdb-api

Serverless IMDB API powered by Cloudflare Worker

cloudflare-worker cloudflare-workers hono honojs imdb imdb-api movie-list web-scraping

Last synced: 20 Nov 2024

https://github.com/roach-php/laravel

Laravel adapter for Roach, the complete web scraping toolkit for PHP.

crawling laravel php web-scraping

Last synced: 30 Dec 2024

https://github.com/justmarkham/trump-lies

Tutorial: Web scraping in Python with Beautiful Soup

beautiful-soup data-science dataset pandas python requests tutorial web-scraping

Last synced: 31 Dec 2024

https://github.com/vdutts7/gpt4V-scraper

AI agent that can SEE 👁️, control, navigate, & do stuff for you on your browser.

ai-agents browser-automation gpt-4-vision puppeteer web-scraping

Last synced: 05 Nov 2024

https://github.com/dirkjbreeuwer/gpt-automated-web-scraper

The GPT-based Universal Web Scraper MVP is a solution that leverages GPT models and web scraping libraries to generate scraper code based on user input and website analysis, simplifying the web scraping process.

automated-web-scraper gpt gpt-based-scraper web-scraping

Last synced: 06 Nov 2024

https://github.com/schasins/helena

A Chrome extension for writing custom web scraping programs and web automation programs. Just demonstrate how to collect the first row of data, then let the extension write the program for collecting all rows.

chrome-extension javascript programming-by-example synthesis web-scraping

Last synced: 16 Nov 2024

https://github.com/n0kovo/fb_friend_list_scraper

OSINT tool to scrape names and usernames from large friend lists on Facebook, without being rate limited.

facebook facebook-bot osint python scraper selenium-webdriver web-automation web-scraping

Last synced: 02 Jan 2025

https://github.com/nazuke/SEOMacroscope

SEO Macroscope is a website scanning tool, to check your website for broken links; including some technical SEO functionality, site scraping, Excel reporting, and more.

broken-links custom-filter duplicate-content extract-pdf-metadata extraction hreflang-checker hreflang-matrix link-checker scan-website seo seo-excel-report seo-macroscope seo-tools web-scraping webmaster

Last synced: 08 Nov 2024

https://github.com/germey/awesomewebscraping

List of libraries, tools and APIs for web scraping and data processing.

aswsome-list awesome captcha golang javascript php proxy python ruby web-scraping

Last synced: 10 Oct 2024

https://github.com/evyatarmeged/humanoid

Node.js package to bypass CloudFlare's anti-bot JavaScript challenges

anti-bot anti-bot-page bot bypass bypass-waf humanoid scrape scraping web-scraping

Last synced: 30 Oct 2024

https://github.com/ayakashi-io/ayakashi

:zap: Ayakashi.io - The next generation web scraping framework

automation data-mining headless-chrome web-crawling web-scraping

Last synced: 07 Nov 2024

https://github.com/galihap76/collector

Collector is a OSINT tool and information gathering. This tool can do information gathering phone numbers, github account, ip address and instagram account.

github-osint information-gathering instagram-osint open-source-intelligence osint osint-ip osint-phonenumbers osint-python osint-tool python3 web-scraping

Last synced: 20 Nov 2024

https://github.com/semanser/jsongenius

Get structured JSON data from any page.

api go golang gpt gpt-3 gpt-4 scraper scraping web-scraping

Last synced: 06 Dec 2024

https://github.com/scrapinghub/scrapy-training

Scrapy Training companion code

python scrapy training web-crawling web-scraping

Last synced: 10 Nov 2024

https://github.com/amerkurev/scrapper

Web scraper with a simple REST API living in Docker and using a Headless browser and Readability.js for parsing.

crawler crawling docker headless readability scraper web-parsers web-parsing web-scraping

Last synced: 04 Jan 2025

https://github.com/wikimedia/html-metadata

MetaData html scraper and parser for Node.js (supports Promises and callback style)

javascript metadata-extraction metadata-extractor node-module nodejs web-scraper web-scraping

Last synced: 04 Jan 2025

https://github.com/algolisted-org/algolisted

Algolisted is an AI-powered platform dedicated to assisting computer science students in preparing for placements and internships. Our services include tracking and analytics across various platforms and topics.

ai css firebase hacktoberfest-2023 javascript mern-stack ml nlp python3 react-js web-scraping

Last synced: 29 Dec 2024