Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

Projects in Awesome Lists tagged with scraper-api

A curated list of projects in awesome lists tagged with scraper-api .

https://github.com/oxylabs/google-news-scraper

Use Google News API to obtain the latest global news for your project, including a wide range of sources, headlines, URLs, and publication dates from the Google News platform.

api-for-google-news api-google-news api-stock-news free-google-news-api google-news-api google-news-scraper google-scraper google-scraper-api google-search-scraper news-scraper news-scraper-api scraper-api

Last synced: 14 May 2025

https://github.com/oxylabs/how-to-scrape-google-scholar

A guide for extracting titles, authors, and citations from Google Scholar using Python and Oxylabs SERP Scraper API.

google-scholar google-scholar-scraper google-scholar-scrapper google-search-scraper python python-scraper scraper-api web-scraper web-scraping

Last synced: 15 May 2025

https://github.com/oxylabs/quick-start-guide

Python quick start guides to get the most out of Oxylabs' Web Scraper API free trial.

oxylabs scraper scraper-api scraper-python scrapers scraping scraping-websites web-scraper web-scraping

Last synced: 06 Jul 2025

https://github.com/oxylabs/how-to-scrape-amazon-prices

A code for extracting best-selling items, search results, and currently available deals from Amazon using Python and Oxylabs E-Commerce Scraper API.

amazon amazon-scraper api python python-scraper scraper-api web-scraper web-scraping

Last synced: 16 May 2025

https://github.com/zfc-digital/cf-clearance-scraper

This library was created for testing and training purposes to retrieve the page source of websites, create Cloudflare Turnstile tokens and create Cloudflare WAF sessions.

bypass cf-clearance cloudflare cloudflare-scrape cloudflare-scraper scraper-api

Last synced: 23 Oct 2025

https://github.com/oxylabs/how-to-bypass-amazon-captcha

The process of bypassing CAPTCHA when extracting public data from Amazon with Oxylabs Amazon Scraper API.

amazon amazon-scraper amazon-scraper-api captcha captcha-solver captcha-solving scraper-api

Last synced: 16 May 2025

https://github.com/oxylabs/how-to-scrape-google-jobs

Learn how to build your own Google Jobs scraper that simultaneously scrapes Google Jobs for multiple search queries and geo-locations with Python and Oxylabs’ Google Jobs Scraper API. https://oxylabs.io/blog/how-to-scrape-google-jobs

google-scraping google-search-api google-search-scraper python scraper scraper-api scraper-python serp-scraper-api serp-scraping web-scraping

Last synced: 24 Oct 2025

https://github.com/oxylabs/how-to-scrape-indeed

A tutorial for collecting job postings from Indeed using Python and Oxylabs Web Scraper API.

api job-posting python scraper-api web-scraper web-scraping

Last synced: 05 Apr 2025

https://github.com/oxylabs/how-to-scrape-google-trends

Learn step-by-step how to scrape Google Trends data and make a result comparison using Python and Oxylabs SERP API. Extract keywords, their popularity, breakdown by region, related queries, and more.

google google-trend google-trend-api google-trend-strategy google-trends google-trends-api google-trends-crawler google-trends-scraper python scraper scraper-api scraping scraping-api web-scraping

Last synced: 14 Apr 2025

https://github.com/danielwte/ebay-kleinanzeigen-api

This tool acts as an API interface, you can use it to get random listings from Ebay-Kleinanzeigen.de, exact data from specific listings and even views of listings.

api docker ebay-api ebay-kleinanzeigen nginx python scraper scraper-api

Last synced: 07 Oct 2025

https://github.com/oxylabs/scrape-google-python

In this tutorial, we showcase how to scrape public Google data with Python and Oxylabs API.

google python python-scraping python-web-scraper scrape scrape-google scraper-api scraping scraping-api web-scraping

Last synced: 21 Sep 2025

https://github.com/flulemon/sneakpeek

Sneakpeek is a framework that helps to quickly and conviniently develop scrapers. It’s the best choice for scrapers that have some specific complex scraping logic that needs to be run on a constant basis

crawler crawler-python crawlers crawling crawling-engine crawling-framework python python3 scraper scraper-api scraper-engine scrapers scraping scraping-framework vue website-crawler

Last synced: 14 Jan 2026

https://github.com/beautifulmoon211/yelp-scraping

Web scraping tool used to record business addresses, phone numbers, website, supported area and other relevant information of companies from Yelp.com

beautifulsoup4 playwright python scraper scraper-api selenium webscraping yelp yelp-scraper yelp-scraping

Last synced: 28 Oct 2025

https://github.com/fongsidev/scraper

@fongsidev/scraper is a library that allows you to download various types of content from various online platforms such as TikTok, Instagram, Capcut, Google Drive, and many more and also provides very useful tools/codes :).

client-server developer-tools discord-js downloader downloader-media-social free-scraper google-api remove-background rest-api rest-api-developer rest-api-scraper scraper scraper-api whatsapp-api whatsapp-scraper

Last synced: 17 Mar 2025

https://github.com/slavyandesu/nekobocc

A simple and lightweight Nekopoi scraper.

anime-scraper hentai nekopoi nekopoi-scraper scraper scraper-api

Last synced: 22 Mar 2025

https://github.com/saluana/ezai-web-scraper

An API that allows you to scrape blog posts and articles and get a list of notes or a summary back.

ai gpt-3 node node-js notes scraper scraper-api summarization web

Last synced: 05 Mar 2025

https://github.com/luminati-io/ecommerce-dataset-samples

A collection of multiple e-commerce dataset samples. Each sample contains over 1,000 records. These datasets are ideal for product trend analysis, pricing strategies, consumer sentiment insights, and more.

amazon amazon-api amazon-scraper datasets ecommerce-data ecommerce-scraper free-datasets products-data scraper-api shein shopee shopee-api walmart web-scraping

Last synced: 25 Jun 2025

https://github.com/botcahx/tiktokdl

Library for downloading videos from TikTok (without watermark)

latest-version nodejs npmjs-packages scraper-api tikdl tiktok-api tiktok-nowatermark tiktokdl

Last synced: 11 Apr 2025

https://github.com/beautifulmoon211/redfin-scraping

Web scraping tool used to get price, number of beds, number of baths, space, address, website link, and phone number - all attributes of real estates from Redfin.com

javascript redfin redfin-scraper requests scraper scraper-api

Last synced: 08 Oct 2025

https://github.com/oxylabs/scraping-real-estate-data-with-python

A guide for extracting public data from Redfin using Python and Oxylabs Web Scraper API. From using the API to processing data and saving the result.

e-commerce-scraper ecommerce-api ecommerce-scraper python python-scraper real-estate scraper-api web-scraping

Last synced: 09 Oct 2025

https://github.com/dgnsrekt/yfs

Yahoo Finance Scraper with a WYSIWYG approach.

finance scraper scraper-api yahoo

Last synced: 01 Apr 2025

https://github.com/neelanjan-chakraborty/kobold-ai-chat-scraper-and-console

🤖💬 Communicate with the Kobold AI website using the Kobold AI Chat Scraper and Console! 🚀 Open-source and easy to configure, this app lets you chat with Kobold AI's server locally or on Colab version. 🌐 Set up the bot, copy the URL, and you're good to go! 🤩 Plus, stay tuned for future plans like a FrontEnd GUI and website interface. 💻🎨🌐

aibot chatbot chatbot-application chatbots colab-notebook kobold koboldai scraper scraper-api tavern-bot tavernai

Last synced: 28 Sep 2025

https://github.com/oxylabs/how-to-scrape-airbnb

Learn how to scrape Airbnb search results and listings on a large scale without blocks.

airbnb-api airbnb-data airbnb-listings airbnb-prices airbnb-scraper airbnb-scraping scrape-airbnb scraper-api

Last synced: 27 Sep 2025

https://github.com/jwc20/waasuapi

A scraper API for fetching available jobs from YCombinator's workatastartup.com

beautifulsoup jobsearch python scraper scraper-api selenium

Last synced: 28 Jan 2026

https://github.com/chaosgen/google-news-scraper

Use Google News API to obtain the latest global news for your project, including a wide range of sources, headlines, URLs, and publication dates from the Google News platform.

api-for-google-news api-google-news api-stock-news free-google-news-api google-news-api google-news-scraper google-scraper google-scraper-api google-search-scraper news-scraper news-scraper-api python scraper-api

Last synced: 06 May 2025

https://github.com/deventerprisesoftware/scrapi-mcp

MCP server for using ScrAPI to scrape web pages.

ai mcp mcp-server scraper-api web-scraping webscraper

Last synced: 16 Jan 2026

https://github.com/njraladdin/newspapers-com-scraper

A Node.js scraper for extracting article data from Newspapers.com based on keywords, dates, and locations.

archive data newspapers scraper scraper-api scraping

Last synced: 06 Apr 2025

https://github.com/nomansiddiqui0000/rozee.pk-jobs-scrapper

This scraper, built in Node.js using Puppeteer and Cheerio, is designed to extract job listings from the Rozee.pk website. It can scrape multiple pages and gather detailed information, including job titles, company names, skills, and more. The output is saved in structured CSV files, with sample datasets for cities like Lahore, Karachi, etc.

automation cheerio javascript jobs jobs-scraping jobs-search jobscraper nodejs puppetter scra scraped-data scraper-api scraping scraping-websites scrapy-crawler

Last synced: 27 Oct 2025

https://github.com/luminati-io/twitter-scraper

Twitter (X) Scraper provides a free scraper for small-scale projects and an enterprise-grade API for large-scale Twitter data extraction , with built-in proxy rotation and CAPTCHA solving.

captcha-solver proxy-server scraper-api twitter twitter-scraper web-scraper web-scraping x x-scraper

Last synced: 10 Feb 2026

https://github.com/selvin-paul-raj/own_amazon_scraper_api

An open-source web scraper for Amazon data. Retrieve product details, reviews, offers, and search results using this Node.js and Express-based API. Ideal for integrating Amazon data into your applications and analysis.

amazon amazon-api expressjs nodejs rapid-development rapidapi scraper-api

Last synced: 30 Dec 2025

https://github.com/pin705/cf-scraper-bypass

A simple tool to scrape Cloudflare clearance cookies (cf_clearance) from websites with Cloudflare challenges.

bypass cloudflare cloudflare-bypass nodejs recaptcha scraper-api

Last synced: 04 Mar 2025

https://github.com/thiti-dev/scraperor-v2

Made scraping easy by the human-understandable $pointer to look for the interested data

dotnet-api dotnet-core dotnet7 htmlagilitypack scraper-api

Last synced: 03 Jul 2025

https://github.com/oxylabs/best-buy-price-tracker

A tutorial for building a scalable price tracker with Python and Oxylabs Best Buy Scraper API to get price change alerts and historical data.

change-monitoring price-tracker price-tracking-system scraper-api web-scraper web-scraping

Last synced: 25 Dec 2025

https://github.com/basemax/kashan-university-phone-directory

This repository contains a scraper and dataset for extracting and publishing the phone directory of employees and other personnel from the University of Kashan. It includes tools to scrape, parse, and export data from a given HTML file into JSON format.

crawler crawlers database html-scraper json kashan kashan-university scraper scraper-api scraper-html scrapers university university-of-kashan

Last synced: 24 Oct 2025

https://github.com/luminati-io/python-web-scraping

Comprehensive Python Web Scraping guide covering libraries, methods, and examples using BeautifulSoup, Selenium, Scrapy, and API solutions.

aiohttp beautifulsoup httpx lxml pyppeteer python python-scraper requests requests-library-python scraper-api scrapy web-scraper web-scraping web-scraping-api

Last synced: 25 Jun 2025

https://github.com/oxylabs/walmart-price-tracker

Tutorial for creating a Walmart price tracker using Python and Oxylabs Walmart Scraper API.

scraper-api walmart walmart-api walmart-scraper web-scraping

Last synced: 08 Dec 2025

https://github.com/kerosiinikone/serverless-scraper

Serverless Problem Scraper with Terraform (AWS Lambda, S3, SQS)

aws-dynamodb aws-lambda aws-s3 aws-sdk golang scraper-api terraform

Last synced: 14 Jun 2025

https://github.com/neotoxic-off/juice

🧃 Scrap the world

api scraper scraper-api

Last synced: 26 Jan 2026

https://github.com/notthestallion/eve-online_deals_finder-web-scraping-

This repository will have different deal finders for EVE Online, offering players ways to discover in-game opportunities like finding rare items, resources and trade routes. With multiple options available, players can optimize their gameplay and achieve their goals.

game game-development games scrape scraper scraper-api scrapers scraping scraping-api scraping-python scraping-web scraping-websites scrapy

Last synced: 02 Mar 2025

https://github.com/vishnuprasad-code/cf_clearance_collector

A simple RESTful API that returns the response cookies after passing cloudflare human verificaton.

python scraper-api

Last synced: 10 Apr 2025

https://github.com/luminati-io/google-hotels-api

Scrape real-time hotel data from Google using a free scraper or Bright Data’s Google Hotels API for large-scale, reliable data collection.

google-hotels hotels-api hotels-scraper html python scraper-api serp-api web-scraper

Last synced: 25 Jun 2025

https://github.com/yinyanfr/dmhy-get-api-unofficial

Yet another DMHY scraper

scraper-api

Last synced: 29 Mar 2025

https://github.com/luminati-io/google-trends-api

Scrape Google Trends data using Python or the Bright Data API. Get real-time search trends, filter by location, category, and time range, and scale effortlessly.

google google-scraper google-trends google-trends-api google-trends-scraper google-trends-scraping-api playwright python scraper-api web-scraper

Last synced: 03 Mar 2025

https://github.com/subhanalii/instagram-scraper

A Python automation tool that logs into Instagram, searches profiles via Bing, scrapes public data like bio, followers, and emails, and saves the results. Demo included. Full script available on request.

automation data-collection data-extraction lead-generation python scraper scraper-api web-scraping-python webscraping webscraping-beautifulsoup webscraping-selenium

Last synced: 18 Oct 2025

https://github.com/luminati-io/zillow-scraper

A Zillow scraper offering a free tool for small-scale data collection and an enterprise-grade API for large-scale, reliable property data extraction.

free-scraper no-code python scraper-api web-scraping zillow zillow-scraper

Last synced: 10 Sep 2025

https://github.com/jwc20/fcisapi

A scraper API for fetching workplace fatality and catastrophe reports from the United States Department of Labor website.

accidents-analysis beautifulsoup beautifulsoup4 fatality labor python requests scraper scraper-api workplace

Last synced: 20 Mar 2025

https://github.com/sora1902/clearance

# 🧹 ClearanceClearance is a tool designed to simplify your workflow. It integrates seamlessly with Go, Docker, and various package managers, making development smoother. 🐙

anti-bot-page async cf-clearance clearance cloudflare cloudflare-bypass cloudflare-reverse cloudflare-scrape docker i18n playwright playwright-python rails-authentication reversed rubygem scraper scraper-api v2-challenge

Last synced: 15 Jun 2025

https://github.com/luminati-io/google-news-scraper

Extract news data with the #1 Google News API, including top stories, publisher details, article summaries, and trending topics. Start your free trial now!

api datasets google-news google-news-api google-news-scraper python scraper-api scraper-python web-scraper

Last synced: 07 Apr 2025

https://github.com/luminati-io/google-reviews-api

A powerful solution for collecting Google Reviews data. Choose between a free scraper for small-scale projects or Bright Data’s enterprise API for large-scale extraction.

google-api google-reviews google-reviews-api google-scraper playwright python scraper-api web-scraper web-scraping

Last synced: 18 Jul 2025

https://github.com/oxylabs/browser-instructions

Browser instructions enable users to execute various Oxylabs' headless browser actions, like clicking, inputting, scrolling, waiting, and fetching resources.

browser-action browser-api browser-automation scraper scraper-api web-scraping

Last synced: 14 Sep 2025

https://github.com/luminati-io/scraping-hotels-google-travel

Scraping hotel data from Google Travel using Selenium or Bright Data's APIs, with practical examples and code snippets

aiohttp datasets google-hotels google-travel proxies proxy-server python scraper-api selenium web-scraper

Last synced: 03 Oct 2025

https://github.com/notfaad/msl-engine

a powerful web scraping engine that uses a custom Domain Specific Language (DSL) to define scraping pipelines.

engine rust rust-lang scraper-api scraper-engine scraping-websites webscraping

Last synced: 23 Sep 2025

https://github.com/oxylabs/airbnb-scraper

Extract Airbnb data like listings, search results, reviews, and host pages with ease using Airbnb Scraper.

airbnb airbnb-api airbnb-data airbnb-scraper booking-api email-crawler scraper scraper-api scraper-python

Last synced: 29 Jul 2025

https://github.com/nycolop/web-scraper-api

A web scraper api that it's a microservice

axios cheerio ecmascript6 expressjs nodejs scrape-websites scraper scraper-api

Last synced: 16 Mar 2025

https://github.com/mineshmelvin/web-scraper-amazon-scala

Web Scraper written in Scala to get product details from Amazon

amazon aws rest-api scala scraper scraper-api

Last synced: 27 Nov 2025

https://github.com/datacollectionspecialist/how-to-scrape-flight-data-from-kayak

Learn how to scrape flight data from Kayak using powerful web scraping tools. This guide provides step-by-step instructions to help you collect accurate and up-to-date flight information for travel analysis, price comparison, and more.

scrape scrapeddata scrapegoogle scraper-api scraper-python

Last synced: 11 Mar 2025

https://github.com/peeepca/skrapster

Scrape tool, so far it can scrape emails via restAPI. Development ll continue.

python python3 scrape scraper scraper-api

Last synced: 26 Feb 2025