Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

Projects in Awesome Lists tagged with scrapper

A curated list of projects in awesome lists tagged with scrapper .

https://github.com/je-suis-tm/web-scraping

Detailed web scraping tutorials for dummies with financial data crawlers on Reddit WallStreetBets, CME (both options and futures), US Treasury, CFTC, LME, MacroTrends, SHFE and alternative data crawlers on Tomtom, BBC, Wall Street Journal, Al Jazeera, Reuters, Financial Times, Bloomberg, CNN, Fortune, The Economist

bloomberg data-scraper data-scraping financial-data financial-times futures futures-historical-data news-scraper news-websites newsletter options-data python-web-scraper reuters scrapper sraping wall-street-journal wallstreetbets web-scraper web-scrapers web-scraping

Last synced: 04 Apr 2025

https://github.com/xonshiz/comic-dl

Comic-dl is a command line tool to download manga and comics from various comic and manga sites. Supported sites : readcomiconline.to, mangafox.me, comic naver and many more.

automation comics debian manga phantomjs python-script python-support scraping scrapper scrapping web youtube-dl

Last synced: 16 May 2025

https://github.com/Xonshiz/comic-dl

Comic-dl is a command line tool to download manga and comics from various comic and manga sites. Supported sites : readcomiconline.to, mangafox.me, comic naver and many more.

automation comics debian manga phantomjs python-script python-support scraping scrapper scrapping web youtube-dl

Last synced: 26 Mar 2025

https://github.com/iw4p/proxy-scraper

scrape proxies from more than 5 different sources and check which ones are still alive

checker grabber http https-proxy proxies proxy proxy-checker proxy-list proxy-scraper proxychecker scraper scrapper socks socks-proxy socks4 socks5 socks5-proxy

Last synced: 08 Apr 2025

https://github.com/baptisteArno/tinking

🧶 Extract data from any website without code, just clicks.

harvesting puppeteer scraping scraping-websites scrapper scrapping

Last synced: 09 Apr 2025

https://github.com/baptistearno/tinking

🧶 Extract data from any website without code, just clicks.

harvesting puppeteer scraping scraping-websites scrapper scrapping

Last synced: 01 Oct 2025

https://github.com/DiegoCaraballo/Email-extractor

The main functionality is to extract all the emails from one or several URLs - La funcionalidad principal es extraer todos los correos electrónicos de una o varias Url

email email-extractor email-marketing emails extraction python scraper scrapers scraping scraping-websites scrapper scrapping scrapy scrapy-spider spyder stractor

Last synced: 11 Jul 2025

https://github.com/voliveirajr/seleniumcrawler

An example using Selenium webdrivers for python and Scrapy framework to create a web scraper to crawl an ASP site

asp-net python scraper scraping scraping-websites scrapper scrapy selenium selenium-webdriver webcrawler webcrawling

Last synced: 11 Oct 2025

https://github.com/filipemeneses/fatec-api

API for SIGA from Centro Paula Souza made for students to create things based in their profile data

api fatec fatec-api profile-data scrapper siga

Last synced: 04 Oct 2025

https://github.com/endenwer/tradingview-ws

TradingView data fetcher through websockets.

scrapper trading tradingview websocket

Last synced: 11 Jun 2025

https://github.com/clouedoc/goquotes

A Go scraper aimed at scrapping quotes.

colly go golang h4ck3rz quotes scrapper scrapping

Last synced: 20 Mar 2025

https://github.com/kant2002/ncrawler

Web Crawler written in C#

crawler scrapper

Last synced: 17 Jul 2025

https://github.com/satyawikananda/waifu-generator

Let's pick up your favorite waifu just from the API

anime nodejs scrapper serverless vercel waifu

Last synced: 04 Jul 2025

https://github.com/anshulrgoyal/imdb-scrapper

This is small project it provide many good api for search ,scrapping ,trending

imdb javascript node scrapper

Last synced: 01 May 2025

https://github.com/codhek/pysoc.js

:sunglasses: Simple gsoc data scraper, search for any keyword and instantly get data about the orgs that match your search criteria, use filters to sort and analyse the data, uses fuzzy-searching to improve user-search experience

analytics bs4 firebase fusejs fuzzy-search gsoc-data-scraper reactjs scrapper thread-pool

Last synced: 19 Jun 2025

https://github.com/owyn/css2rss

scrapper script for RSSGuard to make an RSS feed for any website using CSS

rss scrapper

Last synced: 18 Aug 2025

https://github.com/raphaklaus/chupa-cabra

NodeJS tool to get data from site or API and concatenate to a file.

nodejs scrapper

Last synced: 09 Apr 2025

https://github.com/livelace/gosquito

gosquito ("go" + "mosquito") is a pluggable tool for data gathering, data processing and data transmitting to various destinations.

data-mining jq kafka mattermost minio news prometheus regexp rest rss s3 scrapper slack smtp telegram twitter xpath

Last synced: 22 Aug 2025

https://github.com/o2sh/4chanmarkovtext

Text Generation using Markov Chains fed by 4chan APIs

data-mining markov-chain scrapper

Last synced: 11 Apr 2025

https://github.com/yanncam/kodiwebportal

The Web Portal for Kodi to display, search and download your personal multimedia content.

films kodi movies php scrapper series synology tvshows webportal xbmc

Last synced: 11 Apr 2025

https://github.com/rririanto/shopee-inventory-bot

"I Make dropshiper's job easier" ~ Python Shopee Inventory Bot

python python3 scrapper scrapper-script telegram-api telegram-bot

Last synced: 11 Apr 2025

https://github.com/Py-Contributors/Site-Cloner

Quick & simple Python tool to clone website

hacktoberfest open-source pip-package pypi-package python scrapper

Last synced: 16 Sep 2025

https://github.com/moepoi/nekopoiscrapper

Scrapper for nekopoi

anime hentai nekopoi scrapper

Last synced: 29 Oct 2025

https://github.com/ghpranav/link_preview_generator

A cross-platform flutter package to convert your links into rich beautiful previews.

cards dart flutter flutter-package link link-preview link-scrapper parser preview scrapper

Last synced: 07 Jul 2025

https://github.com/philou/storexplore

Transform online stores into APIs !

api scrapper

Last synced: 12 Oct 2025

https://github.com/mati365/upolujksiazke.pl

📖 Open-source platform that aggregates reviews, book ratings and brochures written in React + TypeScript + NestJS + Redis + ElasticSearch

aggregates-reviews book-ratings bookmeter books bot bots ebooks nest nestjs nodejs open-source opensource-platform react reviews scrapper scrappers typeorm wykop

Last synced: 14 Apr 2025

https://github.com/thatsinewave/spy.pet-info

This repository serves as an index for all info the community has gathered on the Spy.pet situation and as well as my own tables and tools written for these investigations. Spy.pet was taken down by Discord on 11.08.2024, this is just an archive of what bots where in each server.

bot bots database discord discord-api discord-bot discord-data discord-py discord-token scrape scraper scraping scraping-websites scrapper security security-scanner security-tools spy-pet spypet thatsinewave

Last synced: 30 Apr 2025

https://github.com/sammarxz/04-instagram-post-generator

Generates Posts for Instagram with scrapped quotes

automation instagram python quote-generator scrapper

Last synced: 24 Jul 2025

https://github.com/luismr/the-pudim-hunter

The Pudim Hunter 🍮 is a Proof of Concept (PoC) tool to scrape job listings from SimplyHired, analyze them against your resume, and assign a relevance score. Get insights into how well each job matches your skills. Automate your job search smarter! 🚀

openai openai-api playwright playwright-python scrapper scrapping scrapping-python

Last synced: 15 Apr 2025

https://github.com/apgapg/flutter_scrap

Website Scraping with Dart (Flutter)

dart flutter scraper scraping-websites scrapper

Last synced: 15 Apr 2025

https://github.com/ekamid/cricbuzz-live

Unofficial API for data fetching from Cricbuzz.com

cheerio cricbuzz cricket cricket-data cricket-score scrapper

Last synced: 26 Oct 2025

https://github.com/violanes/muskrat_http

Lightweight HTTP C++ library / curl wrapper for web scrappers, session emulators, restful applications inspired by Laravel HTTP client / Guzzle

curl curlwrapper http http-requests libcurl scrapper

Last synced: 12 Apr 2025

https://github.com/victorsouzaleal/fb-downloader-scrapper

Scrap video info and direct urls directly from Facebook

api downloader facebook fb scrapper url

Last synced: 09 Oct 2025

https://github.com/shubhamdutta2000/web-scrapping-from-google-with-scrapy

Python Scrapy spider that searches Google for a particular keyword and extracts all data from the SERP results. The spider will iterate through all pages returned by the keyword query.

google googlesearch pipenv python scrapper scrapping scrapy-crawler

Last synced: 10 Apr 2025

https://github.com/makhuta/homeassistant-duolingo

HomeAssistant Duolingo scraper

duolingo homeassistant parser scrapper

Last synced: 04 Sep 2025

https://github.com/mhdzumair/tamilyogi_stremio_addon

Stremio Addon for TamilYogi Movies.

scrapper stremio stremio-addon tamilyogi

Last synced: 16 Mar 2025

https://github.com/royfrancis/iscrape

A basic web scrapper for Instagram

instagram instagram-scraper r scraping-websites scrapper

Last synced: 17 Aug 2025

https://github.com/rvhonorato/cazy-parser

A way to extract specific information from CAZy

carbohydrates cazy data-mining enzymes scrapper text-mining

Last synced: 29 Jul 2025

https://github.com/mazipan/bumi-langit-scrapper

Scrapper website bumilangit.com

bumi-langit open-data puppeteer scrapper

Last synced: 23 Mar 2025

https://github.com/rocktimsaikia/page-scrapper

:package: node.js scrapper that pulls out all links and images of a given site.

link-scrapper node-scrapper page-scrapper scrapper site-scrapper web-scrapper

Last synced: 11 Jun 2025

https://github.com/gpestana/redonion

:globe_with_meridians: deep web scanning done right

anonymity onion-sites scrapper tor

Last synced: 23 Apr 2025

https://github.com/gabubellon/streamyard-downloader

Download Past Broadcast from StreamYard

downloader hacktoberfest python scrapper streamyard

Last synced: 20 Sep 2025

https://github.com/sajawalfareedi/insta-scrapper

A very simple but useful bot for scrapping users from instagram. It scrapes every single follower, following, and post. Not just that, it also scrapes all the comments and there likes, tagged users, and comments replies

instagram nodejs python scrapper

Last synced: 10 Apr 2025

https://github.com/limebrains/pyolx

Python scrapper for olx

olx python scrapper spider

Last synced: 11 Apr 2025

https://github.com/apanjwani0/scrape-instagram

This is a basic web-scraping tool for Instagram-users.

instagram instagram-users javascript nodejs puppeteer scraper scrapper

Last synced: 09 Oct 2025

https://github.com/voiddevsorg/node-scrapper

A simple search scrapper for Google, Bing, Wikipedia etc. for NodeJS

bing engine google javascript nodejs scrapper search search-engine typescript wikipedia youtube

Last synced: 28 Jul 2025

https://github.com/yfe404/instagram-scraping

Python script to scrap deskgram (Deskgram is a desktop version of Instagram)

instagram instagram-clone instagram-photos scraper scraping scraping-websites scrapper

Last synced: 02 Jul 2025

https://github.com/valentintintin/pixel-downloader

Tools to search on french download websites

downloader jdownloader scrapper telechargement warez

Last synced: 29 Jul 2025

https://github.com/anikhasibul/stackoverflow-scraper-messenger-bot

A messenger bot that answers messages by scraping stackoverflow questions and answers

chatbot crawler messenger-bot scrapper stackoverflow

Last synced: 09 Apr 2025

https://github.com/whiplash5057/insta_scrape_insta

>Extract Instagram User Information, Post Information Without Login in the form of a .json file or a javaScript Object.

extract feed instagram instagram-profile instagram-scraper javascript puppeteer scraping scrapper

Last synced: 07 Sep 2025

https://github.com/lleans/lyricfind-scrapper

Simple API scrapper on LyricFInd 🎹

api pyquery python3 scrapper

Last synced: 13 Aug 2025

https://github.com/ahmard/uticlass

The goal of this project is to let you surf the web without annoying ads.

guzwrap php querylist scrapper scrapping uticlass video-downloader

Last synced: 18 Mar 2025

https://github.com/volkansah/intelilink

InteliLink is a web scraper designed to check publicly accessible websites from a list of domains, extract imprint and contact information, and match this information with an existing CSV database. If the contact information is not in the database, it will be added.

black-python intelilink link python python-tools scrapper scrapping tools webscrapper webscrapper-python

Last synced: 12 Apr 2025

https://github.com/abougouffa/arabic-fonts-scraper

A simple script to download all Arabic fonts from the arfonts.net website

arabic arabic-font arabic-language fonts scrapper scrapping-python

Last synced: 01 Jul 2025

https://github.com/anasaito/semantic-parser

A rich scrapper for semantic scholar papers pages

api research scrapper

Last synced: 23 Apr 2025

https://github.com/chagasaway/nba-players-scrapper

🏀 🔎 A basketball-reference.com NBA players data scrapper

basketball basketball-reference nba nba-stats node scrapper

Last synced: 09 Apr 2025

https://github.com/suiramdev/schoolmouv-scrapper

🚀 A bot to download Schoolmouv lessons, bypassing the premium restriction

bot python schoolmouv scrapper scrapping

Last synced: 06 May 2025

https://github.com/ujen5173/airbnb-clone

Airbnb clone using Nextjs. It has some exciting custom functions / libraries you need to explore. It has awesome animations, and also has a custom date picker. For maps i have used react-leaflet. Do not forget to visit the live demo. Though it is not a full stack but its worth to explore :)

airbnb airbnb-clone airbnb-data-scrapper clone-app custom-build date-picker date-picker-range express nextjs nodejs puppeteer react react-leaflet scrapper tailwind tailwind-css tailwindcss web-scrapper

Last synced: 02 Sep 2025

https://github.com/ahmard/utiweb

A web-based utilities, mainly consist of fzmovies, netnaija, 480mkv download link extractors, written purely in PHP.

php quick-route scrapper scrapping uticlass web

Last synced: 18 Mar 2025

https://github.com/anikhasibul/bing

package bing provides web search functionalities by scraping bing search engine.

bing golang scrapper search-engine

Last synced: 03 Sep 2025

https://github.com/luiseduardobr1/pyolxbrazil

Scrapper for OLX Brazil

brazil olx python scrapper scrapping

Last synced: 14 Dec 2025

https://github.com/rozek1997/otodom-scrapper

Web scrapper for otodom.pl

beautifulsoup4 python3 scrapper

Last synced: 30 Apr 2025

https://github.com/witekbobrowski/humblecritic

💯 Get score for HumbleBundle bundles

goodreads humblebundle python scrapper

Last synced: 08 Nov 2025

https://github.com/8dcc/ndownloader

Generates random ids from nhentai and then downloads them.

automation hentai-bot hentai-downloader porn python scrapper

Last synced: 23 Jun 2025

https://github.com/gustayo/ngexplorer2

Indexes http, ftp, smb, sftp (ssh) providers and local directories in mongodb database with the possibility of synchronizing with ElasticSearch. Backend written in coffeeScript.

coffeescript elasticsearch explorer ftp-client mongodb scrapper sftp-client store webpack

Last synced: 25 Jul 2025

https://github.com/jy95/ytscheduledvideos2ical

Extract your scheduled videos publish date to a ical file so that it allows you to to see your scheduling in your favorite calendar app

calendar chrome-extension scheduled-tasks scraping scrapper scrapper-script youtube youtube-api youtube-video

Last synced: 03 Sep 2025

https://github.com/teodorszeltins/steam-market-crawler

A tool designed to scrape the Steam Market and analyze item data to reveal profit opportunities when purchasing and reselling.

scrapper steam

Last synced: 31 Oct 2025

https://github.com/amajji/web-scraping-with-scrapy-

This project aims to scrap a US government website using the Scrapy framework

scraper scraping scraping-websites scrapper scrapy webscraper webscraping

Last synced: 24 Sep 2025

https://github.com/irevenko/genshin-scrapper

💾 A bunch of scripts for api.genshin.dev

genshin genshin-api genshin-impact genshinimpact parser scrap scrapper

Last synced: 01 May 2025

https://github.com/jatinagrawal0/web_scrapper

Web Text Scraper empowers users to effortlessly extract text elements from web pages with advanced customization options. Its robust error handling ensures a smooth scraping process, enabling efficient data gathering with a human-friendly interface.

beautifulsoup4 python scrapper streamlit-webapp web-scrapper

Last synced: 30 Mar 2025

https://github.com/arutselvan/imgscrapy

A simple and fast CLI for multithreaded image scraping with support for headless scraping of dynamic websites.

cli downloader image-downloader image-downloader-python image-scraper python scrapper

Last synced: 13 Apr 2025

https://github.com/swarajkumarsingh/job-data-digger

Job data digger, scrapes various website and makes it available in a single API, and refreshes every 24hr

api backend cache go golang google job-data-digger jobs redis rest-api scrapper

Last synced: 14 Oct 2025

https://github.com/xooter/esetkeysscrapper

A tool for extracting free licenses from Eset NOD32 antivirus accounts through web scraping

antivirus licenses scrapper

Last synced: 11 Jul 2025

https://github.com/limebrains/pymorizon

Python scrapper for morizon

morizon morizonpl python python3 scrapper

Last synced: 15 Sep 2025

https://github.com/dakrauth/snagit

🔎 Yet another scrapping tool

python scrapper

Last synced: 29 Aug 2025

https://github.com/sanmak/queue-web-crawler

This application is developed to crawl a website with queue that determines no of allowed concurrent connections and find all possible hyperlinks present within it and save it to CSV file.

async chai crawler csv hyperlinks mocha nodejs queue scrapper web

Last synced: 19 Apr 2025

https://github.com/fernandod1/lottery-numbers-scraper

Script to scrap lottery numbers from Lottomatica Italia website and inserts data into MySQL database.

loteria loterias lottery lotto-numbers php php-script php-scripts scraper scraping scraping-websites scrapper scrapping scrapping-php

Last synced: 01 Mar 2025

https://github.com/knightchaser/mitreattackscrapper

A simple scrapper for MITRE ATT&CK information written in Python3.

cti data json package pypi scrapper

Last synced: 07 May 2025

https://github.com/henrique-coder/echosync-project

ECS-P ・ EchoSync Project (All the best for epic music)

downloader multi-downloader music python scrapper terminal-based

Last synced: 13 Jul 2025

https://github.com/guilospanck/dota2-counters

A simple Python webscrapper to gather information of heroes' counters from Dotabuff.

dota2 python scrapper

Last synced: 13 Apr 2025

https://github.com/literateink/ient.js

An awmazing wrapper for iENT.

ient scrapper wrapper

Last synced: 10 Sep 2025