Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
Projects in Awesome Lists tagged with web-scraper
A curated list of projects in awesome lists tagged with web-scraper .
https://github.com/omkarcloud/gitpod-selenium
Run Python Selenium in GitPod
automation botasaurus chrome docker docker-image gitpod gitpod-ready gitpod-selenium gitpod-selenium-python puppeteer python python-selenium selenium selenium-gitpod selenium-webdriver web-scraper web-scrapers web-scraping web-scraping-python webdriver-manager
Last synced: 02 Jan 2025
https://github.com/chandler767/arbys-coupon-scraper
Golang concurrent web scraper for Arby's coupons.
api arby-coupons arbys concurrent-programming coupons email email-marketing food go golang hacking marketing scraper web-scraper
Last synced: 22 Nov 2024
https://github.com/alexpoulsen/vgmusic_midi_scraper
Scrapes vgmusic.com for midi files. Py3
midi midi-files python python3 scraper web-scraper
Last synced: 30 Oct 2024
https://github.com/tahanima/quotes-scraper
Scrapes quotes and the names of their authors from the site - http://quotes.toscrape.com/
quotes ruby selenium-ruby selenium-webdriver web-scraper
Last synced: 14 Nov 2024
https://github.com/gtiwari333/spring-boot-keycloak-angular-quote-app
Spring Boot + Keycloak Backend / Angular Web App
angular heroku java jsoup keycloak spring-boot web-crawler web-scraper
Last synced: 28 Oct 2024
https://github.com/sayyid5416/links-extractor
Extract links from any file or the website.
crawler extract-links extractor links-extraction scraper web-crawler web-scraper
Last synced: 28 Oct 2024
https://github.com/spekulatius/phpscraper-keyword-scraping-example
Simple example of a few ways to extract keywords from a website
keyword keyword-extraction keyword-extractor keyword-scraper php php-scraper php7 phpscraper-example web-scraper web-scraping
Last synced: 12 Nov 2024
https://github.com/JHutter/Aww_for_Reddit
Android app (6.0+) that uses a reddit scraper to deliver cute animal images and gifs in a kid-friendly UI
android-application android-development reddit reddit-scraper ui-design web-scraper
Last synced: 23 Oct 2024
https://github.com/onlyphantom/pricemate
A simple scraper for departure time and prices from Jakarta to Bandung from Tiket.com
beautifulsou beautifulsoup tiket-kereta-api web-scraper web-scraping
Last synced: 13 Dec 2024
https://github.com/edeas123/nairaland_scaping
Web scraper for popular Nigerian website - www.nairaland.com to JSON Lines formatted text file.
Last synced: 22 Nov 2024
https://github.com/ilmalte/quick-dictionary-bot
A super simple Telegram bot that given a word it queries and scrapes a dictionary website and answers back its pronunciation and definition!
beautifulsoup4 dictionary python telegram-bot web-scraper
Last synced: 18 Dec 2024
https://github.com/jacoblee23/sabrmetrics
A library of web-scraping software for popular SABRmetrics websites.
baseball baseball-statistics mlb python python3 sabermetrics sabrmetrics web-scraper
Last synced: 08 Nov 2024
https://github.com/remcostoeten/whatsapp-online-status-tracker
Monitor a WhatsApp user’s online activity with comprehensive reports that include timestamps, duration online, percentage of time spent online, and visual data through charts and graphs. Track statistics and resume insights between sessions for a complete overview.
chromedriver flask osint python scraper selenium web-scraper whatapp whatsapp-tracker
Last synced: 26 Dec 2024
https://github.com/linux08/puppeteer-tut
Getting started with puppeteer
cron-jobs express node-mailer nodejs puppeteer web-scraper
Last synced: 15 Nov 2024
https://github.com/chhayac/apartment_finder
Quest to find an awesome apartment for rent.
apartment-finder apartmentguide beautifulsoup google-maps google-maps-api python3 web-scraper webscraper webscraping
Last synced: 20 Nov 2024
https://github.com/0dist/liveboard
Get latest information of an ongoing live stream
gui gui-application livestream pyqt pyqt5 scraper twitch twitch-tv twitchtv web-scraper web-scraping youtube youtube-live youtube-livestream
Last synced: 02 Dec 2024
https://github.com/cheshire137/webapptestcasegenerators
Web scraper written in Ruby for generating test cases for the QMZ web application testing model. Also another tool, again written in Ruby, for generating test cases for the Atomic Section Model web application testing model for a Ruby on Rails web application.
ruby school-project test-cases web-scraper
Last synced: 20 Dec 2024
https://github.com/vmussa/anpocs-scraper
Raspador dos dados dos Encontros Anuais da ANPOCS
bibliometrics computational-social-science digital-humanities digital-sociology scientometrics web-scraper web-scraping
Last synced: 21 Nov 2024
https://github.com/omkarcloud/gitpod-botasaurus
Run Botasaurus in GitPod
automation botasaurus chrome docker docker-image gitpod gitpod-ready gitpod-selenium gitpod-selenium-python puppeteer python python-selenium selenium selenium-gitpod selenium-webdriver web-scraper web-scrapers web-scraping web-scraping-python webdriver-manager
Last synced: 02 Jan 2025
https://github.com/sarthak-0-sach/amazon_webscraper_application
A Next.js and Bright Data-powered e-commerce product scraping site. Get notified on price drops and stock status. Automate with cron jobs.
bright-data cheerio headless-ui mongodb nextjs nodemailer responsive tailwind-css web-scraper
Last synced: 25 Nov 2024
https://github.com/xdk78/grabbi
grabbi a simple web scraper/crawler
crawler html scraper web-scraper
Last synced: 31 Dec 2024
https://github.com/mystrosto/github-account-creator-bot
GitHub Account Creator is a Python tool that enables automatic account creation on GitHub. Automatically generating your credentials, the script quickly creates a new account.
api-integration authentication automation bot github github-api github-automation github-bot github-integration open-source python security selenium web-driver-chrome web-scraper
Last synced: 01 Nov 2024
https://github.com/kochan4php/kusonime-api
Unofficial Kusonime REST API
anime-scraper cheerio javascript kusonime-scrapper nodejs web-scraper
Last synced: 12 Oct 2024
https://github.com/nexxkinn/downhost
self-hosted gallery archive and reader
deno download-manager web-scraper
Last synced: 17 Dec 2024
https://github.com/teekaytech/scraper
A web scraper application that fetches some specified set of data from one of my favorite JavaScript tutorial websites (https://javascript.info). Built with Ruby and Nokogiri Gem.
microverse nokogiri ruby web-scraper
Last synced: 06 Dec 2024
https://github.com/caimeng2/uniscraper
A universal scraper that grabs text from multiple types of webpages.
Last synced: 12 Oct 2024
https://github.com/farukalamai/yelp-scraper-scrapy-python
Yelp Restaurant data scraping using python, scrapy spider
ai-bot data-extraction data-mining data-scraper data-scraping python python-scraper scrapy scrapy-crawler scrapy-spider web-scraper web-scraping web-scraping-python web-scraping-software yelp yelp-api yelp-restaurants yelp-resturant-data-scraping yelp-scraper
Last synced: 29 Dec 2024
https://github.com/anshu-krishna/html-scraper
A PHP class to simplify data extraction from HTML.
html-scraper html-scraping php php-queryselector scraper web-scraper web-scraping
Last synced: 09 Nov 2024
https://github.com/0xnu/amazon_scraper
Scrape Amazon product data such as Product Name, Product Images, Number of Reviews, Price, Product URL, and ASIN.
amazon amazon-scraper amazon-scraping asin ecommerce product-data products web-scraper web-scraping
Last synced: 15 Dec 2024
https://github.com/kyng-cytro/Screen-Time-Bot
Telegram bot to get Movies and TV-Shows updates
bot mongodb python3 telegram telegram-bot web-scraper web-scraping
Last synced: 20 Nov 2024
https://github.com/ayanhussain81/web_scrapping
Some of my Data Scraping Projects
beautifulsoup4 data-scraping data-scraping-projects python3 selenium web-scraper web-scraping
Last synced: 14 Dec 2024
https://github.com/k9mil/eagle
🦅 A simple, fast, and fun CLI-based application which functions as a helper to find answers to your programming questions! Written in Golang + Cobra.
api-client cli cli-app cobra eagle fmt go golang http json json-api regex scraper scraping-websites stackoverflow stackoverflow-answer stackoverflow-api stackoverflow-questions web-scraper web-scraping
Last synced: 28 Dec 2024
https://github.com/andreybykov/actor-sreality-listings-monitor
apify monitoring-tool web-scraper
Last synced: 09 Nov 2024
https://github.com/evoluteur/web-scraper-sitemaps
Sitemaps for the Web Scraper Chrome extension.
chrome-extension data dataset scraper scraping scrapper scrapping scrapy-crawler sitemap web-scraper web-scraping
Last synced: 13 Nov 2024
https://github.com/arhcoder/school-roll-call-bot
📋 Bot for automatic roll call in for me and my fellas school classes, configurable from Discord.
automation bot discord discord-bot roll-call schoool-roll-call scraper scraping selenium silly web-scraper web-scraping
Last synced: 11 Nov 2024
https://github.com/demon-2-angel/money_control_scrap
Money_Control_Scrap is a Python package for scraping the latest business news from the Money Control website (One of the biggest website on Finance ion India)
finance-news money-control python python-automation web-scraper web-scraping-python
Last synced: 14 Dec 2024
https://github.com/scottgriv/python-pdf_web_scraper
Scrape a web page for pdf files and download them all locally.
pdf pdf-download pdf-downloader pdf-scraper pdf-scraping python utility utility-app utility-application utility-script web-scraper web-scraping
Last synced: 14 Dec 2024
https://github.com/samridhisainii/scrap-it
Scrapper for magicbricks.com
beautifulsoup python selenium web-scraper
Last synced: 29 Nov 2024
https://github.com/mee-to/facebook-checker-api
This Python tool is a powerful Facebook account verification tool used to check Facebook profiles and save checked accounts to .txt file.
api-integration authentication checker checker-account checker-api checker-bot checker-service checkers facebook facebook-api facebook-api-integration facebook-authentication facebook-checker facebook-mass-checker fast-proxy privacy-concerns security social-media web-scraper
Last synced: 14 Dec 2024
https://github.com/chadsr/marktplaats-scraper
Marktplaats.nl (Dutch Classifieds) Listing Scraper
chromedriver dataset-creation dataset-generation dutch-language machine-learning machine-learning-dataset marktplaats scraper selenium web-scraper web-scraping
Last synced: 25 Oct 2024
https://github.com/omkarcloud/multiple-account-generation-template
🚀 THIS WEB SCRAPING TEMPLATE PROVIDES YOU WITH A GREAT STARTING POINT WHEN CREATING MULTIPLE ACCOUNTS ON A WEBSITE. 🤖
beautifulsoup crawler crawling crawling-framework crawling-python crawling-tool headless node-crawler python-crawler scraper scraping scraping-framework scraping-python scraping-tool selenium web-crawler web-crawling web-scraper web-scraping webscraping
Last synced: 02 Jan 2025
https://github.com/andrewtyw/auto-image-classifier
A one-stop solution for image classification: after defining the subjects that need to be classified in a JSON file, the system will automatically scrap images for each class from Bing and use them to train a model for image classification.
deep-learning-application flask python vue web-scraper
Last synced: 26 Dec 2024
https://github.com/dionabazi/web-scraper
information-gathering python script web-scraper website
Last synced: 12 Nov 2024
https://github.com/hydrol0x/amazon-to-ebay
Web scrape amazon product listings then use Ebay API to convert product details into an Ebay listing.
amazon ebay ebay-api web-scraper
Last synced: 14 Nov 2024
https://github.com/tauseedzaman/cool-python-projects
putting my hand on some cool automation projects.
automate automate-dailypakistan automate-geo-news automate-kitploat automate-thenews automation cool-scraping python selenuim web web-scraper web-scraping web-scrapper-python
Last synced: 09 Nov 2024
https://github.com/lyqht/sg-tech-events-finder
Node.js app that locates upcoming tech events in Singapore from Meetup website
jsdom rss-parser tech-events web-scraper
Last synced: 09 Nov 2024
https://github.com/lebrancconvas/web-scraping-playground
Learning about Web Scraping with Puppeteer.
nodejs personal-project playground puppeteer side-project training web-automation web-scraper web-scraping
Last synced: 11 Nov 2024
https://github.com/spekulatius/link-scraping-test-beautifulsoup-vs-phpscraper
Tasking both BeautifulSoup and PHPScraper to extract links - a comparison of code and performance.
beautifulsoup4 link-extractor phpscraper phpscraper-example web-scraper web-scraping
Last synced: 12 Nov 2024
https://github.com/gavindsouza/grab-cli
Light-weight web scraper to download specific types of media from static sites
no-dependencies python web-scraper
Last synced: 15 Nov 2024
https://github.com/msk4862/web-scrappers
Scrapping courses from various popular online course providers using selenium
coursera courses edx seleneum udacity web-scraper
Last synced: 24 Nov 2024
https://github.com/ldilley/gogstash
:video_game: A Gog.com downloader
downloader gog ruby scrape scraper scraping scraping-websites web-scraper web-scrapers web-scraping web-scraping-software
Last synced: 18 Nov 2024
https://github.com/elektrostudios/fhm-crawler-freehardmusic.com
Crawls download urls of albums from freehardmusic.com website
albums crawl crawler crawling desktop-app desktop-application dotnet music web-crawler web-crawling web-scraper web-scraping webcrawler webcrawling webscraper webscraping windows windows-app windowsapp winforms
Last synced: 01 Dec 2024
https://github.com/haywhizzz/scraper
This is a web scraper project build using the ruby language
Last synced: 17 Dec 2024
https://github.com/slyautomation/astar_pathfinding_node_networks
This project extracts the canvas data on https://www.osrsmap.net/ and converts each canvas display as an png file. This is done by exploiting html elements on the website and adding javascript parameter functions such as .toDataURL. .toDataURL() method returns a data URI containing a representation of the image in the format specified by the type parameter (defaults to PNG). WebDriver is an open source tool for automated testing of webapps across many browsers. It provides capabilities for navigating to web pages, user input, JavaScript execution, and more. Download the chrome webdriver here: https://chromedriver.chromium.org/downloads Base64 module allows for the script to This module provides functions for encoding binary data to printable ASCII characters and decoding such encodings back to binary data. This is useful converting the canvas data to a png file. The next function merges those images by looping the interactions of the canvas images, that results in the final product a full osrs map with icons and detailed.
astar-algorithm chrome-webdriver map-generator maps node-networks osrs python web-scraper web-scraping webdriver
Last synced: 22 Nov 2024
https://github.com/omkarcloud/dentalkart-scraper
🚀 SCRAPE 1000'S OF PRODUCTS FROM DENTALKART 🤖
beautifulsoup crawler crawling crawling-framework crawling-python dentalkart dentalkart-product-scraper dentalkart-scraper dentalkart-scraping node-crawler scraper scraping scraping-framework scraping-python selenium web-crawler web-crawling web-scraper web-scraping webscraping
Last synced: 02 Jan 2025
https://github.com/ewliang/cnbc-premarket-web-scraper
A web scraper created using Puppeteer library on the backend to scrape the stock market prediction data off of CNBC's premarket page.
backend-development expressjs javascript node-js nodejs puppeteer stock-market web-scraper web-scraping webscraper
Last synced: 02 Jan 2025
https://github.com/opportus/web-scraper
A web scraper taking as arguments a list of URLs and a list of XPath queries to perform on each document. Returns an instance of DataInterface.
csv json web-scraper web-scraping web-scrapper web-scrapping webscraper webscraping webscrapper webscrapping xpath
Last synced: 10 Nov 2024
https://github.com/blazeinferno64/web-scrapper
Scrap the web using Python web scrapper!
blazeinferno64 python python-3 python-app python-library python-script python3 requets web-scraper web-scraping web-scraping-python
Last synced: 11 Nov 2024
https://github.com/tynandebold/secret-flyer-scraper
Get the first few flight deals on the Secret Flyer website via the command line so we don't have to look at that awful page.
flight-prices javascript nodejs npm npm-package secret-flyer-scraper web-scraper web-scraping
Last synced: 18 Dec 2024
https://github.com/apexal/lector
A super simple Moodle web scraper for my school's Moodle site.
education moodle ruby school scraper web web-scraper web-scraping
Last synced: 15 Dec 2024
https://github.com/justpowerful/masswebdirectorydownloader
Download a mass of files from a website directory
mass mass-downloader python web-scraper web-scraping webscraper webscraping wget
Last synced: 11 Nov 2024
https://github.com/memgonzales/parallel-email-scraper
Multiprocess email address scraper for the De La Salle University website staff directory. Our approach models the scraping task as a multiple producer – multiple consumer problem to achieve a 7.22× superlinear speedup compared to serial execution
email-scraper multiprocessing parallel-programming producer-consumer python queue selenium selenium-webdriver synchronized-queue web-scraper web-scraping
Last synced: 19 Nov 2024
https://github.com/anasfik/user-repositories-scrapper
a puppeteer script to get repositories data from user using puppeteer
javascript puppeteer repositories scrapper web-scraper web-scraping
Last synced: 14 Nov 2024
https://github.com/andersonhsporto/organic-live-api
Api Showing Some of the Latest News About Organic Farming in Brazil.
api axios cheerio express javascript node nodejs organic organic-agriculture organicos rest scraper web-scraper
Last synced: 16 Nov 2024
https://github.com/olivercederborg/energy-notifier
Daily notification with hourly electricity prices.
energy notifications notifier python web-scraper
Last synced: 21 Dec 2024
https://github.com/david-torres/skrapa
A web scraping utility powered by configuration scripts, no coding required!
golang toml web-scraper web-scraping
Last synced: 14 Nov 2024
https://github.com/gnuns/raspa
data mining stuff
crawler robot scraper web-scraper web-scraping web-spider
Last synced: 17 Dec 2024
https://github.com/anthonygharvey/developer_news_cli_app
A Ruby CLI app that scrapes article information from FreeCodeCamp, HackerNoon, Codeburst and A List Apart.
cli-app nokogiri ruby ruby-gem web-scraper
Last synced: 19 Nov 2024
https://github.com/bernso/novelreaderweb
Website made in python that scrapes lightnovelpub.vip for the novel inputted and will create a page for each of the chapters inside of those novels, also it includes features such as text to speech, text opacity and a font selector. All of these settings eill save on your device.
flask flask-application host novel python python-3 python3 web web--scraping web-novel web-scraper web-scraping webscraping website websites
Last synced: 19 Nov 2024
https://github.com/tremainebuchanan/shipme-cli
Unoffical command line interface for the shipme.me freight forwarder.
automation cheeriojs nodejs web-scraper
Last synced: 19 Nov 2024
https://github.com/trixsec/zeuscrawler
The ultimate web crawling powerhouse, striking the web like lightning to harvest data with divine precision.
crawler cybersecurity information-gathering information-retrieval osint python scraper spider web-crawler web-scraper
Last synced: 21 Dec 2024
https://github.com/ahmedshahriar/daraz-scraper
Daraz scraper
daraz e-commerce requests-library-python web-crawler web-crawler-python web-scraper web-scraping
Last synced: 16 Nov 2024
https://github.com/gamemann/web-extract-with-chatgpt
A Python application that extracts web data using Selenium and BeautifulSoup, sends the web data through the ChatGPT API, and outputs the ChatGPT response somewhere.
api beautifulsoup beautifulsoup4 bs4 chatgpt discourse extract post python request selenium web-extractor web-scrape web-scraper
Last synced: 21 Dec 2024
https://github.com/oxylabs/free-trial-guides
Learn how to set up and get the most of your e-commerce, web scraper and serp scraper free trial.
e-commerce scrapeddata scraping-websites serp-scraper web-scraper web-scraping
Last synced: 17 Nov 2024
https://github.com/gabrieledarrigo/curiosities-scraper
A multi thread application to scrape data from curiositadalmondo.it
curiosities curiosity scraper web-scraper
Last synced: 24 Nov 2024
https://github.com/crisszkutnik/gamersclub-spreadsheets
A match web scraper for the GamersClub platform.
counter-strike counter-strike-global-offensive csgo gamersclub python python3 spreadsheets web-scraper
Last synced: 29 Dec 2024
https://github.com/netevert/python-playground
Repository containing miscellaneous python code snippets
caesar-cipher log-parser python web-scraper
Last synced: 30 Nov 2024
https://github.com/bitartisan1/netdigger
A .NET 8.0 C# WPF desktop application for web scraping data into structured databases with a modern UI, comprehensive logging and optimized high performance.
csharp data data-scraper data-scraping database desktop dotnet internet logging scraper ui url web-scraper web-scrapers web-scraping web-scrapping
Last synced: 20 Nov 2024
https://github.com/gayanukabulegoda/web-scraping-starter-kit
Repository designed to help freshers easily grasp the basics of web scripting, offering simple guides and examples to build a strong foundation.
python python-web-scraper python3 scrape scraping scraping-python web-scraper web-scraping web-scraping-python web-scraping-tutorials web-scrapping
Last synced: 17 Nov 2024
https://github.com/alexanyernas/web-scraping-kardex-ucv
Desarrollo de Web Scraper para análisis y generación de reportes para Kardex estudiantil (UCV, Facultad de Ciencias).
python scraping web-scraper web-scraping
Last synced: 23 Nov 2024
https://github.com/schne324/moocher
Web content scraper
scraper scraping scraping-websites web-scraper web-scraping
Last synced: 15 Nov 2024
https://github.com/jbizzlefoshizzle/beer-database-project
jbizzlefoshizzle.github.io
d3-visualization gmaps-api html-css-javascript pandas-python web-scraper
Last synced: 25 Nov 2024
https://github.com/marinoffdev/steamgifts-automated
A Chrome extension for SteamGifts that lets you track and auto-enter selected game giveaways with a single click. ✨
automation browser-extension chrome chrome-extension gaming javascript productivity steam steam-games steamgift steamgifts web-scraper web-scraping
Last synced: 16 Dec 2024
https://github.com/neogeek/raspar
⛏ A simple to use Promise-based web scraper with local caching.
Last synced: 27 Oct 2024
https://github.com/btrevizan/nfe_reader
A Nota Fiscal (NF) reader made in Python 3.
grocery nota-fiscal reader web-scraper
Last synced: 28 Dec 2024
https://github.com/antodata/coronaviruswebscraper
Web scraper that gets information about the corona virus spread from https://www.worldometers.info/coronavirus/ and world population in from https://www.worldometers.info/world-population/population-by-country/ and generates three maps with folium that represent the ratio between cases and population, between recovered people and cases and between death people and cases using circles
beautifulsoup big-data bs4 corona-virus covid-19 covid-19-tracker covid19 covid19-data data-science folium html-parser maps pandas python python-requests python3 requests web-scraper web-scraping webscraper
Last synced: 08 Dec 2024
https://github.com/definetlynotai/web_scraper
Super basic web scraper cli
html-download python scraper side-project simple web web-download web-scraper
Last synced: 28 Nov 2024
https://github.com/stephanlensky/focus-api
Flask server which scrapes and parses pages from the Focus SIS
api python school student-management web-scraper
Last synced: 19 Nov 2024
https://github.com/an0ncer/pwa-sitemap
Репозиторий с веб-скрапером на Node.js и Puppeteer. Создайте sitemap.xml для индексации веб-сайтов.
nodejs puppeteer sitemap web-crawling web-scraper
Last synced: 04 Dec 2024
https://github.com/lynkos/downloader
Basic web scraper to download media from websites. Supports .pdf generation and vertical image stacking; useful for downloading manga, comics, etc.
python python-script python3 scraper scraping scrapping script web-scraper web-scraping web-scraping-python web-scrapping
Last synced: 10 Dec 2024
https://github.com/wkrzywiec/tasty-python
Command Line Tool for finding recipes on Tasty.co (written in Python)
beautifulsoup4 cli pipenv pytest python python-click python3 recipes scraping-python scraping-websites web-scraper
Last synced: 17 Dec 2024
https://github.com/t145/anipop
Short script to download the current season of anime from HorribleSubs
anime anime-downloader python python3 web-scraper web-scraping
Last synced: 12 Dec 2024
https://github.com/milahu/crx4chrome-scraper
scrape crx files from crx4chrome.com - unpack the crx files and add the crx contents to a git repo
chrome-extensions crx crx-scraper crx2git crx4chrome crx4chrome2git scraper web-scraper
Last synced: 13 Dec 2024
https://github.com/ryanfobel/ontario-grid-data
Hourly data on the Ontario electricity grid
Last synced: 13 Nov 2024
https://github.com/ryanfobel/utility-bill-scraper
Download energy usage data and estimate CO2 emissions from utility websites or pdf bills.
carbon-footprint climate-crisis web-scraper
Last synced: 13 Nov 2024