Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

Projects in Awesome Lists tagged with web-scraper

A curated list of projects in awesome lists tagged with web-scraper .

https://github.com/alexpoulsen/vgmusic_midi_scraper

Scrapes vgmusic.com for midi files. Py3

midi midi-files python python3 scraper web-scraper

Last synced: 30 Oct 2024

https://github.com/tahanima/quotes-scraper

Scrapes quotes and the names of their authors from the site - http://quotes.toscrape.com/

quotes ruby selenium-ruby selenium-webdriver web-scraper

Last synced: 14 Nov 2024

https://github.com/JHutter/Aww_for_Reddit

Android app (6.0+) that uses a reddit scraper to deliver cute animal images and gifs in a kid-friendly UI

android-application android-development reddit reddit-scraper ui-design web-scraper

Last synced: 23 Oct 2024

https://github.com/onlyphantom/pricemate

A simple scraper for departure time and prices from Jakarta to Bandung from Tiket.com

beautifulsou beautifulsoup tiket-kereta-api web-scraper web-scraping

Last synced: 13 Dec 2024

https://github.com/edeas123/nairaland_scaping

Web scraper for popular Nigerian website - www.nairaland.com to JSON Lines formatted text file.

nairaland scrapy web-scraper

Last synced: 22 Nov 2024

https://github.com/ilmalte/quick-dictionary-bot

A super simple Telegram bot that given a word it queries and scrapes a dictionary website and answers back its pronunciation and definition!

beautifulsoup4 dictionary python telegram-bot web-scraper

Last synced: 18 Dec 2024

https://github.com/jacoblee23/sabrmetrics

A library of web-scraping software for popular SABRmetrics websites.

baseball baseball-statistics mlb python python3 sabermetrics sabrmetrics web-scraper

Last synced: 08 Nov 2024

https://github.com/remcostoeten/whatsapp-online-status-tracker

Monitor a WhatsApp user’s online activity with comprehensive reports that include timestamps, duration online, percentage of time spent online, and visual data through charts and graphs. Track statistics and resume insights between sessions for a complete overview.

chromedriver flask osint python scraper selenium web-scraper whatapp whatsapp-tracker

Last synced: 26 Dec 2024

https://github.com/linux08/puppeteer-tut

Getting started with puppeteer

cron-jobs express node-mailer nodejs puppeteer web-scraper

Last synced: 15 Nov 2024

https://github.com/cheshire137/webapptestcasegenerators

Web scraper written in Ruby for generating test cases for the QMZ web application testing model. Also another tool, again written in Ruby, for generating test cases for the Atomic Section Model web application testing model for a Ruby on Rails web application.

ruby school-project test-cases web-scraper

Last synced: 20 Dec 2024

https://github.com/sarthak-0-sach/amazon_webscraper_application

A Next.js and Bright Data-powered e-commerce product scraping site. Get notified on price drops and stock status. Automate with cron jobs.

bright-data cheerio headless-ui mongodb nextjs nodemailer responsive tailwind-css web-scraper

Last synced: 25 Nov 2024

https://github.com/xdk78/grabbi

grabbi a simple web scraper/crawler

crawler html scraper web-scraper

Last synced: 31 Dec 2024

https://github.com/mystrosto/github-account-creator-bot

GitHub Account Creator is a Python tool that enables automatic account creation on GitHub. Automatically generating your credentials, the script quickly creates a new account.

api-integration authentication automation bot github github-api github-automation github-bot github-integration open-source python security selenium web-driver-chrome web-scraper

Last synced: 01 Nov 2024

https://github.com/nexxkinn/downhost

self-hosted gallery archive and reader

deno download-manager web-scraper

Last synced: 17 Dec 2024

https://github.com/teekaytech/scraper

A web scraper application that fetches some specified set of data from one of my favorite JavaScript tutorial websites (https://javascript.info). Built with Ruby and Nokogiri Gem.

microverse nokogiri ruby web-scraper

Last synced: 06 Dec 2024

https://github.com/caimeng2/uniscraper

A universal scraper that grabs text from multiple types of webpages.

text-mining web-scraper

Last synced: 12 Oct 2024

https://github.com/anshu-krishna/html-scraper

A PHP class to simplify data extraction from HTML.

html-scraper html-scraping php php-queryselector scraper web-scraper web-scraping

Last synced: 09 Nov 2024

https://github.com/0xnu/amazon_scraper

Scrape Amazon product data such as Product Name, Product Images, Number of Reviews, Price, Product URL, and ASIN.

amazon amazon-scraper amazon-scraping asin ecommerce product-data products web-scraper web-scraping

Last synced: 15 Dec 2024

https://github.com/kyng-cytro/Screen-Time-Bot

Telegram bot to get Movies and TV-Shows updates

bot mongodb python3 telegram telegram-bot web-scraper web-scraping

Last synced: 20 Nov 2024

https://github.com/k9mil/eagle

🦅 A simple, fast, and fun CLI-based application which functions as a helper to find answers to your programming questions! Written in Golang + Cobra.

api-client cli cli-app cobra eagle fmt go golang http json json-api regex scraper scraping-websites stackoverflow stackoverflow-answer stackoverflow-api stackoverflow-questions web-scraper web-scraping

Last synced: 28 Dec 2024

https://github.com/arhcoder/school-roll-call-bot

📋 Bot for automatic roll call in for me and my fellas school classes, configurable from Discord.

automation bot discord discord-bot roll-call schoool-roll-call scraper scraping selenium silly web-scraper web-scraping

Last synced: 11 Nov 2024

https://github.com/demon-2-angel/money_control_scrap

Money_Control_Scrap is a Python package for scraping the latest business news from the Money Control website (One of the biggest website on Finance ion India)

finance-news money-control python python-automation web-scraper web-scraping-python

Last synced: 14 Dec 2024

https://github.com/samridhisainii/scrap-it

Scrapper for magicbricks.com

beautifulsoup python selenium web-scraper

Last synced: 29 Nov 2024

https://github.com/andrewtyw/auto-image-classifier

A one-stop solution for image classification: after defining the subjects that need to be classified in a JSON file, the system will automatically scrap images for each class from Bing and use them to train a model for image classification.

deep-learning-application flask python vue web-scraper

Last synced: 26 Dec 2024

https://github.com/hydrol0x/amazon-to-ebay

Web scrape amazon product listings then use Ebay API to convert product details into an Ebay listing.

amazon ebay ebay-api web-scraper

Last synced: 14 Nov 2024

https://github.com/lyqht/sg-tech-events-finder

Node.js app that locates upcoming tech events in Singapore from Meetup website

jsdom rss-parser tech-events web-scraper

Last synced: 09 Nov 2024

https://github.com/spekulatius/link-scraping-test-beautifulsoup-vs-phpscraper

Tasking both BeautifulSoup and PHPScraper to extract links - a comparison of code and performance.

beautifulsoup4 link-extractor phpscraper phpscraper-example web-scraper web-scraping

Last synced: 12 Nov 2024

https://github.com/gavindsouza/grab-cli

Light-weight web scraper to download specific types of media from static sites

no-dependencies python web-scraper

Last synced: 15 Nov 2024

https://github.com/msk4862/web-scrappers

Scrapping courses from various popular online course providers using selenium

coursera courses edx seleneum udacity web-scraper

Last synced: 24 Nov 2024

https://github.com/haywhizzz/scraper

This is a web scraper project build using the ruby language

ruby scraper web-scraper

Last synced: 17 Dec 2024

https://github.com/slyautomation/astar_pathfinding_node_networks

This project extracts the canvas data on https://www.osrsmap.net/ and converts each canvas display as an png file. This is done by exploiting html elements on the website and adding javascript parameter functions such as .toDataURL. .toDataURL() method returns a data URI containing a representation of the image in the format specified by the type parameter (defaults to PNG). WebDriver is an open source tool for automated testing of webapps across many browsers. It provides capabilities for navigating to web pages, user input, JavaScript execution, and more. Download the chrome webdriver here: https://chromedriver.chromium.org/downloads Base64 module allows for the script to This module provides functions for encoding binary data to printable ASCII characters and decoding such encodings back to binary data. This is useful converting the canvas data to a png file. The next function merges those images by looping the interactions of the canvas images, that results in the final product a full osrs map with icons and detailed.

astar-algorithm chrome-webdriver map-generator maps node-networks osrs python web-scraper web-scraping webdriver

Last synced: 22 Nov 2024

https://github.com/dfalbel/scraper-car

Scraper p/ o site do CAR

web-scraper

Last synced: 16 Nov 2024

https://github.com/ewliang/cnbc-premarket-web-scraper

A web scraper created using Puppeteer library on the backend to scrape the stock market prediction data off of CNBC's premarket page.

backend-development expressjs javascript node-js nodejs puppeteer stock-market web-scraper web-scraping webscraper

Last synced: 02 Jan 2025

https://github.com/opportus/web-scraper

A web scraper taking as arguments a list of URLs and a list of XPath queries to perform on each document. Returns an instance of DataInterface.

csv json web-scraper web-scraping web-scrapper web-scrapping webscraper webscraping webscrapper webscrapping xpath

Last synced: 10 Nov 2024

https://github.com/tynandebold/secret-flyer-scraper

Get the first few flight deals on the Secret Flyer website via the command line so we don't have to look at that awful page.

flight-prices javascript nodejs npm npm-package secret-flyer-scraper web-scraper web-scraping

Last synced: 18 Dec 2024

https://github.com/apexal/lector

A super simple Moodle web scraper for my school's Moodle site.

education moodle ruby school scraper web web-scraper web-scraping

Last synced: 15 Dec 2024

https://github.com/memgonzales/parallel-email-scraper

Multiprocess email address scraper for the De La Salle University website staff directory. Our approach models the scraping task as a multiple producer – multiple consumer problem to achieve a 7.22× superlinear speedup compared to serial execution

email-scraper multiprocessing parallel-programming producer-consumer python queue selenium selenium-webdriver synchronized-queue web-scraper web-scraping

Last synced: 19 Nov 2024

https://github.com/anasfik/user-repositories-scrapper

a puppeteer script to get repositories data from user using puppeteer

javascript puppeteer repositories scrapper web-scraper web-scraping

Last synced: 14 Nov 2024

https://github.com/andersonhsporto/organic-live-api

Api Showing Some of the Latest News About Organic Farming in Brazil.

api axios cheerio express javascript node nodejs organic organic-agriculture organicos rest scraper web-scraper

Last synced: 16 Nov 2024

https://github.com/olivercederborg/energy-notifier

Daily notification with hourly electricity prices.

energy notifications notifier python web-scraper

Last synced: 21 Dec 2024

https://github.com/david-torres/skrapa

A web scraping utility powered by configuration scripts, no coding required!

golang toml web-scraper web-scraping

Last synced: 14 Nov 2024

https://github.com/anthonygharvey/developer_news_cli_app

A Ruby CLI app that scrapes article information from FreeCodeCamp, HackerNoon, Codeburst and A List Apart.

cli-app nokogiri ruby ruby-gem web-scraper

Last synced: 19 Nov 2024

https://github.com/bernso/novelreaderweb

Website made in python that scrapes lightnovelpub.vip for the novel inputted and will create a page for each of the chapters inside of those novels, also it includes features such as text to speech, text opacity and a font selector. All of these settings eill save on your device.

flask flask-application host novel python python-3 python3 web web--scraping web-novel web-scraper web-scraping webscraping website websites

Last synced: 19 Nov 2024

https://github.com/tremainebuchanan/shipme-cli

Unoffical command line interface for the shipme.me freight forwarder.

automation cheeriojs nodejs web-scraper

Last synced: 19 Nov 2024

https://github.com/trixsec/zeuscrawler

The ultimate web crawling powerhouse, striking the web like lightning to harvest data with divine precision.

crawler cybersecurity information-gathering information-retrieval osint python scraper spider web-crawler web-scraper

Last synced: 21 Dec 2024

https://github.com/gamemann/web-extract-with-chatgpt

A Python application that extracts web data using Selenium and BeautifulSoup, sends the web data through the ChatGPT API, and outputs the ChatGPT response somewhere.

api beautifulsoup beautifulsoup4 bs4 chatgpt discourse extract post python request selenium web-extractor web-scrape web-scraper

Last synced: 21 Dec 2024

https://github.com/oxylabs/free-trial-guides

Learn how to set up and get the most of your e-commerce, web scraper and serp scraper free trial.

e-commerce scrapeddata scraping-websites serp-scraper web-scraper web-scraping

Last synced: 17 Nov 2024

https://github.com/gabrieledarrigo/curiosities-scraper

A multi thread application to scrape data from curiositadalmondo.it

curiosities curiosity scraper web-scraper

Last synced: 24 Nov 2024

https://github.com/netevert/python-playground

Repository containing miscellaneous python code snippets

caesar-cipher log-parser python web-scraper

Last synced: 30 Nov 2024

https://github.com/bitartisan1/netdigger

A .NET 8.0 C# WPF desktop application for web scraping data into structured databases with a modern UI, comprehensive logging and optimized high performance.

csharp data data-scraper data-scraping database desktop dotnet internet logging scraper ui url web-scraper web-scrapers web-scraping web-scrapping

Last synced: 20 Nov 2024

https://github.com/gayanukabulegoda/web-scraping-starter-kit

Repository designed to help freshers easily grasp the basics of web scripting, offering simple guides and examples to build a strong foundation.

python python-web-scraper python3 scrape scraping scraping-python web-scraper web-scraping web-scraping-python web-scraping-tutorials web-scrapping

Last synced: 17 Nov 2024

https://github.com/patkub/tvp-pl-proxy

A proxy for vod.tvp.pl

proxy userscript web-scraper

Last synced: 28 Nov 2024

https://github.com/ndom91/pw-web-scraper

🌐 Basic Playwright Web-scraper

nodejs playwright web-scraper

Last synced: 13 Dec 2024

https://github.com/alexanyernas/web-scraping-kardex-ucv

Desarrollo de Web Scraper para análisis y generación de reportes para Kardex estudiantil (UCV, Facultad de Ciencias).

python scraping web-scraper web-scraping

Last synced: 23 Nov 2024

https://github.com/marinoffdev/steamgifts-automated

A Chrome extension for SteamGifts that lets you track and auto-enter selected game giveaways with a single click. ✨

automation browser-extension chrome chrome-extension gaming javascript productivity steam steam-games steamgift steamgifts web-scraper web-scraping

Last synced: 16 Dec 2024

https://github.com/neogeek/raspar

⛏ A simple to use Promise-based web scraper with local caching.

promise web-scraper

Last synced: 27 Oct 2024

https://github.com/btrevizan/nfe_reader

A Nota Fiscal (NF) reader made in Python 3.

grocery nota-fiscal reader web-scraper

Last synced: 28 Dec 2024

https://github.com/antodata/coronaviruswebscraper

Web scraper that gets information about the corona virus spread from https://www.worldometers.info/coronavirus/ and world population in from https://www.worldometers.info/world-population/population-by-country/ and generates three maps with folium that represent the ratio between cases and population, between recovered people and cases and between death people and cases using circles

beautifulsoup big-data bs4 corona-virus covid-19 covid-19-tracker covid19 covid19-data data-science folium html-parser maps pandas python python-requests python3 requests web-scraper web-scraping webscraper

Last synced: 08 Dec 2024

https://github.com/stephanlensky/focus-api

Flask server which scrapes and parses pages from the Focus SIS

api python school student-management web-scraper

Last synced: 19 Nov 2024

https://github.com/an0ncer/pwa-sitemap

Репозиторий с веб-скрапером на Node.js и Puppeteer. Создайте sitemap.xml для индексации веб-сайтов.

nodejs puppeteer sitemap web-crawling web-scraper

Last synced: 04 Dec 2024

https://github.com/lynkos/downloader

Basic web scraper to download media from websites. Supports .pdf generation and vertical image stacking; useful for downloading manga, comics, etc.

python python-script python3 scraper scraping scrapping script web-scraper web-scraping web-scraping-python web-scrapping

Last synced: 10 Dec 2024

https://github.com/wkrzywiec/tasty-python

Command Line Tool for finding recipes on Tasty.co (written in Python)

beautifulsoup4 cli pipenv pytest python python-click python3 recipes scraping-python scraping-websites web-scraper

Last synced: 17 Dec 2024

https://github.com/t145/anipop

Short script to download the current season of anime from HorribleSubs

anime anime-downloader python python3 web-scraper web-scraping

Last synced: 12 Dec 2024

https://github.com/milahu/crx4chrome-scraper

scrape crx files from crx4chrome.com - unpack the crx files and add the crx contents to a git repo

chrome-extensions crx crx-scraper crx2git crx4chrome crx4chrome2git scraper web-scraper

Last synced: 13 Dec 2024

https://github.com/ryanfobel/ontario-grid-data

Hourly data on the Ontario electricity grid

carbon-footprint web-scraper

Last synced: 13 Nov 2024

https://github.com/ryanfobel/utility-bill-scraper

Download energy usage data and estimate CO2 emissions from utility websites or pdf bills.

carbon-footprint climate-crisis web-scraper

Last synced: 13 Nov 2024