Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

Projects in Awesome Lists tagged with web-scraper

A curated list of projects in awesome lists tagged with web-scraper .

https://github.com/lynkos/downloader

Basic web scraper to download media from websites. Supports .pdf generation and vertical image stacking; useful for downloading manga, comics, etc.

python python-script python3 scraper scraping scrapping script web-scraper web-scraping web-scraping-python web-scrapping

Last synced: 10 Dec 2024

https://github.com/wkrzywiec/tasty-python

Command Line Tool for finding recipes on Tasty.co (written in Python)

beautifulsoup4 cli pipenv pytest python python-click python3 recipes scraping-python scraping-websites web-scraper

Last synced: 17 Dec 2024

https://github.com/milahu/crx4chrome-scraper

scrape crx files from crx4chrome.com - unpack the crx files and add the crx contents to a git repo

chrome-extensions crx crx-scraper crx2git crx4chrome crx4chrome2git scraper web-scraper

Last synced: 13 Dec 2024

https://github.com/sarkartanmay393/imdb-scraper-golang

Contains a web scraper that crawls inside `www.imdb.com/name` and stores all celebrity data in given birthday inside a file.

go imdb-webscrapping web-scraper

Last synced: 04 Jan 2025

https://github.com/pawpaw2022/coronavirus-web-scraper-and-voice-assistant

An app that can scrape the COVID-19 cases data from the web by using ParseHub (a web scraper tool). The user can easily access to the data by asking the voice assistant.

covid-19 python voice-assistant web-scraper

Last synced: 24 Nov 2024

https://github.com/afzl210/salespy

A notification bot that watches product prices and alerts users of any drops

amazon cheerio nodejs twilio-sms-api web web-scraper

Last synced: 27 Dec 2024

https://github.com/jillmpla/sentimentanalysis

Comment sentiment analysis of the top 25 posts (from the last 24 hrs) on a subreddit (reddit.com) using a web scraper.

python reddit sentiment-analysis sqlite web-scraper

Last synced: 14 Nov 2024

https://github.com/hayatiyrtgl/wiki_art_crap

Asynchronous web scraper for downloading painting images.

bs4 bs4-requests python python3 web-scraper web-scraping web-scrapper

Last synced: 22 Dec 2024

https://github.com/diusmontenegro/web-scraper-tool

This project is a web scraping program that uses the BeautifulSoup library to extract data from a given URL. It extracts the href and text attributes from all the links on the page and stores them in a list.

python3 tool web-scraper

Last synced: 12 Nov 2024

https://github.com/martinius96/webscraper-esp

WebScraper postavený na mikrokontroléroch ESP8266 a ESP32. Umožňuje scrapovanie údajov z HTML kódu webstránky Minv odcudzené vozidlá.

esp32 esp8266 html parser scraper web-scraper wifi

Last synced: 03 Jan 2025

https://github.com/zahraarshia/cti_crawl

This cyber threat intelligence crawler can be used to gather information from various sources, including open-source and commercial feeds.

crawler cti cyber-news-bot cyber-threat-intelligence mongodb python scrapy sqlite3 web-scraper

Last synced: 11 Nov 2024

https://github.com/boring-dragon/scraper

scraper for avas.mv

python scraper web-scraper

Last synced: 03 Dec 2024

https://github.com/zonggen/markus_scraper

A command line tool to extract grades from Markus

markus python selenium-python terminal-app uoft uwaterloo web-scraper

Last synced: 30 Nov 2024

https://github.com/dahlitzflorian/r-web-scraper

A small web scraper to get the latest articles of a given topic from Techcrunch.com

r web-scraper web-scraping web-scrapper webscraper webscraping webscrapper webscrapping

Last synced: 11 Nov 2024

https://github.com/sky-walkersss/github-account-creator-bot

GitHub Account Creator is a Python tool that enables automatic account creation on GitHub. Automatically generating your credentials, the script quickly creates a new account.

api-integration authentication automation bot github github-api github-automation github-bot github-integration open-source python security selenium web-driver-chrome web-scraper

Last synced: 23 Nov 2024

https://github.com/lakshayd02/web_scraping_python

A program that extracts product information, such as names, prices and ratings from an online ecommerce website and stores the data in a structured format like a CSV file.

beautifulsoup csv csv-parser python python3 pythonlibrarires web-scraper web-scraping

Last synced: 18 Nov 2024

https://github.com/antheta/falcon-php

🌎 An intermediary for web scrapers with built-in parsers.

dynamic gateway scraper scraper-gateway scrapers web-scraper

Last synced: 18 Nov 2024

https://github.com/itachi1621/g2a_scraper

Python script scrapes product information from G2A, extracts pricing, ratings, and seller names, creates an HTML table using ChatGPT, and sends email notifications to recipients specified in the configuration file.

chatgpt chrome chromium g2a mailersend openai python python-3 python-script python3 selenium web-scraper web-scraping web-scraping-python

Last synced: 18 Nov 2024

https://github.com/robin-rpr/harvester

Web Data Extraction Tool 🚜✨

chrome-extension web-actions web-scraper

Last synced: 10 Nov 2024

https://github.com/sirbuig/bacalaureat-2024

Web scraper for the Romanian Baccalaureate website

bac2024 bacalaureat web-scraper

Last synced: 10 Nov 2024

https://github.com/uzairahmednasir/moviedbscrappercpp

a simple movies information scraper to show list of searched movies and displays them on console.

cpp-project object-oriented-programming oop-project web-scraper

Last synced: 10 Nov 2024

https://github.com/bradsec/gofindfiles

Crawl websites attempting to find and download files with matching file types. For use as OSINT or RECON intelligence collection tool.

crawler osint osint-tool recon scraper web-scraper

Last synced: 10 Nov 2024

https://github.com/mulaza/anime-movie-list-web-scraper

🍥A python web-scraper that scrapes the imdb 150 Top anime movie list page for movie ranks, titles, runtimes and descriptions and saves them to a .json file.

automation beautifulsoup beautifulsoup4 python web-scraper webscraper webscrapers webscraping webscrapping

Last synced: 25 Nov 2024

https://github.com/luminati-io/Airbnb-dataset-samples

A sample dataset of over 1000 Airbnb listings, extracted using the Bright Data API, ideal for competitor tracking, brand reputation, and market analysis.

airbnb airbnb-listings api data-analysis datasets web-scraper web-scraper-api web-scraping

Last synced: 06 Nov 2024

https://github.com/oxylabs/chatgpt-web-scraping

Learn to create ChatGPT prompts that generate a web scraping code with proper CSS selectors.

chatgpt chatgpt-scraping gpt-scraping web-scraper web-scraping

Last synced: 21 Nov 2024

https://github.com/luminati-io/Target-dataset-samples

A sample dataset of over 1000 target products, extracted using the Bright Data API, ideal for brand reputation, tracking inventory, and optimizing prices.

api data-analysis data-mining datasets target web-scraper web-scraping

Last synced: 06 Nov 2024

https://github.com/manavk97/web-scrapify

A simple web scraper that can scrape product details from various e-commerce platforms.

ai amazon e-commerce flipkart scraper smart web-scraper

Last synced: 30 Dec 2024

https://github.com/cassiofb-dev/web-scraper-uol-banco-de-redacoes

Um Web Scraper para transformar os dados do banco de radações da UOL em JSON

uol-redacoes web-scraper

Last synced: 28 Dec 2024

https://github.com/infallibless/web-scraper

scrape the html codes and data of any site you want

scraper web web-scraper

Last synced: 22 Nov 2024

https://github.com/airscholar/punchscraper

A News Web scraper Using Python

news python scraping web web-scraper web-scraping

Last synced: 14 Nov 2024

https://github.com/teohrt/abrasion

Collect emails through web scraping

go golang spider web-crawler web-scraper

Last synced: 27 Dec 2024

https://github.com/kecspas/github-account-creator-bot

GitHub Account Creator is a Python tool that enables automatic account creation on GitHub. Automatically generating your credentials, the script quickly creates a new account.

api-integration authentication automation bot github github-api github-automation github-bot github-integration open-source python security selenium web-driver-chrome web-scraper

Last synced: 30 Dec 2024

https://github.com/infinitode/pywebscrapr

An open-source Python web scraping tool. Supports both image scraping and text scraping.

data data-collection data-science open-source pip scraping web-scraper

Last synced: 08 Nov 2024

https://github.com/snehmehta23/writeai

Multi-agent crew that scrapes previously written beehiiv articles and then analyzes the tonage and writes a new article based off a given topic.

agentic-workflows ai ai-agents beehiiv llm ollama python web-scraper

Last synced: 29 Dec 2024

https://github.com/cs-magic-open/scraper_companies

自动化公司信息获取脚本（基于企查查）

aigc jupyter-notebook selenium sqlmodel web-scraper

Last synced: 21 Dec 2024

https://github.com/antodata/homosexualityhistorycurrentstatus

In this project we use folium, pandas and webscraping using requests and beautifulSoup to create a map that contains information about the legal status of homosexuality around the world. We created an API with a function called buildMap to create the map with the colors combining number ranges and static string values to color that map. Also we use request and beautifulSoup to get and parse all the wikipedia pages that contain information about lgbt rights in each country for the html we display when we open a marker (every country has a marker placed in its capital to display with information).

activism beautifulsoup beautifulsoup4 data-science decriminalization-homosexuality folium gay homosexuality html human-rights interactive-maps lgtb pandas python python3 requests web-scraper web-scraping webscraper webscraping

Last synced: 08 Dec 2024

https://github.com/luminati-io/google-news-scraper

Extract news data with the #1 Google News API, including top stories, publisher details, article summaries, and trending topics. Start your free trial now!

api datasets google-news google-news-api google-news-scraper python scraper-api scraper-python web-scraper

Last synced: 21 Dec 2024

https://github.com/rishav-karanjit/watchmen-keep-track-of-your-shopping-list

Desktop app that keeps track of items in shopping list and alerts when the price of the item has changed.

pyqt5 python web-scraper web-scraping

Last synced: 28 Nov 2024

https://github.com/shyguycreate/viva-scrapper

Python web-scrapper with Selenium to get available Viva Aerobus flights.

python scraper selenium vivaaerobus web-scraper

Last synced: 29 Nov 2024

https://github.com/khulnasoft-lab/autospider

AutoSpider: A Smart, Automatic, Fast Web Spider for Python

automatic spider web-scraper web-scraping-python web-scrapping

Last synced: 28 Nov 2024

https://github.com/alizul01/marsland

🌏 Mars Land is a website that is used to place reservations on the world of mars. So if u always see someone do uwu uwu things, get hurry grab your ticket for mars world!

flask python web-scraper web-scraping

Last synced: 11 Nov 2024

https://github.com/tildadares/internships-web-scraper

A NodeJS web scraper that scrapes internship opportunites from LinkedIn.

cheerio expressjs internships nodejs web-scraper

Last synced: 04 Jan 2025

https://github.com/maxmekiska/covid19-webscraper

Simple Covid-19 web scraper.

covid-19 web-scraper

Last synced: 08 Dec 2024

https://github.com/thetardigrade/golang-cachedpagedownloader

Go package to download webpages, or access previously cached versions of them.

downloader go golang web-scraper web-scraping webpage-capture

Last synced: 31 Dec 2024

https://github.com/primetdmomega/webscraper

A data web scraper that looks for jobs on Glassdoor.com

data python web-scraper

Last synced: 03 Dec 2024

https://github.com/noahgorstein/truckin-through-time

Source code to accompany my blog post Truckin' Through Time: Building a Grateful Dead Database

grateful-dead python web-scraper

Last synced: 04 Dec 2024

https://github.com/rmncldyo/firecrawl-toolkit

The Firecrawl Toolkit is the easiest way for developers to interact with web content through crawling, scraping, and mapping capabilities.

ai-batch-scrape ai-crawler ai-scraper ai-toolkit batch-scrape crawl fire-crawl firecrawl firecrawl-ai map scrape sitemap sitemap-crawler sitemap-scraper web-crawler web-scraper

Last synced: 04 Jan 2025

https://github.com/johnwmillr/docscraper

Automatically create functioning Python API wrappers from documentation

api api-wrapper python web-scraper web-scraping

Last synced: 10 Dec 2024

https://github.com/guhan-tofu/web-scraper

This will accept a link and return all text present in the page.

python selenium web-scraper webdriver

Last synced: 13 Nov 2024

https://github.com/thoughtgap/schnell-zulassen-guetersloh

Prüft freigewordene Termine bei der Kfz-Zulassungsstelle Gütersloh und sendet eine Benachrichtigung.

guetersloh web-scraper zulassung

Last synced: 01 Dec 2024

https://github.com/rkstudio585/pyscrape

Web scraper without any 3rd party library for app. Only use required requests, rich library.

pyscraper scrape web-scraper

Last synced: 17 Dec 2024

https://github.com/mee-to/github-account-creator-bot

GitHub Account Creator is a Python tool that enables automatic account creation on GitHub. Automatically generating your credentials, the script quickly creates a new account.

api-integration authentication automation bot github github-api github-automation github-bot github-integration open-source python security selenium web-driver-chrome web-scraper

Last synced: 14 Dec 2024

https://github.com/dmitry-zharinov/books-library

Парсинг и создание оффлайн-библиотеки

beautifulsoup django web-scraper

Last synced: 15 Nov 2024

https://github.com/sesaquecruz/dev-challenge-scraper-service

The Scraper Service solution for a Dev Challenge.

nodejs rabbitmq typescript web-scraper

Last synced: 14 Dec 2024

https://github.com/atia-farha/html-fetcher-script

The HTML Fetcher Script is a Python script that allows users to fetch and optionally save the HTML content from a specified URL using 'requests' library. This script provides user-friendly prompts for input validation, including URL format, redirect options, and timeout settings.

fetch-data fetcher fetching fetching-data fetching-data-from-server fetching-data-from-web html-fetcher html-fetcher-script html-scraper html-scraping python python-scraper python-script python3 pythonscript script web-scraper web-scraper-python web-scraping web-scraping-python

Last synced: 18 Dec 2024

https://github.com/prosenjitjoy/-follow-along-python-web-scraping

It will retrieve all the article over 100 points from Hacker News.

python web-scraper

Last synced: 19 Dec 2024

https://github.com/ffatahillah7/web-scraping-to-csv-using-python

Get data from web using Python and load or export to csv

beautifulsoup python web-scraper

Last synced: 25 Dec 2024

https://github.com/alphabs/navercafeclient

네이버 카페 글 목록 크롤링을 위한 닷넷 라이브러리

crawler crawling dotnet naver naver-api naver-cafe web-scraper web-scraping

Last synced: 29 Nov 2024

https://github.com/tiennhm/web-scraper-with-nodejs-and-typescript

Web scraper with Nodejs and Typescript

axios cheerio nodejs typescript web-scraper

Last synced: 29 Nov 2024

https://github.com/abdoomohamedd/beautifulsoup-web-scraping-projects

A collection of web scraping projects using BeautifulSoup, requests, and CSV modules to extract and analyze data from various websites.

beautifulsoup beautifulsoup4 csv requests web-scraper web-scraping

Last synced: 26 Dec 2024

https://github.com/ewliang/wsj-web-scraper

A simple web scraper to scrape the WSJ website.

data-mining javascript stock-market web-scraper

Last synced: 02 Jan 2025

https://github.com/kshittijagrawal/comparator

A web scraping tool that fetches a desired product from two of the most famous e-commerce platforms and compares the result provided by the latter.

automation web-scraper

Last synced: 13 Nov 2024

https://github.com/codera21/brainyq

Scraping qoutes from brainyqoute website

apify-sdk sdk web-scraper

Last synced: 28 Dec 2024

https://github.com/luminati-io/amazon-scraper

Extract Amazon data with the #1 Amazon Scraper API, including search results, product details, offers, reviews, Q&A, bestsellers, and seller information. Start your free trial now!

amazon amazon-api amazon-data amazon-dataset amazon-product-scraper amazon-reviews amazon-scraper amazon-scraping datasets e-commerce-scraper price-scraper python scraping-amazon web-scraper web-scraping

Last synced: 23 Nov 2024

https://github.com/steveleecode/price-tracker

Simple Amazon Price Tracker using BeautifulSoup4 (Web Scraper) and Tkinter (GUI)

bs4 requests tkinter web-scraper

Last synced: 26 Dec 2024

https://github.com/luminati-io/airbnb-dataset-samples

A sample dataset of over 1000 Airbnb listings, extracted using the Bright Data API, ideal for competitor tracking, brand reputation, and market analysis.

airbnb airbnb-listings api data-analysis datasets web-scraper web-scraper-api web-scraping

Last synced: 23 Nov 2024

https://github.com/luminati-io/target-dataset-samples

A sample dataset of over 1000 target products, extracted using the Bright Data API, ideal for brand reputation, tracking inventory, and optimizing prices.

api data-analysis data-mining datasets target web-scraper web-scraping

Last synced: 23 Nov 2024

https://github.com/k9mil/oculus-monitor

👁️ A scraper for Allegro & OLX to try and catch low priced items at user-set prices. Written in Python, with the help of BeautifulSoup4.

allegro archive camera oculus olx python python3 scraper web-scraper

Last synced: 28 Dec 2024

https://github.com/jaydotmurf/box2box

box2box is a dynamic football data extraction tool that uses rotating proxies to scrape web data

data-analysis python web-scraper

Last synced: 14 Nov 2024

https://github.com/oxylabs/how-to-scrape-amazon-product-data

The process of extracting product data from Amazon using Python, including titles, ratings, prices, images, and descriptions.

amazon amazon-scraper python web-scraper web-scraping web-scraping-python

Last synced: 17 Nov 2024

https://github.com/mauriciovazquezm/a_webscraper_example

Repository of an example of a web scraper designed to collect data from a website

beatifulsoup4 python web-scraper web-scraping

Last synced: 13 Nov 2024

https://github.com/oxylabs/how-to-scrape-amazon-prices

A code for extracting best-selling items, search results, and currently available deals from Amazon using Python and Oxylabs E-Commerce Scraper API.

amazon amazon-scraper api python python-scraper scraper-api web-scraper web-scraping

Last synced: 17 Nov 2024

https://github.com/oxylabs/how-to-scrape-indeed

A tutorial for collecting job postings from Indeed using Python and Oxylabs Web Scraper API.

api job-posting python scraper-api web-scraper web-scraping

Last synced: 17 Nov 2024

https://github.com/sergeypedan/integral-web-scraper

Downloads a set of web pages from your URLs list, saves them into a local SQLite file and calls your transformer class to deal with them, saving the results into the DB. Can further export transformed HTML or create an ePub.

epub web-scraper web-scraping

Last synced: 13 Nov 2024

https://github.com/dvgamerr/cinema-scraper

Now showing and comming soon movie in major cineplex and SF cinema city scraping.

major major-cineplex movie sf sf-cinema web-scraper

Last synced: 12 Nov 2024

https://github.com/oxylabs/best-buy-price-tracker

A tutorial for building a scalable price tracker with Python and Oxylabs Best Buy Scraper API to get price change alerts and historical data.

change-monitoring price-tracker price-tracking-system scraper-api web-scraper web-scraping

Last synced: 17 Nov 2024

https://github.com/sahaavi/web-scraping

Learn Web-Scraping using BeautifulSoup, Selenium and Scrapy with hands on projects!

beautifulsoup4 crawler headless-mode pagination scrapy selenium spider splash web-scraper web-scraping

Last synced: 26 Dec 2024

https://github.com/jonathanlinat/simple-nodejs-amazon-web-scraper

A very simple and automated tool to scrape Amazon.com results

amazon cheerio nodejs puppeteer web-scraper

Last synced: 14 Nov 2024

https://github.com/murtaza-arif/wiki-tree

What happens when you keep following the first link in Wikipedia articles? You almost always get back to the articles on Existence or Awareness

rust web-scraper wiki-links wikipedia

Last synced: 20 Dec 2024