Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

Projects in Awesome Lists tagged with web-scraping-python

A curated list of projects in awesome lists tagged with web-scraping-python .

https://github.com/scrapy/scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.

crawler crawling framework hacktoberfest python scraping web-scraping web-scraping-python

Last synced: 12 May 2025

https://github.com/D4Vinci/Scrapling

🕷️ An undetectable, powerful, flexible, high-performance Python library to make Web Scraping Easy and Effortless as it should be!

ai ai-scraping automation crawler crawling crawling-python data data-extraction hacktoberfest playwright python python3 scraping selectors stealth web-scraper web-scraping web-scraping-python webscraping xpath

Last synced: 13 May 2025

https://github.com/d4vinci/scrapling

🕷️ An undetectable, powerful, flexible, high-performance Python library to make Web Scraping Easy and Effortless as it should be!

ai ai-scraping automation crawler crawling crawling-python data data-extraction hacktoberfest playwright python python3 scraping selectors stealth web-scraper web-scraping web-scraping-python webscraping xpath

Last synced: 13 May 2025

https://github.com/tinyfish-io/agentql

AgentQL is a suite of tools for connecting your AI to the web. Featuring a query language and Playwright integrations for interacting with elements and extracting data quickly, precisely, and at scale. Includes REST API, Python and JavaScript SDKs, browser debugger.

agent ai aiagent automation javascript playwright python rpa scraping web web-scraping web-scraping-colabs web-scraping-javascript web-scraping-python web-scrapping webagent

Last synced: 15 May 2025

https://github.com/oxylabs/python-web-scraping-tutorial

In this Python Web Scraping Tutorial, we will outline everything needed to get started with web scraping. We will begin with simple examples and move on to relatively more complex.

amazon-scraper-python crawler github-python json-database-python python python-projects python-web-crawler python-web-scraper scraper-python scraping web-crawler-python web-scraping web-scraping-api web-scraping-python webscraping

Last synced: 16 May 2025

https://github.com/oxylabs/how-to-scrape-amazon-product-data

The process of extracting product data from Amazon using Python, including titles, ratings, prices, images, and descriptions.

amazon amazon-scraper python web-scraper web-scraping web-scraping-python

Last synced: 14 Apr 2025

https://github.com/pb2204/covid-19

This Is A Web Scraping Projects With Covid-19 Data From 2 Very Popular & Authentic Websites

covid-19 covid19-data web-scraping web-scraping-python web-scrapping

Last synced: 15 Apr 2025

https://github.com/oxylabs/asynchronous-web-scraping-python

A comparison of asynchronous and synchronous web scraping methods with practical examples.

async asynchronous data-acquisition python synchronous tutorial web-scraping web-scraping-python web-scraping-tutorials

Last synced: 27 Sep 2025

https://github.com/lakshayd02/web_scraping_python

This repository contains a Python program that scrapes product information (names, prices, ratings, etc.) from an e-commerce website and stores the data in a CSV file. A useful tool for data collection and analysis! 📊

beautifulsoup csv csv-parser python python3 pythonlibrarires web-scraper web-scraping web-scraping-python

Last synced: 15 Apr 2025

https://github.com/oxylabs/curl-with-python

Master cURL in Python by using the PycURL library. Learn to send GET and POST requests, custom HTTP headers, and how to fix common problems.

curl-library curl-python pycurl web-scraping web-scraping-python

Last synced: 05 Sep 2025

https://github.com/oxylabs/parse-html-pyquery

Learn to parse HTML using PyQuery, a Python library for web scraping and manipulating HTML.

parser parsing pyquery python web-scraping web-scraping-python

Last synced: 17 Feb 2025

https://github.com/demon-2-angel/money_control_scrap

Money_Control_Scrap is a Python package for scraping the latest business news from the Money Control website (One of the biggest website on Finance ion India)

finance-news money-control python python-automation web-scraper web-scraping-python

Last synced: 28 Feb 2025

https://github.com/harshavardh8n/flipkart-scraping

Flipkart-Scraping: A Python-based web scraping project using BeautifulSoup to extract product information from Flipkart. This script retrieves data such as product names, prices, and ratings, providing insights and analysis of Flipkart’s product listings.

beautifulsoup4 python web-scraping-python

Last synced: 18 Oct 2025

https://github.com/lynkos/downloader

Basic web scraper to download media from websites. Supports .pdf generation and vertical image stacking; useful for downloading manga, comics, etc.

python python-script python3 scraper scraping scrapping script web-scraper web-scraping web-scraping-python web-scrapping

Last synced: 14 Oct 2025

https://github.com/breadybred/codewars-rank-fetcher

This Python GUI application simplifies the process of fetching and storing your Codewars ranking data across various categories. It provides a user-friendly interface for configuration and displays retrieved ranks in a clear format.

codewars gui python web-scraping web-scraping-python

Last synced: 21 Mar 2025

https://github.com/samshad/data_scrape_auto_tinder

Data Scrape & Auto‑Swipe for Tinder – Python scripts that authenticate with Tinder’s unofficial API, save profile metadata to CSV, and auto‑like/pass based on simple filters. For educational use only, automation violates Tinder’s ToS.

automation cli data-collection tinder-swiper web-scraping-python

Last synced: 10 Jun 2025

https://github.com/bhattjayd/passbreachfinder

A Python script that checks whether a password has been compromised using the Have I Been Pwned service. The script automates the process of querying the website and retrieving the results for the given password, leveraging Selenium and a headless Firefox browser. It’s a simple tool for testing password security and checking for data breaches.

automation command-line-tool cybersecurity data-breach data-breach-checker geckodriver haveibeenpwned headless-browser password-leak password-security python security-tool selenium web-scraping web-scraping-python

Last synced: 07 Apr 2025

https://github.com/sarthak-0-sach/selenium_web-scraping_project

Automation and web scraping of Instagram just became easier!! This repository is a collection of notebooks related to different purposes in process of Instagram web scraping and automation with well documented code

automation-testing instagram-scraping-bot jupyter-notebook python selenium-python selenium-webdriver web-scraping-python

Last synced: 09 Mar 2025

https://github.com/vlagehj/5chsita_mpcrwrl

community response crawler for MapleStort, Nexon

automation nlp notion-api python selenium transfomers web-scraping-python

Last synced: 10 Oct 2025

https://github.com/frarlo/garfield_bluesky_bot

Simple Python Bluesky bot to post random Garfield comics every four hours.

github-actions python3 web-scraping-python

Last synced: 13 Oct 2025

https://github.com/anonymousx46/advanced-web-scrapping-tool

A web-scrapping-tool I built to automate the process with advanced techniques, ready to use in your Kali Linux Terminal!

amass automation dig kali-linux nikto-features nslookup python web-scraping-python whois-lookup

Last synced: 23 Mar 2025

https://github.com/gauff/belgianelectriccarmarketanalyser

Python tool for analyzing the belgian second hand electric car market by scraping and visualizing data from multiple car listing websites. Features parallel web scraping, price tracking, and interactive dashboards.

automotive beautifulsoup4 car-market dash data-analysis-python data-cleaning data-visualization electric-vehicles market-analysis pandas parallel-processing plotly price-comparison price-monitoring selenium web-scraping-python

Last synced: 22 Jul 2025

https://github.com/sathviknayak123/sentiment-anyalysis

Sentiment Analysis using DistlBERT Transformer from HuggingFace. Also integrated Airflow for end-to-end pipeline

airflow astronomer distilbert flask huggingface-transformer nlp python s3-bucket selenium-webdriver sentiment-analysis tensorflow web-scraping-python

Last synced: 06 Apr 2025

https://github.com/codeterrayt/dare2024.com-solver

Dare2024.com Solver is a Python automation script for seamlessly solving Dare2024.com quizzes. Impress your friends with correct answers effortlessly. Compatible with all dare2024.com versions and future updates.

automation-script automation-scripts dare2024 headless-browser headless-browsers opensource python python3 quiz-solver selenium selenium-python selenium-webdriver web-automation web-automation-with-selenium web-scraping web-scraping-project web-scraping-python web-scraping-software webdriver-manager

Last synced: 23 Mar 2025

https://github.com/jakbin/pcdt-scraper

A PyChromeDevTools based WebScraper and selenium like syntax.

pychromedevtools python-chrome web-scraper web-scraping-python webscraper webscraping

Last synced: 20 Jun 2025

https://github.com/khulnasoft-lab/autospider

AutoSpider: A Smart, Automatic, Fast Web Spider for Python

automatic spider web-scraper web-scraping-python web-scrapping

Last synced: 30 Mar 2025

https://github.com/badranalyst/scraping-data-from-a-real-website

Web-scraped data on the largest U.S. companies by revenue, capturing rank, name, industry, revenue (in USD billions), employees, and headquarters location. Data is structured into a CSV dataset, ready for analysis and insights into major corporate players.

beautifulsoup beautifulsoup4 python requests web-scraping web-scraping-python

Last synced: 26 Feb 2025

https://github.com/abelarduu/global-currency-viewer

Visualizador de moeda global que monitora em tempo real as cotações de moedas como Dólar, Euro, Libra e Iene em relação ao real brasileiro, utilizando web scraping e gráficos interativos. Desenvolvido em Python com Requests, BeautifulSoup, Pandas e Matplotlib.

beautifulsoup data-visualization finance grafico matplotlib matplotlib-python pandas-python python requests-python web-scraping-python web-scrapping

Last synced: 02 Aug 2025

https://github.com/jsem-nerad/strava-cz-python

High level API pro interakci s webovou aplikaci Strava.cz udelane v Pythonu

api czech python python-3 python3 requests requests-library-python web-scraper web-scraping web-scraping-python webscraper webscraping

Last synced: 14 Oct 2025

https://github.com/joyalmpaul/coursicle-ratings

Web Scraping Application using Coursicle to organize Professors ratings

beautifulsoup4 html web-scraping-python

Last synced: 26 Aug 2025

https://github.com/joao-pedro-p-holanda/gh-education-offers-scrapper

Simple python script for extracting all offers from the student pack on Github Education

beautifulsoup4 github-education web-scraping-python

Last synced: 15 Jul 2025

https://github.com/subhanalii/instagram-scraper

A Python automation tool that logs into Instagram, searches profiles via Bing, scrapes public data like bio, followers, and emails, and saves the results. Demo included. Full script available on request.

automation data-collection data-extraction lead-generation python scraper scraper-api web-scraping-python webscraping webscraping-beautifulsoup webscraping-selenium

Last synced: 18 Oct 2025

https://github.com/itachi1621/g2a_scraper

Python script scrapes product information from G2A, extracts pricing, ratings, and seller names, creates an HTML table using ChatGPT, and sends email notifications to recipients specified in the configuration file.

chatgpt chrome chromium g2a mailersend openai python python-3 python-script python3 selenium web-scraper web-scraping web-scraping-python

Last synced: 11 Jul 2025

https://github.com/yahia-kilany/oscar-nominations-database

Oscars Database Project is a comprehensive system designed to store, manage, and query detailed data about the Academy Awards (Oscars). This project includes both terminal-based and web-based applications to interact with the data, which covers Oscar-related information from the 10th to the 96th iteration.

academy-awards data-engineering database database-schema deployment flask flask-application jupyter jupyter-notebook mysql mysql-database python python3 railway railway-app university-course university-project web-scraping web-scraping-python web-scrapper

Last synced: 15 Jun 2025

https://github.com/tinyhiker/web_scraping_amazon_purchase_tracker

Creating an app that allows users to track their product views in python and register items that they might buy in the future. This project incorporates web-scraping and a sqlite3 database

batch-script beautifulsoup4 decorators-python logging-python rich-library sqlite3-database unit-testing web-scraping-python

Last synced: 29 Mar 2025

https://github.com/mtsamm/jupyter-notebooks-web-scraping

Perform web scraping interactively using Jupyter Notebooks, integrating coding, data analysis, and visualization into one seamless workflow.

beer firebase google-scholar-scraper iahoraa jupyter-notebook mongodb numpy pandas-dataframes scraper selenium-python web-data web-scraping web-scraping-python webscra

Last synced: 03 Mar 2025

https://github.com/mr-mudgal/amazon-scrapper

This Python-based Amazon Scraper is designed to efficiently extract detailed product data from Amazon's product pages. The tool leverages powerful libraries like BeautifulSoup4 and csv, along with the Scrapingant API to simulate browser behavior and bypass Amazon’s anti-scraping algorithms.

amazon csv csv-export python3 scraper scraping scraping-framework scraping-websites web-scraper web-scraping web-scraping-python web-scraping-software web-scrapping

Last synced: 26 Mar 2025

https://github.com/madhurimarawat/csvtu-gpt

This repository hosts the CSVTU GPT app, a Streamlit-based interactive application designed to provide efficient access to subject-specific academic information and resources. It supports functionalities like fuzzy matching, exact word matching, and syllabus search capabilities, enabling users to query data conveniently.

code-comments detailed-explanation documentation exact-matching fuzzy-matching fuzzy-matching-algorithm illustrations intermediate-project minor-project project-resources python question-answering-system readme resources streamilt-webapp streamlit streamlit-deployment web-scraping web-scraping-python

Last synced: 03 Apr 2025

https://github.com/harmanveer-2546/movie-industry

Investigate the film industry to gain sufficient understanding of what attributes to success and in turn utilize this analysis to create actionable recommendations for companies to enter the industry.

business business-analytics data-analysis datatime film-industry graphs matplotlib movie-database numpy pandas python scraping-websites seaborn visualization web-scraping-python

Last synced: 28 Feb 2025

https://github.com/marionchaff/windguru-scraper

Python script that scrapes weather forecast from Windguru using Selenium

chromedriver python scraping selenium weather-forecast web-scraping-python windguru

Last synced: 12 Apr 2025

https://github.com/pj-pj-pj/steam-deals-scraper

trying to make a web scraper for getting the current sales or deals on steam (and learning python)

python screen-scrape steam try web-scraping web-scraping-python

Last synced: 06 Jul 2025

https://github.com/00ryanwelzel/minionprofitscalculator

Quick little script to calculate profits in a game I like.

beautifulsoup4 python web-scraping-python

Last synced: 30 Apr 2025

https://github.com/romangw/lukki

Completely free code for a webcrawling bot.

crawler python web-scraping web-scraping-python

Last synced: 08 Oct 2025

https://github.com/code-quang/linkedin-scraping

I scraped the specific company follower's url, name, education and so on.

linkedin profile-scraper python scraping web-scraping web-scraping-python

Last synced: 13 Mar 2025