Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

Projects in Awesome Lists tagged with data-scraping

A curated list of projects in awesome lists tagged with data-scraping .

https://github.com/je-suis-tm/web-scraping

Detailed web scraping tutorials for dummies with financial data crawlers on Reddit WallStreetBets, CME (both options and futures), US Treasury, CFTC, LME, MacroTrends, SHFE and alternative data crawlers on Tomtom, BBC, Wall Street Journal, Al Jazeera, Reuters, Financial Times, Bloomberg, CNN, Fortune, The Economist

bloomberg data-scraper data-scraping financial-data financial-times futures futures-historical-data news-scraper news-websites newsletter options-data python-web-scraper reuters scrapper sraping wall-street-journal wallstreetbets web-scraper web-scrapers web-scraping

Last synced: 04 Apr 2025

https://github.com/scriptsmith/reaper

Social media scraping / data collection tool for the Facebook, Twitter, Reddit, YouTube, Pinterest, and Tumblr APIs

api data-collection data-mining data-scraping facebook gui pinterest reddit scraping socialmedia tumblr twitter youtube

Last synced: 07 Apr 2025

https://github.com/ScriptSmith/reaper

Social media scraping / data collection tool for the Facebook, Twitter, Reddit, YouTube, Pinterest, and Tumblr APIs

api data-collection data-mining data-scraping facebook gui pinterest reddit scraping socialmedia tumblr twitter youtube

Last synced: 04 Apr 2025

https://github.com/toki-plus/video-mover

全自动短视频搬运工具，支持自动下载、去重、AI生成标题+标签、上传，可二开扩展至多平台，例如：TikTok->视频号/抖音/小红书、抖音->TikTok/视频号/小红书......video-processing, automation, tiktok, selenium, pyqt5, ffmpeg, bot, data-scraping, video-deduplication.

automation bot data-scraping ffmpeg pyqt5 selenium tiktok video-deduplication video-processing

Last synced: 29 Apr 2026

https://github.com/dddat1017/Scraping-Youtube-Comments

Scrape comments from any Youtube video

data-scraping python selenium web-scraping

Last synced: 27 Jul 2025

https://github.com/kb22/GitHub-User-Insights-using-API

The project involves using the GitHub API using user authentication to fetch information such as commits and repositories for that specific user and store them as CSV files for data collection and analysis.

api data-analysis data-science data-scraping github-api python

Last synced: 16 Apr 2025

https://github.com/atharva-phatak/analysing-glassdoor-jobs

Data Analysis of Job Postings on Glassdoor.

data-scraping glassdoor

Last synced: 27 Jul 2025

https://github.com/simonw/disaster-data

Data scraped by https://github.com/simonw/disaster-scrapers

data-scraping git-scraping irma-response json

Last synced: 19 Apr 2025

https://github.com/toki-plus/ai-ttv-workflow

本项目专为内容创作者、新媒体运营及开发者设计，致力于简化视频制作流程。无论是文章、脚本，甚至是已有的视频文案，都能被快速转换成可直接发布的短视频，适用于抖音、B站、YouTube Shorts、小红书等平台。video-processing, automation, tiktok, selenium, pyqt5, ffmpeg, bot, data-scraping, video-deduplication.

automation bot data-scraping ffmpeg pyqt5 selenium tiktok video-deduplication video-processing

Last synced: 29 Apr 2026

https://github.com/somdeep/Statball

Statball - Football soccer stats analyser from top 5 european leagues with data obtained by web scraping from Fbref and Statsbomb

csharp data-science data-scraping data-viz dotnet dotnet-core fbref football football-analytics football-data scouting-data scraping soccer soccer-analytics soccer-data statsbomb tableau visualizations

Last synced: 02 Apr 2025

https://github.com/zero-to-mastery/complete-python-developer-manual

Class notes for Andrei Neagoie's Complete Python Developer course through Zero to Mastery.

data-scraping object-oriented-programming python python3 scripting webdevelopment

Last synced: 29 Jun 2025

https://github.com/easonlai/facebook_post_scraping_and_text_analytics

This is demo repo to demostrate how to scrape post data from Facebook by Python with library facebook_scraper. And then use Azure Text Analytics to perform sentiment analysis for post text content.

azure azure-text-analysis azure-text-analytics data-scraping facebook facebook-post facebook-scraper facebook-scraping key-phrase-extraction microsoft-azure microsoft-cognitive-services pandas python python3 scraping-python seaborn text-analytics wordcloud

Last synced: 26 Apr 2025

https://github.com/wingkwong/hk-atm-locator

:atm: 香港自動櫃員機定位器 :atm: Centralising Automated Teller Machine (ATM) Data in Hong Kong in a well-defined yet standardised format and display in a web portal for public use

atm data-enrichment data-scraping data-transformation hk-atm-locator hong-kong-atm open-api

Last synced: 11 Apr 2025

https://github.com/charlywargnier/s4_wiki_topic_grapher

Leverage the power of the Google Natural Language API NLP to retrieve entity relationships from Wikipedia URLs or topics! Get interactive networkx graphs of connected entities!

data-scraping datascience nlp nlproc seo wikipedia

Last synced: 15 May 2025

https://github.com/asoul-sig/acao

🦙 阿草，数据爬取工具人 / acao, the tool man for data scraping of https://asoul.video/.

a-soul data-scraping douyin douyin-api

Last synced: 03 Jul 2025

https://github.com/easonlai/playstore_reviews_scraping_and_text_analytics

This is demo repo to demostrate how to scrape apps review data from Google Play Store by Python with library Google-Play-Scraper. And then use Azure Text Analytics to perform sentiment analysis for reviews content (aka comments).

azure azure-text-analysis azure-text-analytics data-scraping datascraping google-play-store google-play-store-data-analysis google-play-store-scraper microsoft-azure microsoft-cognitive-services pandas python python3 seaborn sentiment-analysis text-analytics

Last synced: 08 Aug 2025

https://github.com/chaitanyarahalkar/financial-info-extractor

Extract financial information in CSV format for companies compliant to the NSE

beautifulsoup csv-parser data-extraction data-scraping financial-data financial-services python selenium

Last synced: 17 Aug 2025

https://github.com/kaustubhgupta/blogathon-analysis

Analytics Vidhya Blogathon Data Analysis: Python Data Extraction with PowerBI dashboard

data-analysis data-mining data-scraping excel pandas powerbi powerbi-report project python tqdm

Last synced: 12 Apr 2025

https://github.com/leabs/misc-python-apps

Miscellaneous Python applications or scripts I've made to help me learn Python and use as productivity tools.

data-scraping nyslrs password-generator retirement-calculator weather-api youtube-analytics

Last synced: 20 Jul 2025

https://github.com/erkurn/locanesia

Laravel Package to populate and searching indonesia locations. From Provinces To Villages. Data Scraped from http://www.nomor.net/

data-scraping laravel localization province village

Last synced: 13 Jan 2026

https://github.com/siddeshsambasivam/ntuoss-datascraping-and-datacleaning-workshop

This repository contains the reference scripts and the content presented in the NTU OSS Data scraping and Data cleaning workshop.

data-cleaning data-crawling data-scraping

Last synced: 12 May 2025

https://github.com/ryanrudes/wikimedia

A dataset comprised of over 40 million images sourced from Wikimedia Commons

computer-vision data-science data-scraping dataset datasets deep-learning gans image images machine-learning wikimedia wikimedia-commons

Last synced: 13 Sep 2025

https://github.com/sondosaabed/palestinian-movies-json-dataset

Github allows me to call a JSON file like a server. So this is my raw data scrapped using apify (imdb-scrapper) 🇵🇸

data-scraping imbd-dataset imdb json-api movies-api palestine

Last synced: 09 Feb 2026

https://github.com/mzubairtahir/latest-twitter-scraper

This python scraper is for latest twitter website structure , that scrapes tweets of an twitter account

data-scraping latest-twitter playwright-python python-automation python3 twitter-bot twitter-scraper web-crawler web-scraping

Last synced: 27 Jun 2025

https://github.com/anonym0uswork1221/python-code-docstring-scraper

A multi-threaded GitHub scraper to collect Python code with docstrings from public repositories, creating a well-documented dataset for the JaraConverse LLM model.

causal-language-modeling data-scraping dataset dataset-generation dataset-scripts docst docstring-generator github-scraper llm llm-training nlp nlp-machine-learning python-code python-dataset python3 scraper script

Last synced: 11 Jan 2026

https://github.com/kishlayjeet/github-topics-data-scraping

This code is a web scraping script that extracts data from GitHub. It creates a CSV file with the top 100 topics from GitHub and the top 20 repositories for each topic.

beautifulsoup data-scraping featured-repo github pandas pandas-dataframe python python-script requests scraping-websites web-scraping

Last synced: 28 Feb 2026

https://github.com/theankurgoswami/linkedin-scraper

Scraping linkedin connections

data-scraping linkedin python scraper

Last synced: 27 May 2026

https://github.com/bitartisan1/netdigger

A .NET 8.0 C# WPF desktop application for web scraping data into structured databases with a modern UI, comprehensive logging and optimized high performance.

csharp data data-scraper data-scraping database desktop dotnet internet logging scraper ui url web-scraper web-scrapers web-scraping web-scrapping

Last synced: 13 Apr 2025

https://github.com/ibizzi-khalid/data-harvest-ecommerce-data-collector-analyzer-

Data Harvest is a tool that helps you make informed purchasing decisions. We analyze product reviews from major e-commerce sites to provide you with detailed information, sentiment analysis, and key insights. Our goal is to simplify your shopping experience and help you find the best products.

data-scraping machine-learning python reactjs sentiment-analysis

Last synced: 19 May 2026

https://github.com/pyrmont/feedstock

A Ruby library for data extraction that can be used to make RSS feeds from webpages

atom-feed data-scraping erb-template rss-feed ruby

Last synced: 02 Aug 2025

https://github.com/sharmadhiraj/free-json-datasets

Collection of free JSON data that are scraped and parsed from different websites.

collection crawler data data-scraping datasets json sports statistics web-scraping

Last synced: 28 Mar 2025

https://github.com/screamman-ssh/football-live-score-chrome-extension

Football Live Score is a Chrome extension that reports football match fixtures, match results, match stats and live scores.

chrome-extension data-scraping expressjs football football-live javascript live-scores manifest-v3 nodejs puppeteer self-hosted sports web-scraping

Last synced: 13 Apr 2026

https://github.com/kuuhaku86/data-scraping-tokopedia

Repository for web scraping article at Medium

data-scraping e-commerce python3 tokopedia web-scraping

Last synced: 30 Jun 2026

https://github.com/watson-clara/logics-database-report-scrapper

This program uses Selenium to scrape data from multiple databases.

data-scraping database logics scraper selenium

Last synced: 30 Apr 2026

https://github.com/easonlai/scraping_data_from_pdf

Code repository sample to demonstrate how to scrape table data from PDF file.

camelot data-analysis-python data-analytics data-scraping pdf python python3

Last synced: 11 Aug 2025

https://github.com/abhisingam/twitter-scraping

A program to scrape tweets from the top hashtag on twitter and analyse it in terms of content, popularity, device, location, etc. Python | Data Scraping | Data Visualization | Tweepy | MatPlotLib

data-scraping data-visualization insights matplotlib python3 tweepy tweets

Last synced: 08 Jul 2025

https://github.com/prashver/salary-prediction-on-scraped-data

This tool predicts data science salaries by scraping Glassdoor job descriptions. Features quantify company valuation of key skills. Regression models include Multiple Linear Regression, Lasso Regression, and Random Forest. Random Forest outperformed others. Technologies/libraries used: Python, Pandas, NumPy, Scikit-learn, Seaborn, Matplotlib.

data-scraping machine-learning-algorithms matplotlib numpy pandas regression-models salary-estimator seaborn selenium selenium-webdriver

Last synced: 06 Apr 2026

https://github.com/cptlobster/aggregation-framework

A Swiss-army knife Scala library for scraping and processing data from the web.

data-scraping http kafka kafka-producer scala selenium

Last synced: 17 May 2026

https://github.com/serhaturtis/tool-libgenscraper

A little script to download books and articles from libgen.

article data-mining data-scraping ebook libgen python scraper scraping

Last synced: 11 Sep 2025

https://github.com/fatihilhan42/web_scraping_football_statistics_per_game_data-main

In this notebook I will describe the process of scraping data from web portal understat.com that has a lot of statistical information about all games in top 5 European football leagues.

data-analysis data-manipulation data-science data-scraping data-visualization jupyter-notebook python

Last synced: 19 May 2026

https://github.com/shahrozatiq/hevy-exercises-data-scraper

This bot automates the process of logging into the Hevy website using your email and password, navigating to the exercises section, and extracting detailed data for each exercise. The extracted attributes include: Name Equipment Primary Muscle Secondary Muscle Source Source Type Thumbnail The data is systematically stored in a CSV file

data-scraper data-scraping dataset exercises exercises-dataset hevy-scraper python selenium selenium-webdriver workout-exercises

Last synced: 15 Aug 2025

https://github.com/tedoaba/kaim-w7

Building a Data Warehouse to Store Data on Ethiopian Medical Business Data Scraped from Telegram Channels

data-scraping data-warehousing dbt kaim medical-business-data postgresql sqlalchemy store-data telegram telegram-api telethon yolov5

Last synced: 20 May 2026

https://github.com/georift/australian-vaccine-rollout

Australian vaccine rollout data

covid-19 data-scraping git-scraping

Last synced: 18 Jul 2025

https://github.com/ch1nmay7898/inspire-posts-downloader

Inspire.com is a community forum to engage in discussions about diseases. Select/search from a list of Inspire communities and use this scraper to scrape publicly available posts made by users.

data-science data-scraping diseases healthcare python3 research scraper

Last synced: 22 Sep 2025

https://github.com/kaymen99/google-maps-lead-generator

Extract Google Maps business leads and enrich contact details using AI & web scraping

ai-agents ai-scraping data-scraping google-maps google-maps-api lead-generation web-scraping

Last synced: 04 Oct 2025

https://github.com/aymane-maghouti/sentiment-analysis-for-jumia-reviews-and-smartphone-price-prediction-system

The project focuses on customer sentiment analysis for Jumia, aiding informed online decisions. It collects and analyzes product comments to determine sentiments and implements a decision-making algorithm. Additionally, it includes product price prediction system using regression techniques.

beutifulsoup data-analysis data-cleaning data-collection data-preprocessing data-scraping data-visualization eda falsk machine-learning python web-application

Last synced: 18 Apr 2026

https://github.com/maxinexiong/acme-work-items-rpa

This repository contains a robust UiPath automation solution that utilises the UiPath REFramework to fulfill the specified requirements, which includes automating data scraping from acme-test.com, filtering specific records, and appending the results into an Excel worksheet.

acme-challenge data-scraping datatable excel-operations reframework robotic-enterprise-framework robotic-process-automation rpa table-extraction uipath uipath-modern-design uipath-reframework uipath-studio web-scraping

Last synced: 21 Jan 2026

https://github.com/mzubairtahir/amazon-search-result-scraper

A scraper that scrapes products from the search results of Amazon

amazon-bot amazon-scraper data-scraping product-scraper python scraper scraping web-scraping

Last synced: 09 Apr 2025

https://github.com/mzubairtahir/ebay-search-result-scraper

It is a python scraper scraping search results from 'ebay.com' website

data-mining data-scraping ebay ebay-scraper ecommerce-scrapper python3 web-scraping

Last synced: 31 Aug 2025

https://github.com/satoshi-sh/leaderboard-tracker

Tracking the Kaggle LeaderBoard progress and visualize the data with Stramlit.

data-scraping python selenium visualization

Last synced: 29 Apr 2026

https://github.com/iamgmujtaba/github-python-daily-trending

This repository provides an automated, daily-updated list of the top trending Python repositories on GitHub. Using a GitHub Actions workflow, it scrapes data from GitHub's trending page, sorts the results by total stars, and generates a clean, well-structured README file

data data-scraping github-actions tranding tranding-bot

Last synced: 13 Oct 2025

https://github.com/dagiteferi/ethiopian-medical-datawarehouse

Building a data warehouse for Ethiopian medical businesses by scraping data from Telegram channels. The project integrates object detection using YOLO and provides CRUD operations via FastAPI to enable comprehensive data analysis and user interaction.

data-scraping data-warehouse dpt fastapi postgresql yolo

Last synced: 17 Apr 2026

https://github.com/sebastianbrzustowicz/data-scraping-webapp

Django + Pandas. A straightforward static web page for scraping tables from a chosen site.

data-scraping django pandas python

Last synced: 29 Apr 2026

https://github.com/lixx21/coursera-data-scarping

End-to-end project to scraping courses in coursera

data-science-projects data-scraping scrapy streamlit

Last synced: 01 May 2026

https://github.com/bernta/norwegiandemographics

Norwegian Demographics using data from 1900-1910, analyzed with Hadoop & Spark.

ancestry census data-science data-scraping datascience demography hadoop norway python spark

Last synced: 02 May 2026

https://github.com/lixx21/spotify-scrapping

Scraping data from Spotify Playlist URL using Python and Selenium

data-engineer data-scraping python scraping-websites selenium spotify-playlist

Last synced: 03 May 2026

https://github.com/mahmoodhamdi/esports-flask

A comprehensive Flask REST API providing real-time data for the Esports World Cup, including tournaments, teams, players, matches, and news. Features advanced search, live data scraping, and a full admin interface.

api beautifulsoup4 cors cs2 data-scraping esports esports-api flask football liquipedia openapi python rest-api sqlite swagger valorant

Last synced: 08 May 2026

https://github.com/parvshah22/rag_free_cources_assisstant

This is a RAG based smart search tool for recommending free online courses for upskilling

data-scraping generative-ai groq huggingface llama3 pinecone python rag retrival-augmented-generation vector-database

Last synced: 13 Apr 2026

https://github.com/paologazzola/etf-open-analytics

ETF Open Analytics is an open-source project for scraping and analyzing historical ETF (Exchange-Traded Funds) data.

data-scraping etf finance financial-data github-actions machine-learning open-data portfolio-analysis postgresql python yfinance

Last synced: 20 Jan 2026

https://github.com/sridharyadav07/data-science-project-british-airways-reviews-

This project showcases skills in web scraping, data preprocessing, sentiment analysis, machine learning, and data visualization, making it a comprehensive example of applying data science techniques to real-world problems.

data-scraping jupyter-notebook matplotlib nltk pandas python seaborn sentiment-analysis sklearn wordcloud

Last synced: 06 May 2026

https://github.com/arya-io/flipkart-data-scraping

A data scraping project to extract product information such as names, prices, descriptions, and ratings from Flipkart using Selenium and BeautifulSoup.

beautifulsoup data-science data-scraping flipkart pandas python selenium web-scraping

Last synced: 07 May 2026

https://github.com/simonpierreboucher/crawler

A robust, modular web crawler built in Python for extracting and saving content from websites. This crawler is specifically designed to extract text content from both HTML and PDF files, saving them in a structured format with metadata.

concurrent-crawling content-extraction data-collection data-extraction-pipeline data-preservation-and-recovery data-scraping error-handling html-parsing http-requests metadata-storage modular-design pdf-text-extraction python-crawler rate-limiting structured-data-storage text-processing url-normalization web-crawling yaml-configuration

Last synced: 30 Mar 2025

https://github.com/imshakil/torrent-peers-scrape

A simple python script to scrape peers from torrent files

data-scraping peers python scraper torrent

Last synced: 01 Apr 2025

https://github.com/mfa/waterlevel-bw

Get waterlevels in Baden-Württemberg

data-scraping git-scraping

Last synced: 13 Jun 2025

https://github.com/engkinandatama/ncbi-sequence-fetcher

NCBI Sequence Fetcher is a Python desktop app for downloading nucleotide sequences and extracting metadata from NCBI. It features an easy-to-use GUI, supports FASTA and GenBank formats, and helps researchers students and bioinformaticians efficiently collect DNA sequences and store metadata in Excel files.

academic bioinformatics bioinformatics-tool data-scraping fasta genbank metadata metadata-extraction molecular-biology ncbi nucleotide-sequences python tkinter

Last synced: 23 Jul 2025

https://github.com/williamd1k0/metacritic-games

Distribution of Metacritic scores for console games.

data-scraping data-visualization metacritic web-scraping

Last synced: 26 Jun 2025

https://github.com/patmaz/publicapi

API with JWT authentication + websockets + data scraping

api bcrypt cors crud data-scraping firebase graphql jwt jwt-authentication nodejs real-time websockets

Last synced: 17 Apr 2026

https://github.com/drisskhattabi6/data-scraping-tasks

This repository showcases my data scraping Tasks, where I have used Python libraries to extract, process, and analyze data from the web.

data-scraping data-scraping-projects etsy scraper scrapethissite scraping toscrape wikipedia worldometers

Last synced: 22 Apr 2026

https://github.com/nickklos10/seriea_machine_learning_predictions_2025

This project involves scraping data, processing the data, and building machine learning models to predict the standings for the 2024-2025 Serie-A season.

beatifulsoup data-scraping keras matplotlib pandas scikit-learn shap tensorflow

Last synced: 13 Apr 2026

https://github.com/ekamid/goodreads-scraper-api

A modern, RESTful API for accessing Goodreads data. Created as an alternative to the deprecated official Goodreads API.

book-apps book-community data-scraping goodreads goodreads-api-alternative goodreads-scraper goodreads-scraper-api

Last synced: 13 Sep 2025

https://github.com/valpere/datascrapexter

Universal web scraper built with Go featuring advanced anti-detection, ethical compliance, and configuration-driven operation for any website.

anti-detection captcha-solver chromedp cobra-cli colly configuration-driven data-extraction data-scraping ethical-scraping go golang goquery headless-browser high-performance legal-compliance proxy-rotation viper web-crawling web-scraping

Last synced: 11 Jul 2025

https://github.com/syscall97/fruit-recognizer

Fruit image classifier with ResNet50, 83% accuracy, deployed via Hugging Face & GitHub Pages.

data-scraping fastai huggingface-spaces image-classification python

Last synced: 30 May 2026