Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

Projects in Awesome Lists tagged with scrapy-spider

A curated list of projects in awesome lists tagged with scrapy-spider .

https://github.com/TheWebScrapingClub/webscraping-from-0-to-hero

The web scraping open project repository aims to share knowledge and experiences about web scraping with Python

playwright python scrapy scrapy-spider scrapysplash webscraping

Last synced: 14 Mar 2025

https://github.com/thewebscrapingclub/webscraping-from-0-to-hero

The web scraping open project repository aims to share knowledge and experiences about web scraping with Python

playwright python scrapy scrapy-spider scrapysplash webscraping

Last synced: 08 Apr 2025

https://github.com/kezhenxu94/house-renting

Possibly the best practice of Scrapy 🕷 and renting a house 🏡

docker python scrapy scrapy-crawler scrapy-spider scrapyd

Last synced: 21 Nov 2024

https://github.com/djm/python-scrapyd-api

A Python wrapper for working with Scrapyd's API.

api-wrapper python scrapy-spider

Last synced: 13 Apr 2025

https://github.com/DiegoCaraballo/Email-extractor

The main functionality is to extract all the emails from one or several URLs - La funcionalidad principal es extraer todos los correos electrónicos de una o varias Url

email email-extractor email-marketing emails extraction python scraper scrapers scraping scraping-websites scrapper scrapping scrapy scrapy-spider spyder stractor

Last synced: 21 Nov 2024

https://github.com/accordbox/scrapy-spider-example

Scrapy spider example for Scrapy Tutorial Series

python3 scrapy-spider scrapy-tutorial

Last synced: 12 Apr 2025

https://github.com/chinesehuazhou/scrapyproject

Scrapy项目（mysql+mongodb豆瓣top250电影）

mongodb mysql python scrapy scrapy-spider

Last synced: 11 Apr 2025

https://github.com/zoranpandovski/prodirectscraper

:necktie: Web scraper for http://www.prodirectselect.com/ :shoe:

python scraper scrapy scrapy-crawler scrapy-spider spider webscraper webscraping

Last synced: 10 Dec 2024

https://github.com/jumbaeric/woocommerce_shop_scraper_to_csv

The WooCommerce Shop Scraper to CSV tool is a powerful python tool designed to simplify the process of extracting product data from a WooCommerce-powered online shop and generating a comprehensive CSV file. Built on the foundations of WordPress and WooCommerce,

data python scraping-websites scrapy-spider woocommerce-products wordpress

Last synced: 13 Apr 2025

https://github.com/activatedgeek/winemag-dataset

Dataset of Wine Reviews from Wine Enthusiast Magazine :grapes: :wine_glass: :earth_asia:

crawler dataset python3 scrapy scrapy-spider vega-lite visualization wine wine-tasting

Last synced: 12 Apr 2025

https://github.com/ekaputra07/kodepos

Scrapper Kode POS Indonesia berbasis Python Scrapy. Sumber kode POS: http://kodepos.nomor.net

json kodepos python scrapy-spider

Last synced: 03 Jan 2025

https://github.com/rggh/scrapy14

Scraping News Stories From Multiple Sources

mysql scraping-news-stories scrapy scrapy-spider

Last synced: 07 Dec 2024

https://github.com/nguyenda18/portland-jail-data-crawler

Scraper used for recording changes to Portland jail database

dataframe datasette python python3 scrapy scrapy-crawler scrapy-spider

Last synced: 23 Apr 2025

https://github.com/santhoshse7en/imdb-scrapy

A fun projects made using Scrapy. The Spiders included in this are able to extract Movie, TV-Series, TV-Movies based on year and title type. A lot more to come ahead

beautifulsoup4 imdb imdb-dataset imdb-scraper scrapy scrapy-crawler scrapy-framework scrapy-spider

Last synced: 03 Mar 2025

https://github.com/ayemunhossain/scrapy-web-crawler-projects

Python crawler with Scrapy framework, which may help you to learn webscrapping.

python-web-scraping python-webscraping-application scrapy scrapy-crawler scrapy-python scrapy-spider webscraping

Last synced: 24 Mar 2025

https://github.com/franck-gaspoz/moviedbassistant

Generates Movie documents (HTML, movie database,...) using Web Crawlers ( IMDb, ... )

cli database gui imdb movie scraper scrapy-crawler scrapy-spider scrawler

Last synced: 10 Apr 2025

https://github.com/noseparte/hero-kills-reptile

:snowflake: 英雄杀三部曲：英雄杀爬虫项目, 爬取英雄资料以及卡牌信息

game python3 scrapy-spider

Last synced: 15 May 2025

https://github.com/adamfisher/scrapyrt.client

A C# client to make calls to a scrapyrt (Scrapy real-time) HTTP endpoint.

crawler scraper scrapy scrapy-crawler scrapy-framework scrapy-spider

Last synced: 21 Mar 2025

https://github.com/aa-nadim/trip-crawler

This project is designed for web scraping and managing data using Docker containers. It includes services for a PostgreSQL database, a web scraper, pgAdmin, and a testing setup.

scrapy scrapy-crawler scrapy-spider

Last synced: 19 Feb 2025

https://github.com/aquatiko/craigslist-spider

A python spider to scrape jobs list and details form https://newyork.craigslist.org.

craigslist dynamic jobseeker python3 scrapy-spider

Last synced: 14 Mar 2025

https://github.com/fmgarciaa/scrapy

Notes on Platzi's Scrapy course. I learned how to extract data from the internet in an automated way by creating spiders

platzi-course python scrapy-spider

Last synced: 07 Apr 2025

https://github.com/xinlc/scrapyjiandan

福利 [手动滑稽]

scrapy-crawler scrapy-spider

Last synced: 14 Mar 2025

https://github.com/fernandotorresl/platzi-intelligence-agency_bak

Platzi Intelligence Agency Website (Project from Scrapy Course - Platzi.com)

python scrapy scrapy-crawler scrapy-spider

Last synced: 26 Mar 2025

https://github.com/muneeb1030/webscrapper_mastodon

The Mastodon Social Platform Scraper is a Python-based web scraping tool designed to explore and extract valuable data from the Mastodon social platform.

data-analysis data-collection mastodon python3 scrapy scrapy-spider selenium-python webscraping

Last synced: 08 Feb 2025

https://github.com/buihdk/scrapy-books

A demo of scraping book data from the website https://books.toscrape.com using Scrapy

ipython scrapy scrapy-spider

Last synced: 13 Mar 2025

https://github.com/amiriiw/coin_crawler

A cryptocurrency data scraping and web application built with Flask and Scrapy. This project collects cryptocurrency data from CoinMarketCap and serves it through a Flask-based web application, allowing users to view various crypto trends, newly listed coins, top gainers/losers, and specific coin details.

coin-crawler css3 flask-application html5 python3 requests scrapy scrapy-spider scrapy-splash spider

Last synced: 13 Mar 2025

https://github.com/ahmadhuss/scrapy_universal

A universal, plug-and-play Scrapy boilerplate for scalable web scraping projects.

python scrapy scrapy-crawler scrapy-spider

Last synced: 07 May 2025

https://github.com/damikaalwis-gif/adscrapex

AdScrapeX is a web scraping project built with Scrapy and Scrapy Playwright to extract data from popular classified ad websites in Sri Lanka, including vehicles, properties, and job listings.

classifieds ikman-lk playwright python scrapy scrapy-spider webscraping

Last synced: 13 Dec 2024

https://github.com/marc7666/prac1-tcvd-web-scraping

PRAC1 of the subject "Data typology and life cycle" of the MSc in Data Science at Universitat oberta de Catalunya

csv-export csv-files python scrapy scrapy-crawler scrapy-spider webscraping

Last synced: 09 May 2025

https://github.com/intel00000/web_scraper_hasmo

A python web scraper designed to collect data from competitor websites, news articles, and market research reports.

google-api google-api-python-client gspread-api openai-api scrapy-crawler scrapy-framework scrapy-spider scrapy-splash

Last synced: 13 Mar 2025

https://github.com/sachnaror/scrapy_scraper

Using Scrapy: it scrapes quotes from [quotes.toscrape.com] and saves them to a CSV file

scrapy scrapy-crawler scrapy-spider

Last synced: 20 Feb 2025

https://github.com/loglux/scan_co_uk

This repository contains a Scrapy spider designed to scrape product information from Scan.co.uk based on provided search terms and filters.

filtering-data python scrapy scrapy-spider search webscraping

Last synced: 25 Feb 2025

https://github.com/loglux/amazon_uk

A Scrapy spider to scrape Amazon UK product details based on search terms, categories, and filters. Features deduplication filter for unique listings and optional sponsored link exclusion.

amazon filtering-data python scraper scrapy-spider search webscraping

Last synced: 25 Feb 2025

https://github.com/dyslab/spy-sample

Scrapy Learning... 🕷🕸🕸🕷

python3 samples scrapy-spider

Last synced: 05 Apr 2025

https://github.com/danielsiepmann/typo3-docs-search

Full Stack to allow searching docs.typo3.org

docs scrapy-spider search typo3

Last synced: 22 Mar 2025

https://github.com/vaibhavyadav-dev/codeforces-problemset-scrapper

Web Scrapper that scrap the whole problemset of Codeforces into csv or json file.

codeforces competative competative-programming crawler problemset programming python scrapy-crawler scrapy-spider

Last synced: 06 Mar 2025

https://github.com/joannescode/quotestoscrape

Raspagem de conteúdo com a biblioteca Scrapy...

python3 scraping-websites scrapy-spider

Last synced: 26 Feb 2025

https://github.com/joannescode/metacritic-crawler

Raspagem de conteúdo com a biblioteca Scapy...

python3 scraping-websites scrapy-spider

Last synced: 26 Feb 2025

https://github.com/tknishh/scrapy-redis-data-pipeline

Building a highly scalable data streaming pipeline in python

datapipeline python redis-client redis-server scrapy-spider

Last synced: 18 Mar 2025

https://github.com/vlada-pv/scrapy-parser-wiki-movies

The Wiki Movies Scraper is a Scrapy project designed to collect information on movies from Wikipedia, including their Title, Genre, Director, Country and Year.

scrapy scrapy-spider wikipedia

Last synced: 15 Mar 2025

https://github.com/steadyxlr8/scrapy_practice

Web scrapping using Scrapy framework

python scrapy-spider web-scraping

Last synced: 31 Mar 2025

https://github.com/team-synergy-resource-mapper/classified-ad-scraper

AdScrapeX is a web scraping project built with Scrapy and Scrapy Playwright to extract data from popular classified ad websites in Sri Lanka, including vehicles, properties, and job listings.

classifieds ikman-lk playwright python scrapy scrapy-spider webscraping

Last synced: 31 Mar 2025

https://github.com/aman-dutta/oscar_winners

Scraping IMDB website to find out the Oscar winners using web-crawler and Python

python3 scrapy-crawler scrapy-spider

Last synced: 05 Mar 2025

https://github.com/chyornyy/funpay-parser

The Funpay Parser is a web scraper built with Scrapy that extracts information about lots on the https://funpay.com/ marketplace.

python3 scrapy scrapy-crawler scrapy-spider

Last synced: 31 Mar 2025

https://github.com/muneeb1030/webscrapper_altnews

The project utilizes a combination of Python, Scrapy, and Selenium to navigate through the dynamic content of AltNews.in and collect valuable information for analysis and verification.

data-analysis data-collection python3 scrapy scrapy-spider selenium selenium-python

Last synced: 28 Feb 2025

https://github.com/muneeb1030/webscrapper_politifact

This initiative seeks to extract and analyze fact-checking data from Politifact.com, providing valuable insights into political statements, rulings, and the evolving information landscape.

data data-collection dataanalysis python3 scrapy scrapy-spider webscraping

Last synced: 28 Feb 2025

https://github.com/vikram-bhati/quotes_scrapy

simple spider to quotes using scrapy

python scraping-websites scrapy scrapy-crawler scrapy-spider

Last synced: 20 Feb 2025

https://github.com/karantan/scrapy_scout

Web crawler for scraping varstvo-konkurence.si

python3 scrapy scrapy-spider

Last synced: 02 Apr 2025

https://github.com/elymsyr/bimproject_mongo

Web Crawling with Scrapy (bimobject.com)

scrapy scrapy-crawler scrapy-spider web-crawler web-crawler-python

Last synced: 21 Feb 2025

https://github.com/zuoez02/konachan-scrapy

Konachan.com scrapy

anime python3 scrapy-spider

Last synced: 12 Mar 2025

https://github.com/fernandotorresl/scraping-conferencias

Word cloud analysis of daily government conferences from México president using scrapy

jupyter-notebook mexico python scrapy scrapy-spider wip

Last synced: 26 Mar 2025