Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
Projects in Awesome Lists tagged with scrapy
A curated list of projects in awesome lists tagged with scrapy .
https://github.com/crawlab-team/crawlab
Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架
crawlab crawler crawling-tasks docker go platform scrapy scrapyd-ui spider spiders-management web-crawler webcrawler webspider
Last synced: 25 Sep 2024
https://github.com/rmax/scrapy-redis
Redis-based components for Scrapy.
crawler distributed redis scrapy
Last synced: 27 Sep 2024
https://github.com/rolando/scrapy-redis
Redis-based components for Scrapy.
crawler distributed redis scrapy
Last synced: 05 Aug 2024
https://github.com/spiderclub/haipproxy
:sparkling_heart: High available distributed ip proxy pool, powerd by Scrapy and Redis
crawler distributed high-availability ipproxy redis scheduler scrapy spider
Last synced: 27 Sep 2024
https://github.com/SpiderClub/haipproxy
:sparkling_heart: High available distributed ip proxy pool, powerd by Scrapy and Redis
crawler distributed high-availability ipproxy redis scheduler scrapy spider
Last synced: 31 Jul 2024
https://github.com/dropsdevopsorg/ecommercecrawlers
实战🐍多种网站、电商数据爬虫🕷。包含🕸：淘宝商品、微信公众号、大众点评、企查查、招聘网站、闲鱼、阿里任务、博客园、微博、百度贴吧、豆瓣电影、包图网、全景网、豆瓣音乐、某省药监局、搜狐新闻、机器学习文本采集、fofa资产采集、汽车之家、国家统计局、百度关键词收录数、蜘蛛泛目录、今日头条、豆瓣影评、携程、小米应用商店、安居客、途家民宿❤️❤️❤️。微信爬虫展示项目:
alitask baidu baidu-tieba baotu boss crawler ctrip dazhong-spider douban-movie douban-music fofa lagou python3 quanjing scrapy sohu taobao-spider wechat xianyu zhilianzhaopin
Last synced: 02 Oct 2024
https://github.com/DropsDevopsOrg/ECommerceCrawlers
实战🐍多种网站、电商数据爬虫🕷。包含🕸：淘宝商品、微信公众号、大众点评、企查查、招聘网站、闲鱼、阿里任务、博客园、微博、百度贴吧、豆瓣电影、包图网、全景网、豆瓣音乐、某省药监局、搜狐新闻、机器学习文本采集、fofa资产采集、汽车之家、国家统计局、百度关键词收录数、蜘蛛泛目录、今日头条、豆瓣影评、携程、小米应用商店、安居客、途家民宿❤️❤️❤️。微信爬虫展示项目:
alitask baidu baidu-tieba baotu boss crawler ctrip dazhong-spider douban-movie douban-music fofa lagou python3 quanjing scrapy sohu taobao-spider wechat xianyu zhilianzhaopin
Last synced: 30 Jul 2024
https://github.com/nghuyong/weibospider
持续维护的新浪微博采集工具🚀🚀🚀
python scrapy weibo weibospider
Last synced: 30 Sep 2024
https://github.com/nghuyong/WeiboSpider
持续维护的新浪微博采集工具🚀🚀🚀
python scrapy weibo weibospider
Last synced: 31 Jul 2024
https://github.com/my8100/scrapydweb
Web app for Scrapyd cluster management, Scrapy log analysis & visualization, Auto packaging, Timer tasks, Monitor & Alert, and Mobile UI. DEMO :point_right:
dashboard log-analysis log-parsing scrapy scrapy-log-analysis scrapy-visualization scrapyd scrapyd-admin scrapyd-api scrapyd-cluster-management scrapyd-control scrapyd-keeper scrapyd-log-analysis scrapyd-manage scrapyd-monitor scrapyd-ui scrapyd-visualization spider
Last synced: 30 Sep 2024
https://github.com/scrapy-plugins/scrapy-splash
Scrapy+Splash for JavaScript integration
Last synced: 31 Jul 2024
https://github.com/luckyzxl2016/movie_recommend
基于Spark的电影推荐系统，包含爬虫项目、web网站、后台管理系统以及spark推荐系统
hadoop hive mysql nginx scala scrapy spark-mllib spark-streaming ssm-maven
Last synced: 27 Sep 2024
https://github.com/dormymo/spiderkeeper
admin ui for scrapy/open source scrapinghub
dashboard scrapy scrapy-ui scrapyd scrapyd-dashboard scrapyd-ui spider
Last synced: 30 Sep 2024
https://github.com/LuckyZXL2016/Movie_Recommend
基于Spark的电影推荐系统，包含爬虫项目、web网站、后台管理系统以及spark推荐系统
hadoop hive mysql nginx scala scrapy spark-mllib spark-streaming ssm-maven
Last synced: 31 Jul 2024
https://github.com/DormyMo/SpiderKeeper
admin ui for scrapy/open source scrapinghub
dashboard scrapy scrapy-ui scrapyd scrapyd-dashboard scrapyd-ui spider
Last synced: 31 Jul 2024
https://github.com/Boris-code/feapder
🚀🚀🚀feapder is an easy to use, powerful crawler framework | feapder是一款上手简单，功能强大的Python爬虫框架。内置AirSpider、Spider、TaskSpider、BatchSpider四种爬虫解决不同场景的需求。且支持断点续爬、监控报警、浏览器渲染、海量数据去重等功能。更有功能强大的爬虫管理系统feaplat为其提供方便的部署及调度
crawler feapder feaplat python scrapy spider
Last synced: 31 Jul 2024
https://github.com/boris-code/feapder
🚀🚀🚀feapder is an easy to use, powerful crawler framework | feapder是一款上手简单，功能强大的Python爬虫框架。内置AirSpider、Spider、TaskSpider、BatchSpider四种爬虫解决不同场景的需求。且支持断点续爬、监控报警、浏览器渲染、海量数据去重等功能。更有功能强大的爬虫管理系统feaplat为其提供方便的部署及调度
crawler feapder feaplat python scrapy spider
Last synced: 01 Oct 2024
https://github.com/qianyantech/image-downloader
Download images from Google, Bing, Baidu. 谷歌、百度、必应图片下载.
baidu bing google google-images image-downloader pyqt scrapy spider
Last synced: 30 Sep 2024
https://github.com/QianyanTech/Image-Downloader
Download images from Google, Bing, Baidu. 谷歌、百度、必应图片下载.
baidu bing google google-images image-downloader pyqt scrapy spider
Last synced: 01 Aug 2024
https://github.com/librauee/reptile
🏀 Python3 网络爬虫实战（部分含详细教程）猫眼 腾讯视频 豆瓣 研招网 微博 笔趣阁小说 百度热点 B站 CSDN 网易云阅读 阿里文学 百度股票 今日头条 微信公众号 网易云音乐 拉勾 有道 unsplash 实习僧 汽车之家 英雄联盟盒子 大众点评 链家 LPL赛程 台风 梦幻西游、阴阳师藏宝阁 天气 牛客网 百度文库 睡前故事 知乎 Wish
python3 requests scrapy spider
Last synced: 01 Oct 2024
https://github.com/thewebscrapingclub/webscraping-from-0-to-hero
The web scraping open project repository aims to share knowledge and experiences about web scraping with Python
playwright python scrapy scrapy-spider scrapysplash webscraping
Last synced: 30 Sep 2024
https://github.com/TheWebScrapingClub/webscraping-from-0-to-hero
The web scraping open project repository aims to share knowledge and experiences about web scraping with Python
playwright python scrapy scrapy-spider scrapysplash webscraping
Last synced: 30 Jul 2024
https://github.com/istresearch/scrapy-cluster
This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.
distributed kafka python redis scraping scrapy
Last synced: 30 Sep 2024
https://github.com/holgerd77/django-dynamic-scraper
Creating Scrapy scrapers via the Django admin interface
django python scraper scraping scrapy spider webscraping
Last synced: 03 Oct 2024
https://github.com/eliasdabbas/advertools
advertools - online marketing productivity and analysis tools
advertising adwords digital-marketing google-ads keywords log-analysis logfile-parser marketing online-marketing python robots-txt scrapy search-engine-marketing search-engine-optimization seo seo-crawler serp social-media twitter-api youtube
Last synced: 01 Oct 2024
https://github.com/kkoooqq/fakebrowser
🤖 Fake fingerprints to bypass anti-bot systems. Simulate mouse and keyboard operations to make behavior like a real person.
anti-bot-detection anti-fingerprinting automation bot browser-fingerprint cheat crawler fake headless puppeteer puppeteer-extra puppeteer-extra-plugin scrapy spoof stealth
Last synced: 27 Sep 2024
https://github.com/bytebuff/jspider
JSpider会每周更新至少一个网站的JS解密方式，欢迎 Star，交流微信：13298307816
javascript nodejs python3 scrapy spider
Last synced: 30 Sep 2024
https://github.com/juancarlospaco/faster-than-requests
Faster requests on Python 3
curl cython download-file faster-than-requests high-performance http-requests ndjson open-data python python-library python-requests python3 requests-toolbelt requests3 scrapy speed urllib urllib3 web-scraper web-scraping
Last synced: 30 Sep 2024
https://github.com/bytebuff/JSpider
JSpider会每周更新至少一个网站的JS解密方式，欢迎 Star，交流微信：13298307816
javascript nodejs python3 scrapy spider
Last synced: 01 Aug 2024
https://github.com/vifreefly/kimuraframework
Kimurai is a modern web scraping framework written in Ruby which works out of box with Headless Chromium/Firefox, PhantomJS, or simple HTTP requests and allows to scrape and interact with JavaScript rendered websites
crawler headless-chrome kimurai scraper scrapy
Last synced: 30 Sep 2024
https://github.com/jonbakerfish/TweetScraper
TweetScraper is a simple crawler/spider for Twitter Search without using API
scrapy tweets twitter twitter-search
Last synced: 01 Aug 2024
https://github.com/mtianyan/FunpySpiderSearchEngine
Word2vec 千人千面 个性化搜索 + Scrapy2.3.0(爬取数据) + ElasticSearch7.9.1(存储数据并提供对外Restful API) + Django3.1.1 搜索
django elasticsearch elasticsearch-analysis-ik lagou mysql python redis scrapy search-engine spider zhihu
Last synced: 30 Jul 2024
https://github.com/clemfromspace/scrapy-selenium
Scrapy middleware to handle javascript pages using selenium
Last synced: 28 Sep 2024
https://github.com/scrapy-plugins/scrapy-playwright
🎭 Playwright integration for Scrapy
chrome-headless firefox-headless hacktoberfest headless-browser javascript-renderer playwright playwright-python python python-asyncio python3 scrapy webkit-headless
Last synced: 30 Sep 2024
https://github.com/alanchn31/data-engineering-projects
Personal Data Engineering Projects
airflow aws-redshift cassandra data-engineering data-engineering-nanodegree data-lake data-modeling data-warehouse ingest-data mongodb postgres scrapy spark star-schema
Last synced: 29 Sep 2024
https://github.com/scrapinghub/scrapyrt
HTTP API for Scrapy spiders
crawler crawling hacktoberfest hacktoberfest2021 python scraper scrapy twisted webcrawler webcrawling
Last synced: 01 Aug 2024
https://github.com/kezhenxu94/house-renting
Possibly the best practice of Scrapy 🕷 and renting a house 🏡
docker python scrapy scrapy-crawler scrapy-spider scrapyd
Last synced: 04 Aug 2024
https://github.com/lb2281075105/python-spider
豆瓣电影top250、斗鱼爬取json数据以及爬取美女图片、淘宝、有缘、CrawlSpider爬取红娘网相亲人的部分基本信息以及红娘网分布式爬取和存储redis、爬虫小demo、Selenium、爬取多点、django开发接口、爬取有缘网信息、模拟知乎登录、模拟github登录、模拟图虫网登录、爬取多点商城整站数据、爬取微信公众号历史文章、爬取微信群或者微信好友分享的文章、itchat监听指定微信公众号分享的文章
beautifulsoup4 crawlspider django itchat mongodb mysql pymysql python redis scrapy selenium spider weichat xpath
Last synced: 28 Sep 2024
https://github.com/lb2281075105/Python-Spider
豆瓣电影top250、斗鱼爬取json数据以及爬取美女图片、淘宝、有缘、CrawlSpider爬取红娘网相亲人的部分基本信息以及红娘网分布式爬取和存储redis、爬虫小demo、Selenium、爬取多点、django开发接口、爬取有缘网信息、模拟知乎登录、模拟github登录、模拟图虫网登录、爬取多点商城整站数据、爬取微信公众号历史文章、爬取微信群或者微信好友分享的文章、itchat监听指定微信公众号分享的文章
beautifulsoup4 crawlspider django itchat mongodb mysql pymysql python redis scrapy selenium spider weichat xpath
Last synced: 30 Jul 2024
https://github.com/alanchn31/Data-Engineering-Projects
Personal Data Engineering Projects
airflow aws-redshift cassandra data-engineering data-engineering-nanodegree data-lake data-modeling data-warehouse ingest-data mongodb postgres scrapy spark star-schema
Last synced: 01 Aug 2024
https://github.com/TeamHG-Memex/scrapy-rotating-proxies
use multiple proxies with Scrapy
Last synced: 03 Aug 2024
https://github.com/alecxe/scrapy-fake-useragent
Random User-Agent middleware based on fake-useragent
Last synced: 03 Oct 2024
https://github.com/tb0hdan/domains
World’s single largest Internet domains dataset
colly dataset internet-domains scrapy search-engines yacy
Last synced: 01 Aug 2024
https://github.com/eracle/linkedin
Linkedin Scraper using Selenium Web Driver, Chromium headless, Docker and Scrapy
bot chromium-browser docker docker-compose linkedin scraper scraping scrapy selenium-webdriver
Last synced: 28 Sep 2024
https://github.com/TurboWay/spiderman
基于 scrapy-redis 的通用分布式爬虫框架
hbase hive kafka rdbm scapy-redis scrapy spiderman
Last synced: 31 Jul 2024
https://github.com/spekulatius/phpscraper
A universal web-util for PHP.
beautifulsoup chromium headless-chrome php php-crawler php-scraper php-spider php-spiders puppeteer pyppeteer scraper scraping scraping-websites scrapy web-scraper web-scraping
Last synced: 27 Sep 2024
https://github.com/alltheplaces/alltheplaces
A set of spiders and scrapers to extract location information from places that post their location on the internet.
geojson hacktoberfest python scrapers scrapy spider
Last synced: 12 Aug 2024
https://github.com/spekulatius/PHPScraper
A universal web-util for PHP.
beautifulsoup chromium headless-chrome php php-crawler php-scraper php-spider php-spiders puppeteer pyppeteer scraper scraping scraping-websites scrapy web-scraper web-scraping
Last synced: 30 Jul 2024
https://github.com/abhisharma404/vault
swiss army knife for hackers
crawler fuzzing hacking hacking-tool information-gathering lfi networking offensive-security osint pentesting port-scanner python rfi scanner scrapy security sqlite ssl-inspection vault xss-vulnerability
Last synced: 01 Aug 2024
https://github.com/AlexMathew/scrapple
A framework for creating semi-automatic web content extractors
beautifulsoup crawler css-selector extractor lxml python scrapers scraping scrapy selector selector-expression tutorial web-scraper web-scraping xpath-expression
Last synced: 31 Jul 2024
https://github.com/sangaline/advanced-web-scraping-tutorial
The Zipru scraper developed in the Advanced Web Scraping Tutorial.
python scraper scrapy tutorial-code
Last synced: 03 Aug 2024
https://github.com/my8100/files
Docs and files for ScrapydWeb, Scrapyd, Scrapy, and other projects
Last synced: 31 Jul 2024
https://github.com/MarwanDebbiche/post-tuto-deployment
Build and deploy a machine learning app from scratch 🚀
api aws character-level-cnn deployment docker machine-learning pytorch scraping scrapy selenium
Last synced: 07 Aug 2024
https://github.com/scrapy-plugins/scrapy-zyte-smartproxy
Zyte Smart Proxy Manager (formerly Crawlera) middleware for Scrapy
crawler crawler-detection plugin proxy scraping scrapy
Last synced: 30 Jul 2024
https://github.com/scrapy-plugins/scrapy-crawlera
Zyte Smart Proxy Manager (formerly Crawlera) middleware for Scrapy
crawler crawler-detection plugin proxy scraping scrapy
Last synced: 05 Sep 2024
https://github.com/lkuffo/web-scraping
Más de 50 ejemplos de web scraping utilizando: Requests | Scrapy | Selenium | LXML | BeautifulSoup
beautifulsoup beautifulsoup4 lxml-etree scraping scraping-python scraping-websites scrapping-python scrapy scrapy-crawler scrapy-spider selenium selenium-python selenium-webdriver web-scraping webscraping
Last synced: 28 Sep 2024
https://github.com/City-Bureau/city-scrapers
Scrape, standardize and share public meetings from local government websites
city-scrapers open-data python scrapy web-scraping
Last synced: 01 Aug 2024
https://github.com/glaucocustodio/tanakai
Tanakai is a modern web scraping framework written in Ruby. A fork of Kimurai.
chrome-headless crawler kimurai scraper scrapy webscraping
Last synced: 31 Jul 2024
https://github.com/TikHubIO/TikHub-API-Python-SDK
High-performance asynchronous Douyin(抖音) TikTok Xiaohongshu(小红书) Kuaishou(快手) Weibo(微博) Instagram YouTube(油管) Twitter(X) Captcha Solver(验证码解决器) Temp Mail(临时邮箱) API(接口).
api captcha-solver crawler data-api douyin douyin-tiktok-api instagram kuaishou netease-cloud-music private-api scrapy tiktok twitter weibo xiaohongshu xiguashipin
Last synced: 31 Jul 2024
https://github.com/ttonys/Scrapy-CVE-CNVD
漏洞监控，基于scrapy，scrapy-redis，获取每日最新的CVE和CNVD漏洞，邮件通知
Last synced: 04 Aug 2024
https://github.com/DiegoCaraballo/Email-extractor
The main functionality is to extract all the emails from one or several URLs - La funcionalidad principal es extraer todos los correos electrónicos de una o varias Url
email email-extractor email-marketing emails extraction python scraper scrapers scraping scraping-websites scrapper scrapping scrapy scrapy-spider spyder stractor
Last synced: 04 Aug 2024
https://github.com/voliveirajr/seleniumcrawler
An example using Selenium webdrivers for python and Scrapy framework to create a web scraper to crawl an ASP site
asp-net python scraper scraping scraping-websites scrapper scrapy selenium selenium-webdriver webcrawler webcrawling
Last synced: 28 Sep 2024
https://github.com/alash3al/scraply
Scraply a simple dom scraper to fetch information from any html based website
crawler crawling dom golang scraper scrapers scraping-websites scrapy server
Last synced: 01 Aug 2024
https://github.com/my8100/scrapyd-cluster-on-heroku
Set up free and scalable Scrapyd cluster for distributed web-crawling with just a few clicks. DEMO :point_right:
cluster heroku logparser python scrapy scrapyd scrapydweb web-crawling web-scraping
Last synced: 05 Aug 2024
https://github.com/dangsh/hive
lots of spider (很多爬虫）
beautifulsoup python3 scrapy selenium-webdriver spider
Last synced: 28 Sep 2024
https://github.com/lan-ce-lot/pythorch-text-classification
对豆瓣影评进行文本分类情感分析，利用爬虫豆瓣爬取评论，进行数据清洗，分词，采用BERT、CNN、LSTM等模型进行训练，采用tensorboardX可视化训练过程，自然语言处理项目\A project for text classification, based on torch 1.7.1
bert cnn douban lstm natural-language-processing nlp qt qt5 qt6 rnn scrapy sentiment-analysis tensorboard tensorboardx text-classification ui
Last synced: 28 Sep 2024
https://github.com/dyweb/scrala
Unmaintained :whale: :coffee: :spider: Scala crawler(spider) framework, inspired by scrapy, created by @gaocegege
actor-model docker scala scrapy spider
Last synced: 31 Jul 2024
https://github.com/datawizard1337/ARGUS
ARGUS is an easy-to-use web scraping tool. The program is based on the Scrapy Python framework and is able to crawl a broad range of different websites. On the websites, ARGUS is able to perform tasks like scraping texts or collecting hyperlinks between websites. See: https://link.springer.com/article/10.1007/s11192-020-03726-9
crawling python scraping scrapy scrapyd webcrawling webscraping
Last synced: 31 Jul 2024
https://github.com/TeamHG-Memex/scrapy-crawl-once
Scrapy middleware which allows to crawl only new content
Last synced: 30 Jul 2024
https://github.com/EasyPi/docker-scrapyd
🕷️ Scrapyd is an application for deploying and running Scrapy spiders.
Last synced: 01 Aug 2024
https://github.com/windrises/bgmtools
Bangumi小工具
bangumi django scrapy tampermonkey
Last synced: 30 Sep 2024
https://github.com/orangain/scrapy-s3pipeline
Scrapy pipeline to store chunked items into Amazon S3 or Google Cloud Storage bucket.
Last synced: 03 Aug 2024
https://github.com/scrapy/itemadapter
Common interface for data container classes
hacktoberfest metadata python python-attrs python-dataclasses python3 scrapy
Last synced: 03 Aug 2024
https://github.com/yashpatel7025/pricetracker
Amazon Price tracker using Python, Django, Celery Task Queue and Scrapy Frameworks. Front end is developed using HTML, CSS and Bootstrap
amazon amazon-price-tracker celery django django-background-tasks django-framework price-tracker price-tracking-system pricetrack python python3 scrapy scrapy-framework tracker
Last synced: 27 Sep 2024
https://github.com/alash3al/scrapyr
a simple & tiny scrapy clustering solution, considered a drop-in replacement for scrapyd
clustering golang python scrapy scrapyd-server
Last synced: 01 Aug 2024