Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
Projects in Awesome Lists tagged with scrapy
A curated list of projects in awesome lists tagged with scrapy .
https://github.com/crawlab-team/crawlab
Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架
crawlab crawler crawling-tasks docker go platform scrapy scrapyd-ui spider spiders-management web-crawler webcrawler webspider
Last synced: 17 Dec 2024
https://github.com/rmax/scrapy-redis
Redis-based components for Scrapy.
crawler distributed redis scrapy
Last synced: 16 Dec 2024
https://github.com/SpiderClub/haipproxy
:sparkling_heart: High available distributed ip proxy pool, powerd by Scrapy and Redis
crawler distributed high-availability ipproxy redis scheduler scrapy spider
Last synced: 29 Oct 2024
https://github.com/spiderclub/haipproxy
:sparkling_heart: High available distributed ip proxy pool, powerd by Scrapy and Redis
crawler distributed high-availability ipproxy redis scheduler scrapy spider
Last synced: 18 Dec 2024
https://github.com/dropsdevopsorg/ecommercecrawlers
实战🐍多种网站、电商数据爬虫🕷。包含🕸：淘宝商品、微信公众号、大众点评、企查查、招聘网站、闲鱼、阿里任务、博客园、微博、百度贴吧、豆瓣电影、包图网、全景网、豆瓣音乐、某省药监局、搜狐新闻、机器学习文本采集、fofa资产采集、汽车之家、国家统计局、百度关键词收录数、蜘蛛泛目录、今日头条、豆瓣影评、携程、小米应用商店、安居客、途家民宿❤️❤️❤️。微信爬虫展示项目:
alitask baidu baidu-tieba baotu boss crawler ctrip dazhong-spider douban-movie douban-music fofa lagou python3 quanjing scrapy sohu taobao-spider wechat xianyu zhilianzhaopin
Last synced: 19 Dec 2024
https://github.com/DropsDevopsOrg/ECommerceCrawlers
实战🐍多种网站、电商数据爬虫🕷。包含🕸：淘宝商品、微信公众号、大众点评、企查查、招聘网站、闲鱼、阿里任务、博客园、微博、百度贴吧、豆瓣电影、包图网、全景网、豆瓣音乐、某省药监局、搜狐新闻、机器学习文本采集、fofa资产采集、汽车之家、国家统计局、百度关键词收录数、蜘蛛泛目录、今日头条、豆瓣影评、携程、小米应用商店、安居客、途家民宿❤️❤️❤️。微信爬虫展示项目:
alitask baidu baidu-tieba baotu boss crawler ctrip dazhong-spider douban-movie douban-music fofa lagou python3 quanjing scrapy sohu taobao-spider wechat xianyu zhilianzhaopin
Last synced: 26 Oct 2024
https://github.com/nghuyong/weibospider
持续维护的新浪微博采集工具🚀🚀🚀
python scrapy weibo weibospider
Last synced: 17 Dec 2024
https://github.com/nghuyong/WeiboSpider
持续维护的新浪微博采集工具🚀🚀🚀
python scrapy weibo weibospider
Last synced: 29 Oct 2024
https://github.com/my8100/scrapydweb
Web app for Scrapyd cluster management, Scrapy log analysis & visualization, Auto packaging, Timer tasks, Monitor & Alert, and Mobile UI. DEMO :point_right:
dashboard log-analysis log-parsing scrapy scrapy-log-analysis scrapy-visualization scrapyd scrapyd-admin scrapyd-api scrapyd-cluster-management scrapyd-control scrapyd-keeper scrapyd-log-analysis scrapyd-manage scrapyd-monitor scrapyd-ui scrapyd-visualization spider
Last synced: 17 Dec 2024
https://github.com/scrapy-plugins/scrapy-splash
Scrapy+Splash for JavaScript integration
Last synced: 18 Dec 2024
https://github.com/luckyzxl2016/movie_recommend
基于Spark的电影推荐系统，包含爬虫项目、web网站、后台管理系统以及spark推荐系统
hadoop hive mysql nginx scala scrapy spark-mllib spark-streaming ssm-maven
Last synced: 20 Dec 2024
https://github.com/LuckyZXL2016/Movie_Recommend
基于Spark的电影推荐系统，包含爬虫项目、web网站、后台管理系统以及spark推荐系统
hadoop hive mysql nginx scala scrapy spark-mllib spark-streaming ssm-maven
Last synced: 29 Oct 2024
https://github.com/dormymo/spiderkeeper
admin ui for scrapy/open source scrapinghub
dashboard scrapy scrapy-ui scrapyd scrapyd-dashboard scrapyd-ui spider
Last synced: 19 Dec 2024
https://github.com/DormyMo/SpiderKeeper
admin ui for scrapy/open source scrapinghub
dashboard scrapy scrapy-ui scrapyd scrapyd-dashboard scrapyd-ui spider
Last synced: 30 Oct 2024
https://github.com/Boris-code/feapder
🚀🚀🚀feapder is an easy to use, powerful crawler framework | feapder是一款上手简单，功能强大的Python爬虫框架。内置AirSpider、Spider、TaskSpider、BatchSpider四种爬虫解决不同场景的需求。且支持断点续爬、监控报警、浏览器渲染、海量数据去重等功能。更有功能强大的爬虫管理系统feaplat为其提供方便的部署及调度
crawler feapder feaplat python scrapy spider
Last synced: 31 Oct 2024
https://github.com/boris-code/feapder
🚀🚀🚀feapder is an easy to use, powerful crawler framework | feapder是一款上手简单，功能强大的Python爬虫框架。内置AirSpider、Spider、TaskSpider、BatchSpider四种爬虫解决不同场景的需求。且支持断点续爬、监控报警、浏览器渲染、海量数据去重等功能。更有功能强大的爬虫管理系统feaplat为其提供方便的部署及调度
crawler feapder feaplat python scrapy spider
Last synced: 18 Dec 2024
https://github.com/qianyantech/image-downloader
Download images from Google, Bing, Baidu. 谷歌、百度、必应图片下载.
baidu bing google google-images image-downloader pyqt scrapy spider
Last synced: 18 Dec 2024
https://github.com/QianyanTech/Image-Downloader
Download images from Google, Bing, Baidu. 谷歌、百度、必应图片下载.
baidu bing google google-images image-downloader pyqt scrapy spider
Last synced: 08 Nov 2024
https://github.com/librauee/reptile
🏀 Python3 网络爬虫实战（部分含详细教程）猫眼 腾讯视频 豆瓣 研招网 微博 笔趣阁小说 百度热点 B站 CSDN 网易云阅读 阿里文学 百度股票 今日头条 微信公众号 网易云音乐 拉勾 有道 unsplash 实习僧 汽车之家 英雄联盟盒子 大众点评 链家 LPL赛程 台风 梦幻西游、阴阳师藏宝阁 天气 牛客网 百度文库 睡前故事 知乎 Wish
python3 requests scrapy spider
Last synced: 22 Dec 2024
https://github.com/thewebscrapingclub/webscraping-from-0-to-hero
The web scraping open project repository aims to share knowledge and experiences about web scraping with Python
playwright python scrapy scrapy-spider scrapysplash webscraping
Last synced: 20 Dec 2024
https://github.com/TheWebScrapingClub/webscraping-from-0-to-hero
The web scraping open project repository aims to share knowledge and experiences about web scraping with Python
playwright python scrapy scrapy-spider scrapysplash webscraping
Last synced: 26 Oct 2024
https://github.com/kkoooqq/fakebrowser
🤖 Fake fingerprints to bypass anti-bot systems. Simulate mouse and keyboard operations to make behavior like a real person.
anti-bot-detection anti-fingerprinting automation bot browser-fingerprint cheat crawler fake headless puppeteer puppeteer-extra puppeteer-extra-plugin scrapy spoof stealth
Last synced: 21 Dec 2024
https://github.com/istresearch/scrapy-cluster
This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.
distributed kafka python redis scraping scrapy
Last synced: 19 Dec 2024
https://github.com/eliasdabbas/advertools
advertools - online marketing productivity and analysis tools
advertising adwords digital-marketing google-ads keywords log-analysis logfile-parser marketing online-marketing python robots-txt scrapy search-engine-marketing search-engine-optimization seo seo-crawler serp social-media twitter-api youtube
Last synced: 17 Dec 2024
https://github.com/holgerd77/django-dynamic-scraper
Creating Scrapy scrapers via the Django admin interface
django python scraper scraping scrapy spider webscraping
Last synced: 20 Dec 2024
https://github.com/juancarlospaco/faster-than-requests
Faster requests on Python 3
curl cython download-file faster-than-requests high-performance http-requests ndjson open-data python python-library python-requests python3 requests-toolbelt requests3 scrapy speed urllib urllib3 web-scraper web-scraping
Last synced: 19 Dec 2024
https://github.com/bytebuff/JSpider
JSpider会每周更新至少一个网站的JS解密方式，欢迎 Star，交流微信：13298307816
javascript nodejs python3 scrapy spider
Last synced: 01 Nov 2024
https://github.com/bytebuff/jspider
JSpider会每周更新至少一个网站的JS解密方式，欢迎 Star，交流微信：13298307816
javascript nodejs python3 scrapy spider
Last synced: 16 Dec 2024
https://github.com/scrapy-plugins/scrapy-playwright
🎭 Playwright integration for Scrapy
chrome-headless firefox-headless hacktoberfest headless-browser javascript-renderer playwright playwright-python python python-asyncio python3 scrapy webkit-headless
Last synced: 18 Dec 2024
https://github.com/vifreefly/kimuraframework
Kimurai is a modern web scraping framework written in Ruby which works out of box with Headless Chromium/Firefox, PhantomJS, or simple HTTP requests and allows to scrape and interact with JavaScript rendered websites
crawler headless-chrome kimurai scraper scrapy
Last synced: 18 Dec 2024
https://github.com/jonbakerfish/TweetScraper
TweetScraper is a simple crawler/spider for Twitter Search without using API
scrapy tweets twitter twitter-search
Last synced: 08 Nov 2024
https://github.com/moyada/stealer
抖音、快手、火山、皮皮虾，视频去水印程序
bilibili bilibili-download douyin douyin-download kuaishou pipixia scrapy
Last synced: 15 Dec 2024
https://github.com/clemfromspace/scrapy-selenium
Scrapy middleware to handle javascript pages using selenium
Last synced: 18 Dec 2024
https://github.com/mtianyan/FunpySpiderSearchEngine
Word2vec 千人千面 个性化搜索 + Scrapy2.3.0(爬取数据) + ElasticSearch7.9.1(存储数据并提供对外Restful API) + Django3.1.1 搜索
django elasticsearch elasticsearch-analysis-ik lagou mysql python redis scrapy search-engine spider zhihu
Last synced: 26 Oct 2024
https://github.com/alanchn31/data-engineering-projects
Personal Data Engineering Projects
airflow aws-redshift cassandra data-engineering data-engineering-nanodegree data-lake data-modeling data-warehouse ingest-data mongodb postgres scrapy spark star-schema
Last synced: 18 Dec 2024
https://github.com/hellock/icrawler
A multi-thread crawler framework with many builtin image crawlers provided.
bing-image crawler flickr-api google-images python scrapy spider
Last synced: 18 Dec 2024
https://github.com/alanchn31/Data-Engineering-Projects
Personal Data Engineering Projects
airflow aws-redshift cassandra data-engineering data-engineering-nanodegree data-lake data-modeling data-warehouse ingest-data mongodb postgres scrapy spark star-schema
Last synced: 08 Nov 2024
https://github.com/scrapinghub/scrapyrt
HTTP API for Scrapy spiders
crawler crawling hacktoberfest hacktoberfest2021 python scraper scrapy twisted webcrawler webcrawling
Last synced: 15 Dec 2024
https://github.com/eracle/linkedin
Linkedin Scraper using Selenium Web Driver, Chromium headless, Docker and Scrapy
bot chromium-browser docker docker-compose linkedin scraper scraping scrapy selenium-webdriver
Last synced: 20 Dec 2024
https://github.com/kezhenxu94/house-renting
Possibly the best practice of Scrapy 🕷 and renting a house 🏡
docker python scrapy scrapy-crawler scrapy-spider scrapyd
Last synced: 21 Nov 2024
https://github.com/morvanzhou/easy-scraping-tutorial
Simple but useful Python web scraping tutorial code.
asyncio beautifulsoup crawler crawling distributed-scraper regex requests scraping scrapy urllib
Last synced: 15 Dec 2024
https://github.com/lb2281075105/python-spider
豆瓣电影top250、斗鱼爬取json数据以及爬取美女图片、淘宝、有缘、CrawlSpider爬取红娘网相亲人的部分基本信息以及红娘网分布式爬取和存储redis、爬虫小demo、Selenium、爬取多点、django开发接口、爬取有缘网信息、模拟知乎登录、模拟github登录、模拟图虫网登录、爬取多点商城整站数据、爬取微信公众号历史文章、爬取微信群或者微信好友分享的文章、itchat监听指定微信公众号分享的文章
beautifulsoup4 crawlspider django itchat mongodb mysql pymysql python redis scrapy selenium spider weichat xpath
Last synced: 18 Dec 2024
https://github.com/lb2281075105/Python-Spider
豆瓣电影top250、斗鱼爬取json数据以及爬取美女图片、淘宝、有缘、CrawlSpider爬取红娘网相亲人的部分基本信息以及红娘网分布式爬取和存储redis、爬虫小demo、Selenium、爬取多点、django开发接口、爬取有缘网信息、模拟知乎登录、模拟github登录、模拟图虫网登录、爬取多点商城整站数据、爬取微信公众号历史文章、爬取微信群或者微信好友分享的文章、itchat监听指定微信公众号分享的文章
beautifulsoup4 crawlspider django itchat mongodb mysql pymysql python redis scrapy selenium spider weichat xpath
Last synced: 26 Oct 2024
https://github.com/teamhg-memex/scrapy-rotating-proxies
use multiple proxies with Scrapy
Last synced: 21 Dec 2024
https://github.com/TeamHG-Memex/scrapy-rotating-proxies
use multiple proxies with Scrapy
Last synced: 18 Nov 2024
https://github.com/alecxe/scrapy-fake-useragent
Random User-Agent middleware based on fake-useragent
Last synced: 20 Dec 2024
https://github.com/tb0hdan/domains
World’s single largest Internet domains dataset
colly dataset internet-domains scrapy search-engines yacy
Last synced: 08 Nov 2024
https://github.com/alltheplaces/alltheplaces
A set of spiders and scrapers to extract location information from places that post their location on the internet.
geojson hacktoberfest python scrapers scrapy spider
Last synced: 02 Dec 2024
https://github.com/turboway/spiderman
基于 scrapy-redis 的通用分布式爬虫框架
hbase hive kafka rdbm scapy-redis scrapy spiderman
Last synced: 21 Dec 2024
https://github.com/TurboWay/spiderman
基于 scrapy-redis 的通用分布式爬虫框架
hbase hive kafka rdbm scapy-redis scrapy spiderman
Last synced: 29 Oct 2024
https://github.com/mouday/spider-admin-pro
spider-admin-pro 一个集爬虫Scrapy+Scrapyd爬虫项目查看 和 爬虫任务定时调度的可视化管理工具，SpiderAdmin的升级版
Last synced: 17 Dec 2024
https://github.com/spekulatius/phpscraper
A universal web-util for PHP.
beautifulsoup chromium headless-chrome php php-crawler php-scraper php-spider php-spiders puppeteer pyppeteer scraper scraping scraping-websites scrapy web-scraper web-scraping
Last synced: 20 Dec 2024
https://github.com/spekulatius/PHPScraper
A universal web-util for PHP.
beautifulsoup chromium headless-chrome php php-crawler php-scraper php-spider php-spiders puppeteer pyppeteer scraper scraping scraping-websites scrapy web-scraper web-scraping
Last synced: 25 Oct 2024
https://github.com/abhisharma404/vault
swiss army knife for hackers
crawler fuzzing hacking hacking-tool information-gathering lfi networking offensive-security osint pentesting port-scanner python rfi scanner scrapy security sqlite ssl-inspection vault xss-vulnerability
Last synced: 03 Nov 2024
https://github.com/AlexMathew/scrapple
A framework for creating semi-automatic web content extractors
beautifulsoup crawler css-selector extractor lxml python scrapers scraping scrapy selector selector-expression tutorial web-scraper web-scraping xpath-expression
Last synced: 31 Oct 2024
https://github.com/sangaline/advanced-web-scraping-tutorial
The Zipru scraper developed in the Advanced Web Scraping Tutorial.
python scraper scrapy tutorial-code
Last synced: 17 Dec 2024
https://github.com/my8100/files
Docs and files for ScrapydWeb, Scrapyd, Scrapy, and other projects
Last synced: 01 Dec 2024
https://github.com/MarwanDebbiche/post-tuto-deployment
Build and deploy a machine learning app from scratch 🚀
api aws character-level-cnn deployment docker machine-learning pytorch scraping scrapy selenium
Last synced: 27 Nov 2024
https://github.com/marwandebbiche/post-tuto-deployment
Build and deploy a machine learning app from scratch 🚀
api aws character-level-cnn deployment docker machine-learning pytorch scraping scrapy selenium
Last synced: 15 Dec 2024
https://github.com/kingname/sourcecodeofbook
《Python爬虫开发 从入门到实战》配套源代码。
python python3 requests scrapy webcrawler
Last synced: 16 Dec 2024
https://github.com/scrapy-plugins/scrapy-zyte-smartproxy
Zyte Smart Proxy Manager (formerly Crawlera) middleware for Scrapy
crawler crawler-detection plugin proxy scraping scrapy
Last synced: 21 Dec 2024
https://github.com/scrapy-plugins/scrapy-crawlera
Zyte Smart Proxy Manager (formerly Crawlera) middleware for Scrapy
crawler crawler-detection plugin proxy scraping scrapy
Last synced: 05 Sep 2024
https://github.com/lkuffo/web-scraping
Más de 50 ejemplos de web scraping utilizando: Requests | Scrapy | Selenium | LXML | BeautifulSoup
beautifulsoup beautifulsoup4 lxml-etree scraping scraping-python scraping-websites scrapping-python scrapy scrapy-crawler scrapy-spider selenium selenium-python selenium-webdriver web-scraping webscraping
Last synced: 21 Dec 2024
https://github.com/City-Bureau/city-scrapers
Scrape, standardize and share public meetings from local government websites
city-scrapers open-data python scrapy web-scraping
Last synced: 06 Nov 2024
https://github.com/yangjianxin1/qqmusicspider
基于Scrapy的QQ音乐爬虫(QQ Music Spider)，爬取歌曲信息、歌词、精彩评论等，并且分享了QQ音乐中排名前6400名的内地和港台歌手的49万+的音乐语料
crawler music musicspider qqmusic scrapy
Last synced: 16 Dec 2024
https://github.com/TikHubIO/TikHub-API-Python-SDK
High-performance asynchronous Douyin(抖音) TikTok Xiaohongshu(小红书) Kuaishou(快手) Weibo(微博) Instagram YouTube(油管) Twitter(X) Captcha Solver(验证码解决器) Temp Mail(临时邮箱) API(接口).
api captcha-solver crawler data-api douyin douyin-tiktok-api instagram kuaishou netease-cloud-music private-api scrapy tiktok twitter weibo xiaohongshu xiguashipin
Last synced: 29 Oct 2024
https://github.com/windrises/dialogue.moe
dialogue django elasticsearch scrapy vue
Last synced: 19 Nov 2024
https://github.com/zhupingqi/RuiJi.Net
crawler framework, distributed crawler extractor
crawler extractor headless-chrome netcore owin scraper scrapy
Last synced: 13 Nov 2024
https://github.com/glaucocustodio/tanakai
Tanakai is a modern web scraping framework written in Ruby. A fork of Kimurai.
chrome-headless crawler kimurai scraper scrapy webscraping
Last synced: 31 Oct 2024
https://github.com/xyntax/filesensor
Dynamic file detection tool based on crawler 基于爬虫的动态敏感文件探测工具
crawler fuzzing pentesting scrapy
Last synced: 18 Dec 2024
https://github.com/crawlab-team/crawlab-lite
Lite version of Crawlab. 轻量版 Crawlab 爬虫管理平台
crawlab crawler crawler-management crawling-tasks platform scrapy scrapy-ui scrapyd scrapyd-ui spider web-crawler
Last synced: 17 Nov 2024
https://github.com/ttonys/Scrapy-CVE-CNVD
漏洞监控，基于scrapy，scrapy-redis，获取每日最新的CVE和CNVD漏洞，邮件通知
Last synced: 21 Nov 2024
https://github.com/henryhaohao/wenshu_spider
:rainbow:Wenshu_Spider-Scrapy框架爬取中国裁判文书网案件数据(2019-1-9最新版)
abuyun decrypt judgement proxy-server scrapy wenshu
Last synced: 19 Dec 2024
https://github.com/DiegoCaraballo/Email-extractor
The main functionality is to extract all the emails from one or several URLs - La funcionalidad principal es extraer todos los correos electrónicos de una o varias Url
email email-extractor email-marketing emails extraction python scraper scrapers scraping scraping-websites scrapper scrapping scrapy scrapy-spider spyder stractor
Last synced: 21 Nov 2024
https://github.com/mehmetozkaya/dotnetcrawler
DotnetCrawler is a straightforward, lightweight web crawling/scrapying library for Entity Framework Core output based on dotnet core. This library designed like other strong crawler libraries like WebMagic and Scrapy but for enabling extandable your custom requirements. Medium link : https://medium.com/@mehmetozkaya/creating-custom-web-crawler-with-dotnet-core-using-entity-framework-core-ec8d23f0ca7c
crawler crawling csharp ddd-architecture dotnetcore entity-framework-core htmlagilitypack scraping scrapy scrapy-crawler webcrawler webcrawler-htmlagilitypack webcrawling webscraper webscraping
Last synced: 17 Nov 2024
https://github.com/mehmetozkaya/DotnetCrawler
DotnetCrawler is a straightforward, lightweight web crawling/scrapying library for Entity Framework Core output based on dotnet core. This library designed like other strong crawler libraries like WebMagic and Scrapy but for enabling extandable your custom requirements. Medium link : https://medium.com/@mehmetozkaya/creating-custom-web-crawler-with-dotnet-core-using-entity-framework-core-ec8d23f0ca7c
crawler crawling csharp ddd-architecture dotnetcore entity-framework-core htmlagilitypack scraping scrapy scrapy-crawler webcrawler webcrawler-htmlagilitypack webcrawling webscraper webscraping
Last synced: 09 Nov 2024
https://github.com/scrapinghub/scrapy-training
Scrapy Training companion code
python scrapy training web-crawling web-scraping
Last synced: 10 Nov 2024
https://github.com/brucedone/scrapy_demo
all kinds of scrapy demo
cnbeta demo douban-image example imagespipeline kafak kafka mongodb oss pipeline scrapy scrapy-demo spider sqlalchemy
Last synced: 18 Dec 2024
