Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
Projects in Awesome Lists tagged with spider
A curated list of projects in awesome lists tagged with spider .
https://github.com/naibowang/easyspider
A visual no-code/code-free web crawler/spider易采集：一个可视化浏览器自动化测试/数据采集/爬虫软件，可以无代码图形化的设计和执行爬虫任务。别名：ServiceWrapper面向Web应用的智能化服务封装系统。
batch-processing batch-script code-free crawler data-collection frontend gui html input-parameters layman parameters robotics rpa scraper spider visual visualization visualprogramming web www
Last synced: 18 Nov 2024
https://github.com/NaiboWang/EasySpider
A visual no-code/code-free web crawler/spider易采集：一个可视化浏览器自动化测试/数据采集/爬虫软件，可以无代码图形化的设计和执行爬虫任务。别名：ServiceWrapper面向Web应用的智能化服务封装系统。
batch-processing batch-script code-free crawler data-collection frontend gui html input-parameters layman parameters robotics rpa scraper spider visual visualization visualprogramming web www
Last synced: 27 Oct 2024
https://github.com/shengqiangzhang/examples-of-web-crawlers
一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、微信读书、豆瓣、QQ等网站。(Some interesting examples of python crawlers that are friendly to beginners. )
agent-pool crawler example fund multithreading pyquery python selenium spider stock taobao tmall wechat wechat-report wereader
Last synced: 19 Nov 2024
https://github.com/crawlab-team/crawlab
Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架
crawlab crawler crawling-tasks docker go platform scrapy scrapyd-ui spider spiders-management web-crawler webcrawler webspider
Last synced: 18 Nov 2024
https://github.com/s0md3v/photon
Incredibly fast crawler designed for OSINT.
crawler information-gathering osint python spider
Last synced: 18 Nov 2024
https://github.com/s0md3v/Photon
Incredibly fast crawler designed for OSINT.
crawler information-gathering osint python spider
Last synced: 28 Oct 2024
https://github.com/ssssssss-team/spider-flow
新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。
crawler jsoup spider spider-flow web-crawler web-spider webcrawler webspider xpath
Last synced: 19 Nov 2024
https://github.com/guyueyingmu/avbook
AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database
adult adult-video avmoo crawler database guzzlehttp javbus javlibrary laravel magnet magnet-link scraper spider
Last synced: 18 Nov 2024
https://github.com/Evil0ctal/Douyin_TikTok_Download_API
🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、TikTok、Bilibili数据爬取工具，支持API调用，在线批量解析及下载。
api async crawler douyin douyin-api douyin-scraper douyin-tiktok-api douyin-tiktok-download fastapi no-watermark online-parsing python pywebio scraper spider tiktok tiktok-api tiktok-scraper tiktok-signature web-scraping
Last synced: 29 Oct 2024
https://github.com/evil0ctal/douyin_tiktok_download_api
🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、TikTok、Bilibili数据爬取工具，支持API调用，在线批量解析及下载。
api async crawler douyin douyin-api douyin-scraper douyin-tiktok-api douyin-tiktok-download fastapi no-watermark online-parsing python pywebio scraper spider tiktok tiktok-api tiktok-scraper tiktok-signature web-scraping
Last synced: 18 Nov 2024
https://github.com/kangvcar/infospider
INFO-SPIDER 是一个集众多数据源于一身的爬虫工具箱🧰，旨在安全快捷的帮助用户拿回自己的数据，工具代码开源，流程透明。支持数据源包括GitHub、QQ邮箱、网易邮箱、阿里邮箱、新浪邮箱、Hotmail邮箱、Outlook邮箱、京东、淘宝、支付宝、中国移动、中国联通、中国电信、知乎、哔哩哔哩、网易云音乐、QQ好友、QQ群、生成朋友圈相册、浏览器浏览历史、12306、博客园、CSDN博客、开源中国博客、简书。
automation chrome crawl csdn hotmail outlook python3 selenium spider tkinter wxpython
Last synced: 18 Nov 2024
https://github.com/kangvcar/InfoSpider
INFO-SPIDER 是一个集众多数据源于一身的爬虫工具箱🧰，旨在安全快捷的帮助用户拿回自己的数据，工具代码开源，流程透明。支持数据源包括GitHub、QQ邮箱、网易邮箱、阿里邮箱、新浪邮箱、Hotmail邮箱、Outlook邮箱、京东、淘宝、支付宝、中国移动、中国联通、中国电信、知乎、哔哩哔哩、网易云音乐、QQ好友、QQ群、生成朋友圈相册、浏览器浏览历史、12306、博客园、CSDN博客、开源中国博客、简书。
automation chrome crawl csdn hotmail outlook python3 selenium spider tkinter wxpython
Last synced: 29 Oct 2024
https://github.com/andeya/pholcus
Pholcus is a distributed high-concurrency crawler software written in pure golang
Last synced: 16 Nov 2024
https://github.com/luyishisi/anti-anti-spider
越来越多的网站具有反爬虫特性，有的用图片隐藏关键数据，有的使用反人类的验证码，建立反反爬虫的代码仓库，通过与不同特性的网站做斗争（无恶意）提高技术。（欢迎提交难以采集的网站）（因工作原因，项目暂停）
Last synced: 19 Nov 2024
https://github.com/luyishisi/Anti-Anti-Spider
越来越多的网站具有反爬虫特性，有的用图片隐藏关键数据，有的使用反人类的验证码，建立反反爬虫的代码仓库，通过与不同特性的网站做斗争（无恶意）提高技术。（欢迎提交难以采集的网站）（因工作原因，项目暂停）
Last synced: 24 Oct 2024
https://github.com/bda-research/node-crawler
Web Crawler/Spider for NodeJS + server-side jQuery ;-)
cheerio crawler extract-data javascript jquery nodejs spider
Last synced: 18 Nov 2024
https://github.com/spiderclub/haipproxy
:sparkling_heart: High available distributed ip proxy pool, powerd by Scrapy and Redis
crawler distributed high-availability ipproxy redis scheduler scrapy spider
Last synced: 19 Nov 2024
https://github.com/SpiderClub/haipproxy
:sparkling_heart: High available distributed ip proxy pool, powerd by Scrapy and Redis
crawler distributed high-availability ipproxy redis scheduler scrapy spider
Last synced: 29 Oct 2024
https://github.com/ihmily/DouyinLiveRecorder
可循环值守和多人录制的直播录制软件，支持抖音、TikTok、快手、虎牙、斗鱼、B站、小红书、pandatv、afreecatv、flextv、popkontv、twitcasting、winktv、百度、微博、酷狗、花椒、Twitch、Acfun、CHZZK等平台直播录制
acfun-live afreecatv douyin douyin-api douyin-live douyu douyulive flextv huya live-recorder pandatv showroom-live spider tiktok tiktok-api tiktoklive twitcasting twitch video-downloader weibo-live
Last synced: 29 Oct 2024
https://github.com/tophubs/toplist
今日热榜，一个获取各大热门网站热门头条的聚合网站，使用Go语言编写，多协程异步快速抓取信息，预览:https://mo.fish
golang hot hotlist spider today-s-hot-list
Last synced: 14 Oct 2024
https://github.com/tophubs/TopList
今日热榜，一个获取各大热门网站热门头条的聚合网站，使用Go语言编写，多协程异步快速抓取信息，预览:https://mo.fish
golang hot hotlist spider today-s-hot-list
Last synced: 30 Oct 2024
https://github.com/niespodd/browser-fingerprinting
Analysis of Bot Protection systems with available countermeasures 🚿. How to defeat anti-bot system 👻 and get around browser fingerprinting scripts 🕵️♂️ when scraping the web?
automation bot bot-detection browser-fingerprinting chromedriver chromium chromium-browser crawler detection fingerprinting puppeteer recaptcha scraper spider stealth web webscraping
Last synced: 19 Nov 2024
https://github.com/wechatsync/Wechatsync
一键同步文章到多个内容平台，支持今日头条、WordPress、知乎、简书、掘金、CSDN、typecho各大平台，一次发布，多平台同步发布。解放个人生产力
blog chrome chrome-extension markdown multiplatform spider vue wechat-official-account writer
Last synced: 29 Oct 2024
https://github.com/wechatsync/wechatsync
一键同步文章到多个内容平台，支持今日头条、WordPress、知乎、简书、掘金、CSDN、typecho各大平台，一次发布，多平台同步发布。解放个人生产力
blog chrome chrome-extension markdown multiplatform spider vue wechat-official-account writer
Last synced: 13 Oct 2024
https://github.com/my8100/scrapydweb
Web app for Scrapyd cluster management, Scrapy log analysis & visualization, Auto packaging, Timer tasks, Monitor & Alert, and Mobile UI. DEMO :point_right:
dashboard log-analysis log-parsing scrapy scrapy-log-analysis scrapy-visualization scrapyd scrapyd-admin scrapyd-api scrapyd-cluster-management scrapyd-control scrapyd-keeper scrapyd-log-analysis scrapyd-manage scrapyd-monitor scrapyd-ui scrapyd-visualization spider
Last synced: 19 Nov 2024
https://github.com/dedsecinside/torbot
Dark Web OSINT Tool
algorithm crawler dark-web dedsec-inside deepweb go hacking hacktoberfest osint projects psnappz python python-web-crawler python3 security security-tools spider tor tor-network torbot
Last synced: 19 Nov 2024
https://github.com/DedSecInside/TorBot
Dark Web OSINT Tool
algorithm crawler dark-web dedsec-inside deepweb go hacking hacktoberfest osint projects psnappz python python-web-crawler python3 security security-tools spider tor tor-network torbot
Last synced: 02 Nov 2024
https://github.com/JAVClub/core
🔞 JAVClub - 让你的大姐姐不再走丢
adult adult-content google-drive japanese jav javbus javiewer magnet porn spider video-streaming
Last synced: 19 Nov 2024
https://github.com/wnma3mz/wechat_articles_spider
微信公众号文章的爬虫
officialaccounts python36 spider wechat wechat-official-account
Last synced: 09 Oct 2024
https://github.com/DormyMo/SpiderKeeper
admin ui for scrapy/open source scrapinghub
dashboard scrapy scrapy-ui scrapyd scrapyd-dashboard scrapyd-ui spider
Last synced: 30 Oct 2024
https://github.com/dormymo/spiderkeeper
admin ui for scrapy/open source scrapinghub
dashboard scrapy scrapy-ui scrapyd scrapyd-dashboard scrapyd-ui spider
Last synced: 14 Oct 2024
https://github.com/shiyanhui/dht
BitTorrent DHT Protocol && DHT Spider.
bittorrent-dht-protocol dht go spider
Last synced: 14 Oct 2024
https://github.com/jae-jae/querylist
:spider: The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。
crawler querylist scraper spider
Last synced: 18 Nov 2024
https://github.com/jae-jae/QueryList
:spider: The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。
crawler querylist scraper spider
Last synced: 25 Oct 2024
https://github.com/boris-code/feapder
🚀🚀🚀feapder is an easy to use, powerful crawler framework | feapder是一款上手简单，功能强大的Python爬虫框架。内置AirSpider、Spider、TaskSpider、BatchSpider四种爬虫解决不同场景的需求。且支持断点续爬、监控报警、浏览器渲染、海量数据去重等功能。更有功能强大的爬虫管理系统feaplat为其提供方便的部署及调度
crawler feapder feaplat python scrapy spider
Last synced: 19 Nov 2024
https://github.com/Boris-code/feapder
🚀🚀🚀feapder is an easy to use, powerful crawler framework | feapder是一款上手简单，功能强大的Python爬虫框架。内置AirSpider、Spider、TaskSpider、BatchSpider四种爬虫解决不同场景的需求。且支持断点续爬、监控报警、浏览器渲染、海量数据去重等功能。更有功能强大的爬虫管理系统feaplat为其提供方便的部署及调度
crawler feapder feaplat python scrapy spider
Last synced: 31 Oct 2024
https://github.com/5ime/video_spider
短视频去水印：抖音,皮皮虾,火山,微视,微博,绿洲,最右,轻视频,快手,全民小视频,巴塞电影,陌陌,Before避风,开眼,Vue Vlog 小咖秀,皮皮搞笑,全民K歌,西瓜视频,逗拍,虎牙,6间房,梨视频,新片场,acfun,美拍...
Last synced: 19 Nov 2024
https://github.com/lorien/grab
Web Scraping Framework
asynchronous crawler crawling framework http-client network pycurl python python-library python3 scraping spider urllib3 web-scraping
Last synced: 19 Nov 2024
https://github.com/sjdirect/abot
Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.
abot abot-nuget c-sharp crawler cross-platform csharp csharp-library javascript-renderer netcore netcore2 netcore3 netsta netstandard20 netstandard21 parsing pluggable spider spiders unit-testing web-crawler
Last synced: 19 Nov 2024
https://github.com/qianyantech/image-downloader
Download images from Google, Bing, Baidu. 谷歌、百度、必应图片下载.
baidu bing google google-images image-downloader pyqt scrapy spider
Last synced: 20 Nov 2024
https://github.com/QianyanTech/Image-Downloader
Download images from Google, Bing, Baidu. 谷歌、百度、必应图片下载.
baidu bing google google-images image-downloader pyqt scrapy spider
Last synced: 08 Nov 2024
https://github.com/nemo2011/bilibili-api
哔哩哔哩常用API调用。支持视频、番剧、用户、频道、音频等功能。原仓库地址：https://github.com/MoyuScript/bilibili-api
api bilibili bilibili-api python spider
Last synced: 19 Nov 2024
https://github.com/Nemo2011/bilibili-api
哔哩哔哩常用API调用。支持视频、番剧、用户、频道、音频等功能。原仓库地址：https://github.com/MoyuScript/bilibili-api
api bilibili bilibili-api python spider
Last synced: 27 Oct 2024
https://github.com/JayBizzle/Crawler-Detect
🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent
bots crawler detect hacktoberfest php spider user-agent
Last synced: 03 Nov 2024
https://github.com/zorlan/skycaiji
蓝天采集器是一款开源免费的爬虫系统，仅需点选编辑规则即可采集数据，可运行在本地、虚拟主机或云服务器中，几乎能采集所有类型的网页，无缝对接各类CMS建站程序，免登录实时发布数据，全自动无需人工干预！是网页大数据采集软件中完全跨平台的云端爬虫系统
crawler crawling php spider webcrawler
Last synced: 15 Oct 2024
https://github.com/jaybizzle/crawler-detect
🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent
bots crawler detect hacktoberfest php spider user-agent
Last synced: 18 Nov 2024
https://github.com/xianhu/pspider
简单易用的Python爬虫框架，QQ交流群：597510560
crawler multi-threading multiprocessing proxies python python-spider spider web-crawler web-spider
Last synced: 12 Nov 2024
https://github.com/hu17889/go_spider
[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized crawler easily or you can use the default crawl components only.
crawler go pipeline schedule spider
Last synced: 29 Oct 2024
https://github.com/xianhu/PSpider
简单易用的Python爬虫框架，QQ交流群：597510560
crawler multi-threading multiprocessing proxies python python-spider spider web-crawler web-spider
Last synced: 29 Oct 2024
https://github.com/howie6879/ruia
Async Python 3.6+ web scraping micro-framework based on asyncio
aiohttp asyncio asyncio-spider crawler crawling-framework middlewares python python-ruia ruia spider uvloop
Last synced: 15 Oct 2024
https://github.com/howie6879/aspider
Async Python 3.6+ web scraping micro-framework based on asyncio
aiohttp asyncio asyncio-spider crawler crawling-framework middlewares python python-ruia ruia spider uvloop
Last synced: 05 Aug 2024
https://github.com/librauee/reptile
🏀 Python3 网络爬虫实战（部分含详细教程）猫眼 腾讯视频 豆瓣 研招网 微博 笔趣阁小说 百度热点 B站 CSDN 网易云阅读 阿里文学 百度股票 今日头条 微信公众号 网易云音乐 拉勾 有道 unsplash 实习僧 汽车之家 英雄联盟盒子 大众点评 链家 LPL赛程 台风 梦幻西游、阴阳师藏宝阁 天气 牛客网 百度文库 睡前故事 知乎 Wish
python3 requests scrapy spider
Last synced: 29 Oct 2024
https://github.com/coder-hxl/x-crawl
Flexible Node.js AI-assisted crawler library
ai ai-crawl chromium crawl crawler fingerprint flexible javascript multifunction nodejs puppeteer spider typescript
Last synced: 19 Nov 2024
https://github.com/1N3/BlackWidow
A Python based web application scanner to gather OSINT and fuzz for OWASP vulnerabilities on a target website.
active application automated bugbounty csrf fuzzer lfi osint owasp passive python rce rfi scan scanner spider sqli vulnerability web xss
Last synced: 01 Nov 2024
https://github.com/1n3/blackwidow
A Python based web application scanner to gather OSINT and fuzz for OWASP vulnerabilities on a target website.
active application automated bugbounty csrf fuzzer lfi osint owasp passive python rce rfi scan scanner spider sqli vulnerability web xss
Last synced: 15 Oct 2024
https://github.com/shuncai/qzoneexport
QQ空间导出助手，用于备份QQ空间的说说、日志、私密日记、相册、视频、留言板、QQ好友、收藏夹、分享、最近访客为文件，便于迁移与保存
backup chrome chrome-extension chromium crx export qq qqzone qzone qzone-spider spider
Last synced: 25 Sep 2024
https://github.com/ShunCai/QZoneExport
QQ空间导出助手，用于备份QQ空间的说说、日志、私密日记、相册、视频、留言板、QQ好友、收藏夹、分享、最近访客为文件，便于迁移与保存
backup chrome chrome-extension chromium crx export qq qqzone qzone qzone-spider spider
Last synced: 29 Oct 2024
https://github.com/keenwon/antcolony
Nodejs实现的一个磁力链接爬虫 https://findit.keenwon.com (原域名http://findit.so )
antcolony bencode bittorrent dht javascript nodejs spider torrent
Last synced: 06 Nov 2024
https://github.com/0xHJK/dumpall
一款信息泄漏利用工具，适用于.git/.svn/.DS_Store泄漏和目录列出
bug-bounty dumpall githack hacking pentesting python3 scanner security spider svn tools
Last synced: 03 Nov 2024
https://github.com/0xhjk/dumpall
一款信息泄漏利用工具，适用于.git/.svn/.DS_Store泄漏和目录列出
bug-bounty dumpall githack hacking pentesting python3 scanner security spider svn tools
Last synced: 14 Oct 2024
https://github.com/u3c3/bt-btt
磁力網站U3C3介紹以及域名更新
adult avmoo bittorrent bt btsow crawler download jav javbus javlibrary magnet magnet-link nyaa porn rarbg spider sukebei tracker u3c3
Last synced: 15 Oct 2024
https://github.com/kiddyuchina/Beanbun
Beanbun 是用 PHP 编写的多进程网络爬虫框架，具有良好的开放性、高可扩展性，基于 Workerman。
Last synced: 01 Nov 2024
https://github.com/kiddyuchina/beanbun
Beanbun 是用 PHP 编写的多进程网络爬虫框架，具有良好的开放性、高可扩展性，基于 Workerman。
Last synced: 14 Oct 2024
https://github.com/srx-2000/spider_collection
python爬虫，目前库存：网易云音乐歌曲爬取，B站视频爬取，知乎问答爬取，壁纸爬取，xvideos视频爬取，有声书爬取，微博爬虫，安居客信息爬取+数据可视化，哔哩哔哩视频封面提取器，ip代理池封装，知乎百万级用户爬虫+数据分析，github用户爬虫
Last synced: 15 Oct 2024
https://github.com/holgerd77/django-dynamic-scraper
Creating Scrapy scrapers via the Django admin interface
django python scraper scraping scrapy spider webscraping
Last synced: 21 Oct 2024
https://github.com/filamentgroup/glyphhanger
Your web font utility belt. It can subset web fonts. It can find unicode-ranges for you automatically. It makes julienne fries.
font glyphs spider subset subsetting unicode web-fonts webfonts
Last synced: 10 Nov 2024
https://github.com/0xinfection/xsrfprobe
The Prime Cross Site Request Forgery (CSRF) Audit and Exploitation Toolkit.
audit crafted-tokens crawler csrf csrf-attacks csrf-poc csrf-scanner csrf-tokens spider token-generation xsrf
Last synced: 19 Nov 2024
https://github.com/0xInfection/XSRFProbe
The Prime Cross Site Request Forgery (CSRF) Audit and Exploitation Toolkit.
audit crafted-tokens crawler csrf csrf-attacks csrf-poc csrf-scanner csrf-tokens spider token-generation xsrf
Last synced: 28 Oct 2024
https://github.com/bytebuff/JSpider
JSpider会每周更新至少一个网站的JS解密方式，欢迎 Star，交流微信：13298307816
javascript nodejs python3 scrapy spider
Last synced: 01 Nov 2024
https://github.com/bytebuff/jspider
JSpider会每周更新至少一个网站的JS解密方式，欢迎 Star，交流微信：13298307816
javascript nodejs python3 scrapy spider
Last synced: 12 Nov 2024
https://github.com/okfn-brasil/querido-diario
📰 Diários oficiais brasileiros acessíveis a todos | 📰 Brazilian government gazettes, accessible to everyone.
civic-tech data-science governments-gazettes govtech hacktoberfest open-data politics scraping spider
Last synced: 14 Oct 2024
https://github.com/s045pd/DarkNet_ChineseTrading
🚇暗网中文网监控爬虫(DEEPMIX)
darknet darknet-chinesetrading grafana grafana-dashboard python python3 spider telegram tor
Last synced: 03 Nov 2024
https://github.com/s045pd/darknet_chinesetrading
🚇暗网中文网监控爬虫(DEEPMIX)
darknet darknet-chinesetrading grafana grafana-dashboard python python3 spider telegram tor
Last synced: 27 Sep 2024
https://github.com/yutto-dev/bilili
:beers: bilibili video (including bangumi) and danmaku downloader | B站视频（含番剧）、弹幕下载器
bilibili crawler danmaku download downloader multithread python3 requests spider subtitle video
Last synced: 14 Oct 2024
https://github.com/zhangyd-c/OneBlog
:alien: OneBlog，一个简洁美观、功能强大并且自适应的Java博客
blog blog-hunter bootstrap dblog justauth oneblog oss qiniu redis seo shiro spider spring-boot springboot wangeditor websockets
Last synced: 13 Nov 2024
https://github.com/zhangyd-c/oneblog
:alien: OneBlog，一个简洁美观、功能强大并且自适应的Java博客
blog blog-hunter bootstrap dblog justauth oneblog oss qiniu redis seo shiro spider spring-boot springboot wangeditor websockets
Last synced: 11 Oct 2024
https://github.com/QIN2DIM/V2RSS
:rocket: 采集|免费|优质|的-订?阅<；
chromedriver flask python3 selenium spider v2rss
Last synced: 08 Nov 2024
https://github.com/qin2dim/v2rss
:rocket: 采集|免费|优质|的-订?阅<；
chromedriver flask python3 selenium spider v2rss
Last synced: 11 Oct 2024