Projects in Awesome Lists tagged with spiders
A curated list of projects in awesome lists tagged with spiders .
https://github.com/sjdirect/abot
Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.
abot abot-nuget c-sharp crawler cross-platform csharp csharp-library javascript-renderer netcore netcore2 netcore3 netsta netstandard20 netstandard21 parsing pluggable spider spiders unit-testing web-crawler
Last synced: 13 May 2025
https://github.com/FioraLove/Net-Spider
新手小白~~持续更新中〜随意Fork，Star，大佬萌多多issue哦 (目前更新: 1.短视频视频无水印解析（支持超过30多个平台 e.g., 抖音，快手，微博...） 2.bilibili音视频解析🐱👓 3.知音漫客图片加密 🔥 4. 世纪佳缘信息👨👩👧👦 5.小鸡词典🐥🐥。。。。。。
pixiv pornhub python-script python3 spiders
Last synced: 28 Mar 2025
https://github.com/lixi5338619/lxbook
《爬虫逆向进阶实战》书籍代码库
android-resever crawler frida java javascript python smali spiders unidbg xposed
Last synced: 13 Apr 2025
https://github.com/scrapinghub/spidermon
Scrapy Extension for monitoring spiders execution.
crawling hacktoberfest monitoring monitoring-tool scraping scrapinghub spiders testing
Last synced: 14 May 2025
https://github.com/TRHX/Python3-Spider-Practice
Python3 各种爬虫实战练习，JS 逆向、反反爬、验证码处理、登录签到抽奖、数据可视化，Python 3 practice of various spiders.
jsreverse python python3-spider-practice spider spiders
Last synced: 12 May 2025
https://github.com/hoochanlon/scripts
平台：Windows/Mac/Linux。脚本语言：多种、不限；我流，按需编写。涉及：桌面基线排查、软件激活破解、免杀及特权执行、渗透式支援固件识别读写、主机账户密码空值检测、Wi-Fi密码扫描、云主机终端安全加固、主机系统日志分析、自然语言处理、人文社科信息数据分析等。
automation ban-ip clawer linux macos security spiders wi-fi windows-update
Last synced: 04 Apr 2025
https://github.com/sjdirect/abotx
Cross Platform C# Web crawler framework, headless browser, parallel crawler. Please star this project! +1.
abotx abotx-website cross-platform csharp csharp-library framework headless headless-br headless-browser javascript-renderer netcore netcore3 netstan netstandard netstandard-libraries netstandard20 spider spiders spiders- web-crawler
Last synced: 09 Apr 2025
https://github.com/zhangyingwei/cockroach
又一个号称高性能的 java 爬虫工具/爬虫框架
cockroach java java-8 java-spider spider spiders
Last synced: 10 Apr 2025
https://github.com/viveckh/lilhomie
A Machine Learning Project implemented from scratch which involves web scraping, data engineering, exploratory data analysis and machine learning to predict housing prices in New York Tri-State Area.
data-engineering eda housing-price-analysis housing-price-prediction machine-learning machine-learning-projects predictions random-forest-regressor scrapy-crawler spiders trulia web-crawler
Last synced: 09 Sep 2025
https://github.com/yaleimeng/free_proxy_pool
对免费代理IP网站进行爬取，收集汇总为自己的代理池。关键是验证代理的有效性、匿名性、去重复
Last synced: 11 May 2025
https://github.com/andsonder/space.keter.top
这里是sonder的有点又没有太多用的笔记本 “一个人只有不停的写作，才不会被人海淹没” 你可以通过这个链接来访问网页版：https://space.keter.top
algorithms deep-learning math spiders
Last synced: 30 Oct 2025
https://github.com/dotnetage/scrapy_plus
scrapy 常用爬网必备工具包
middlewares python scrapy scrapy-extension scrapy-spider spiders tor
Last synced: 15 Oct 2025
https://github.com/python-world/joble
This Platform Search Thousands Of Job Boards In Different Technologies From Over The World .
hacktoberfest hacktoberfest2020 jobsearch jobseeker scraping spiders
Last synced: 24 Jul 2025
https://github.com/fooock/robots.txt
:robot: robots.txt as a service. Crawls robots.txt files, downloads and parses them to check rules through an API
antlr4 api crawler crawler-engine docker docker-compose gradle java kotlin makefile postgresql redis redis-stream redis-streams robots-parser robots-txt spiders spring-boot
Last synced: 18 Mar 2025
https://github.com/hhhrrrttt222111/selenium_python
beatifulsoup beautifulsoup-library beautifulsoup4 chromedriver geckodriver hacktoberfest pycharm-ide python python-parser python-requests python-scraper requests-html scraping-python scraping-websites selenium selenium-python selenium-webdriver spiders webscraper webscraping
Last synced: 23 Oct 2025
https://github.com/ikp4success/shopasource
Easiest way to find best lowest price products online.
async celery collection css data data-mining flask flask-sqlalchemy html javascript json postgresql python python3 quart scrapy spider spiders webscraper webscraping
Last synced: 07 Sep 2025
https://github.com/just5ky/spidertrap
Traps web crawlers
active-defense cybersecurity docker spiders
Last synced: 12 Apr 2025
https://github.com/harhao/dushecommunity
利用Fiddler抓包分析毒舌影评社区的APP api接口。单机版的scrapy爬虫，基于scrapy-redis
python scrapy scrapy-redis spiders
Last synced: 14 Aug 2025
https://github.com/snowplow/iab-spiders-and-robots-java-client
Java 8+ client library for the IAB and ABC International Spiders and Robots list
abc abc-international-spiders bots iab iab-spiders java-library robots spiders
Last synced: 21 Apr 2025
https://github.com/chanran/distributedcrawling
分布式爬虫系统
distributed python scrapy spiders
Last synced: 05 Jan 2026
https://github.com/excalibra/scripts
Platform: Windows/Mac/Linux. Scripting: Various, no restrictions; customized as needed. Tasks: Desktop checks, software cracking, AV evasion, privilege execution, firmware read/write, password checks, Wi-Fi scans, cloud security, log analysis, NLP, and data analysis in humanities/social sciences.
automation ban-ip clawer linux macos security spiders update wi-fi windows
Last synced: 13 Apr 2025
https://github.com/croqaz/scrapy-quotes
Simple spiders to scraping popular "quotes"
Last synced: 25 Feb 2025
https://github.com/excalibra/script-draft
Platform: Windows/Mac/Linux. Scripting: Various, no restrictions; customized as needed. Tasks: Desktop checks, software cracking, AV evasion, privilege execution, firmware read/write, password checks, Wi-Fi scans, cloud security, log analysis, NLP, and data analysis in humanities/social sciences.
automation ban-ip clawer linux macos security spiders update wi-fi windows
Last synced: 26 Mar 2025
https://github.com/stormwind99/webslinger
Minecraft mod: Spiders shoot webbing at you - slinging webs from a distance, or when they hit you in melee!
forge forge-mod minecraft minecraft-forge-mod minecraft-mod spiders
Last synced: 30 Mar 2025
https://github.com/arkaniightt/scraper_vagas_bot
Este repositório contém um script para varrer um site específico em busca de vagas de emprego. O script coleta e organiza informações sobre oportunidades disponíveis neste site, facilitando a busca por emprego para os candidatos interessados. Os usuários podem personalizar a busca de acordo com suas preferências e qualificações específicas.
scraper scrapers scraping-python scraping-websites scrapy selenium selenium-python selenium-webdriver spiders
Last synced: 03 Nov 2025
https://github.com/devliusir/python
This is some experience and case in Python3 learning.
examples python python-library python3 pythonapplication spiders turtle
Last synced: 12 Mar 2025
https://github.com/asanka-j/post-scraper
Scraper of the blog https://blog.scrapinghub.com/, implemented in Python with Scrapy
Last synced: 24 Aug 2025
https://github.com/niconoe/arabel
🕷️ Source code for the future Atlas of spiders of Belgium based on the ARABEL database
biodiversity biodiversity-data biodiversity-informatics spiders
Last synced: 03 Oct 2025
https://github.com/gyumeijie/quotesspider
A simpler spider which scrapes http://quotes.toscrape.com, a website that lists quotes from famous authors.
python scrapy scrapy-crawler scrapy-demo spiders
Last synced: 28 Mar 2025
https://github.com/tiagodongo/kriolspider
A guide to arachnids in Cape Verde
cape-verde catalog guide landing-page spiders
Last synced: 09 Apr 2025
https://github.com/bashkirtsevich/tg-facts-bot
Telegram bot for publishing interesting facts
aiocron aiohttp aiohttp-client aiosqlite asyncio scorpions spiders sql sqlite sqlite3 telegram telegram-api telegram-bot telegram-bot-api telegram-bots ticks
Last synced: 16 Mar 2025
https://github.com/manu-sh/http_normalizer_parts
http url normalization utilities for web crawlers
http-url library normalization spiders web-crawling web-scraping
Last synced: 05 Aug 2025
https://github.com/excalibra/script
Platform: Windows/Mac/Linux. Scripting: Various, no restrictions; customized as needed. Tasks: Desktop checks, software cracking, AV evasion, privilege execution, firmware read/write, password checks, Wi-Fi scans, cloud security, log analysis, NLP, and data analysis in humanities/social sciences.
automation ban-ip clawer linux macos security spiders update wi-fi windows
Last synced: 17 Mar 2025
https://github.com/palwisha-18/quotes-scraper
Scrape Quote, its Author and Tags from qoutes.toscrape.com (Scrapy Tutorial)
python scrapy spiders web-scraping
Last synced: 05 Apr 2025
