Projects in Awesome Lists tagged with spider-framework
A curated list of projects in awesome lists tagged with spider-framework .
https://github.com/projectdiscovery/katana
A next-generation crawling and spidering framework.
cli crawler gocrawler headless spider-framework web-spider
Last synced: 06 Jan 2026
https://github.com/Symbo1/wsltools
Web Scan Lazy Tools - Python Package
crawling-framework package python-package scanner-web security security-audit security-automation security-scanner security-tool security-tools spider spider-framework web-vulnerability-scanner
Last synced: 11 Jul 2025
https://github.com/mangenotwork/gathertool
gathertool是golang脚本化开发库，目的是提高对应场景程序开发的效率；轻量级爬虫库，接口测试&压力测试库，DB操作库等。
crawl gathertool golang grpc-go proxy-server spider spider-framework
Last synced: 26 Jan 2026
https://github.com/miroshnikov/scrapyteer
Web crawling & scraping framework for Node.js on top of headless Chrome browser
crawer crawling crawling-framework crawling-sites crawling-tool headless scrape scraper scraping scraping-websites scrapy scrapy-crawler spider spider-framework web-crawler web-crawling web-scraping web-scraping-nodejs
Last synced: 26 Oct 2025
https://github.com/bkeepers/spiderman
your friendly neighborhood web crawler
crawler crawler-engine http httprb nokogiri ruby spider spider-framework web-crawler web-scraping webcrawler webscraping
Last synced: 14 Oct 2025
https://github.com/hubertroy/seen
A lightweight crawling/spider framework for everyone(support JavaScript!).:sparkles:
easy-to-use javasciprt lightweight-framework python3 spider-framework support-javascript web-crawling
Last synced: 25 Mar 2025
https://github.com/xunzhuo/airspider
A Fast and Light Python Spider Framework 🕷️
asynchronous crawler crawler-python distributed python3 redis spider spider-framework web
Last synced: 23 Mar 2025
https://github.com/windfarer/biu
biubiubiu~~ I'm a tiny web crawler framework
crawler python spider spider-framework web-crawler
Last synced: 23 Mar 2025
https://github.com/konghayao/jspider
This is a JavaScript toolkit for browser crawler testing.
browser crawler-engine js-jspider spider-framework website
Last synced: 12 Oct 2025
https://github.com/showx/phpspiderman
php爬虫
php phpspider spider spider-framework
Last synced: 04 Jul 2025
