Projects in Awesome Lists tagged with crawling-framework
A curated list of projects in awesome lists tagged with crawling-framework .
https://github.com/howie6879/ruia
Async Python 3.6+ web scraping micro-framework based on asyncio
aiohttp asyncio asyncio-spider crawler crawling-framework middlewares python python-ruia ruia spider uvloop
Last synced: 14 May 2025
https://github.com/Symbo1/wsltools
Web Scan Lazy Tools - Python Package
crawling-framework package python-package scanner-web security security-audit security-automation security-scanner security-tool security-tools spider spider-framework web-vulnerability-scanner
Last synced: 11 Jul 2025
https://github.com/crawlzone/crawlzone
Crawlzone is a fast asynchronous internet crawling framework for PHP.
automated-testing crawler crawling-framework middleware php web-scraping web-search
Last synced: 25 Mar 2025
https://github.com/omkarcloud/botasaurus-starter
🚀 OFFICIAL STARTER TEMPLATE FOR BOTASAURUS SCRAPING FRAMEWORK 🤖
beautifulsoup crawler crawling crawling-framework crawling-python crawling-tool headless node-crawler python-crawler scraper scraping scraping-framework scraping-python scraping-tool selenium web-crawler web-crawling web-scraper web-scraping webscraping
Last synced: 23 Apr 2025
https://github.com/ruedigervoigt/exoskeleton
A Python framework to build polite, but tenacious crawlers / scrapers with a MariaDB backend
crawler crawling-framework database machine-learning mariadb network python python-3 scraping
Last synced: 13 Apr 2025
https://github.com/RuedigerVoigt/exoskeleton
A Python framework to build polite, but tenacious crawlers / scrapers with a MariaDB backend
crawler crawling-framework database machine-learning mariadb network python python-3 scraping
Last synced: 17 Apr 2025
https://github.com/tokenmill/crawling-framework
Easily crawl news portals or blog sites using Storm Crawler.
crawler crawling crawling-framework elasticsearch java scraping storm storm-crawler vaadin
Last synced: 22 Apr 2025
https://github.com/miroshnikov/scrapyteer
Web crawling & scraping framework for Node.js on top of headless Chrome browser
crawer crawling crawling-framework crawling-sites crawling-tool headless scrape scraper scraping scraping-websites scrapy scrapy-crawler spider spider-framework web-crawler web-crawling web-scraping web-scraping-nodejs
Last synced: 26 Oct 2025
https://github.com/basemax/stockexchangecrawler
A crawler program to extract all of the data and the price for symbols in the global stock exchange.
crawl-pages crawler crawlers crawling crawling-framework crawling-sites crawlspider stock stock-analysis stock-data stock-exchange stock-exchange-crawler stock-exchange-platform stock-exchange-simulator stock-exchanges stock-market stock-prediction stock-price-prediction stock-prices stock-trading
Last synced: 02 Sep 2025
https://github.com/omkarcloud/web-scraping-template
🚀 THIS WEB SCRAPING TEMPLATE PROVIDES YOU WITH A GREAT STARTING POINT WHEN CREATING WEB SCRAPING BOTS. 🤖
beautifulsoup crawler crawling crawling-framework crawling-python crawling-tool headless node-crawler python-crawler scraper scraping scraping-framework scraping-python scraping-tool selenium web-crawler web-crawling web-scraper web-scraping webscraping
Last synced: 24 Oct 2025
https://github.com/basemax/netphp
Useful functions for connecting to the network in the PHP based applications.
crawl-pages crawler-engine crawling-framework crawling-sites curl curl-commands curl-cookie curl-header curl-library curl-php curl-request curl-wrapper network php php-cookie php-http php-https php-httpstatuscode php-network php-request
Last synced: 05 May 2025
https://github.com/jimmylaurent/node-crawling-framework
✨ NodeJs crawling & scraping framework heavily inspired by Scrapy
crawler crawling crawling-framework elasticsearch headless-chrome middleware mongodb nodejs-framework scraper scraping scraping-framework scrapy spider
Last synced: 15 Mar 2025
https://github.com/crwlrsoft/laravel-crawler
Laravel adapter for the crwlr/crawler package.
crawler crawling crawling-framework hacktoberfest laravel laravel-package php scraper scraping web-crawler web-crawling web-scraping
Last synced: 28 Feb 2025
https://github.com/tokenmill/crawling-framework-example
Demonstration on how to use the Crawling Framework to setup a simple science news crawler and store results in ElasticSearch. Use this configuration to set up your own crawler.
crawler crawling-framework elasticsearch storm-crawler
Last synced: 24 Feb 2025
https://github.com/omkarcloud/multiple-account-generation-template
🚀 THIS WEB SCRAPING TEMPLATE PROVIDES YOU WITH A GREAT STARTING POINT WHEN CREATING MULTIPLE ACCOUNTS ON A WEBSITE. 🤖
beautifulsoup crawler crawling crawling-framework crawling-python crawling-tool headless node-crawler python-crawler scraper scraping scraping-framework scraping-python scraping-tool selenium web-crawler web-crawling web-scraper web-scraping webscraping
Last synced: 21 Feb 2025
https://github.com/omkarcloud/dentalkart-scraper
🚀 SCRAPE 1000'S OF PRODUCTS FROM DENTALKART 🤖
beautifulsoup crawler crawling crawling-framework crawling-python dentalkart dentalkart-product-scraper dentalkart-scraper dentalkart-scraping node-crawler scraper scraping scraping-framework scraping-python selenium web-crawler web-crawling web-scraper web-scraping webscraping
Last synced: 07 Sep 2025
https://github.com/surister/scrupy
Python library to create web Crawlers which aims to be powerful yet simple.
crawler crawling-framework crawling-python http library python scraping
Last synced: 25 Dec 2025
https://github.com/vivekg13186/lucas
A web crawler
crawler crawler-engine crawling-framework java
Last synced: 22 Jul 2025
https://github.com/matheusfaustino/phrawl
Phrawl: A web crawling framework in PHP (or it seems so)
crawler crawling crawling-framework php scraper wip
Last synced: 08 Sep 2025
