Projects in Awesome Lists tagged with website-crawler
A curated list of projects in awesome lists tagged with website-crawler .
https://github.com/x-slayer/website-cloner
It allows you to download a website from the Internet to a local directory, building recursively all directories, getting HTML, images, and other files from the server to your computer.
clone css front-end front-end-clone html images js website-clone website-cloner website-crawler
Last synced: 07 Apr 2025
https://github.com/flulemon/sneakpeek
Sneakpeek is a framework that helps to quickly and conviniently develop scrapers. It’s the best choice for scrapers that have some specific complex scraping logic that needs to be run on a constant basis
crawler crawler-python crawlers crawling crawling-engine crawling-framework python python3 scraper scraper-api scraper-engine scrapers scraping scraping-framework vue website-crawler
Last synced: 14 Jan 2026
https://github.com/sammwyy/spearcopy
A universal and local phishing toolkit for audit purposes
audit pentest pentest-tool pentesting phishing phishing-kit phishing-page phishing-script phishing-tool python web-clone web-crawler webscraping website-clone website-crawler
Last synced: 24 Jul 2025
https://github.com/oxylabs/web-scraping-php
A tutorial and code samples of web scraping with PHP
email-scraper email-scraper-with-proxy php screen-scraping url-scraper web-scraping website-crawler wikipedia-scraper
Last synced: 15 Jun 2025
https://github.com/zebbern/reconx
🕷️ | ReconX is a Live-Website Crawler made to gather critical information with an option to take a picture of each site crawled!
crawler hacking information-gathering information-retrieval information-security livedata opsec osint osint-tool pentest python python-crawler search-engine security security-tools website website-crawler website-scraper website-security
Last synced: 03 Jul 2025
https://github.com/foomo/walker
Crawls website and collect SEO relevant data
apache-benchmark benchmarking foomo foomo-walker siege spider website-crawler
Last synced: 14 May 2026
https://github.com/deependra-patel/websitecrawler
Crawls a website to generate insights
golang sitemap-generator website-crawler
Last synced: 16 Mar 2025
https://github.com/1970mr/link-crawler
Web Link Crawler: A Python script to crawl websites and collect links based on a regex pattern. Efficient and customizable.
clawler crawler crawler-python link-crawler link-crawler-python link-scraper link-scraper-python links python scraper scraper-python website-crawler website-scraper
Last synced: 06 Feb 2026
https://github.com/dyzio18/java-web-bot-library
Java website crawler - library for analyze and testing websites
crawler-engine web-bot website-crawler
Last synced: 16 May 2025
https://github.com/mattmoony/image-grabber
Grabs images off webpages.
downloader images internet pictures python python3 python36 webcrawler webpages website-crawler
Last synced: 04 Jun 2026
https://github.com/reineimi/va2crawl
Website crawler, validator and SEO optimizer
crawler seo-optimization seotools validator website-crawler
Last synced: 07 Jul 2025
https://github.com/radityaharya/sitesweeper
Sitesweeper is a python package to help you automate your web scraping process, outputting pages to a file
crawler pdf python website-crawler
Last synced: 27 Mar 2025
https://github.com/mishqatabid/domain-email-harvesting-tool
Email Harvesting Tool designed to efficiently gather and validate emails from specified websites
cybersecurity-tool email email-harvester website-crawler
Last synced: 15 May 2025
https://github.com/ursnj/seo-master
SEO Master is a powerful all-in-one tool developed to boost your website's visibility and rankings. With features like automatic sitemap generation, customizable robots.txt creation, SEO-optimized metadata, Image assets generation and seamless integration with major search engines.
meta-tags-generator robots-txt search-engine seo-optimization sitemap-builder sitemap-generator sitemaps website-change-monitor website-crawler
Last synced: 20 Feb 2026
