https://github.com/anonymousx46/advanced-web-scrapping-tool
A web-scrapping-tool I built to automate the process with advanced techniques, ready to use in your Kali Linux Terminal!
https://github.com/anonymousx46/advanced-web-scrapping-tool
amass automation dig kali-linux nikto-features nslookup python web-scraping-python whois-lookup
Last synced: about 1 month ago
JSON representation
A web-scrapping-tool I built to automate the process with advanced techniques, ready to use in your Kali Linux Terminal!
- Host: GitHub
- URL: https://github.com/anonymousx46/advanced-web-scrapping-tool
- Owner: ANONYMOUSx46
- Created: 2025-03-22T23:36:46.000Z (about 1 month ago)
- Default Branch: main
- Last Pushed: 2025-03-23T00:14:57.000Z (about 1 month ago)
- Last Synced: 2025-03-23T01:19:16.451Z (about 1 month ago)
- Topics: amass, automation, dig, kali-linux, nikto-features, nslookup, python, web-scraping-python, whois-lookup
- Language: Python
- Homepage:
- Size: 12.7 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# 🔍Advanced Web Scraping Tool (AWT)🐍
## Created by ANONYMOUSx46
## Overview
The **Advanced Web Scraping Tool (AWT)** is a powerful Python-based utility designed to scrape dynamic web content and perform comprehensive reconnaissance on websites. It leverages tools like Selenium, Nikto, Amass, WhatWeb, and more to gather data efficiently. Whether you're extracting product details, performing security audits, or conducting research, AWT provides a streamlined workflow.
---
## Features
- **Dynamic Content Handling**: Automatically scrolls through pages to load all dynamic content.
- **Customizable Selectors**: Supports both XPath and CSS selectors for flexible data extraction.
- **Headless Mode**: Option to run the browser in headless mode for faster execution.
- **Comprehensive Reconnaissance**:
- WHOIS Lookup
- NSLookup
- DIG Queries (A, AAAA, MX, NS, TXT records)
- WhatWeb Scan
- Curl Headers Collection
- Nikto Vulnerability Scan
- Amass Subdomain Enumeration
- **Log Management**: Saves all scraped data and reconnaissance results into a customizable log file.
- **User-Friendly Interface**: Interactive prompts guide users through the setup and execution process.
---
## Installation
### Prerequisites
Before running the script, ensure you have the following installed:
1. **Python 3.x**: [Download Python](https://www.python.org/downloads/)
2. **Firefox Browser**: [Download Firefox](https://www.mozilla.org/firefox/)
3. **Geckodriver**: [Download Geckodriver](https://github.com/mozilla/geckodriver/releases)
- Add `geckodriver` to your system's PATH. OR use the Chrome Driver, just make sure to change the path in the script.
4. **Required Tools**:
- `nikto`: Install via package manager (e.g., `sudo apt install nikto` on Ubuntu).
- `amass`: [Install Amass](https://github.com/OWASP/Amass)
- `whatweb`: Install via package manager (e.g., `sudo apt install whatweb` on Ubuntu).
5. **Python Dependencies**:
```bash
pip install selenium
## Set Up
### Windows:
```bash
git clone https://github.com/ANONYMOUSx46/Advanced-Web-Scraping-Too.git
cd Advanced-Web-Scraping-Tool
```
- Ensure all external tools (nikto, amass, whatweb, etc.) are installed and added to your system's PATH. You can verify this by running the tool names (e.g., nikto, amass) in Command Prompt.
- Run the script:
```bash
python scrape.py
```
## Cross-Platform Notes:
The script has been tested on both Windows and Kali Linux. However, some tools like dig and curl may require additional installations on Windows. Use tools like Cygwin or WSL (Windows Subsystem for Linux) to access these utilities.
Ensure all required tools are accessible from your terminal or command prompt before running the script.