Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/geeknik/test-proxy

Advanced test for proxy & waf
https://github.com/geeknik/test-proxy

cybersecurity devops infosec proxy quality-assurance security testing waf

Last synced: 16 days ago
JSON representation

Advanced test for proxy & waf

Awesome Lists containing this project

README 

          
# test-proxy: An advanced Proxy and WAF Detection Tool



Welcome to the **Advanced Proxy and WAF Detection Tool**! This powerful and flexible tool is designed to analyze potential proxy servers, load balancers, and Web Application Firewalls (WAFs) by examining open ports, SSL certificates, HTTP headers, and various other indicators.



## Overview



This advanced script performs a comprehensive analysis of target hosts, including:



1. **Asynchronous Port Scanning with IPv6 Support**: Rapidly scans custom port ranges on both IPv4 and IPv6 addresses using asynchronous I/O.

2. **SSL/TLS Certificate Analysis**: Retrieves detailed SSL/TLS certificate information, including cipher suites, protocol versions, and validity checks.

3. **HTTP/HTTPS Header Inspection**: Sends requests to both HTTP and HTTPS endpoints and thoroughly examines the headers.

4. **Proxy/Load Balancer Detection**: Analyzes headers for a wide range of proxy and load balancer indicators, loaded dynamically from external files.

5. **Web Application Firewall (WAF) Detection**: Identifies potential WAFs based on specific header signatures, also loaded dynamically.

6. **Redirect Chain Analysis**: Tracks and reports on HTTP and HTTPS redirect chains.

7. **GeoIP Lookup**: Provides geolocation information for target IP addresses.

8. **Banner Grabbing**: Retrieves service banners on open ports to identify running services.

9. **Customizable Output Formats**: Supports text, JSON, and CSV output formats for flexibility in data analysis.

10. **Advanced Logging Control**: Allows setting of logging levels and offers verbose output for in-depth analysis.



## Features



- **High-Performance Asynchronous Scanning**: Utilizes `asyncio` for efficient port scanning and analysis.

- **IPv4 and IPv6 Support**: Capable of analyzing both IPv4 and IPv6 addresses.

- **Advanced Rate Limiting**: Implements configurable rate limiting to prevent overwhelming target servers (1-100 concurrent connections).

- **Custom Port Ranges**: Allows users to specify custom port ranges or additional common ports with comprehensive validation.

- **Secure SSL/TLS Analysis**: Provides detailed SSL/TLS certificate data with optional strict verification to detect certificate issues.

- **Advanced Input Validation**: Comprehensive security validation for hostnames, ports, and file paths to prevent injection attacks.

- **Advanced HTTP(S) Header Analysis**: Examines a wide range of headers to detect proxies, load balancers, and WAFs, with dynamic lists.

- **Banner Grabbing**: Retrieves service banners to identify running services on open ports with rate limiting.

- **Flexible Output Options**: Supports text, JSON, and CSV output formats.

- **Redirect Chain Tracking**: Follows and reports on HTTP and HTTPS redirects.

- **WAF Detection**: Identifies common Web Application Firewalls based on specific headers, with dynamic lists.

- **Verbose Logging and Logging Levels**: Offers detailed logging options and allows setting of logging levels.

- **Multiple Target Support**: Can analyze multiple targets provided via a file.

- **GeoIP Lookup**: Provides geolocation information for target IP addresses.

- **Modular Design**: Code is organized into functions and modules for better readability and maintainability.

- **Dynamic Indicator Lists**: Loads proxy and WAF indicators from external files for easy updates.

- **Security Hardening**: Built-in protection against common security vulnerabilities including directory traversal and DoS attacks.



## Requirements



- **Python 3.10+** (for optimal type hinting support)

- **Required Python libraries**:

  - `requests`

  - `urllib3`

  - `cryptography`

  - `asyncio` (built-in with Python 3.4+)

  - `csv` (built-in)

  - `datetime` (built-in)

  - `typing` (built-in)



Install the required libraries using:



```bash

pip install -r requirements.txt

```



**`requirements.txt`:**



```text

requests

cryptography

urllib3

```



## Setup



It's recommended to use a virtual environment to manage dependencies:



```bash

python3 -m venv venv

source venv/bin/activate  # On Windows use `venv\Scripts\activate`

pip install -r requirements.txt

```



## Usage



Clone the repository and navigate to the project directory:



```bash

git clone https://github.com/geeknik/test-proxy.git

cd test-proxy

```



Run the script with various options:



1. **Basic usage**:



   ```bash

   python testproxy.py -t example.com

   ```



2. **Analyze multiple targets from a file**:



   ```bash

   python testproxy.py -T targets.txt

   ```



3. **Specify custom port ranges**:



   ```bash

   python testproxy.py -t example.com -p 80,443,8000-8100

   ```



4. **JSON output**:



   ```bash

   python testproxy.py -t example.com -of json

   ```



5. **CSV output**:



   ```bash

   python testproxy.py -T targets.txt -of csv -f results.csv

   ```



6. **Save results to a file**:



   ```bash

   python testproxy.py -t example.com -of json -f results.json

   ```



7. **Verbose output**:



   ```bash

   python testproxy.py -t example.com -v

   ```



8. **Set logging level to DEBUG**:



   ```bash

   python testproxy.py -t example.com -l DEBUG

   ```



## Command-line Arguments



- `-t, --target`: The IP address or hostname to analyze.

- `-T, --target-file`: File containing a list of targets to analyze.

- `-p, --ports`: Comma-separated list of ports or port ranges (e.g., `80,443,8000-8100`).

- `-o, --output`: Output format, either 'text' (default) or 'json'.

- `-of, --output-format`: Output format, choices are 'text', 'json', or 'csv'.

- `-f, --file`: Output file path to save results.

- `-l, --log-level`: Set the logging level, choices are 'DEBUG', 'INFO', 'WARNING', 'ERROR' (default: 'INFO').

- `-v, --verbose`: Enable verbose output (equivalent to `--log-level DEBUG`).

- `--verify-ssl`: Enable SSL certificate verification (default: disabled).

- `--rate-limit`: Maximum concurrent connections (default: 5, range: 1-100).

- `--rate-window`: Rate limiting time window in seconds (default: 1.0).

- `-h, --help`: Show help message and exit.



**Note**: You must specify either `-t/--target` or `-T/--target-file`.



## Example Output



```plaintext

Analyzing www.mapbox.com...

Resolved www.mapbox.com to IP: 146.75.104.143

Geolocation info: {'country': 'Sweden', 'state': None, 'city': None, 'latitude': 59.3247, 'longitude': 18.056}

Open ports: []



HTTP Headers (Status: 200):

  Connection: keep-alive

  Content-Type: text/html

  Content-Encoding: gzip

  CF-Ray: 97e96c3e8c00f07a-DFW

  CF-Cache-Status: DYNAMIC

  Age: 74245

  Content-Language: en

  Link: ; rel="canonical"

  content-security-policy: frame-ancestors 'self'

  processed-by: Weglot

  Weglot: id.97e96c3e8c00f07a, p.cf, cs

  weglot-translated: true

  x-lambda-id: e3b8133e-4123-4b1d-8685-e9c5ed9ce63d

  Server: cloudflare

  Accept-Ranges: bytes

  Date: Sat, 13 Sep 2025 17:39:14 GMT

  Via: 1.1 varnish

  X-Frame-Options: SAMEORIGIN

  X-Served-By: cache-dfw-kdfw8210095-DFW, cache-dfw-kdfw8210066-DFW

  X-Cache: HIT, MISS

  X-Cache-Hits: 2, 0

  X-Timer: S1757785154.316774,VS0,VE164

  Vary: x-wf-forwarded-proto, Accept-Encoding

  Cross-Origin-Opener-Policy: same-origin

  Strict-Transport-Security: max-age=31536000; includeSubDomains; preload

  X-Content-Type-Options: nosniff

  X-Download-Options: noopen

  X-XSS-Protection: 1; mode=block



HTTPS Headers (Status: 200):

  Connection: keep-alive

  Content-Type: text/html

  Content-Encoding: gzip

  CF-Ray: 97e96c40ff096a08-DFW

  CF-Cache-Status: DYNAMIC

  Age: 74245

  Content-Language: en

  Link: ; rel="canonical"

  content-security-policy: frame-ancestors 'self'

  processed-by: Weglot

  Weglot: id.97e96c40ff096a08, p.cf, cs

  weglot-translated: true

  x-lambda-id: e3b8133e-4123-4b1d-8685-e9c5ed9ce63d

  Server: cloudflare

  Accept-Ranges: bytes

  Date: Sat, 13 Sep 2025 17:39:14 GMT

  Via: 1.1 varnish

  X-Frame-Options: SAMEORIGIN

  X-Served-By: cache-dfw-kdfw8210029-DFW, cache-dfw-kdfw8210167-DFW

  X-Cache: HIT, MISS

  X-Cache-Hits: 0, 0

  X-Timer: S1757785155.700663,VS0,VE109

  Vary: x-wf-forwarded-proto, Accept-Encoding

  Cross-Origin-Opener-Policy: same-origin

  Strict-Transport-Security: max-age=31536000; includeSubDomains; preload

  X-Content-Type-Options: nosniff

  X-Download-Options: noopen

  X-XSS-Protection: 1; mode=block



HTTP Redirects:

http://www.mapbox.com/



Potential proxy/load balancer detected. Indicators found: Via, CF-RAY, X-Served-By, X-Cache, X-Timer, X-Cache



WAF detected: Generic WAF, Cloudflare WAF



Summary of findings:

  Host: www.mapbox.com

  IP: 146.75.104.143

  Open ports: []

  Proxy/load balancer indicators: Via, CF-RAY, X-Served-By, X-Cache, X-Timer, X-Cache

  WAF detected: Generic WAF, Cloudflare WAF

  Redirects detected: 1



Analysis completed in 7.89 seconds.

```



## Security Features



This tool includes comprehensive security hardening to protect against common vulnerabilities and ensure safe scanning:



### Input Validation & Sanitization

- **Hostname Validation**: Prevents malicious hostnames, blocks localhost/private IPs (127.0.0.1, ::1), validates IP formats, and checks IDNA encoding

- **Port Security**: Restricts to valid port ranges (1-65535), prevents oversized ranges, blocks duplicates, and limits range sizes to 1000 ports

- **File Path Protection**: Prevents directory traversal attacks, validates path lengths, and resolves symbolic links safely



### Advanced Rate Limiting

- **Configurable Concurrency**: Set maximum concurrent connections (1-100) using `--rate-limit`

- **Sliding Window Control**: Adjust time window for rate limiting using `--rate-window` (default: 1.0 seconds)

- **Thread-Safe Implementation**: Uses advanced rate limiting class with proper locking mechanisms

- **DOS Protection**: Built-in protection against resource exhaustion through controlled concurrent operations



### SSL/TLS Security

- **Selective SSL Verification**: Enable/disable SSL certificate verification with `--verify-ssl` flag

- **Enhanced Error Reporting**: Differentiates between SSL verification failures and connection errors

- **Certificate Analysis**: Maintains full SSL/TLS certificate information with optional strict validation



### Security Hardening

- **Resource Limits**: Prevents DoS through input size restrictions (1000 char ports, 50 max port ranges)

- **Memory Protection**: Bounded buffer sizes and controlled memory usage

- **Injection Prevention**: Multiple layers of input validation to prevent command injection

- **DNS Rebinding Protection**: Hostname validation prevents DNS rebinding attacks



### Usage Examples with Security Options



```bash

# Enable SSL verification for production scanning

python testproxy.py -t example.com --verify-ssl



# Custom rate limiting for large-scale scanning

python testproxy.py -t example.com --rate-limit 20 --rate-window 2.0



# Secure scanning with combined security options

python testproxy.py -t example.com --verify-ssl --rate-limit 10 --rate-window 1.5 -v

```



### Best Practices for Security



1. **Always use `--verify-ssl`** in production environments

2. **Adjust rate limiting** based on network capacity and target tolerance

3. **Validate inputs** - the tool provides extensive built-in validation

4. **Use appropriate logging levels** - avoid verbose output in sensitive environments

5. **Keep dependencies updated** - security updates for cryptography libraries are crucial



## Dynamic Indicator Lists



The script uses external files for proxy and WAF indicators, allowing for easy updates:



- **Proxy Indicators File (`proxy_indicators.txt`)**: Contains a list of proxy indicator headers, one per line.

- **WAF Indicators File (`waf_indicators.txt`)**: Contains WAF indicator headers and their corresponding WAF names in the format `Header:WAF Name`.



Ensure these files are placed in the same directory as the script.



## Contribution



We welcome contributions! If you have ideas for improvements, new features, or bug fixes, please open an issue or submit a pull request. Make sure to follow the existing code style and add tests for new functionality.



## License



This project is licensed under the GPLv3 License. See the [LICENSE](LICENSE) file for details.



## Acknowledgments



- Thanks to the `requests`, `urllib3`, and `cryptography` libraries for their powerful features.

- Inspired by various cybersecurity tools and the need for comprehensive proxy and WAF detection.

- Thanks to [cryptoscuttlebutt](https://github.com/cryptoscuttlebutt) for their contributions.



## Disclaimer



This tool is for educational and informational purposes only. Ensure you have permission before scanning any networks or systems you do not own or have explicit permission to test.



## Recent Improvements



### v2.x.x Code Quality Enhancements

- **Comprehensive Type Hints**: Full Python type annotations for better IDE support and code maintainability

- **Performance Optimizations**: Parallelized banner grabbing using asyncio and thread pools for faster scanning

- **Enhanced Error Handling**: Improved exception management and null safety throughout the codebase

- **Security Hardening**: Better SSL certificate validation and input sanitization

- **Configuration Management**: Extracted hardcoded constants to centralized, configurable defaults



### v2.x.x Features

- **Improved Performance**: Concurrent banner scanning reduces analysis time by up to 60%

- **Enhanced Reliability**: Better timeout handling and connection management

- **Code Maintainability**: Modular design with clear type annotations makes future development easier



## Notes



- Always ensure you have proper authorization before scanning or analyzing targets to comply with legal and ethical guidelines.

- Be cautious when scanning multiple targets or using custom port ranges to avoid potential network issues or abuse reports.