https://github.com/asadullah-nadeem/block-web-scraping-tools

Block Web Scraping Tools using htaccess
https://github.com/asadullah-nadeem/block-web-scraping-tools

htaccess

Last synced: 4 months ago
JSON representation

Block Web Scraping Tools using htaccess

• Host: GitHub
• URL: https://github.com/asadullah-nadeem/block-web-scraping-tools
• Owner: Asadullah-nadeem
• Created: 2024-12-19T16:38:46.000Z (over 1 year ago)
• Default Branch: main
• Last Pushed: 2025-01-15T16:44:20.000Z (over 1 year ago)
• Last Synced: 2025-02-14T11:20:09.611Z (over 1 year ago)
• Topics: htaccess
• Homepage:
• Size: 3.91 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

          
# Block Specific User Agents and Enhance Website Security in `.htaccess`

This `.htaccess` snippet is designed to block specific user agents commonly associated with web scraping, unauthorized access attempts, and direct IP access. It also includes protection for the `.htaccess` file itself to prevent unauthorized viewing or modification.

## Rules Description

1. **Enable Rewrite Engine**: Activates the mod_rewrite module.

2. **Block User Agents**:

   - Prevents access from tools and bots like:

     - HTTrack

     - wget

     - curl

     - libwww

     - python

     - WinHTTrack

     - Nmap Scripting Engine

     - Go-http-client

     - apachebench

     - Any User-Agent containing "crawler", "spider", or "scanner".

3. **Block Direct IP Access**:

   - Ensures the website is only accessible via the domain name (e.g., `yourwebsite.com`).

4. **Protect `.htaccess`**:

   - Prevents unauthorized access to the `.htaccess` file itself.

## Usage

1. **Copy and Paste the Snippet**  

   Add the following code to the `.htaccess` file located in the root directory of your website:

   ```apache

   # Enable Rewrite Engine

   RewriteEngine On

   # Block specific user agents

   RewriteCond %{HTTP_USER_AGENT} HTTrack [NC,OR]

   RewriteCond %{HTTP_USER_AGENT} wget [NC,OR]

   RewriteCond %{HTTP_USER_AGENT} curl [NC,OR]

   RewriteCond %{HTTP_USER_AGENT} libwww [NC,OR]

   RewriteCond %{HTTP_USER_AGENT} python [NC,OR]

   RewriteCond %{HTTP_USER_AGENT} WinHTTrack [NC,OR]

   RewriteCond %{HTTP_USER_AGENT} "Nmap Scripting Engine" [NC,OR]

   RewriteCond %{HTTP_USER_AGENT} Go-http-client [NC,OR]

   RewriteCond %{HTTP_USER_AGENT} apachebench [NC,OR]

   RewriteCond %{HTTP_USER_AGENT} ^.*(crawler|spider|scanner).* [NC]

   RewriteRule .* - [F,L]

   

   # Block direct IP access

   RewriteCond %{HTTP_HOST} !^yourwebsite\.com [NC]

   RewriteRule .* - [F,L]

   # Protect .htaccess file

   

     Order Allow,Deny

     Deny from all