Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/fabriziosalmi/patterns
Automated OWASP CRS and Bad Bot Detection for Nginx, Apache, Traefik and HaProxy
https://github.com/fabriziosalmi/patterns
apache bad-requests bot-detection caddy caddyserver crs firewall-configuration firewall-rules malicious-url-detection mod-security nginx owasp waf web-application-firewall
Last synced: 3 days ago
JSON representation
Automated OWASP CRS and Bad Bot Detection for Nginx, Apache, Traefik and HaProxy
- Host: GitHub
- URL: https://github.com/fabriziosalmi/patterns
- Owner: fabriziosalmi
- License: mit
- Created: 2024-12-21T00:00:15.000Z (19 days ago)
- Default Branch: main
- Last Pushed: 2025-01-05T00:29:20.000Z (4 days ago)
- Last Synced: 2025-01-05T01:22:05.875Z (4 days ago)
- Topics: apache, bad-requests, bot-detection, caddy, caddyserver, crs, firewall-configuration, firewall-rules, malicious-url-detection, mod-security, nginx, owasp, waf, web-application-firewall
- Language: Python
- Homepage:
- Size: 861 KB
- Stars: 196
- Watchers: 4
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Security: SECURITY.md
Awesome Lists containing this project
README
# 🔒 Patterns: OWASP CRS and Bad Bot Detection for Web Servers
Automate the scraping of **OWASP Core Rule Set (CRS)** patterns and convert them into **Apache, Nginx, Traefik, and HAProxy** WAF configurations.
Additionally, **Bad Bot/User-Agent detection** is integrated to block malicious web crawlers and scrapers.
> 🚀 **Protect your servers against SQL Injection (SQLi), XSS, RCE, LFI, and malicious bots – with automated daily updates.**
---
## 📌 Project Highlights
- **🛡️ OWASP CRS Protection** – Leverages OWASP Core Rule Set for web application firewall (WAF) defense.
- **🤖 Bad Bot Blocking** – Blocks known malicious bots using public bot lists.
- **⚙️ Multi-Web Server Support** – Generates WAF configs for **Apache, Nginx, Traefik, and HAProxy**.
- **🔄 Automatic Updates** – GitHub Actions fetch new rules **daily** and push updated configs.
- **📦 Pre-Generated Configurations** – Download ready-to-use WAF configurations from [GitHub Releases](https://github.com/fabriziosalmi/patterns/releases).
- **🧩 Scalable and Modular** – Easily extendable to support other web servers or load balancers.
---
## 🌐 Supported Web Servers
- **🔵 Nginx**
- **🟠 Apache (ModSecurity)**
- **🟣 Traefik**
- **🔴 HAProxy**
> [!NOTE]
> If you are using Caddy, check the [caddy-waf](https://github.com/fabriziosalmi/caddy-waf) project.
---
## 📂 Project Structure
```
patterns/
├── waf_patterns/ # 🔧 Generated WAF config files
│ ├── nginx/ # Nginx WAF configs
│ ├── apache/ # Apache WAF configs (ModSecurity)
│ ├── traefik/ # Traefik WAF configs
│ └── haproxy/ # HAProxy WAF configs
│── import_apache_waf.py
│── import_haproxy_waf.py
│── import_nginx_waf.py
│── import_traefik_waf.py
├── owasp.py # 🕵️ OWASP scraper (fetch CRS rules)
├── owasp2nginx.py # 🔄 Convert OWASP JSON to Nginx WAF configs
├── owasp2apache.py # 🔄 Convert OWASP JSON to Apache ModSecurity configs
├── owasp2haproxy.py # 🔄 Convert OWASP JSON to HAProxy WAF configs
├── badbots.py # 🤖 Generate WAF configs to block bad bots
├── requirements.txt # 📄 Required dependencies
└── .github/workflows/ # 🤖 GitHub Actions for automation
└── update_patterns.yml
```
---
## 🛠️ How It Works
### 🔹 1. Scraping OWASP Rules
- **`owasp.py`** scrapes the latest OWASP CRS patterns from GitHub.
- Extracts **SQLi, XSS, RCE, LFI** patterns from OWASP CRS `.conf` files.
### 🔹 2. Generating WAF Configs for Each Platform
- **`owasp2nginx.py`** – Generates **Nginx WAF** configurations.
- **`owasp2apache.py`** – Outputs **Apache ModSecurity** rules.
- **`owasp2traefik.py`** – Creates **Traefik WAF** rules.
- **`owasp2haproxy.py`** – Builds **HAProxy ACL** files.
### 🔹 3. Bad Bot/User-Agent Detection
- **`badbots.py`** fetches public bot lists and generates bot-blocking configs.
- Supports fallback lists to ensure reliable detection.
---
## ⚙️ Installation
### Option 1: Download Pre-Generated Configurations
You can download the latest pre-generated WAF configurations directly from the [GitHub Releases](https://github.com/fabriziosalmi/patterns/releases) page.
1. Go to the [Releases](https://github.com/fabriziosalmi/patterns/releases) section.
2. Download the zip file for your web server (e.g., `nginx_waf.zip`, `apache_waf.zip`).
3. Extract the files and follow the integration instructions below.
### Option 2: Build from Source
If you prefer to generate the configurations yourself:
**1. Clone the Repository:**
```bash
git clone https://github.com/fabriziosalmi/patterns.git
cd patterns
```
**2. Install Dependencies:**
```bash
pip install -r requirements.txt
```
**3. Run Manually (Optional):**
```bash
python owasp.py
python owasp2nginx.py
python owasp2apache.py
python owasp2haproxy.py
python owasp2traefik.py
python badbots.py
```
---
## 🚀 Usage (Web Server Integration)
### 🔹 1. Nginx WAF Integration
1. Download the `nginx_waf.zip` file from the [Releases](https://github.com/fabriziosalmi/patterns/releases) page.
2. Extract the files to your Nginx configuration directory.
3. Include the generated `.conf` files in your Nginx configuration:
```nginx
include /path/to/waf_patterns/nginx/*.conf;
```
### 🔹 2. Apache WAF Integration
1. Download the `apache_waf.zip` file from the [Releases](https://github.com/fabriziosalmi/patterns/releases) page.
2. Extract the files to your Apache configuration directory.
3. Include the generated `.conf` files in your Apache configuration:
```apache
Include /path/to/waf_patterns/apache/*.conf
```
### 🔹 3. Traefik WAF Integration
1. Download the `traefik_waf.zip` file from the [Releases](https://github.com/fabriziosalmi/patterns/releases) page.
2. Extract the files and use the `middleware.toml` file in your Traefik configuration.
### 🔹 4. HAProxy WAF Integration
1. Download the `haproxy_waf.zip` file from the [Releases](https://github.com/fabriziosalmi/patterns/releases) page.
2. Extract the files and include the `waf.acl` file in your HAProxy configuration.
---
## 🔧 Example Output (Bot Blocker – Nginx)
```nginx
map $http_user_agent $bad_bot {
"~*AhrefsBot" 1;
"~*SemrushBot" 1;
"~*MJ12bot" 1;
default 0;
}
if ($bad_bot) {
return 403;
}
```
---
## 🤖 Automation (GitHub Workflow)
- **🕛 Daily Updates** – GitHub Actions fetch the latest OWASP CRS rules every day.
- **🔄 Auto Deployment** – Pushes new `.conf` files directly to `waf_patterns/`.
- **📦 Release Automation** – Automatically creates a new release with pre-generated configurations.
- **🎯 Manual Trigger** – Updates can also be triggered manually.
---
## 🤝 Contributing
1. **Fork** the repository.
2. Create a **feature branch** (`feature/new-patterns`).
3. **Commit** and push changes.
4. Open a **Pull Request**.
---
## 📄 License
This project is licensed under the **MIT License**.
See the [LICENSE](LICENSE) file for details.
---
## 📞 Need Help?
- **Issues?** Open a ticket in the [Issues Tab](https://github.com/fabriziosalmi/patterns/issues).
---
## 🌐 Resources
- [OWASP CRS](https://github.com/coreruleset/coreruleset)
- [Apache ModSecurity](https://modsecurity.org/)
- [Nginx](https://nginx.org/)
- [Traefik](https://github.com/traefik/traefik)
- [HaProxy](https://www.haproxy.org/)