https://github.com/nevil-ing/apache_log_agent

A lightweight, cross-platform Python-based agent for collecting, parsing, and forwarding Apache web server logs via an API endpoint to security analytics platforms.
https://github.com/nevil-ing/apache_log_agent

apache apache-logs bash filebeat log-aggregation logs logstash logstash-outputs python

Last synced: 3 days ago
JSON representation

A lightweight, cross-platform Python-based agent for collecting, parsing, and forwarding Apache web server logs via an API endpoint to security analytics platforms.

• Host: GitHub
• URL: https://github.com/nevil-ing/apache_log_agent
• Owner: nevil-ing
• Created: 2025-07-23T19:33:12.000Z (11 months ago)
• Default Branch: main
• Last Pushed: 2025-09-29T10:52:34.000Z (9 months ago)
• Last Synced: 2025-09-29T12:32:15.856Z (9 months ago)
• Topics: apache, apache-logs, bash, filebeat, log-aggregation, logs, logstash, logstash-outputs, python
• Language: Python
• Homepage:
• Size: 15.6 KB
• Stars: 0
• Watchers: 0
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.MD

Awesome Lists containing this project

README

          
# Apache Log Agent

A lightweight, cross-platform **Python-based agent** for collecting, parsing, and forwarding **Apache web server logs** via an API endpoint to security analytics platforms.

This agent wraps and orchestrates the powerful **ELK Stack components (Filebeat and Logstash)** to provide enterprise-grade log processing capabilities with minimal overhead.

---

## Purpose

This agent serves as a complete **log collection and forwarding solution** specifically designed for **Apache HTTP Server** environments.

Rather than reinventing log parsing logic, it **leverages battle-tested tools** while providing a simplified deployment and management interface through Python.

---

##  Core Components

- **Filebeat**  

  Lightweight log shipper that tails Apache access and error logs in real-time

- **Logstash**  

  Powerful log parser and transformer that applies Grok patterns, GeoIP lookups, and custom field mappings

- **Python Wrapper**  

  Central management interface for installation, configuration, and service orchestration

---

##  Key Features

### Smart Log Processing

- Parses Apache Combined Log Format with precision

- Extracts client IP, user agent, response codes, and request details

- Performs automatic GeoIP enrichment for threat analysis

- Normalizes timestamps to UTC with millisecond precision

---

### Security-Oriented Payload Construction

- Builds structured **JSON payloads optimized for SIEM/XDR ingestion**

- Includes security metadata fields like `is_anomaly`, `threat_type`, and `anomaly_score`

- Creates nested data object containing raw log fields for detailed analysis

- Standardizes source identification and timestamp formatting

---

### Easy Deployment & Management

- **Single-command** installation and configuration

- Automatic service management (`start` / `stop` / `status`)

- Cross-distribution compatibility (**systemd-based Linux systems**)

- Self-contained package with all required configurations

---

### Flexible Output Options

- Forwards parsed logs to **remote APIs/SIEM solutions via HTTP**

- Console output for debugging and development

- Ready for integration with **Sentinel XDR**, **Elastic SIEM**, or custom backends

---

## Configuration Highlights

### Filebeat Input Configuration

- Monitors `/var/log/httpd/*.log` for Apache logs

- Automatically tags events with Apache module metadata

- Supports log rotation and multi-line log handling

---

### Logstash Processing Pipeline

- Grok parsing using `%{COMBINEDAPACHELOG}` pattern

- Date parsing and `@timestamp` normalization

- GeoIP enrichment for client IP addresses

- Ruby-based payload restructuring for security use cases

- HTTP output to configured backend APIs