{"id":21344470,"url":"https://github.com/s00d/logutil","last_synced_at":"2026-02-08T23:34:37.328Z","repository":{"id":243696193,"uuid":"813175966","full_name":"s00d/logutil","owner":"s00d","description":null,"archived":false,"fork":false,"pushed_at":"2024-07-16T14:45:45.000Z","size":5945,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-28T10:41:11.792Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/s00d.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-06-10T15:59:39.000Z","updated_at":"2025-01-22T10:37:37.000Z","dependencies_parsed_at":"2024-06-10T18:42:14.419Z","dependency_job_id":"bdcf21c4-2d2b-49bf-a4f2-7180a5b0dd72","html_url":"https://github.com/s00d/logutil","commit_stats":null,"previous_names":["s00d/logutil"],"tags_count":20,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/s00d%2Flogutil","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/s00d%2Flogutil/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/s00d%2Flogutil/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/s00d%2Flogutil/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/s00d","download_url":"https://codeload.github.com/s00d/logutil/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242015872,"owners_count":20058106,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-22T01:19:11.184Z","updated_at":"2026-02-08T23:34:32.285Z","avatar_url":"https://github.com/s00d.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n![Logo](https://github.com/s00d/logutil/blob/main/assets/logo.png?raw=true)\n\n\u003c/div\u003e\n# LogUtil - Real-time Nginx Log Analyzer\n\n\u003cdiv align=\"center\"\u003e\n    \u003ca href=\"https://crates.io/crates/logutil\" target=\"_blank\"\u003e\n        \u003cimg src=\"https://img.shields.io/crates/v/logutil?style=for-the-badge\" alt=\"crates.io version\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://crates.io/crates/logutil\" target=\"_blank\"\u003e\n        \u003cimg src=\"https://img.shields.io/crates/d/logutil?style=for-the-badge\" alt=\"crates.io downloads\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://github.com/s00d/logutil/blob/master/LICENSE\" target=\"_blank\"\u003e\n        \u003cimg src=\"https://img.shields.io/crates/l/logutil?style=for-the-badge\" alt=\"crates.io license\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://github.com/s00d/logutil\" target=\"_blank\"\u003e\n        \u003cimg src=\"https://img.shields.io/github/stars/s00d/logutil?style=for-the-badge\" alt=\"GitHub stars\"\u003e\n    \u003c/a\u003e\n\u003c/div\u003e\n\n## Overview\n\n**LogUtil** is a powerful, real-time log analysis tool written in Rust that provides an interactive terminal interface for monitoring and analyzing Nginx access logs. It offers comprehensive log parsing, real-time monitoring, and detailed analytics with a beautiful TUI (Terminal User Interface).\n\n![LogUtil in action](https://github.com/s00d/logutil/blob/main/assets/img.gif?raw=true)\n\n## Key Features\n\n### 🔍 **Real-time Log Monitoring**\n- Live tail functionality that monitors log files as they grow\n- Automatic detection and processing of new log entries\n- Real-time updates without manual refresh\n\n### 📊 **Comprehensive Analytics**\n- **IP Address Analysis**: Track requests by IP addresses with detailed statistics\n- **URL Analysis**: Monitor most accessed URLs and their patterns\n- **Request Type Tracking**: Categorize requests by HTTP methods (GET, POST, etc.)\n- **Domain Analysis**: Track requests by domain names\n- **Time-based Analytics**: Analyze request patterns over time\n\n### 🎨 **Interactive TUI Interface**\n- **9 Main Tabs**: Overview, Requests, Detailed, Sparkline, Heatmap, Security, Performance, Errors, and Bots\n- **Overview Tab**: Shows top IPs and URLs with real-time statistics\n- **Requests Tab**: Searchable log entries with pagination\n- **Detailed Tab**: Drill-down view for specific IP addresses\n- **Sparkline Tab**: Real-time request timeline visualization\n- **Heatmap Tab**: Hourly request patterns across multiple days\n- **Security Tab**: Detection of suspicious IPs and attack patterns\n- **Performance Tab**: Response time analysis and slow request identification\n- **Errors Tab**: HTTP error code analysis and categorization\n- **Bots Tab**: Bot and crawler detection and classification\n\n### ⚡ **Performance Optimizations**\n- Efficient memory management with automatic cleanup\n- Optimized for large log files (handles 10,000+ entries)\n- Minimal CPU usage during real-time monitoring\n- Configurable cleanup policies\n\n### 🔧 **Flexible Configuration**\n- Customizable regex patterns for different log formats\n- Support for various date formats\n- Configurable top N entries display\n- Optional automatic cleanup of outdated entries\n\n### 📁 **Interactive File Selection Mode**\nWhen you run LogUtil without specifying a log file, it launches an interactive file selection mode:\n\n**File Selector Features:**\n- **📂 Directory Navigation**: Browse through folders with intuitive navigation\n- **📄 File Browser**: View all files with clear icons (📁 for folders, 📄 for files, ⬆️ for parent directory)\n- **🔍 Visual Selection**: Highlighted current selection with clear visual feedback\n- **⚡ Quick Access**: Navigate with arrow keys and select with Enter\n\n**Settings Configuration:**\nAfter selecting a log file, you'll see an interactive settings screen where you can configure:\n- **📊 Analysis Parameters**: Enable/disable specific analysis tabs (Security, Performance, Errors, Bots, Sparkline, Heatmap)\n- **🔧 Processing Options**: Set count limits, regex patterns, date formats\n- **⚙️ Real-time Settings**: Configure cleanup policies and monitoring options\n- **▶️ Start Analysis**: Launch the analysis with your custom configuration\n\n**Usage:**\n```bash\n# Launch interactive mode\nlogutil\n\n# Or specify a file directly\nlogutil /path/to/access.log\n```\n\n### 🎮 **Keyboard Shortcuts**\n\n**Navigation:**\n- **Tab** / **T**: Switch to next tab\n- **Shift+Tab** / **Shift+T**: Switch to previous tab\n- **↑/↓**: Navigate through lists and options\n- **←/→**: Switch between panels (in tabs with multiple panels)\n- **Enter**: Select item or toggle boolean values\n- **Esc**: Go back or cancel\n\n**File Selector:**\n- **↑/↓**: Navigate through files and folders\n- **Enter**: Select file or enter folder\n- **Esc**: Go back to parent directory or exit\n\n**Settings:**\n- **↑/↓**: Navigate through settings\n- **Enter**: Edit setting or toggle boolean values\n- **Esc**: Cancel editing or go back\n\n**TUI Controls:**\n- **Q** / **Ctrl+C**: Quit application\n- **Enter**: Copy selected item to clipboard (Overview tab)\n\n### 📋 **Command Line Examples**\n\n**Interactive mode with pre-configured parameters:**\n```bash\n# Launch file selector with all analysis tabs enabled\ncargo run \"\" --enable-security --enable-performance --enable-errors --enable-bots --enable-sparkline --enable-heatmap --count=1000\n\n# Launch with custom settings\ncargo run \"\" --enable-security --enable-performance --top=20 --count=500\n\n# Launch with specific analysis tabs only\ncargo run \"\" --enable-security --enable-errors --count=2000\n```\n\n**Direct file analysis with all features:**\n```bash\n# Analyze with all tabs enabled\nlogutil access.log --enable-security --enable-performance --enable-errors --enable-bots --enable-sparkline --enable-heatmap --count=1000\n\n# Security-focused analysis\nlogutil access.log --enable-security --enable-errors --top=50\n\n# Performance monitoring\nlogutil access.log --enable-performance --enable-sparkline --enable-heatmap\n```\n\n## Installation\n\n### Quick Install (Recommended)\n\nDownload the latest release for your platform:\n\n**Linux (x86_64):**\n```bash\ncurl -L -o /usr/local/bin/logutil https://github.com/s00d/logutil/releases/latest/download/logutil-x86_64-unknown-linux-gnu\nchmod +x /usr/local/bin/logutil\n```\n\n**Linux (ARM64):**\n```bash\ncurl -L -o /usr/local/bin/logutil https://github.com/s00d/logutil/releases/latest/download/logutil-aarch64-unknown-linux-gnu\nchmod +x /usr/local/bin/logutil\n```\n\n**macOS:**\n```bash\ncurl -L -o /usr/local/bin/logutil https://github.com/s00d/logutil/releases/latest/download/logutil-x86_64-apple-darwin\nchmod +x /usr/local/bin/logutil\n```\n\n### Build from Source\n\n1. **Install Rust:**\n```bash\ncurl --proto '=https' --tlsv1.2 -sSf https://sh.rust-lang.org | sh\n```\n\n2. **Clone and Build:**\n```bash\ngit clone https://github.com/s00d/logutil.git\ncd logutil\ncargo build --release\n```\n\n3. **Install:**\n```bash\nsudo cp target/release/logutil /usr/local/bin/\n```\n\n## Usage\n\n### Basic Usage\n\n**Monitor a log file in real-time:**\n```bash\nlogutil /var/log/nginx/access.log\n```\n\n**Analyze the entire log file:**\n```bash\nlogutil /var/log/nginx/access.log --count=-1\n```\n\n**Show only the last 1000 lines:**\n```bash\nlogutil /var/log/nginx/access.log --count=1000\n```\n\n### Advanced Usage\n\n**Custom regex pattern:**\n```bash\nlogutil /var/log/nginx/access.log --regex='^(\\S+) - - \\[(.*?)\\] \"(\\S+) (\\S+) HTTP/\\d+\\.\\d+\" (\\d+) (\\d+) \"([^\"]*)\" \"([^\"]*)\"$'\n```\n\n**Show top 20 entries:**\n```bash\nlogutil /var/log/nginx/access.log --top=20\n```\n\n**Disable automatic cleanup:**\n```bash\nlogutil /var/log/nginx/access.log --no-clear\n```\n\n**Custom date format:**\n```bash\nlogutil /var/log/nginx/access.log --date-format=\"%d/%b/%Y:%H:%M:%S %z\"\n```\n\n**Load regex from file:**\n```bash\nlogutil /var/log/nginx/access.log --regex=/path/to/regex.txt\n```\n\n### Tab Management\n\nBy default, only the core tabs (Overview, Requests, Detailed) are enabled. Additional tabs can be enabled using command-line flags:\n\n**Enable Security tab (detect suspicious activity, attacks, etc.):**\n```bash\nlogutil /var/log/nginx/access.log --enable-security\n```\n\n**Enable Performance tab (monitor response times, slow requests):**\n```bash\nlogutil /var/log/nginx/access.log --enable-performance\n```\n\n**Enable Errors tab (track error codes and failed requests):**\n```bash\nlogutil /var/log/nginx/access.log --enable-errors\n```\n\n**Enable Bots tab (detect bot traffic and crawlers):**\n```bash\nlogutil /var/log/nginx/access.log --enable-bots\n```\n\n**Enable Sparkline tab (real-time request rate visualization):**\n```bash\nlogutil /var/log/nginx/access.log --enable-sparkline\n```\n\n**Enable Heatmap tab (hourly traffic patterns visualization):**\n```bash\nlogutil /var/log/nginx/access.log --enable-heatmap\n```\n\n**Enable all tabs:**\n```bash\nlogutil /var/log/nginx/access.log --enable-security --enable-performance --enable-errors --enable-bots --enable-sparkline --enable-heatmap\n```\n\n### Console Output Mode\n\n**Show top URLs in console:**\n```bash\nlogutil /var/log/nginx/access.log --show-urls --top=10\n```\n\n**Show top IPs in console:**\n```bash\nlogutil /var/log/nginx/access.log --show-ips --top=10\n```\n\n## Interactive Interface\n\n### Navigation\n\n- **`Tab` or `t`**: Switch between tabs (Overview → Requests → Detailed → Sparkline → Heatmap)\n- **`↑/↓`**: Navigate through lists\n- **`←/→`**: Switch between panels or pages\n- **`Enter`**: Copy selected item to clipboard (in Overview tab)\n- **`q` or `Ctrl+C`**: Quit the application\n\n### Tabs Overview\n\n#### 1. **Overview Tab**\n- **Left Panel**: Top IP addresses with request counts and last update times\n- **Right Panel**: Top URLs with request types, domains, and statistics\n- **Bottom Panel**: Full URL display for selected URL\n- **Navigation**: Use arrow keys to switch between panels\n\n#### 2. **Requests Tab**\n- **Search Functionality**: Type to filter log entries\n- **Pagination**: Navigate through large result sets\n- **Real-time Updates**: New requests appear automatically\n\n#### 3. **Detailed Tab**\n- **IP List**: Select an IP address to view its details\n- **Request Details**: View all requests from the selected IP\n- **Drill-down Analysis**: Understand traffic patterns per IP\n\n#### 4. **Sparkline Tab**\n- **Real-time Timeline**: Visual representation of request patterns\n- **Statistics**: Min, max, average, and current request counts\n- **Time Range**: Shows the time span of the data\n\n#### 5. **Heatmap Tab**\n- **Hourly Patterns**: Color-coded request intensity by hour\n- **Multi-day View**: Track patterns across multiple days\n- **Legend**: Blue (low) → Green → Red (high) intensity\n\n#### 6. **Security Tab**\n- **Advanced Threat Detection**: SQL Injection, XSS, Path Traversal, Command Injection, Brute Force\n- **Log Analysis**: View highlighted suspicious patterns in log entries with visual indicators\n- **IP Blocking**: Block/unblock suspicious IP addresses with Enter key\n- **Threat Levels**: Visual indicators (🔴🟡🟢) for threat severity assessment\n- **Pattern Highlighting**: Suspicious patterns are highlighted with icons in log details\n- **Security Summary**: Comprehensive overview of all security threats and violations\n- **Log Detail View**: Press Enter to view detailed logs with highlighted suspicious patterns\n- **Real-time Monitoring**: Continuous monitoring and detection of security threats\n\n#### 7. **Performance Tab**\n- **Response Time Analysis**: Tracks average, min, and max response times\n- **Slow Request Identification**: Highlights requests taking longer than 1 second\n- **Requests Per Second**: Real-time RPS calculation and monitoring\n- **Performance Tracking**: Detailed tracking of slow requests with timestamps\n- **Performance Metrics**: Real-time performance statistics\n- **Resource Usage**: Total response size and throughput analysis\n\n#### 8. **Errors Tab**\n- **HTTP Error Analysis**: Categorizes and counts error codes (4xx, 5xx)\n- **Error Pattern Detection**: Identifies common error sources\n- **Error Distribution**: Shows which URLs and IPs generate most errors\n- **Error Trends**: Tracks error patterns over time\n\n#### 9. **Bots Tab**\n- **Bot Detection**: Identifies crawlers, scrapers, and automated traffic\n- **Bot Classification**: Categorizes different types of bots (Googlebot, Bingbot, etc.)\n- **Bot Activity Analysis**: Tracks bot behavior patterns\n- **Bot Traffic Statistics**: Quantifies bot vs human traffic\n\n## Configuration\n\n### Command Line Options\n\n| Option | Description | Default |\n|--------|-------------|---------|\n| `file` | Path to the log file | Required |\n| `--count` | Lines to read from end (0=tail, -1=all) | `0` |\n| `--regex` | Regex pattern or file path | Nginx default |\n| `--date-format` | Date parsing format | `%d/%b/%Y:%H:%M:%S %z` |\n| `--top` | Number of top entries to show | `10` |\n| `--no-clear` | Disable automatic cleanup | `false` |\n| `--show-urls` | Output top URLs to console | `false` |\n| `--show-ips` | Output top IPs to console | `false` |\n| `--log-to-file` | Enable logging to app.log | `false` |\n| `--enable-security` | Enable Security tab | `false` |\n| `--enable-performance` | Enable Performance tab | `false` |\n| `--enable-errors` | Enable Errors tab | `false` |\n| `--enable-bots` | Enable Bots tab | `false` |\n| `--enable-sparkline` | Enable Sparkline tab | `false` |\n| `--enable-heatmap` | Enable Heatmap tab | `false` |\n\n### Supported Log Formats\n\n#### 1. **Nginx Access Log (Default)**\n```regex\n^(\\S+) - \".+\" \\[(.*?)\\] \\d+\\.\\d+ \"(\\S+)\" \"(\\S+) (\\S+?)(?:\\?.*?)? \"\n```\n**Date Format:** `%d/%b/%Y:%H:%M:%S %z`\n\n#### 2. **Apache Access Log**\n```regex\n^(\\S+) \\S+ \\S+ \\[.*?\\] \"\\S+ (\\S+?)(?:\\?.*?)? \\S+\" \\d+ \\d+\n```\n**Date Format:** `%d/%b/%Y:%H:%M:%S %z`\n\n#### 3. **Custom Format Example**\n```regex\n^(\\S+) - - \\[(.*?)\\] \"(\\S+) (\\S+) HTTP/\\d+\\.\\d+\" (\\d+) (\\d+) \"([^\"]*)\" \"([^\"]*)\"$\n```\n**Date Format:** `%d/%b/%Y:%H:%M:%S %z`\n\n## Examples\n\n### Basic Monitoring\n```bash\n# Monitor nginx access logs in real-time\nlogutil /var/log/nginx/access.log\n\n# Analyze entire log file\nlogutil /var/log/nginx/access.log --count=-1 --top=20\n```\n\n### Custom Log Formats\n```bash\n# Apache access logs\nlogutil /var/log/apache2/access.log --regex='^(\\S+) \\S+ \\S+ \\[.*?\\] \"\\S+ (\\S+?)(?:\\?.*?)? \\S+\" \\d+ \\d+'\n\n# Custom application logs\nlogutil /var/log/app/access.log --regex='^(\\S+) \\[(.*?)\\] (\\S+) (\\S+)'\n```\n\n### Console Output\n```bash\n# Get top URLs for reporting\nlogutil /var/log/nginx/access.log --show-urls --top=10\n\n# Get top IPs for security analysis\nlogutil /var/log/nginx/access.log --show-ips --top=20\n```\n\n### Advanced Configuration\n```bash\n# Custom regex from file\nlogutil /var/log/nginx/access.log --regex=/etc/logutil/patterns.txt\n\n# Disable cleanup for long-term analysis\nlogutil /var/log/nginx/access.log --no-clear --count=-1\n\n# Custom date format\nlogutil /var/log/nginx/access.log --date-format=\"%Y-%m-%d %H:%M:%S\"\n```\n\n## Performance Considerations\n\n### Memory Management\n- **Automatic Cleanup**: Removes entries older than 20 minutes when over 10,000 entries\n- **Configurable**: Use `--no-clear` to disable automatic cleanup\n- **Efficient**: Minimal memory footprint even with large log files\n\n### Processing Speed\n- **Real-time**: Processes new lines as they appear\n- **Optimized**: Efficient regex matching and data structures\n- **Scalable**: Handles high-traffic logs without performance degradation\n\n### File Handling\n- **Smart Reading**: Only processes new lines when tailing\n- **Error Recovery**: Gracefully handles file rotation and truncation\n- **Progress Tracking**: Shows loading progress for large files\n\n## Troubleshooting\n\n### Common Issues\n\n**1. \"No match for line\" errors**\n- Check your regex pattern with `--regex` option\n- Verify log format matches the expected pattern\n- Use `--log-to-file` to debug parsing issues\n\n**2. High memory usage**\n- Enable automatic cleanup (default behavior)\n- Use `--count` to limit initial processing\n- Consider using `--no-clear` only for short-term analysis\n\n**3. Slow performance with large files**\n- Use `--count=1000` to limit initial processing\n- Ensure regex pattern is optimized\n- Check system resources (CPU, memory)\n\n### Debug Mode\n```bash\n# Enable debug logging\nlogutil /var/log/nginx/access.log --log-to-file\n\n# Check the generated app.log file for errors\ntail -f app.log\n```\n\n## Contributing\n\nWe welcome contributions! Please see our [Contributing Guidelines](CONTRIBUTING.md) for details.\n\n### Development Setup\n```bash\ngit clone https://github.com/s00d/logutil.git\ncd logutil\ncargo build\ncargo test\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fs00d%2Flogutil","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fs00d%2Flogutil","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fs00d%2Flogutil/lists"}