Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/saqqdy/kafka-log-analyzer

Intelligent Kafka log analysis with real-time monitoring, anomaly detection, and source code linking
https://github.com/saqqdy/kafka-log-analyzer

claude-code claude-code-plugin kafka kafka-log-analyzer log-analyzer mcp

Last synced: 6 days ago
JSON representation

Intelligent Kafka log analysis with real-time monitoring, anomaly detection, and source code linking

Awesome Lists containing this project

README 

          
# πŸ” Kafka Log Analyzer



> AI-powered Kafka log analysis β€” intelligent diagnostics for Kafka clusters with Claude Code



[![npm version](https://img.shields.io/npm/v/kafka-log-analyzer.svg)](https://www.npmjs.com/package/kafka-log-analyzer)

[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)



[δΈ­ζ–‡ζ–‡ζ‘£](README_CN.md)



---



## 🎯 The Problem It Solves



| Scenario | Traditional Debugging | Kafka Log Analyzer |

|----------|----------------------|-------------------|

| Log Analysis | Manually grep thousands of lines | AI extracts events, detects anomalies, prioritizes issues |

| Consumer Lag | Check multiple dashboards | Single command shows lag across all consumer groups |

| Error Diagnosis | Correlate logs with source code manually | AI links errors to configuration and implementation |

| Incident Response | Copy-paste logs to Slack/Feishu | Auto-generate formatted reports ready to share |



**Core insight**: Kafka troubleshooting requires understanding log semantics, not just keyword matching.



---



## ✨ Core Features



### πŸ“Š Intelligent Log Analysis (v0.1.0)



Structured parsing with zero configuration:



- **Multi-format Support** β€” Parse both text and JSON Kafka logs

- **11 Event Types** β€” send_success, send_failure, consumer_lag, rebalance, commit_failure, buffer_exhausted, leader_change, offset_out_of_range, serialization_error, network_error, auth_error

- **Anomaly Detection** β€” 7 built-in detectors for error spikes, rebalance storms, lag spikes, leader instability, replica lag, serialization issues, network problems



### πŸ”Ž Enhanced Filtering & Reports (v0.2.0)



Parameterized control over analysis output:



- **`--focus` Filter** β€” Filter by component (`producer/consumer/broker`) or semantics (`lag/error`)

- **`--priority` Filter** β€” Show only anomalies at specified severity (P0-P3)

- **`--timeline` Analysis** β€” Time-windowed event distribution with component breakdown

- **`--report` Formats** β€” Unified `markdown/json/slack` output for CLI and MCP

- **`--debug` Mode** β€” Verbose logging for troubleshooting



### πŸ“‘ Prometheus & Data Source Integration (v0.3.0)



Real-time metrics from Prometheus/Kafka Exporter:



- **Prometheus Connector** β€” Query Kafka Exporter metrics via PromQL (`queryInstant`, `queryRange`)

- **Consumer Lag Monitoring** β€” Real-time `kafka_consumergroup_lag` from Prometheus with group/topic/partition filtering

- **Exporter Source Analysis** β€” `analyze_log --source exporter` fetches live metrics and detects anomalies from lag data

- **Loki Log Source** β€” `analyze_log --source loki` queries Kafka logs via LogQL and reuses the Python analysis pipeline

- **MCP Resources** β€” Subscribe to live metric streams (`kafka://metrics/{cluster}`, `kafka://lag/{cluster}`)

- **Graceful Degradation** β€” All data source tools return empty data + friendly warnings when Prometheus/Loki are unreachable

- **Config Validators** β€” `isPrometheusConfigured()` / `isLokiConfigured()` / `validateSourceConfig()` helpers



### πŸ”” Hook Automation & Push Integration (v0.4.0)



Automatic alert triggering and team notifications:



- **Grafana Alert Hook** β€” Receive Grafana webhook payloads, auto-trigger analysis

- **PagerDuty Alert Hook** β€” Parse PagerDuty Events API v2 alerts into normalized analysis triggers

- **Hook HTTP Server** β€” Lightweight server with `POST /hooks/grafana`, `POST /hooks/pagerduty`, `GET /hooks/health`

- **Hook Dedup Engine** β€” Fingerprint-based in-memory dedup with configurable TTL window and merge count

- **Feishu Webhook Push** β€” Send analysis results as interactive cards to Feishu groups

- **Slack Webhook Push** β€” Send results as Block Kit messages to Slack channels

- **JIRA Integration** β€” Auto-create JIRA issues from analysis with priority mapping

- **Push Dispatcher** β€” Parallel push to all configured targets (Feishu/Slack/JIRA)

- **CLI: hooks** β€” `npx kafka-log-analyze hooks [--port P]` to start the hook server

- **CLI: push-test** β€” `npx kafka-log-analyze push-test [--target feishu|slack|all]` to test push



### πŸ“œ Historical Trends & Analysis (v0.5.0)



Persistent storage and trend comparison:



- **SQLite Persistence** β€” Automatic database creation with WAL mode, storing all analysis records, anomaly details, lag snapshots, and baselines

- **Event-Driven Architecture** β€” Decoupled pipeline: analysis results trigger persistence and automatic baseline updates via `EventEmitter`

- **Historical Query** β€” `query_history` tool filters past analysis by time range, source, and cluster

- **Trend Comparison** β€” `compare_trend` tool compares metrics (error rate, lag, anomaly count) across configurable time windows (`1h`/`6h`/`1d`/`7d`) with `previous`/`last_week`/`last_month` comparison

- **Anomaly Multiplier Detection** β€” Automatically annotates anomalies when values are β‰₯ 2Γ— baseline (e.g. "ζ―”εŸΊηΊΏε€š 3.5 倍")

- **Baseline Management** β€” Auto baselines from 7-day rolling averages; manual overrides via `set_baseline` / `list_baselines` tools

- **Data Cleanup** β€” Configurable retention policy (default 30 days) with `cleanup_storage` tool and automatic daily cleanup



### πŸ”’ Security Hardening (v1.0.0)



Production-ready security and reliability:



- **Webhook URL Validation** β€” SSRF protection: enforces HTTPS and rejects private IP ranges for Feishu and Slack integrations

- **Path Traversal Prevention** β€” `sanitizePath()` validates file paths, blocks `../` and null byte injection

- **SQLite Parameter Safety** β€” Full audit confirms all queries use prepared statements (no SQL injection)

- **Secret Redaction in Logs** β€” `redact()` function automatically masks sensitive values (token, secret, password, key, auth, credential)

- **Test Coverage β‰₯85%** β€” MCP server integration, event listeners, CLI, dedup, get_lag, JIRA, security, benchmark tests

- **Stdin Piping** β€” Paste mode pipes content via `child.stdin` instead of temp file I/O for better performance



### 🌏 User Experience Optimization (v1.1.0)



Internationalization, visualization, and smart diagnostics:



- **i18n Support (EN/ZH)** β€” Full internationalization with `--locale` / `-l` flag or `LOCALE` env var (~180 keys each). All user-visible strings use `t()` function.

- **In-Memory Analysis Cache** β€” TTL-based cache with per-source expiry (paste/file: 24h, exporter: 5m, loki: 15m). Prevents redundant re-analysis of identical inputs.

- **ASCII Timeline Chart** β€” Stacked bar chart with severity-segmented bars (`β–ˆ` ERROR, `Β·` WARN, `β–‘` INFO) and auto-generated legend. Sparkline mini-charts in push notifications.

- **Shell Completion** β€” `npx kafka-log-analyze completion ` generates shell completion scripts with enum suggestions for flags.

- **MCP Schema Enhancement** β€” Dynamic `analyze_log` inputSchema with cluster/consumer_group/topic enums auto-populated from history.

- **Folded Markdown Reports** β€” New `--report folded-markdown` format with collapsible `/` blocks.

- **Interactive TUI Browser** β€” `--interactive` / `-i` flag for keyboard-driven section navigation (↑/↓, Enter/Space, e, c, q). Zero additional dependencies.

- **Diagnostic Templates** β€” 5 built-in templates (`lag-diagnosis`, `rebalance-storm`, `producer-errors`, `broker-health`, `full-audit`). `--template ` flag and `templates` CLI command.

- **Smart Template Recommendation** β€” Templates ranked by historical anomaly patterns via `recommendTemplates()`. New MCP tools `diagnose` and `list_templates`.



### 🎯 Priority Classification



Automatic severity grading (P0-P3):



| Priority | Severity | Examples |

|----------|----------|----------|

| 🟒 **P0 (Critical)** | Cluster down, data loss risk | Complete broker failure, unrecoverable corruption |

| 🟑 **P1 (High)** | Consumer lag > 10K, frequent rebalances | Sustained high lag, rebalance storms |

| 🟠 **P2 (Medium)** | Leader changes, transient errors | Leader election, temporary network issues |

| πŸ”΄ **P3 (Low)** | Warnings, informational events | Configuration warnings, debug logs |



### πŸ”„ Multiple Output Formats



- **Markdown** β€” Detailed report with sections and recommendations

- **JSON** β€” Structured data for programmatic processing

- **Slack** β€” Compact format optimized for team channels

- **Folded Markdown** β€” Collapsible sections with `/` for GitHub/Slack



### πŸ–₯️ Claude Code Integration



- **MCP Tools** β€” Call analysis from Claude Code or any MCP client

- **Slash Commands** β€” `/kafka-analyze` for quick analysis

- **Source Code Linking** β€” AI correlates errors with your Kafka configuration



---



## πŸš€ Getting Started



### Option 1: Claude Code Plugin (Recommended)



This project is a **Claude Code Plugin**. Install via marketplace for one-click setup.



#### Method A: Plugin Marketplace (Recommended)



```bash

# In Claude Code, run:

/plugin marketplace add saqqdy/kafka-log-analyzer

/plugin install kafka-log-analyzer

```



#### Method B: Local Install



```bash

# 1. Go to your project

cd your-project



# 2. Install npm package

pnpm add -D kafka-log-analyzer



# 3. Copy plugin files

mkdir -p .claude/skills

cp -r node_modules/kafka-log-analyzer/.claude/skills/kafka-log-analyzer .claude/skills/

```



#### Available Commands



Type these commands in Claude Code:



| Command | Description | Example |

|---------|-------------|---------|

| `/kafka-analyze` | Analyze Kafka logs | `/kafka-analyze --source file --path server.log` |



#### Output Example



```

/kafka-analyze --source file --path server.log



πŸ“Š Analyzing server.log...



πŸ“‹ Summary:

  Total Events: 847

  P0 (Critical): 3

  P1 (High): 12



⚠️ Anomalies:

  πŸ”΄ send_failure_spike (P0)

     β†’ Error rate 15% exceeds threshold (5%)

     β†’ Check broker availability and network connectivity



  🟠 consumer_lag (P1)

     β†’ Consumer group order-processor lag: 15,000 messages

     β†’ Consider scaling consumer instances



πŸ’‘ Recommendations:

  1. Check broker availability

  2. Review consumer group configuration

```



### Option 2: Programmatic Usage



```bash

pnpm add kafka-log-analyzer

```



```typescript

import { analyzeLog } from 'kafka-log-analyzer'



const result = await analyzeLog({

  source: 'file',

  path: '/var/log/kafka/server.log',

  focus: ['producer', 'error'],

  timeline: '1h'

})



console.log(`Total Events: ${result.summary.total}`)

console.log(`P0: ${result.summary.byPriority.P0}`)

```



### Option 3: CLI (Zero-Install)



```bash

# Analyze log file

npx kafka-log-analyze analyze /var/log/kafka/server.log



# Paste log content via stdin

cat error.log | npx kafka-log-analyze analyze-paste



# Fetch consumer lag from Prometheus

npx kafka-log-analyze lag --cluster production --group order-processor



# Analyze from Kafka Exporter via Prometheus

npx kafka-log-analyze analyze-exporter --cluster production



# Analyze from Loki log system

npx kafka-log-analyze analyze-loki --cluster production --limit 500



# Start hook server for receiving Grafana/PagerDuty alerts

npx kafka-log-analyze hooks --port 3100



# Test push integration

npx kafka-log-analyze push-test --target feishu



# Generate shell completion script

npx kafka-log-analyze completion bash > /etc/bash_completion.d/kafka-log-analyze



# List diagnostic templates

npx kafka-log-analyze templates



# Analyze with a diagnostic template

npx kafka-log-analyze analyze server.log --template lag-diagnosis



# Analyze with interactive TUI browser

npx kafka-log-analyze analyze server.log --interactive



# Use Chinese locale

npx kafka-log-analyze analyze server.log --locale zh

```



### Option 4: Clone and Run Examples



```bash

git clone https://github.com/saqqdy/kafka-log-analyzer.git

cd kafka-log-analyzer

pnpm install



# Run examples

node dist/cli.js --source file --path tests/fixtures/sample-kafka-log.txt

```



### πŸ“Š Basic Usage



#### 1. Analyze Logs from File



```bash

# Analyze a Kafka log file

node dist/commands/kafka-analyze.js --source file --path /var/log/kafka/server.log



# With timeline analysis

node dist/commands/kafka-analyze.js --source file --path server.log --timeline 1h



# Focus on specific areas

node dist/commands/kafka-analyze.js --source file --path server.log --focus producer,consumer

```



#### 2. Analyze Pasted Logs



```bash

# Paste logs via stdin

cat kafka-error.log | node dist/commands/kafka-analyze.js --source paste



# Or use heredoc

node dist/commands/kafka-analyze.js --source paste <