{"id":40650396,"url":"https://github.com/softwaremill/kcpilot","last_synced_at":"2026-01-21T08:24:06.100Z","repository":{"id":319262026,"uuid":"1041256085","full_name":"softwaremill/kcpilot","owner":"softwaremill","description":"CLI tool to help you investigate your Kafka clusters","archived":false,"fork":false,"pushed_at":"2025-11-27T08:15:26.000Z","size":537,"stargazers_count":8,"open_issues_count":2,"forks_count":2,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-11-30T02:11:59.869Z","etag":null,"topics":["cli","kafka","rust"],"latest_commit_sha":null,"homepage":"https://softwaremill.github.io/kcpilot/","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/softwaremill.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-08-20T08:07:03.000Z","updated_at":"2025-11-27T08:15:30.000Z","dependencies_parsed_at":"2025-10-23T11:28:24.905Z","dependency_job_id":"5ec05875-9982-4f32-a650-b282e100650e","html_url":"https://github.com/softwaremill/kcpilot","commit_stats":null,"previous_names":["softwaremill/kafkapilot","softwaremill/kcpilot"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/softwaremill/kcpilot","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/softwaremill%2Fkcpilot","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/softwaremill%2Fkcpilot/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/softwaremill%2Fkcpilot/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/softwaremill%2Fkcpilot/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/softwaremill","download_url":"https://codeload.github.com/softwaremill/kcpilot/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/softwaremill%2Fkcpilot/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28630164,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-21T04:47:28.174Z","status":"ssl_error","status_checked_at":"2026-01-21T04:47:22.943Z","response_time":86,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","kafka","rust"],"created_at":"2026-01-21T08:24:05.420Z","updated_at":"2026-01-21T08:24:06.090Z","avatar_url":"https://github.com/softwaremill.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# KCPilot\n\nA CLI-first Apache Kafka® cluster health diagnostics tool that automatically collects cluster signals, identifies issues, and provides actionable remediation guidance. KCPilot combines SSH-based data collection with AI-powered analysis using LLM integration.\n\n\u003e ⚠️ **Innovation Hub Project**: This project is part of SoftwareMill's Innovation Hub and is currently in MVP stage. While functional, it may contain bugs and has significant room for improvement. We welcome feedback and contributions!\n\u003e\n\u003e Treat this project as a base, invent your own tasks and share them with the world by opening a pull request.\n\n\n## Features\n\n- **SSH-based data collection** - Collects comprehensive cluster data through SSH bastion hosts\n- **AI-powered analysis** - Uses LLMs to analyze collected data and identify issues\n- **Configurable analysis tasks** - YAML-defined analysis prompts for customizable diagnostics\n- **Per-broker analysis** - Intelligent chunking to handle large clusters without token limits\n- **Zero configuration** - Works with standard SSH config aliases\n- **Multiple output formats** - Terminal, JSON, and Markdown reports\n- **Organized output** - Timestamped directories with structured data organization\n- **Summary reports** - Automatic generation of collection summaries\n\n## Installation\n\n```bash\n# Build from source\ncargo build --release\n\n# The binary will be available at target/release/kcpilot\n```\n\n## Environment Configuration\n\n### Required Environment Variables\n```bash\n# For AI-powered analysis (required for LLM features)\nexport OPENAI_API_KEY=your_openai_api_key_here\n\n# Optional LLM debugging\nexport LLM_DEBUG=true\n```\n\n## Quick Start\n\nYou can run KCPilot directly from source using `cargo run`:\n\n```bash\n# 1. Scan a cluster (data collection)\ncargo run --bin kcpilot -- scan --bastion kafka-poligon --broker kafka-broker-1.internal:9092 --output test-scan\n\n# 2. Analyze the collected data with AI\ncargo run --bin kcpilot -- analyze ./test-scan --report terminal\n\n# 3. Generate markdown report\ncargo run --bin kcpilot -- analyze ./test-scan --report markdown\n```\n\n## Commands Overview\n\n### Data Collection\n\n#### Local Scan (running directly on bastion host)\n```bash\n# Basic local scan - must specify at least one broker\ncargo run --bin kcpilot -- scan --broker kafka-broker-1.internal:9092\n\n# Local scan with custom output directory\ncargo run --bin kcpilot -- scan --broker kafka-broker-1.internal:9092 --output my-cluster-scan\n\n# With debug logging\nRUST_LOG=kcpilot=debug cargo run --bin kcpilot -- scan --broker kafka-broker-1.internal:9092\n```\n\n#### Remote Scan (via SSH bastion)\n```bash\n# Basic remote scan - requires both bastion and broker\ncargo run --bin kcpilot -- scan --bastion kafka-poligon --broker kafka-broker-1.internal:9092\n\n# Remote scan with custom output directory\ncargo run --bin kcpilot -- scan --bastion kafka-poligon --broker kafka-broker-1.internal:9092 --output test-scan\n\n# With debug logging\nRUST_LOG=kcpilot=debug cargo run --bin kcpilot -- scan --bastion kafka-poligon --broker kafka-broker-1.internal:9092\n```\n\n### Analysis\n```bash\n# Analyze collected data (terminal output)\ncargo run --bin kcpilot -- analyze ./test-scan\n\n# Generate JSON report\ncargo run --bin kcpilot -- analyze ./test-scan --report json\n\n# Generate markdown report\ncargo run --bin kcpilot -- analyze ./test-scan --report markdown\n\n# Enable LLM debug logging\ncargo run --bin kcpilot -- analyze ./test-scan --llmdbg\n\n# Custom LLM timeout (default: 300s)\ncargo run --bin kcpilot -- analyze ./test-scan --llm-timeout 600\n```\n\n### Analysis Task Management\n```bash\n# List all available analysis tasks\ncargo run --bin kcpilot -- task list\n\n# List tasks with detailed information\ncargo run --bin kcpilot -- task list --detailed\n\n# Test a specific analysis task\ncargo run --bin kcpilot -- task test recent_log_errors ./test-scan\n\n# Show details of a specific task\ncargo run --bin kcpilot -- task show recent_log_errors\n\n```\n\n### Utility Commands\n```bash\n# Test SSH connectivity to brokers\ncargo run --bin kcpilot -- test-ssh --bastion kafka-poligon --broker kafka-broker-1.internal:9092\n\n# Show configuration\ncargo run --bin kcpilot -- config\n\n# Show help\ncargo run --bin kcpilot -- --help\n\n# Show version and info\ncargo run --bin kcpilot -- info\n```\n\n## Using the Compiled Binary\n\nAfter building with `cargo build --release`, you can use the binary directly:\n\n```bash\n# Add to PATH (optional)\nexport PATH=\"$PATH:$(pwd)/target/release\"\n\n# Complete workflow: scan + analyze (remote)\nkcpilot scan --bastion kafka-poligon --broker kafka-broker-1.internal:9092\nkcpilot analyze ./my-cluster-scan --report terminal\n\n# Complete workflow: scan + analyze (local)\nkcpilot scan --broker kafka-broker-1.internal:9092 --output my-local-scan\nkcpilot analyze ./my-local-scan --report terminal\n\n# Generate reports\nkcpilot analyze ./my-cluster-scan --report markdown \n```\n\n### Prerequisites\n\n#### For Local Scan (running on bastion)\n- Direct SSH access to Kafka brokers\n- `kafkactl` installed\n- Optional: `kafka_exporter` for Prometheus metrics\n\n#### For Remote Scan (running from your local machine)\n\n1. **SSH Configuration**: Your SSH bastion should be configured in `~/.ssh/config`:\n   ```\n   Host kafka-poligon\n       HostName your-bastion-host.example.com\n       User your-username\n       ForwardAgent yes\n       IdentityFile ~/.ssh/your-key\n   ```\n\n2. **SSH Agent**: Ensure your SSH key is loaded:\n   ```bash\n   ssh-add ~/.ssh/your-key\n   ```\n\n3. **Bastion Requirements**: The bastion host should have:\n   - `kafkactl` for cluster metadata\n   - SSH access to individual Kafka brokers\n   - Optional: `kafka_exporter` for Prometheus metrics\n\n## Output Structure\n\n```\nkafka-scan-TIMESTAMP/\n├── brokers/           # Per-broker data\n│   ├── broker_11/\n│   │   ├── configs/   # Configuration files\n│   │   ├── logs/      # Log files\n│   │   ├── metrics/   # JVM metrics\n│   │   ├── system/    # System information\n│   │   └── data/      # Data directory info\n│   └── ...\n├── cluster/           # Cluster-wide data\n│   └── kafkactl/      # Broker lists, topics, consumer groups\n├── metrics/           # Prometheus metrics\n├── system/            # Bastion system info\n├── COLLECTION_SUMMARY.md\n└── scan_metadata.json\n```\n\n## Data Collected\n\n### Cluster Level (from bastion)\n- Broker list and configurations\n- Topic list and details\n- Consumer groups\n- Prometheus metrics (if available)\n- Bastion system information\n\n### Per Broker\n- System information (CPU, memory, disk)\n- Java/JVM information and metrics\n- Configuration files (server.properties, log4j, etc.)\n- **Enhanced log collection** - Dynamically discovers log files from any Kafka deployment:\n  - Automatic process discovery and service analysis\n  - AI-powered log4j configuration parsing\n  - Both file-based logs and systemd journal logs\n  - Works with any log directory structure\n- Data directory information and sizes\n- Network connections and ports\n\n## Analysis Tasks System\n\nKCPilot uses a sophisticated analysis system powered by YAML-defined tasks that leverage AI for intelligent cluster diagnostics.\n\n### Built-in Analysis Tasks\n\n- **Recent Log Error Detection** - Scans broker logs for ERROR/FATAL messages and connectivity issues\n- **JVM Heap Configuration** - Validates heap size settings and memory allocation\n- **Thread Configuration** - Checks broker thread pool settings\n- **Authentication \u0026 Authorization** - Verifies security configuration\n- **High Availability Checks** - Ensures proper HA setup (broker count, ISR, etc.)\n- **Network Configuration** - Validates listener and encryption settings\n- **And many more...**\n\n### Creating Custom Tasks\n\nCreate custom analysis tasks by adding YAML files to the `analysis_tasks/` directory:\n\n```yaml\nid: my_custom_check\nname: My Custom Analysis\ndescription: Custom analysis for specific requirements\ncategory: performance\n\nprompt: |\n  Analyze the Kafka configuration and logs for custom requirements:\n  Configuration: {config}\n  Logs: {logs}\n  \n  Look for specific patterns and provide recommendations.\n\ninclude_data:\n  - config\n  - logs\n\n# For large clusters, enable per-broker analysis to avoid token limits\nper_broker_analysis: true\nmax_tokens_per_request: 80000\n\nseverity_keywords:\n  \"critical issue\": \"high\"\n  \"minor issue\": \"low\"\n\ndefault_severity: medium\nenabled: true\n```\n\n### Per-Broker Analysis\n\nFor large clusters that exceed LLM token limits, KCPilot automatically:\n- Processes each broker individually when `per_broker_analysis: true`\n- Filters data to only include relevant information (logs + configs for log analysis)\n- Combines findings from all brokers into a comprehensive report\n- Handles any cluster size without token limit errors\n\n## Commercial Support\n\nWe offer commercial support for Kafka and related technologies, as well as development services. Contact us to learn more about our offer!\n\n## Trademark Notice\n\nApache®, Apache Kafka®, and Kafka® are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. This project is not affiliated with, endorsed by, or sponsored by the Apache Software Foundation. For more information about Apache trademarks, please see the [Apache Trademark Policy](https://www.apache.org/foundation/marks/).\n\n## Copyright\n\nCopyright (C) 2025 SoftwareMill https://softwaremill.com.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsoftwaremill%2Fkcpilot","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsoftwaremill%2Fkcpilot","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsoftwaremill%2Fkcpilot/lists"}