{"id":50221457,"url":"https://github.com/rahadbhuiya/cnsl","last_synced_at":"2026-05-26T12:05:39.528Z","repository":{"id":353551696,"uuid":"1219953513","full_name":"rahadbhuiya/cnsl","owner":"rahadbhuiya","description":"Intent-driven security correlation for Linux servers. Detects coordinated attacks across SSH, web, and database logs — catches what Fail2ban misses.","archived":false,"fork":false,"pushed_at":"2026-05-05T08:19:52.000Z","size":190,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-05T09:36:41.901Z","etag":null,"topics":["brute-force","cybersecurity","fail2ban-alternative","honeypot","ids","intrusion-detection","linux","python","security","self-hosted","siem","ssh"],"latest_commit_sha":null,"homepage":"https://github.com/rahadbhuiya/cnsl#quick-start","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rahadbhuiya.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-24T11:47:58.000Z","updated_at":"2026-05-05T08:19:56.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/rahadbhuiya/cnsl","commit_stats":null,"previous_names":["rahadbhuiya/cnsl"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/rahadbhuiya/cnsl","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rahadbhuiya%2Fcnsl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rahadbhuiya%2Fcnsl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rahadbhuiya%2Fcnsl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rahadbhuiya%2Fcnsl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rahadbhuiya","download_url":"https://codeload.github.com/rahadbhuiya/cnsl/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rahadbhuiya%2Fcnsl/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33519234,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T03:12:49.672Z","status":"ssl_error","status_checked_at":"2026-05-26T03:12:47.976Z","response_time":63,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["brute-force","cybersecurity","fail2ban-alternative","honeypot","ids","intrusion-detection","linux","python","security","self-hosted","siem","ssh"],"created_at":"2026-05-26T12:05:37.909Z","updated_at":"2026-05-26T12:05:39.522Z","avatar_url":"https://github.com/rahadbhuiya.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n# CNSL — Correlated Network Security Layer\n\n\u003cp\u003e\n  \u003ca href=\"https://github.com/rahadbhuiya/cnsl/actions\"\u003e\u003cimg src=\"https://github.com/rahadbhuiya/cnsl/actions/workflows/ci.yml/badge.svg\" alt=\"CI\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://www.python.org\"\u003e\u003cimg src=\"https://img.shields.io/badge/python-3.10%2B-blue\" alt=\"Python 3.10+\"\u003e\u003c/a\u003e\n  \u003ca href=\"LICENSE\"\u003e\u003cimg src=\"https://img.shields.io/badge/License-MIT-green.svg\" alt=\"MIT License\"\u003e\u003c/a\u003e\n  \u003cimg src=\"https://img.shields.io/badge/tests-48%20passing-brightgreen\" alt=\"48 Tests Passing\"\u003e\n  \u003cimg src=\"https://img.shields.io/badge/version-1.1.0-blue\" alt=\"Version 1.1.0\"\u003e\n  \u003cimg src=\"https://img.shields.io/badge/platform-Linux-lightgrey\" alt=\"Linux\"\u003e\n\u003c/p\u003e\n\n**A lightweight SIEM for Linux — correlation, ML, honeypot, and search.**\n\nCorrelates SSH, web, database, and firewall signals to detect attacks  \nthat no single log source can see alone — then blocks them automatically.\n\n[Quick Start](#quick-start) · [Features](#features) · [Dashboard](#dashboard) · [API](#rest-api) · [Docs](#how-to-run-step-by-step) · [Changelog](#changelog)\n\n\u003c/div\u003e\n\n---\n\n## Why CNSL?\n\nMost security tools watch **one log** and count failures. That is not enough.\n\nA real attacker does not just hammer SSH — they scan your web server, probe your database, then log in with stolen credentials. **CNSL sees the full picture.**\n\n```\nWeb scan      from 45.33.32.1  --+\nSSH brute     from 45.33.32.1  --+---\u003e  HIGH alert + auto-block\nDB auth fail  from 45.33.32.1  --+\n```\n\n---\n\n## CNSL vs Fail2ban vs SSHGuard\n\n| Feature | Fail2ban | SSHGuard | CNSL |\n|:---|:---:|:---:|:---:|\n| Multi-source log correlation | No | No | Yes |\n| Credential breach detection | No | No | Yes |\n| ML anomaly detection | No | No | Yes |\n| Honeypot (fake SSH shell) | No | No | Yes |\n| Live web dashboard | No | No | Yes |\n| GeoIP + threat intelligence | No | No | Yes |\n| File integrity monitoring | No | No | Yes |\n| Telegram / Discord / Slack | No | No | Yes |\n| Redis distributed blocklist | No | No | Yes |\n| Prometheus + Grafana | No | No | Yes |\n| SOC2 / PCI-DSS compliance reports | No | No | Yes |\n| Privilege escalation detection | No | No | Yes |\n| PDF export from dashboard | No | No | Yes |\n| Remote syslog ingestion (UDP/TCP) | No | No | Yes |\n| ECS / CEF event normalization | No | No | Yes |\n| Full-text search + KQL queries | No | No | Yes |\n| Elasticsearch / OpenSearch export | No | No | Yes |\n| Auto-unblock timer | Yes | Yes | Yes |\n| Language | Python | C | Python |\n\n---\n\n## What CNSL Detects\n\n| Threat | How |\n|:---|:---|\n| SSH brute-force | Threshold-based failure counting per IP |\n| Credential breach | SSH success after repeated failures (stolen password) |\n| Credential stuffing | Many different usernames tried from one IP |\n| Web scanner | Nikto, sqlmap, gobuster User-Agent and path detection |\n| Web exploit attempts | `/.env`, `/wp-admin`, `/phpmyadmin`, path traversal |\n| Database brute-force | MySQL auth failure spikes |\n| Honeypot port probe | Any connection to port 23 / 3389 / 6379 — instant block |\n| Privilege escalation | sudo/su failure after successful SSH login |\n| File tampering | `/etc/passwd`, `authorized_keys`, `sshd_config`, any watched directory |\n| Behavioral anomaly | Unusual login hour, new username, frequency spike (ML) |\n| Coordinated attack | Same IP across SSH + web + DB simultaneously |\n| Remote device events | Syslog from routers, switches, firewalls (RFC 3164 / RFC 5424) |\n\n---\n\n## Features\n\n| Category | Capability |\n|:---|:---|\n| Detection | SSH brute-force, credential stuffing, credential breach |\n| Detection | Web scanner + exploit paths (false positive resistant) |\n| Detection | Database brute-force (MySQL), firewall honeypot ports |\n| Detection | Privilege escalation (sudo/su after login) |\n| Correlation | 6 cross-source rules — web+SSH, multi-service, kill chain |\n| Response | iptables / ipset auto-block with configurable auto-unblock timer |\n| Response | Honeypot redirect — attacker lands on a fake Ubuntu shell (40+ commands) |\n| Response | Redis distributed blocklist — sync blocks across a server cluster |\n| Intelligence | GeoIP enrichment (MaxMind offline or ip-api.com fallback) |\n| Intelligence | AbuseIPDB threat score lookup |\n| Intelligence | Behavioral baseline + ML anomaly detection (IsolationForest) |\n| Ingestion | Remote syslog receiver — UDP/TCP 514 or 5514 (RFC 3164 / RFC 5424) |\n| Normalization | ECS-compatible event schema, CEF export for ArcSight/Splunk |\n| Search | KQL-like full-text search, time-range filters, aggregations |\n| Export | Elasticsearch/OpenSearch bulk push, NDJSON and CEF file export |\n| Monitoring | File Integrity Monitoring — watches files and directories recursively |\n| Monitoring | Passive asset inventory via network events |\n| Visibility | Live web dashboard with tabbed UI and real-time SSE feed |\n| Visibility | Prometheus metrics + Grafana dashboard template |\n| Reporting | PDF export from dashboard (no extra tools needed) |\n| Reporting | PDF / HTML compliance reports (SOC2, ISO27001, PCI-DSS) |\n| Access | JWT authentication + Role-Based Access Control (4 roles) |\n| Notifications | Telegram, Discord, Slack, custom webhook (plain text, no emoji) |\n| Persistence | SQLite incident history, FIM baseline, block records |\n| Ops | Dry-run safe by default, systemd ready, Docker ready |\n\n---\n\n## Quick Start\n\n```bash\n# 1. Clone\ngit clone https://github.com/rahadbhuiya/cnsl.git\ncd cnsl\n\n# 2. Install (use a virtualenv)\npython3 -m venv venv\nsource venv/bin/activate\npip install -e \".[full]\"\n\n# 3. Run in safe dry-run mode (no real blocks)\nsudo venv/bin/python -m cnsl --no-tcpdump\n\n# 4. Run with live dashboard\nsudo venv/bin/python -m cnsl --dashboard --no-tcpdump\n# Open http://127.0.0.1:8765\n# Default login: admin / cnsl-change-me\n\n# 5. Enable real blocking when ready\nsudo venv/bin/python -m cnsl --execute --dashboard\n```\n\n\u003e Always use the virtualenv Python (`venv/bin/python`) with `sudo`.\n\u003e Running `sudo python` uses the system Python which may not have all packages (e.g. scikit-learn).\n\n---\n\n## Installation\n\n```bash\npip install -e .            # core only\npip install -e \".[full]\"    # everything recommended\npip install -e \".[dev]\"     # + testing tools\n```\n\n| Extra | Packages | Required for |\n|:---|:---|:---|\n| `full` | aiohttp, aiosqlite, pyyaml, bcrypt, sklearn, numpy, reportlab | Dashboard, DB, auth, ML, PDF reports |\n| `auth` | bcrypt, PyJWT | Dashboard login |\n| `db` | aiosqlite | SQLite persistence |\n| `geoip` | geoip2 | MaxMind offline GeoIP |\n| `ml` | scikit-learn, numpy | ML anomaly detection |\n| `reports` | reportlab | PDF compliance reports |\n| `dev` | pytest + all above | Running tests |\n\n---\n\n## Usage\n\n```\nsudo venv/bin/python -m cnsl [options]\n\nCore:\n  --config FILE        Config file (.json or .yaml)\n  --authlog PATH       Auth log path  (default: /var/log/auth.log)\n  --iface IFACE        Network interface for tcpdump  (default: any)\n  --execute            Enable real blocking  (default: dry-run)\n  --backend BACKEND    Blocking backend: iptables or ipset  (default: iptables)\n\nFeatures:\n  --dashboard          Enable web dashboard on http://127.0.0.1:8765\n  --no-tcpdump         Auth.log only, lower CPU\n  --no-geoip           Disable GeoIP enrichment\n  --no-db              Disable SQLite persistence\n\nReports:\n  --report FORMAT      Generate report and exit  (html | pdf | json)\n  --report-days N      Report period in days  (default: 30)\n  --grafana-export     Export Grafana dashboard JSON and exit\n```\n\n### Auth log paths by OS\n\n| OS | Default path |\n|:---|:---|\n| Ubuntu / Debian | `/var/log/auth.log` |\n| CentOS / RHEL / Fedora | `/var/log/secure` |\n| OpenSUSE | `/var/log/messages` |\n\n---\n\n## How to Run (Step by Step)\n\n### Step 1 — Minimal run (dry-run, no config needed)\n\n```bash\nsudo venv/bin/python -m cnsl --no-tcpdump\n```\n\n### Step 2 — With dashboard\n\n```bash\nsudo venv/bin/python -m cnsl --dashboard --no-tcpdump\n```\n\nOpen `http://127.0.0.1:8765` — Login: `admin` / `cnsl-change-me`\n\n---\n\n### Step 3 — Create a config file\n\n```bash\nsudo mkdir -p /etc/cnsl\nsudo cp config/config.example.json /etc/cnsl/config.json\nsudo nano /etc/cnsl/config.json\n```\n\nMinimum required changes:\n\n```json\n{\n  \"authlog_path\": \"/var/log/auth.log\",\n\n  \"allowlist\": [\n    \"127.0.0.1\",\n    \"YOUR_OWN_IP_HERE\"\n  ],\n\n  \"actions\": {\n    \"dry_run\": false,\n    \"block_duration_sec\": 900\n  },\n\n  \"store\": {\n    \"db_path\": \"/var/lib/cnsl/cnsl_state.db\"\n  },\n\n  \"fim\": {\n    \"db_path\": \"/var/lib/cnsl/cnsl_fim.db\"\n  }\n}\n```\n\n\u003e Always add your own IP to `allowlist` before setting `dry_run: false`.\n\u003e Remove `::1` from `allowlist` if you want to detect attacks from localhost.\n\u003e Use absolute paths for `db_path` — relative paths cause baselines to reset on restart.\n\n---\n\n### Step 4 — Enable live blocking\n\n```bash\nsudo venv/bin/python -m cnsl \\\n  --config /etc/cnsl/config.json \\\n  --execute \\\n  --dashboard\n```\n\n---\n\n### Step 5 — Add more log sources (optional)\n\n```json\n\"log_sources\": {\n  \"nginx\":  \"/var/log/nginx/access.log\",\n  \"apache\": \"/var/log/apache2/access.log\",\n  \"mysql\":  \"/var/log/mysql/error.log\",\n  \"ufw\":    \"/var/log/ufw.log\",\n  \"syslog\": \"/var/log/syslog\"\n}\n```\n\n---\n\n### Step 6 — Enable File Integrity Monitoring (optional)\n\n```json\n\"fim\": {\n  \"enabled\": true,\n  \"db_path\": \"/var/lib/cnsl/cnsl_fim.db\",\n  \"watch_paths\": [\n    \"/etc/passwd\",\n    \"/etc/ssh/\",\n    \"/var/www/\"\n  ],\n  \"scan_interval_sec\": 60\n}\n```\n\nFIM watches both individual files and entire directories recursively. Any file created, modified, deleted, or permission-changed fires an alert.\n\nTest FIM:\n```bash\nsudo touch /etc/ssh/test_cnsl.txt\n# wait 60 seconds\ngrep \"fim_alert\" /var/log/cnsl.jsonl | tail -3\nsudo rm /etc/ssh/test_cnsl.txt\n```\n\n---\n\n### Step 7 — Enable ML anomaly detection (optional)\n\n```json\n\"ml\": {\n  \"enabled\": true,\n  \"min_samples\": 100,\n  \"retrain_interval_sec\": 3600,\n  \"contamination\": 0.05,\n  \"anomaly_score_threshold\": -0.1\n}\n```\n\nML uses IsolationForest from scikit-learn — no pre-trained model needed. CNSL trains on your own traffic automatically after collecting `min_samples` events.\n\nCheck training status: `http://127.0.0.1:8765/api/ml-status`\n\n---\n\n### Step 8 — Enable remote syslog ingestion (optional)\n\nReceive logs from routers, switches, firewalls, and other Linux servers:\n\n```json\n\"syslog_receiver\": {\n  \"enabled\":     true,\n  \"host\":        \"0.0.0.0\",\n  \"udp_port\":    5514,\n  \"tcp_port\":    5514,\n  \"udp_enabled\": true,\n  \"tcp_enabled\": true\n}\n```\n\n\u003e Use port 5514 (not 514) to avoid needing root. Configure remote devices to send to `CNSL_IP:5514`.\n\nConfigure remote devices:\n```bash\n# Linux rsyslog\necho \"*.* @CNSL_IP:5514\" \u003e\u003e /etc/rsyslog.conf\nsystemctl restart rsyslog\n\n# Cisco IOS\nlogging host CNSL_IP transport udp port 5514\n```\n\nTest:\n```bash\necho \"\u003c38\u003eJan  1 00:00:00 router sshd[1]: Failed password for root from 9.9.9.9 port 22 ssh2\" \\\n  | nc -u 127.0.0.1 5514\n```\n\n---\n\n### Step 9 — Enable Telegram alerts (optional)\n\n```json\n\"notifications\": {\n  \"min_severity\": \"MEDIUM\",\n  \"telegram\": {\n    \"enabled\":   true,\n    \"bot_token\": \"YOUR_BOT_TOKEN\",\n    \"chat_id\":   \"YOUR_CHAT_ID\"\n  }\n}\n```\n\nGet a bot token from `@BotFather` on Telegram. Get your chat ID from `@userinfobot`.\n\n---\n\n### Step 10 — Run as a systemd service\n\n```bash\nsudo nano /etc/systemd/system/cnsl.service\n```\n\n```ini\n[Unit]\nDescription=CNSL — Correlated Network Security Layer\nAfter=network.target redis.service\nWants=redis.service\n\n[Service]\nType=simple\nUser=root\nExecStart=/opt/cnsl/venv/bin/python -m cnsl \\\n  --config /etc/cnsl/config.json \\\n  --execute \\\n  --dashboard\nWorkingDirectory=/opt/cnsl\nRestart=always\nRestartSec=5\nStandardOutput=append:/var/log/cnsl/service.log\nStandardError=append:/var/log/cnsl/service.log\n\n[Install]\nWantedBy=multi-user.target\n```\n\n```bash\nsudo systemctl daemon-reload\nsudo systemctl enable --now cnsl\nsudo journalctl -u cnsl -f\n```\n\n---\n\n## Test Without a Real Server\n\n```bash\n# Run all scenarios\npython simulate.py\n\n# Run a specific scenario\npython simulate.py brute        # SSH brute-force\npython simulate.py breach       # credential breach (HIGH severity)\npython simulate.py stuffing     # credential stuffing\npython simulate.py web          # web scanner + exploit attempt\npython simulate.py db           # database brute-force\npython simulate.py priv         # privilege escalation (SSH to sudo)\npython simulate.py honeypot     # honeypot port probe\npython simulate.py correlation  # multi-source coordinated attack\npython simulate.py unblock      # auto-unblock + Prometheus verify\npython simulate.py allowlist    # allowlist protection test\npython simulate.py metrics      # metrics and DB stats\npython simulate.py notify       # notification pipeline test\n\n# Interactive mode\npython simulate.py live\n```\n\n---\n\n## Dashboard\n\nEnable with `--dashboard`. Access at `http://127.0.0.1:8765`\n\n| Tab | What it shows |\n|:---|:---|\n| Overview | Stat cards, timeline chart (last 24h), severity doughnut, top attackers |\n| Incidents | Full incident table with time, IP, location, severity, fail count, reasons |\n| Blocks | Active blocks with unblock button, manual block form |\n| Honeypot | Status, active redirects, session table (IP, duration, auth attempts, commands typed) |\n| FIM | Watched paths list, file integrity alerts |\n| ML | Enabled/trained status, training progress bar, samples collected, last trained time |\n| Live Feed | Every event streamed in real time via SSE |\n\nExport PDF button in the header generates a full security report from live data — uses browser print, no extra tools needed.\n\n\u003e Dashboard binds to `127.0.0.1` only. For remote access use an SSH tunnel:\n\u003e ```bash\n\u003e ssh -L 8765:127.0.0.1:8765 user@yourserver\n\u003e ```\n\n---\n\n## Dashboard Authentication\n\n```json\n\"auth\": {\n  \"enabled\": true,\n  \"secret_key\": \"REPLACE_WITH_RANDOM_SECRET\"\n}\n```\n\nGenerate a secure key:\n```bash\npython3 -c \"import secrets; print(secrets.token_hex(32))\"\n```\n\nDefault credentials: `admin` / `cnsl-change-me` — change before deploying to production.\n\n### Roles\n\n| Role | Permissions |\n|:---|:---|\n| `viewer` | Read stats, incidents, blocks, metrics |\n| `analyst` | viewer + manual block / unblock |\n| `auditor` | analyst + generate reports + view asset inventory |\n| `admin` | Full access |\n\n---\n\n## Honeypot Mode\n\nInstead of blocking attackers, redirect them to a fake SSH server:\n\n```json\n\"honeypot\": {\n  \"enabled\":                true,\n  \"mode\":                   \"redirect\",\n  \"honeypot_host\":          \"127.0.0.1\",\n  \"honeypot_port\":          2222,\n  \"fake_hostname\":          \"ubuntu-server\",\n  \"fake_version\":           \"Ubuntu 22.04.3 LTS\",\n  \"log_commands\":           true,\n  \"auto_redirect_severity\": \"HIGH\"\n}\n```\n\nThe built-in fake shell simulates a real Ubuntu system:\n\n| What the attacker can do | What actually happens |\n|:---|:---|\n| `ls`, `cd`, `pwd`, `cat` | Full fake filesystem — `/etc`, `/root`, `/var`, `/proc`, `/home` |\n| `touch`, `mkdir`, `rm`, `cp`, `mv` | Works in a session-local virtual filesystem |\n| `echo \"x\" \u003e file` | Writes to virtual filesystem (`\u003e\u003e` append also works) |\n| `cat /etc/passwd`, `/etc/shadow`, `/etc/sudoers` | Returns realistic fake content |\n| `ps`, `top`, `df`, `free`, `netstat` | Returns realistic fake system info |\n| `wget`, `curl` | Simulates DNS timeout after a delay |\n| `python3`, `perl` | Interactive prompt or silent run |\n| `sudo`, `passwd` | Password prompts — logs what the attacker types |\n| `systemctl status` | Returns fake service status |\n\n\u003e Use `honeypot_port` (not `ports`). Make sure port 2222 is free before starting.\n\n---\n\n## Event Normalization\n\nEvery CNSL event is automatically normalized to an ECS-compatible schema:\n\n```json\n{\n  \"@timestamp\": \"2026-05-04T07:33:34Z\",\n  \"event\": {\n    \"kind\": \"event\",\n    \"category\": [\"authentication\", \"network\"],\n    \"outcome\": \"failure\",\n    \"severity\": 40\n  },\n  \"source\": { \"ip\": \"1.2.3.4\" },\n  \"cnsl\": {\n    \"kind\": \"SSH_FAIL\",\n    \"threat_score\": 3,\n    \"reasons\": [\"brute_force: 5 fails in 60s\"]\n  }\n}\n```\n\nExport formats:\n```bash\n# Elasticsearch bulk NDJSON\ncurl http://127.0.0.1:8765/api/export/ecs -H \"Authorization: Bearer $TOKEN\" -o events.ndjson\n\n# CEF (ArcSight / Splunk)\ncurl http://127.0.0.1:8765/api/export/cef -H \"Authorization: Bearer $TOKEN\"\n```\n\nPush to Elasticsearch:\n```bash\ncurl -X POST http://localhost:9200/_bulk \\\n  -H \"Content-Type: application/x-ndjson\" \\\n  --data-binary @events.ndjson\n```\n\n---\n\n## Search and Query\n\nFull-text search over incidents using a KQL-like syntax:\n\n```bash\n# Search by severity\nGET /api/search?q=severity:HIGH\n\n# Search by IP\nGET /api/search?q=1.2.3.4\n\n# Search by country\nGET /api/search?q=country:China\n\n# Search by reason\nGET /api/search?q=reasons:brute_force\n\n# Time range\nGET /api/search?q=severity:HIGH\u0026since=1700000000\u0026until=1800000000\n\n# Aggregations — top IPs, countries, hourly buckets\nGET /api/aggregate\n```\n\n---\n\n## Notifications\n\n```json\n\"notifications\": {\n  \"min_severity\": \"MEDIUM\",\n  \"telegram\": { \"enabled\": true, \"bot_token\": \"...\", \"chat_id\": \"...\" },\n  \"discord\":  { \"enabled\": true, \"webhook_url\": \"...\" },\n  \"slack\":    { \"enabled\": true, \"webhook_url\": \"...\" }\n}\n```\n\nMessages use clean plain text — no emoji. ISP names and detection reasons with special characters are automatically escaped so Telegram formatting never breaks.\n\n---\n\n## Reports\n\nFrom the dashboard — click Export PDF in the header to generate a full security report from live data.\n\nFrom the CLI:\n\n```bash\npython -m cnsl --report html --report-days 30\npython -m cnsl --report pdf\npython -m cnsl --report json\npython -m cnsl --grafana-export\n```\n\nReports include: executive summary, top attackers, recent incidents, FIM alerts, honeypot sessions, ML status, and SOC2 / ISO27001 / PCI-DSS compliance mapping.\n\n---\n\n## Grafana\n\n```bash\npython -m cnsl --grafana-export\n```\n\nImport in Grafana: `Dashboards \u003e Import \u003e Upload cnsl_grafana_dashboard.json`\n\nAdd to `prometheus.yml`:\n\n```yaml\nscrape_configs:\n  - job_name: cnsl\n    static_configs:\n      - targets: ['localhost:8765']\n    metrics_path: /api/metrics\n    authorization:\n      credentials: YOUR_JWT_TOKEN_HERE\n```\n\n---\n\n## REST API\n\nAll endpoints available when `--dashboard` is active.\n\n```\nGET  /api/stats                  Engine summary\nGET  /api/incidents              Recent incidents (?limit=50, max 500)\nGET  /api/events                 Raw recent events\nGET  /api/blocks                 Currently active blocks\nGET  /api/top-attackers          Top attacker IPs with geo info\nGET  /api/timeline               Incident counts per hour (last 24h)\nGET  /api/search                 Full-text search (?q=severity:HIGH\u0026limit=50)\nGET  /api/aggregate              Aggregations: by_severity, top_ips, top_countries, hourly\nGET  /api/events/normalized      ECS-normalized incident documents\nGET  /api/export/ecs             Elasticsearch bulk NDJSON download\nGET  /api/export/cef             CEF text download (ArcSight/Splunk)\nGET  /api/ml-status              ML detector status and training progress\nGET  /api/honeypot               Honeypot status and recent sessions\nGET  /api/fim                    FIM alerts and watched paths\nGET  /api/system                 Uptime, SSH fails, events processed, blocks total\nGET  /api/assets                 Passive network asset inventory\nGET  /api/metrics                Prometheus metrics (auth required)\nGET  /api/debug                  Module wiring status\nGET  /api/search/es-status       Elasticsearch cluster health\n\nPOST /api/login                  {\"username\": \"...\", \"password\": \"...\"}\nPOST /api/logout\nPOST /api/block                  {\"ip\": \"1.2.3.4\"}  analyst+ only\nPOST /api/unblock                {\"ip\": \"1.2.3.4\"}  analyst+ only\nPOST /api/report                 {\"format\": \"html\", \"days\": 30}\nPOST /api/search/es-push         Push incidents to Elasticsearch\n```\n\n---\n\n## JSON Log Format\n\nEvery event is a newline-delimited JSON record in `cnsl.jsonl`:\n\n```json\n{\n  \"ts\": 1713260000.0,\n  \"time\": \"2024-04-16T10:00:00Z\",\n  \"type\": \"incident\",\n  \"payload\": {\n    \"src_ip\": \"1.2.3.4\",\n    \"severity\": \"HIGH\",\n    \"reasons\": [\"credential_breach: success after 6 fails (threshold=5)\"],\n    \"fail_count\": 6,\n    \"geo\": { \"country\": \"China\", \"city\": \"Beijing\" }\n  }\n}\n```\n\n```bash\n# Stream all events live\ntail -f /var/log/cnsl.jsonl | jq .\n\n# HIGH severity incidents only\ntail -f /var/log/cnsl.jsonl | jq 'select(.type==\"incident\" and .payload.severity==\"HIGH\")'\n\n# ML training events\ngrep \"ml_retrained\\|ml_error\" /var/log/cnsl.jsonl | tail -5\n\n# FIM alerts\ngrep \"fim_alert\" /var/log/cnsl.jsonl | tail -10\n```\n\nCompatible with: Grafana Loki, Elasticsearch, Splunk, Vector, Fluentd, Datadog\n\n---\n\n## Docker\n\n```bash\ndocker build -t cnsl .\n\ndocker run --rm \\\n  --cap-add NET_ADMIN \\\n  --cap-add NET_RAW \\\n  --network host \\\n  -v /var/log:/var/log:ro \\\n  -v /etc/cnsl:/etc/cnsl:ro \\\n  cnsl --config /etc/cnsl/config.json --execute --dashboard\n```\n\n---\n\n## Testing\n\n```bash\npip install -e \".[dev]\"\npytest tests/ -v --timeout=60\n# 48 passed\n```\n\n---\n\n## Project Structure\n\n```\ncnsl/\n├── cnsl/\n│   ├── __init__.py          package version (1.1.0)\n│   ├── __main__.py          python -m cnsl entrypoint\n│   │\n│   ├── models.py            Event, Detection dataclasses\n│   ├── config.py            config loading and all defaults\n│   ├── validator.py         startup config validation\n│   ├── logger.py            async JSON logger (text prefixes, no emoji)\n│   │\n│   ├── parsers.py           auth.log + tcpdump parsers (sshd + sshd-session)\n│   ├── log_sources.py       nginx, apache, mysql, ufw, syslog parsers\n│   ├── sources.py           async log file tailers\n│   ├── syslog_receiver.py   UDP/TCP syslog server (RFC 3164/5424)\n│   │\n│   ├── normalizer.py        ECS/CEF event normalization\n│   ├── search_engine.py     KQL search, aggregations, Elasticsearch push\n│   │\n│   ├── detector.py          stateful per-IP detection engine\n│   ├── correlator.py        cross-source correlation rules (6 rules)\n│   ├── ml_detector.py       ML anomaly detection (IsolationForest, auto-trains)\n│   ├── threat_intel.py      AbuseIPDB + behavioral baseline\n│   │\n│   ├── blocker.py           iptables / ipset blocking backend\n│   ├── honeypot.py          fake SSH server (40+ commands, virtual filesystem)\n│   ├── redis_sync.py        distributed blocklist via Redis pub/sub\n│   │\n│   ├── geoip.py             GeoIP (MaxMind offline + ip-api.com)\n│   ├── assets.py            passive asset inventory\n│   ├── fim.py               file integrity monitoring (files + directories)\n│   │\n│   ├── auth.py              JWT authentication\n│   ├── rbac.py              role-based access control (4 roles)\n│   ├── dashboard.py         web dashboard + REST API + SSE feed\n│   ├── metrics.py           Prometheus metrics\n│   ├── grafana.py           Grafana dashboard template generator\n│   ├── reporter.py          PDF / HTML compliance reports\n│   ├── notify.py            Telegram, Discord, Slack, webhook\n│   ├── store.py             SQLite persistence (aiosqlite)\n│   └── engine.py            main async loop + CLI argument parser\n│\n├── tests/\n│   └── test_cnsl.py         48 unit tests\n│\n├── config/\n│   └── config.example.json  annotated example config\n│\n├── .github/workflows/ci.yml\n├── simulate.py              local test simulator (12 scenarios)\n├── Dockerfile\n├── setup.py\n├── requirements.txt\n└── README.md\n```\n\n---\n\n## Roadmap\n\n- [ ] Alert Rule Engine — Sigma rule support, custom thresholds, enable/disable toggle\n- [ ] Case Management — incident tickets, assign to analyst, status tracking\n- [ ] Full UEBA — per-user behavior profiles, lateral movement detection\n- [ ] Country-based blocking (`block_countries: [\"CN\", \"RU\"]`)\n- [ ] Email notifications (SMTP)\n- [ ] 2FA for dashboard login\n- [ ] Community threat feed — opt-in shared blocklist\n- [ ] Kafka support for high-volume environments\n- [ ] Zeek log ingestion\n- [ ] Multi-tenant support\n- [ ] Agent system for multi-server log collection\n- [ ] WebSocket instead of SSE for bidirectional dashboard control\n\n---\n\n## Safety\n\n\u003e `--execute` flag modifies live firewall rules.\n\nBefore enabling real blocking:\n\n1. Add your management IP to `allowlist` in config\n2. Test in dry-run mode first (this is the default)\n3. Ensure you have console or out-of-band access to the server\n4. The authors are not responsible for accidental self-lockouts\n\n---\n\n## Contributing\n\n1. Fork and create a feature branch\n2. Add or update tests in `tests/test_cnsl.py`\n3. Run `pytest tests/ -v --timeout=60` — all 48 must pass\n4. Submit a pull request\n\nCode style: type hints on all public functions, docstrings on all public methods, no external dependencies in `cnsl/` core modules.\n\n---\n\n## Changelog\n\n### v1.1.0 — Remote ingestion, ECS normalization, search engine\n\n**New modules**\n\n- `syslog_receiver.py` — UDP/TCP syslog server (RFC 3164 and RFC 5424). Routers, switches, firewalls, and remote Linux servers can ship logs directly to CNSL without a file agent. Parsed events go through the same detection pipeline as local logs. Configure with `syslog_receiver.enabled: true` and `udp_port: 5514` (no root required for ports above 1024).\n\n- `normalizer.py` — ECS-compatible event normalization. Every CNSL event is automatically normalized to a consistent schema regardless of source (SSH, web, MySQL, UFW, syslog, remote syslog). Supports ECS JSON export for Elasticsearch and CEF format for ArcSight/Splunk. CEF `rt=` field uses the event's actual timestamp.\n\n- `search_engine.py` — KQL-like full-text search, time-range filtering, and aggregations over the SQLite incident store. Optional Elasticsearch/OpenSearch push. New endpoints: `/api/search`, `/api/aggregate`, `/api/events`, `/api/export/ecs`, `/api/export/cef`, `/api/search/es-push`, `/api/search/es-status`.\n\n**Bug fixes**\n\n- `parsers.py` — IPv6-mapped IPv4 addresses (`::ffff:1.2.3.4`) are now stripped to plain IPv4 (`1.2.3.4`). Zone IDs (`fe80::1%eth0`) and brackets (`[::1]`) are also cleaned. The word `invalid` is no longer mistakenly extracted as a username from \"Failed password for invalid user admin\" lines.\n\n- `log_sources.py` — Web log parser completely rewritten. A bare 404 on a normal path is no longer flagged as `WEB_SCAN` — only scanner user-agents or exploit paths trigger alerts. Googlebot, Bingbot, and other known legitimate crawlers are whitelisted. Duplicate `parse_web_access` function removed.\n\n- `logger.py` — File write errors (disk full, file removed) are now handled gracefully instead of crashing the engine. Parent directories are created automatically if missing.\n\n- `store.py` — Added `kind` column to the incidents table. Automatic migration for existing databases (`ALTER TABLE ADD COLUMN` on startup). `save_incident()` now infers the event kind from reasons — `credential_breach` incidents correctly store `SSH_SUCCESS`, not `SSH_FAIL`. Old databases can be fixed manually: `UPDATE incidents SET kind='SSH_SUCCESS' WHERE reasons LIKE '%credential_breach%'`.\n\n- `search_engine.py` — Hourly aggregation now defaults to the last 30 days instead of last 24 hours, so data is visible even when there are no recent incidents.\n\n- `dashboard.py` — `kind` field is now read from the database column instead of being hardcoded as `SSH_FAIL` in normalized event endpoints. Added `search_engine` and `es_pusher` parameters to `start_dashboard()`.\n\n**Tests**\n\n- 48 tests total (was 26 in v1.0.0).\n- Added `TestSshdSessionParser` — 5 tests covering modern OpenSSH sshd-session format and IPv6.\n- Added `TestTelegramEscape` — 6 tests covering Markdown v1 special character escaping.\n- Added `TestStoreLowCount` — 2 tests verifying LOW severity is counted correctly in SQL.\n- Added `TestFIMDirectoryScanning` — 3 tests verifying directories in `watch_paths` are scanned recursively.\n- Added `TestMLRetrain` — 2 tests verifying ML timer logic (no training without data).\n- Added `TestDashboardSignature` — 2 tests verifying `ml_detector` and `fim` parameters are present.\n- `TestConfig` updated — tests `DEFAULT_CONFIG` directly instead of `load_config(None)` which now auto-discovers `/etc/cnsl/config.json`.\n- `TestParseAuthEvent.test_ipv6_address` updated — `::ffff:1.2.3.4` correctly strips to `1.2.3.4`.\n- sklearn pre-imported at module level so ML tests do not timeout on first import.\n\n---\n\n### v1.0.4 — Honeypot overhaul, FIM fix, emoji removed\n\n- `honeypot.py` — Full shell simulation rewrite. 40+ commands, persistent virtual filesystem, dynamic prompt, realistic file contents.\n- `fim.py` — Directories in `watch_paths` now scanned recursively with `os.walk()`.\n- `notify.py` — All emoji removed, plain text messages, Telegram escaping fixed.\n- `logger.py` — Emoji prefixes replaced with aligned text labels.\n- `reporter.py` — HTML uses inline SVG, PDF uses plain text.\n\n---\n\n### v1.0.3 — Critical runtime fixes\n\n- `parsers.py` — `sshd-session[PID]` regex added for modern OpenSSH.\n- `config.py` — `/etc/cnsl/config.json` now auto-discovered on startup.\n- Default `allowlist` — `::1` removed, `fails_threshold` lowered to 5.\n\n---\n\n### v1.0.2 — Dashboard overhaul\n\n- Tabbed UI with 7 tabs, 8 stat cards, timeline chart, PDF export, SVG icons.\n- ML tab, Honeypot tab, FIM tab, manual block form, live feed improvements.\n- New endpoints: `/api/timeline`, `/api/ml-status`, `/api/honeypot`, `/api/fim`, `/api/system`, `/api/debug`.\n- `engine.py` — `ml_detector` and `fim` parameters added to `start_dashboard()`.\n- `store.py` — LOW severity now counted correctly.\n\n---\n\n### v1.0.1 — Bug fixes\n\n- `engine_loop()` — `NameError` crash on first event fixed.\n- `start_dashboard()` — missing parameters fixed.\n- RBAC enforced on block/unblock endpoints.\n- Prometheus gauge decrements on unblock.\n- Redis unblock propagation fixed.\n- Subprocess resource leaks fixed.\n\n---\n\n### v1.0.0 — Initial release\n\n---\n\n## License\n\nMIT — see [LICENSE](LICENSE).\n\n---\n\n\u003cdiv align=\"center\"\u003e\n\nMade with by \u003ca href=\"https://github.com/rahadbhuiya\"\u003eRahad Bhuiya\u003c/a\u003e\n\n\u003c/div\u003e","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frahadbhuiya%2Fcnsl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frahadbhuiya%2Fcnsl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frahadbhuiya%2Fcnsl/lists"}