{"id":28412581,"url":"https://github.com/virtualisp/ai-powered-threat-detection-system","last_synced_at":"2026-04-11T11:01:39.595Z","repository":{"id":293331439,"uuid":"983697392","full_name":"virtualISP/AI-Powered-Threat-Detection-System","owner":"virtualISP","description":"A real-time security monitoring system that combines the ELK Stack (Elasticsearch, Logstash, Kibana) with Ollama's AI models (like Phi-3-mini) to automatically detect and analyze security threats in logs. Ideal for identifying brute-force attacks, SQLi, XSS, and more—with actionable insights visualized in Kibana dashboards.","archived":false,"fork":false,"pushed_at":"2025-05-14T20:10:20.000Z","size":194,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-03T11:37:43.023Z","etag":null,"topics":["cybersecurity","devsecops","docker","docker-compose","elasticsearch","elk","elk-stack","filebeat","kibana","llama3","llm","log-analysis","log-processing","logstash","nginx","ollama","phi3","python","security-tools","threat-detection"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/virtualISP.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-05-14T19:19:58.000Z","updated_at":"2025-05-15T03:32:13.000Z","dependencies_parsed_at":"2025-05-14T20:45:05.115Z","dependency_job_id":null,"html_url":"https://github.com/virtualISP/AI-Powered-Threat-Detection-System","commit_stats":null,"previous_names":["virtualisp/ai-powered-threat-detection-system"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/virtualISP/AI-Powered-Threat-Detection-System","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/virtualISP%2FAI-Powered-Threat-Detection-System","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/virtualISP%2FAI-Powered-Threat-Detection-System/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/virtualISP%2FAI-Powered-Threat-Detection-System/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/virtualISP%2FAI-Powered-Threat-Detection-System/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/virtualISP","download_url":"https://codeload.github.com/virtualISP/AI-Powered-Threat-Detection-System/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/virtualISP%2FAI-Powered-Threat-Detection-System/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261685339,"owners_count":23194090,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cybersecurity","devsecops","docker","docker-compose","elasticsearch","elk","elk-stack","filebeat","kibana","llama3","llm","log-analysis","log-processing","logstash","nginx","ollama","phi3","python","security-tools","threat-detection"],"created_at":"2025-06-02T23:14:32.511Z","updated_at":"2026-04-11T11:01:34.554Z","avatar_url":"https://github.com/virtualISP.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# 🛡️ AI-Powered Threat Detection System\n\n![ELK Stack with AI](https://img.shields.io/badge/ELK-8.12.0-orange?logo=elasticsearch)\n![Docker](https://img.shields.io/badge/Docker-20.10.24-blue?logo=docker)\n![Ollama](https://img.shields.io/badge/Ollama-0.4.8-red?logo=ollama)\n![Python](https://img.shields.io/badge/Python-3.9-green?logo=python)\n![License](https://img.shields.io/badge/License-MIT-blue)\n\n**Next-gen security monitoring** that combines the power of **ELK Stack** with **AI analysis** using Ollama's model to detect threats in real-time. Like a cybersecurity guard dog 🐕 with a PhD in machine learning! \n\n## ✨ Key Features\n\n| Feature | Benefit |\n|---------|---------|\n| **🧠 AI-Powered Analysis** | Uses Phi-3-mini/TinyLLaMA models for accurate threat classification |\n| **⏱️ Real-time Processing** | Detects threats as they happen (not after the breach!) |\n| **📊 Threat Dashboard** | Beautiful Kibana visualizations with threat heatmaps |\n| **🔌 Multi-source Support** | Works with Nginx, Apache, system logs, and more |\n| **⚙️ Customizable Rules** | Teach it to recognize new threats in 5 minutes |\n\n## 🏗️ System Architecture\n\n```mermaid\ngraph LR\n    A[Raw Logs] --\u003e B[Filebeat]\n    B --\u003e C[Logstash]\n    C --\u003e D[(Raw Log Index)]\n    D --\u003e E[AI Analyzer]\n    E --\u003e F[(AI-Threats Index)]\n    D --\u003e G[Kibana Log Dashboard]\n    F --\u003e H[Kibana Threat Dashboard]\n```\n\n## 📜 Log Feed (Raw Logs)\n\n**📍 Location:** Elasticsearch indices matching pattern `logs-*`\n\n### ⚙️ Kibana Configuration\n\n1. Create index pattern: `logs-*`, Time field: `@timestamp`\n2. Create index pattern: `ai-threats`, Time field: `@timestamp`\n\n\n## 🤖 AI-Threat Feed (Analyzed Threats)\n\n**📍 Location:** Elasticsearch index `ai-threats`\n\n### 🧾 Sample Document Structure\n\n```json\n{\n  \"@timestamp\": \"2025-05-15T12:00:05Z\",\n  \"log\": \"POST /wp-login.php HTTP/1.1 401 532\",\n  \"source_ip\": \"192.168.1.100\",\n  \"threat\": \"brute_force\",\n  \"confidence\": 92,\n  \"evidence\": \"Multiple rapid POST requests to login endpoint\",\n  \"recommendation\": \"Implement rate limiting and CAPTCHA\",\n  \"analysis_timestamp\": \"2025-05-15T12:00:10Z\"\n}\n```\n\n## 🔍 Feed Comparison\n\n| 🧩 Feature           | 📜 Log Feed         | 🤖 AI-Threat Feed               |\n| -------------------- | ------------------- | ------------------------------- |\n| **Content**          | Raw log data 🗒️    | Analyzed threats 🚨             |\n| **Index Pattern**    | `logs-*` 📁         | `ai-threats` 🧠                 |\n| **Update Frequency** | Real-time ⚡         | Near-real-time (5–10s delay) ⏱️ |\n| **Primary Use**      | Troubleshooting 🛠️ | Threat detection 🛡️            |\n| **Data Volume**      | High 📈             | Lower (only threats) 📉         |\n\n## 🔗 Accessing the Feeds\n\n### 1. 🧭 Via Kibana\n\n* **Log Feed**: `Discover → logs-*` 🔍\n* **AI-Threat Feed**: `Discover → ai-threats` 🧠\n\n### 2. 💻 Via API\n\n```bash\n# 🗒️ Get raw logs\ncurl \"http://localhost:9200/logs-*/_search?q=tags:security\u0026pretty\"\n\n# 🚨 Get analyzed threats\ncurl \"http://localhost:9200/ai-threats/_search?q=threat:brute_force\u0026pretty\"\n```\n\n## 🖥️ Using the Dashboard\n\n1. **Access Kibana** at `http://localhost:5601`  \n   ![Kibana Dashboard](resources/Dashboard-screenshot.png)\n\n2. **Recommended Visualizations**:\n   - 🍕 Threat Type Pie Chart\n   - 📈 Confidence Level Histogram\n   - 🕰️ Events Timeline\n   - 🗺️ Threat Origin World Map\n\n## 🚀 Getting Started\n\n### Prerequisites\n\n- Docker 🐳 20.10+\n- Docker Compose 🎭\n- Python 🐍\n- Hardware: \n  - **Recommended**: 16GB RAM, 8 cores (for AI analysis)  \n  - Storage: 20GB SSD (for log retention)\n\n### Installation\n\n1. **Clone the repo**  \n   ```bash\n   git clone https://github.com/virtualISP/AI-Powered-Threat-Detection-System.git\n   cd AI-Powered-Threat-Detection-System\n   ```\n\n2. **Download AI model** (takes 2-5 minutes)  \n   ```bash\n   docker-compose up ollama \u0026\u0026 docker exec -it ollama ollama pull phi3:mini\n   ```\n\n3. **Launch the system**  \n   ```bash\n   docker-compose up --build -d\n   ```\n\n4. **Verify services**  \n   ```bash\n   docker-compose ps\n   ```\n\n## ⚙️ Configuration Guide\n\n### 🔧 Key Files\n\n| File | Purpose | Icon |\n|------|---------|------|\n| `docker-compose.yml` | Main service configuration | 🐋 |\n| `filebeat/filebeat.yml` | Log collection settings | 📡 |\n| `logstash/pipeline/*.conf` | Log processing pipelines | ⚙️ |\n| `analyzer.py` | AI analysis brain | 🧠 |\n\n\n## 🧪 Testing Your Setup\n\nTry these test logs (run in terminal):\n\n```bash\n# SQL Injection 🪓\ncurl -X POST \"http://localhost:9200/logs-*/_doc\" -H 'Content-Type: application/json' -d'\n{\n  \"@timestamp\": \"'$(date -u +\"%Y-%m-%dT%H:%M:%SZ\")'\",\n  \"message\": \"GET /products?id=1%20UNION%20SELECT%20*%20FROM%20users--\"\n}'\n\n# Brute Force 🔑\ncurl -X POST \"http://localhost:9200/logs-*/_doc\" -H 'Content-Type: application/json' -d'\n{\n  \"@timestamp\": \"'$(date -u +\"%Y-%m-%dT%H:%M:%SZ\")'\",\n  \"message\": \"10 failed SSH attempts for root from 45.33.21.10\"\n}'\n```\n\n## 🚨 Troubleshooting\n\n| Symptom | Fix |\n|---------|-----|\n| Ollama model not loading 🐘 | Increase memory in `docker-compose.yml` |\n| Kibana shows no data 🕵️ | Check index pattern matches `ai-threats` |\n| High CPU usage 🔥 | Reduce `POLL_INTERVAL` or switch to `tinyllama` |\n\n**Pro Tip**: Monitor system health with:\n```bash\nwatch -n 1 'docker stats --no-stream'\n```\n\n## 🤝 How to Contribute\n1. Fork the repository\n2. Create your feature branch (`git checkout -b cool-new-feature`)\n3. Commit changes (`git commit -m 'Add some feature'`)\n4. Push to branch (`git push origin cool-new-feature`)\n5. Open a Pull Request\n\n## 📜 License\nMIT © VirtualISP – **[⚖️ See LICENSE](resources/LICENSE)**\n\n---\n\n**Happy Threat Hunting!** 👾🔍  \n*\"Sleep better knowing your logs are being watched by AI\"*\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvirtualisp%2Fai-powered-threat-detection-system","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvirtualisp%2Fai-powered-threat-detection-system","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvirtualisp%2Fai-powered-threat-detection-system/lists"}