{"id":30654874,"url":"https://github.com/ebwi11/agentsmith-hub","last_synced_at":"2026-04-02T00:29:46.082Z","repository":{"id":307411305,"uuid":"983536570","full_name":"EBWi11/AgentSmith-HUB","owner":"EBWi11","description":"Enterprise Security Data Pipeline Platform (SDPP) with Integrated Real-Time Threat Detection Engine","archived":false,"fork":false,"pushed_at":"2025-08-21T08:21:35.000Z","size":80097,"stargazers_count":49,"open_issues_count":0,"forks_count":12,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-08-21T10:31:12.854Z","etag":null,"topics":["cybersecurity","detection-engine","rules-engine","sdpp","security-data-pipeline-platform"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/EBWi11.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-05-14T14:25:32.000Z","updated_at":"2025-08-21T08:21:02.000Z","dependencies_parsed_at":"2025-08-07T11:22:14.555Z","dependency_job_id":null,"html_url":"https://github.com/EBWi11/AgentSmith-HUB","commit_stats":null,"previous_names":["ebwi11/agentsmith-hub"],"tags_count":10,"template":false,"template_full_name":null,"purl":"pkg:github/EBWi11/AgentSmith-HUB","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EBWi11%2FAgentSmith-HUB","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EBWi11%2FAgentSmith-HUB/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EBWi11%2FAgentSmith-HUB/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EBWi11%2FAgentSmith-HUB/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/EBWi11","download_url":"https://codeload.github.com/EBWi11/AgentSmith-HUB/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EBWi11%2FAgentSmith-HUB/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272959666,"owners_count":25022094,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-31T02:00:09.071Z","response_time":79,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cybersecurity","detection-engine","rules-engine","sdpp","security-data-pipeline-platform"],"created_at":"2025-08-31T09:08:54.748Z","updated_at":"2026-04-02T00:29:46.067Z","avatar_url":"https://github.com/EBWi11.png","language":"Go","readme":"# AgentSmith-HUB\n\n[![GitHub release](https://img.shields.io/github/v/release/EBWi11/AgentSmith-HUB)](https://github.com/EBWi11/AgentSmith-HUB/releases)\n[![License](https://img.shields.io/badge/license-Apache%202.0%20with%20Commons%20Clause-blue)](./LICENSE)\n\n\n**A high-performance security data pipeline with a real-time rules engine and deeply integrated LLM agents — built for modern SOC and detection engineering teams.**\n\nProcess, enrich, detect, and respond at scale — with simple XML-based rules, CEP, rich plugins, and AI-powered analysis wired directly into the stream.\n\n![Dashboard](docs/png/Dashboard.png)\n\n---\n\n## Why AgentSmith-HUB?\n\nIf you work in security operations, you probably deal with massive volumes of raw logs and alerts every day. You need to normalize, enrich, correlate, and route them — and ideally detect threats in real time, not in batch jobs. AgentSmith-HUB is built to handle all of this in a single, opinionated platform:\n\n- **High-signal detections, not dashboards** — Design real-time detections and data transformations with simple, readable XML rules instead of ad‑hoc scripts\n- **Blazing fast at scale** — 3.90M messages/sec on just 2 vCPUs ([benchmark](docs/performance-testing-report.md)); built to sit directly in front of your SIEM / lake\n- **All-in-one pipeline** — Input, normalization, enrichment, correlation, and output in one flow; no more glue scripts between Kafka, ES, ClickHouse, and “rule engines”\n- **First-class CEP** — Detect ordered event sequences, absence patterns, and multi-source correlations over time with `\u003csequence\u003e`, `\u003cthreshold\u003e`, `\u003citerator\u003e`, and `\u003cchecklist\u003e`\n- **LLM agents in the stream** — Drop LLM-powered agents into the same pipeline for alert triage, enrichment, rule authoring, and auto-whitelisting\n- **Comment-to-memory learning loop** — Convert reviewer comments from Agent Tools Logs into durable `memory_notes`, auto-commit updates, and continuously improve agent behavior\n- **Skills system** — Attach knowledge bases and operational tools to agents via Skills, with progressive disclosure so prompts stay small and fast\n- **Rich plugin ecosystem** — Threat intel (VirusTotal, ThreatBook, Shodan), GeoIP, encoding, regex, time/window helpers, LLM calls, and more\n- **Production features out of the box** — Cluster mode, health checks, daily stats, sample data, Push Changes / review workflow, and a modern Web UI for rule and project orchestration\n\n### Who is this for?\n\n- **SOC / CERT / CSIRT teams** that want an opinionated place to run detections, triage alerts, and reduce false positives without building their own engine from scratch.\n- **Detection engineers / threat hunters** who care about CEP, thresholds, and precise control over when an alert fires (and when it must not).\n- **Security platform / data teams** who already own Kafka / ES / ClickHouse and want a thin, fast, open platform to orchestrate security data flows and LLM-powered analysis.\n\n## How It Works\n\nAgentSmith-HUB uses a straightforward pipeline model:\n\n```\nINPUT (Kafka / SLS / ...) → RULESET / AGENT → RULESET / AGENT → OUTPUT (Kafka / ES / ClickHouse / SLS / ...)\n```\n\nRulesets and agents can be freely chained within a **Project**, giving you full control over data flow and allowing you to mix “hard” rules with “soft” LLM judgement in the same stream:\n\n![ExampleProject](docs/png/ExampleProject.png)\n\n### Core Components at a Glance\n\n- **INPUT**: Connects to streaming sources like **Kafka**, Aliyun **SLS**, and cloud-managed Kafka variants; supports Grok parsing and JSON so you normalize once and reuse everywhere.\n- **RULESET**: XML-based real-time rules engine with checks, checklists, thresholds (count / SUM / CLASSIFY), CEP sequences, iterators, and data append/modify/del — all executed strictly in the order you write them.\n- **AGENT**: LLM-powered node that runs in the same pipeline as rulesets; for each event it can call an LLM (with tools and skills) to score, enrich, or auto-generate rules/whitelists, then forward the enriched event downstream.\n- **OUTPUT**: Sends processed data to **Kafka**, **Elasticsearch** (v7/v8/v9), **ClickHouse**, or simple print, with batching, time-based flush, TLS/auth, and idempotent Kafka producers for safe delivery.\n- **SKILL**: Reusable capability module for agents — knowledge skills provide on‑demand reference content, builtin skills expose Go-implemented tools like `hub_ruleset_editor` for ruleset CRUD.\n- **PLUGIN**: Extensible function system powering checks, enrichment, and actions: GeoIP, URL parsing, encoding, time window helpers, threat intelligence lookups, single-shot LLM calls, and more — all composable directly in rules.\n\n### Web UI \u0026 API Highlights\n\n- **Visual rule and project editing**: Rich browser UI for editing rulesets with syntax help, validation, and GIF-level feedback; drag-style project orchestration to define `INPUT → RULESET / AGENT → OUTPUT` flows.\n- **One-click testing everywhere**: Built-in test runners for **Output**, **Ruleset**, **Plugin**, **Agent**, and **Project** components (including sample data capture), so you can validate changes before they hit real outputs.\n- **Operations, errors, and cluster view**: Dedicated views for error logs, operations history (project start/stop/restart, config changes, agent tool calls), and basic cluster status so you can see what is running where.\n- **Safe change management**: All edits go through temporary configs, diff \u0026 review, and then **Push Changes** to apply — the platform automatically figures out affected projects and restarts them safely.\n- **HTTP API for automation**: JSON APIs mirror the UI capabilities (component CRUD, project lifecycle, testing), so you can integrate AgentSmith-HUB into CI/CD, internal portals, or automation scripts.\n\n### Rules Engine in 60 Seconds\n\nAt the heart of AgentSmith-HUB is a streaming rules engine designed for security detections:\n\n- **Checks \u0026 checklists**: Match on strings, numbers, regex, and plugins; combine conditions with AND/OR/NOT using logical expressions.\n- **Thresholds \u0026 windows**: Detect frequency, sums, or distinct counts over sliding time windows (e.g. brute-force, spray, exfil).\n- **CEP sequences**: Express ordered multi-event patterns and absence (e.g. `login -\u003e !mfa`, `recon -\u003e exploit -\u003e exfil`) with `\u003csequence\u003e`.\n- **Data shaping**: Enrich, modify, or delete fields in place, and call plugins to pull in external context or compute derived fields.\n\nA minimal example that enriches with threat intel and then detects on the enriched field:\n\n```xml\n\u003crule id=\"enrich_and_detect\" name=\"Enrich with TI then alert\"\u003e\n    \u003cappend type=\"PLUGIN\" field=\"threat_info\"\u003ethreatbook(src_ip)\u003c/append\u003e\n    \u003ccheck type=\"EQU\" field=\"threat_info.severity\"\u003ehigh\u003c/check\u003e\n    \u003cappend field=\"alert_level\"\u003ecritical\u003c/append\u003e\n\u003c/rule\u003e\n```\n\nFor the full syntax (all operations, modes, and best practices), see the [Complete Guide](docs/agentsmith-hub-guide.md).\n\n### LLM Agents \u0026 Skills\n\nAgents are LLM-powered components that sit in the pipeline alongside rulesets. They process events independently, call an LLM with tool-use support, and forward enriched results downstream.\n\n```yaml\n# Agent: AI-powered alert triage\nmodel: gpt-4o-mini\nsystem_prompt: |\n  For each alert, add llm_confidence (0-1) and llm_analysis fields.\nskills:\n  - hub_ruleset_expert    # Knowledge skill: rules engine reference\ntools: all                # Expose all plugins as LLM tools\nmax_rounds: 3\ntimeout: 30s\n\n# Optional long-term memory (recommended as YAML sequence)\nmemory_notes:\n  - Keep output JSON compact and stable.\n  - Treat routine CI scanner traffic as lower priority unless other signals exist.\n```\n\n**Skills** provide modular capabilities to agents:\n- **Knowledge skills** — Reference docs loaded on-demand (progressive disclosure)\n- **Builtin skills** — Go-implemented tools (e.g., `hub_ruleset_editor` for reading/writing rulesets)\n\nQuick production tips:\n- Prefer `tools: []` by default and allowlist only needed plugin tools.\n- Use `tools: all` only for broad assistant agents (rule-authoring / deep triage).\n- In cluster mode, memory write/generate actions must go to the **leader** node.\n\nUse agents in your project like any other component:\n\n```yaml\ncontent: |\n  INPUT.kafka_alerts -\u003e AGENT.alert_reviewer\n  AGENT.alert_reviewer -\u003e OUTPUT.enriched_alerts\n```\n\nFor full agent details (fields like `reasoning_mode`, `reasoning_budget_tokens`, `memory_notes`, and memory workflow in UI/API), see the [Complete Guide](docs/agentsmith-hub-guide.md#14-agent-syntax-description).\n\n## Built-in Detection Rulesets\n\nAgentSmith-HUB ships with production-ready detection rulesets that you can deploy immediately — no rule-writing required. All rules are mapped to [MITRE ATT\u0026CK](https://attack.mitre.org/) for seamless integration with your security workflows.\n\n### Built-in K8s Ruleset Files\n\nAgentSmith-HUB includes Kubernetes security rulesets out of the box. You can use them directly without writing custom XML first:\n\n- `config/ruleset/k8s_security/k8s_audit_baseline.xml`\n- `config/ruleset/k8s_security/k8s_audit_intrusion.xml`\n\nRecommended onboarding flow:\n\n1. Import both built-in rulesets.\n2. Route Kubernetes audit logs to these rulesets in your Project.\n3. Verify detections in test mode with real sample events.\n4. Tune thresholds (if needed) for your cluster's normal behavior.\n\n### Sysmon Endpoint Security (Windows)\n\nTwo Sysmon rulesets are provided for medium/high-confidence endpoint detection use cases:\n\n- `config/ruleset/sysmon_security/sysmon_baseline.xml`\n- `config/ruleset/sysmon_security/sysmon_intrusion.xml`\n- `config/ruleset/sysmon_security/sysmon_exclude.xml` (strict allowlist template)\n\nRecommended onboarding flow for Sysmon:\n\n1. Ensure your input normalizes core Sysmon fields used by rulesets.\n2. Import `sysmon_baseline.xml` first and validate behavior in test mode.\n3. Import `sysmon_intrusion.xml` and tune based on your endpoint baseline.\n4. Add environment-specific allowlists with a separate EXCLUDE ruleset if needed.\n\nMore built-in rulesets for additional data sources are on the roadmap. Contributions are welcome!\n\n## Features at a Glance\n\n\u003ctable\u003e\n\u003ctr\u003e\n\u003ctd width=\"50%\"\u003e\n\n**Rule Editing**\n\n![RuleEdit](docs/GIF/RuleEdit.gif)\n\n\u003c/td\u003e\n\u003ctd width=\"50%\"\u003e\n\n**Rule Testing**\n\n![RuleTest](docs/GIF/RuleTest.gif)\n\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\n\n**Project Orchestration**\n\n![ProjectEdit](docs/GIF/ProjectEdit.gif)\n\n\u003c/td\u003e\n\u003ctd\u003e\n\n**Plugin Testing**\n\n![Plugintest](docs/GIF/Plugintest.gif)\n\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\n\n**Input Connection Check**\n\n![InputEditConnectCheck](docs/GIF/InputEditConnectCheck.gif)\n\n\u003c/td\u003e\n\u003ctd\u003e\n\n**Search**\n\n![Search](docs/GIF/Search.gif)\n\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\n\n**Error Logs \u0026 Operations History**\n\n![ErrlogOperations](docs/GIF/ErrlogOperations.gif)\n\n\u003c/td\u003e\n\u003ctd width=\"50%\"\u003e\n\n**Comment-to-memory learning loop**\n\n![Comment-to-memory](docs/png/Memory.png)\n\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n\n\n## Deployment\n\n1. Download and extract the release archive to `/opt/agentsmith-hub`\n2. Copy the config folder: `cp -r /opt/agentsmith-hub/config /opt/hub_config`\n3. Configure Redis in `/opt/hub_config/config.yaml`\n4. Start the service:\n   ```bash\n   # Leader mode (default)\n   ./start.sh\n\n   # Follower mode (uses the same Redis as leader)\n   ./start.sh --follower\n\n   # See all options\n   ./start.sh --help\n   ```\n5. Access token is generated at `/etc/hub/.token` on first run\n6. Install and configure Nginx:\n   ```bash\n   sudo cp /opt/agentsmith-hub/nginx/nginx.conf /etc/nginx/\n   sudo nginx -s reload\n   ```\n7. Open `http://your-host` in your browser (port 80)\n\n## Documentation\n\n- [Complete Guide](docs/agentsmith-hub-guide.md) | [Guide (Chinese)](docs/agentsmith-hub-guide-zh.md)\n- [Performance Testing Report](docs/performance-testing-report.md)\n\n## License\n\nAgentSmith-HUB is licensed under the [Apache License 2.0](./LICENSE) with the Commons Clause restriction.\n\nYou are free to use, modify, and deploy this software — the restriction only prevents selling the software itself as a commercial product or service. Internal enterprise use is fully permitted.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Febwi11%2Fagentsmith-hub","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Febwi11%2Fagentsmith-hub","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Febwi11%2Fagentsmith-hub/lists"}