{"id":21672844,"url":"https://github.com/dogoncouch/siemstress","last_synced_at":"2025-04-12T04:09:23.715Z","repository":{"id":82059580,"uuid":"90201144","full_name":"dogoncouch/siemstress","owner":"dogoncouch","description":"Very basic CLI SIEM (Security Information and Event Management system).","archived":false,"fork":false,"pushed_at":"2018-03-24T05:48:51.000Z","size":1628,"stargazers_count":39,"open_issues_count":0,"forks_count":7,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-12T04:09:14.455Z","etag":null,"topics":["cli","command-line","forensics","log","log-analysis","log-analytics","log-analyzer","log-collector","log-management","log-monitoring","parser","parsing","secops","security","security-analysis","security-awareness","security-tools","siem","syslog"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dogoncouch.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-05-03T23:13:17.000Z","updated_at":"2025-03-30T13:32:21.000Z","dependencies_parsed_at":null,"dependency_job_id":"0c2761cc-f62d-44f0-8dbb-c4a7bfc701ce","html_url":"https://github.com/dogoncouch/siemstress","commit_stats":null,"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dogoncouch%2Fsiemstress","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dogoncouch%2Fsiemstress/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dogoncouch%2Fsiemstress/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dogoncouch%2Fsiemstress/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dogoncouch","download_url":"https://codeload.github.com/dogoncouch/siemstress/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248514223,"owners_count":21116903,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","command-line","forensics","log","log-analysis","log-analytics","log-analyzer","log-collector","log-management","log-monitoring","parser","parsing","secops","security","security-analysis","security-awareness","security-tools","siem","syslog"],"created_at":"2024-11-25T13:33:14.299Z","updated_at":"2025-04-12T04:09:23.703Z","avatar_url":"https://github.com/dogoncouch.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# siemstress\nA very basic Security Information and Event Management system (SIEM)\n\n## Index\n\n- [Introduction](#introduction)\n  - [Description](#description)\n  - [Overview](#overview)\n    - [Design principles](#design-principles)\n    - [Database](#database)\n    - [JSON output](#json-output)\n- [Installing](#installing)\n  - [Database setup](#database-setup)\n  - [Config](#config)\n- [Tools](#tools)\n  - [siemparse](#siemparse)\n  - [siemquery](#siemquery)\n  - [siemtrigger](#siemtrigger)\n  - [siemmanage](#siemmanage)\n- [Copyright](#copyright)\n\n## Introduction\n\n### Description\nSiemstress is a lightweight but powerful security information and event management (SIEM) system. It uses a database and a suite of CLI tools for managing log events, and automating event analysis. It comes with four programs: siemparse, siemquery, siemtrigger, and siemmanage.\n\n### Overview\n\n#### Design principles\nSiemstress is designed to parse data, and organize it into prioritized, manageable streams of relevant information. The goal is a streamlined open source information management system that embodies unix design principles. It should be simple, modular, and useful beyond its original scope.\n\nParsing data from specific applications is left to user configurable parse helpers, and user interfaces beyond the command line are left to other programs. Core functions are contained in modules that are meant to be as useful to other programs as they are to ours.\n\n#### Database\nSiemstress uses a database (MariaDB) to store the following information:\n\n- Parsed events - Parsed events are syslog (or other) events from files that are being watched by siemparse. Each event represents one line in a log file. Note: `siemquery` does not show all fields.\n![Parsed event table screenshot](https://github.com/dogoncouch/siemstress/blob/master/doc/images/events.png)\n\n- Helpers - Helpers are used to help siemparse pull dynamically configurable attributes from events. Events created by helpers are stored in the `Extended` column of parsed events. This is useful for identifying IP addresses, user names, temperatures, file names, or other data that siemstress doesn't parse automatically.\n![Helper table screenshot](https://github.com/dogoncouch/siemstress/blob/master/doc/images/helpers.png)\n\n- Rules - Rules are conditions that are used by siemtrigger to evaluate tables of parsed events. So far there is only one type of rule: the limit rule. Limit rules have a set of criteria, a time interval, and an event limit. If there are more events in a time interval that meet these criteria than the set limit, a SIEM event is created using the message defined by the rule. Rules also have a severity attribute, which works like syslog severity; 0 is the most severe, and 7 is the least (the specific syslog severity levels are not used).\n![Rule table screenshot](https://github.com/dogoncouch/siemstress/blob/master/doc/images/rules.png)\n\n- SIEM events - SIEM events are created by rules that evaluate events. They are stored in separate tables that are meant to be monitored. Each SIEM event has a magnitude, which is calculated using rule severity, and the ratio of event count to the rule's event limit. SIEM events also contain a list of source event IDs.\n![SIEM event table screenshot](https://github.com/dogoncouch/siemstress/blob/master/doc/images/SIEMevents.png)\n\n#### JSON output\n`siemquery` terminal output does not show all attributes. To view all attributes, including the extended attributes parsed by helpers, use the `--json` option to store the output in a JSON file. This can also be useful for data visualization and manipulation using programs like Jupyter.\n![JSON file screenshot](https://github.com/dogoncouch/siemstress/blob/master/doc/images/events-json.png)\n\n## Installing\n\nSee the latest instructions on the [releases page](https://github.com/dogoncouch/siemstress/releases).\n\n### Database Setup\nsiemstress is developed and tested using MariaDB as an SQL server. Refer to instructions for installing MariaDB on your operating system or distribution. You will need to create a database, and a user with permissions on it.\n\n### Config\nThe default siemstress config file location is `/etc/siemstress/db.conf` (`config/db.conf` if working in the repository). This file contains information needed to connect to the database, and should be updated with the proper server location, credentials, and database instance name.\n\nThere is also a section config file in the same directory called `sections.conf` that contains information about tables. This file needs more documentation.\n\n## Tools\n\n### Siemparse\n`siemparse` is a CLI tool to parse log lines from a log file or standard input into a siemstress database.\n\n#### Options\n\n```\n\nusage: siemparse [-h] [--version] [-c CONFIG] [-s SECTION] [-z TZONE]\n                 [file]\n\npositional arguments:\n  file        set a file to follow\n\noptional arguments:\n  -h, --help  show this help message and exit\n  --version   show program's version number and exit\n  -c CONFIG   set the config file\n  -s SECTION  set the config section\n  -z TZONE    set the offset to UTC (e.g. '+0500')\n\n```\n\n#### Examples\n    siemparse /var/log/messages\n    siemparse -s auth /var/log/auth.log\n\n### Siemquery\n`siemquery` is a CLI tool to query a siemstress database.\n\n#### Options\n\n```\n\nusage: siemquery [-h] [--version] [-c CONFIG] [-s SECTION] [--verbose]\n                 [--silent] [--rule] [--json FILE] [--table TABLE]\n                 [--last LAST] [--range START-FINISH] [--id ID]\n                 [--shost HOST] [--sport PORT] [--dhost HOST]\n                 [--dport PORT] [--process PROCESS] [--pid PID]\n                 [--protocol PROTOCOL] [--grep PATTERN] [--rshost HOST]\n                 [--rsport PORT] [--rdhost HOST] [--rdport PORT]\n                 [--rprocess PROCESS] [--rpid PID] [--rprotocol PROTOCOL]\n                 [--rgrep PATTERN]\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --version             show program's version number and exit\n  -c CONFIG             set the config file\n  -s SECTION            set the config section\n  --verbose             print SQL statement used for query\n  --silent              silence table output to terminal\n  --rule                set rule query mode\n  --json FILE           set a JSON output file\n\nquery options:\n  --table TABLE         set a table to query\n  --last LAST           match a preceeding time range (5m, 24h, etc)\n  --range START-FINISH  match a date range (format: YYmmddHHMMSS)\n  --id ID               match an event ID\n  --shost HOST          match a source host\n  --sport PORT          match a source port\n  --dhost HOST          match a destination host\n  --dport PORT          match a destination port\n  --process PROCESS     match a source process\n  --pid PID             match a source Process ID\n  --protocol PROTOCOL   match a protocol\n  --grep PATTERN        match a pattern\n  --rshost HOST         filter out a source host\n  --rsport PORT         filter out a source port\n  --rdhost HOST         filter out a destination host\n  --rdport PORT         filter out a destination port\n  --rprocess PROCESS    filter out a source process\n  --rpid PID            filter out a source Process ID\n  --rprotocol PROTOCOL  filter out a protocol\n  --rgrep PATTERN       filter out a pattern\n\n```\n\n#### Examples\n    siemquery --last 6h --json events.json\n    siemquery --last 20m -s auth --process sshd --process systemd-logind --grep fail\n    siemquery --range 20170726020000-20170726050000 -s auth --grep fail\n\n#### Notes\nArguments under `query options` that are not time-related can be used more than once.\n\n### Siemtrigger\n`siemtrigger` is a CLI tool to trigger SIEM events based on siemstress database analysis.\n\n#### Options\n\n```\n\nusage: siemtrigger [-h] [--version] [-c CONFIG] [--table TABLE] [--oneshot]\n\noptional arguments:\n  -h, --help     show this help message and exit\n  --version      show program's version number and exit\n  -c CONFIG      set the config file\n  --table TABLE  set a rule table\n  --oneshot      check database once and exit\n\n```\n\n#### Examples\n\n    siemtrigger --table RULESAuth\n\n### Siemmanage\n`siemmanage` is a tool for clearing SIEM tables, and importing and exporting rules and helpers.\n\n#### Options\n\n```\n\nusage: siemmanage [-h] [--version] [-c CONFIG] [--clear] [--force]\n                  [--importrules FILE] [--exportrules FILE]\n                  [--importhelpers FILE] [--exporthelpers FILE]\n                  [--table TABLE]\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --version             show program's version number and exit\n  -c CONFIG             set the config file\n  --clear               delete the selected SQL table\n  --force               really delete the table\n  --importrules FILE    set a JSON file to import rules\n  --exportrules FILE    set a JSON file to export rules\n  --importhelpers FILE  set a JSON file to import helpers\n  --exporthelpers FILE  set a JSON file to export helpers\n  --table TABLE         set a table\n\n```\n\n#### Examples\n\n    siemmanage --importhelpers config/example_helpers.json\n    siemmanage --exportrules rules.json --table RULESAuth\n    siemmanage --clear --force --table Auth\n\n## Copyright\nMIT License\n\nCopyright (c) 2017 Dan Persons (dpersonsdev@gmail.com)\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdogoncouch%2Fsiemstress","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdogoncouch%2Fsiemstress","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdogoncouch%2Fsiemstress/lists"}