{"id":40943450,"url":"https://github.com/stratosphereips/idps-comparison-tool","last_synced_at":"2026-01-22T04:37:39.365Z","repository":{"id":223706199,"uuid":"662212329","full_name":"stratosphereips/IDPS-Comparison-Tool","owner":"stratosphereips","description":null,"archived":false,"fork":false,"pushed_at":"2025-01-09T21:37:05.000Z","size":49383,"stargazers_count":1,"open_issues_count":1,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-09-04T18:40:39.300Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/stratosphereips.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"docs/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-07-04T15:44:34.000Z","updated_at":"2025-01-09T21:36:48.000Z","dependencies_parsed_at":"2024-02-21T17:28:34.635Z","dependency_job_id":"224af434-f815-4dfd-9598-bd97e467d87f","html_url":"https://github.com/stratosphereips/IDPS-Comparison-Tool","commit_stats":null,"previous_names":["stratosphereips/idps-comparison-tool"],"tags_count":153,"template":false,"template_full_name":"stratosphereips/awesome-code-template","purl":"pkg:github/stratosphereips/IDPS-Comparison-Tool","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stratosphereips%2FIDPS-Comparison-Tool","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stratosphereips%2FIDPS-Comparison-Tool/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stratosphereips%2FIDPS-Comparison-Tool/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stratosphereips%2FIDPS-Comparison-Tool/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/stratosphereips","download_url":"https://codeload.github.com/stratosphereips/IDPS-Comparison-Tool/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stratosphereips%2FIDPS-Comparison-Tool/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28654897,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-22T01:17:37.254Z","status":"online","status_checked_at":"2026-01-22T02:00:07.137Z","response_time":144,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-22T04:37:38.638Z","updated_at":"2026-01-22T04:37:39.359Z","avatar_url":"https://github.com/stratosphereips.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Installation\n\npip3 install -r requirements.txt\n\n---\n\n# Usage \n\npython3 main.py -s \u003cslips_db_abs_path\u003e -e \u003ceve.json_abs_path\u003e -g \u003cground_truth_labeled_zeek_dir_abs_path\u003e\n\nfor testing, use this command:\n\n***Example of using labeled ground truth dir***\n\npython3 main.py -gtd $(pwd)/dataset/CTU-Malware-Capture-Botnet-4/ground_truth/ -s $(pwd)/dataset/CTU-Malware-Capture-Botnet-4/slips/flows.sqlite -e $(pwd)/dataset/CTU-Malware-Capture-Botnet-4/suricata/eve.json  \n\n***Example of using ground truth file***\n\npython3 main.py -gtf $(pwd)/dataset/CTU-Malware-Capture-Botnet-4/ground_truth/conn.log.labeled -s $(pwd)/dataset/CTU-Malware-Capture-Botnet-4/slips/flows.sqlite -e $(pwd)/dataset/CTU-Malware-Capture-Botnet-4/suricata/eve.json  \n\npython3 main.py -s $(pwd)/dataset/2023-02-20/slips/flows.sqlite -e $(pwd)/dataset/2023-02-20/suricata/eve.json -gtf $(pwd)/dataset/2023-02-20/zeek_labeled/conn.log.labeled\n\n\n# Comparison Tool Input\n\nThe tool needs the following 3 to run:\n1. Slips db\n2. suricata eve.json\n3. a labeled conn.log file\n\n\n#### Slips DB \n\nSlips stores the AID hash for each conn.log flow in the sqlite db\n\nThe SQL table with the AID and label in Slips is called 'flows' inside the ```flows.sqlite``` db\n\nThis tool reads the ```flows.sqlite``` db, extracts the labels and AIDs,\nand stores them in its' own db stored in ```output/\u003cdate-time\u003e/db.sqlite```\n\nit calculates the aid of each read flow on the fly using the (zeek's community_id + ts) combination by using the aid_hash \nlib from pypi\nhttps://pypi.org/project/aid-hash\n\n\n#### suricata eve.json\n\nThis tool reads Suricata's eve.json file \n\nif the field event_type is set to 'alert', this tool marks this flow as malicious by suricata.\n\nit calculates the aid of each read flow on the fly using the (community_id + ts) combination by using the aid_hash \nlib from pypi\nhttps://pypi.org/project/aid-hash\n\n\n---\n\n# Comparison tool output\n\nThe output directory of this tool can be specified using -o, for example:\n\n```\npython3 main.py -e suricata/eve.json -s slips/flows.sqlite -gtf zeek/conn.log.labeled -o some_dir\n```\n\nif -o is not given, it creates a new dir inside ```output/``` with the date and time of the run.\n\nThe output of this tool consists of:\n\n1. a sqlite db with labels per flow, and labels per time window, it also has the performance errors and, total flows count and the files read of each tool. \n\nThe sqlite db created by this tool is stored in a subdir in the output/ dir\nfor example\n```output/2023-07-10-14:04:16```\n\n2. the metrics printed in the CLI and to the output directory in results.txt at the end of the analysis\n\n3. a metadata file with the versions and files used\n\n\n---\n\n# How it works\n\n\u003cimg src=\"docs/images/workflow.png\"\u003e\n\n1. This tool consists of 3 parsers, the ground truth parser runs first, once it's done the slips and suricata parsers start in parallel\n2. The 3 parsers job is to store the label for each tool in the sqlite database\n3. labels are stored per flow and per timewindow\n4. the tool then retrieves the actual and predicted value of each of the given tools and passes them to the calculator for calculating the metrics\n\n\n## Parsers\n\nThe below diagram shows how we parse each tool and the ground truth given file. \n\n\u003cimg src=\"docs/images/how_parsers_work.jpg\"\u003e\n\n\nParsers result in the following tables\n\n![image](https://github.com/user-attachments/assets/fc5e0eb0-a19b-4f41-8901-278f19d307fe)\n\n## Comparison\n![image](https://github.com/user-attachments/assets/4b82f601-e8ca-486d-8711-acad350219d4)\n\n\n\n### Comparison Method 1: Comparison per timewindow\n\n##### How labels per timewindow are calculated\nTimewindow labels are detected in the following way:\n\nA timewindow is 1h interval, the given pcap is split into as many 1h intervals as needed and each interval (timewindow) has 1 label, either malicious or benign\n\n#### for slips\nthe slips database given to this tool using -s contains a table called alerts where slips stores the malcious timewindows with their label, start and end date.\n\n\n#### Applying the timewindow concept for the ground truth\nwe read 1h worth of \nflows, once we find one 'malicious' label, we consider their entire timewindow as malicious, if there is no malicious flows in there, we mark that timewindow as benign\n\n\n#### Applying the timewindow concept for suricata\n\nSame as the ground truth. we read 1h worth of flows, once we find one 'malicious' label, we consider their entire timewindow as malicious, if there is no malicious flows in there, we mark that timewindow as benign\n\nIf a timewindow was detected by one of the tools, and not detected by the ground truth, for example negative timewindows in slips, \nwe consider the gt label of it as \"benign\"\n\n\n### Comparison Method2: labels flow by flow\n\n#### for slips\n\nThe slips database given to this tool using -s contains a table called flows where each flow is stored with its label. \nThe flow is considered malicious by slips if it was part of an alert.\nSlips detects alerts based on a complex ensembling algorithm, check it Slips documentation for more about this.\n\n#### for suricata\nThe eve.json given to this tool using -e contains flows and event_type = 'alert'.\nEach alerts is marked as malicious and each flow is marked as benign\n\n#### for the ground truth\nGround truth flows are labeled using the netflow labeler. so each flow has a label either benign or malicious\n\n---\n\n# Limitations\n\n* the labels in ground truth zeek dir have to be 'Malicious' or 'Benign' only. if any other label is present this tool will completely discard the flow. \n* ground truth dirs can either be json or tab separated zeek dir or conn.log file\n\n* all paths given as parameters to this tool must be absolute paths.\n* if any flow doesn't have a label by suricata or slips, this tool considers the flow as benign \n\n* slips now labels conn.log flows only, just like zeek does when zeek's community_id is enabled as a plugin\n\n* all flows and timewindows read by a tool, that don't have a matching flow/timewindow in the ground truth file, \nare discarded. The number of discarded flows is written in the cli at the end of the analysis.\n\n* we only read event_type= \"flow\" or \"alert\" in suricata eve.json files\n\n* the flows read by suricata, slips and the gt don't have to be the same, meaning that, the final flows count don't have to match because each tool reads the pcap differently\n\n* timewindow numbers may be negative if a flow is found with a flow timestamp \u003c timestamp of the first flow seen\n* if a slips alert exists in parts of 2 timewindows\n* what we're doing here is marking bith timewindows as malicious \n```\n\n                1:30                   2:30\n                 │      slips alert     │\n                 ├──────────────────────┤\n 1:00                       2:00                         3:00\n ├───────────────────────────┼────────────────────────────┤\n │             tw 1                 tw 2                  │\n\n```\n\n\n---\n\n# FAQ\n\n- When given a PCAP with 24hs of flows, why are the sum of the ByTimewindow comparison not equal 24?\nBecause each IP seen in the traffic, will have a label in each of the timewindows. Check the labels_per_tw table in the generated db.sqlite\nand sort by IPs for verification.\n\n\n\n\n---\n\n\n# Used cmds\n\n* command for generating all zeek files in the dataset/\n\n``` zeek -C -r \u003cpcap\u003e  tcp_inactivity_timeout=60mins tcp_attempt_delay=1min```\n\n\n* command for labeling conn.log files\n\n``` python3 netflowlabeler.py -c labels.config -f /path/to/generated/conn.log ```\n\n* (optional) To label the rest of the Zeek files using an already labeled conn.log file (conn.log.labeled)\n\n```zeek-files-labeler.py -l conn.log.labeled -f folder-with-zeek-log-files```\n\nCheckout the scripts used for the comparison \n@ https://github.com/stratosphereips/Scripts-Used-for-Comparison\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstratosphereips%2Fidps-comparison-tool","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstratosphereips%2Fidps-comparison-tool","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstratosphereips%2Fidps-comparison-tool/lists"}