{"id":13531064,"url":"https://github.com/FrankHassanabad/suricata-sample-data","last_synced_at":"2025-04-01T19:31:10.671Z","repository":{"id":151013497,"uuid":"163012712","full_name":"FrankHassanabad/suricata-sample-data","owner":"FrankHassanabad","description":"Repository of creating different example suricata data sets","archived":false,"fork":false,"pushed_at":"2019-01-02T07:25:51.000Z","size":1332,"stargazers_count":32,"open_issues_count":0,"forks_count":9,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-11-02T17:36:38.586Z","etag":null,"topics":["idp","ids","sample-data","secops","suricata"],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/FrankHassanabad.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2018-12-24T17:45:52.000Z","updated_at":"2024-10-27T14:43:56.000Z","dependencies_parsed_at":"2023-04-25T07:35:24.697Z","dependency_job_id":null,"html_url":"https://github.com/FrankHassanabad/suricata-sample-data","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FrankHassanabad%2Fsuricata-sample-data","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FrankHassanabad%2Fsuricata-sample-data/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FrankHassanabad%2Fsuricata-sample-data/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FrankHassanabad%2Fsuricata-sample-data/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/FrankHassanabad","download_url":"https://codeload.github.com/FrankHassanabad/suricata-sample-data/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246700241,"owners_count":20819844,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["idp","ids","sample-data","secops","suricata"],"created_at":"2024-08-01T07:00:59.525Z","updated_at":"2025-04-01T19:31:05.663Z","avatar_url":"https://github.com/FrankHassanabad.png","language":"Shell","funding_links":[],"categories":["Data Sets"],"sub_categories":[],"readme":"# suricata-sample-data\n\nThis is a project and repository of different outputs of Suricata run against different\n[pcap](https://en.wikipedia.org/wiki/Pcap) data sets. You can download the Suricata data\nsets from the releases page of this repository.\n\n[eve.json files](https://github.com/FrankHassanabad/suricata-sample-data/releases/download/v4.0.0/release.zip)\n\n# Select samples of alerts from the zip\n\nSamples README and starting folder of the files generated from the eve files  \n[samples](samples)\n\n## The mission of the Collegiate Cyber Defense Competition (CCDC) system 2018\n\n[samples/wrccdc-2018](samples/wrccdc-2018) are generated from http://www.wrccdc.org/\nmirrored [here](https://archive.wrccdc.org/pcaps/2018/)\n\nA sampling of 1 of each alert from its eve.json  \n[samples/wrccdc-2018/alerts-only.json](samples/wrccdc-2018/alerts-only.json)\n\nA list of id's, signature names, and hyperlinks from the rules references section  \n[samples/wrccdc-2018/references.md](samples/wrccdc-2018/references.md)\n\nA list of id's, sampling of 1 of each alert from its eve.json  \n[samples/wrccdc-2018/alerts-only.json](samples/wrccdc-2018/alerts-only.json)\n\nA unique list of sids (Suricata Id's)  \n[samples/wrccdc-2018/ids-list.txt](samples/wrccdc-2018/ids-list.txt)\n\nA list of signatures that map 1-1 with the ids-list  \n[samples/wrccdc-2018/signature-list.txt](samples/wrccdc-2018/signature-list.txt)\n\n![topology](img/wrccdc2018-topology.png?raw=true)\n\n## The mission of the Collegiate Cyber Defense Competition (CCDC) system 2017\n\n[samples/wrccdc-2017](samples/wrccdc-2017) are generated from http://www.wrccdc.org/\nmirrored [here](https://archive.wrccdc.org/pcaps/2017/)\n\nA sampling of 1 of each alert from its eve.json  \n[samples/wrccdc-2017/alerts-only.json](samples/wrccdc-2017/alerts-only.json)\n\nA list of id's, signature names, and hyperlinks from the rules references section  \n[samples/wrccdc-2017/references.md](samples/wrccdc-2017/references.md)\n\nA unique list of sids (Suricata Id's)  \n[samples/wrccdc-2017/ids-list.txt](samples/wrccdc-2017/ids-list.txt)\n\nA list of signatures that map 1-1 with the ids-list  \n[samples/wrccdc-2017/signature-list.txt](samples/wrccdc-2017/signature-list.txt)\n\n![topology](img/wrccdc2017-topology.png?raw=true)\n\n## Hands-on Network Forensics - Training PCAP dataset from FIRST 2015\n\n[samples/first-org-conf-2015](samples/first-org-conf-2015) are generated from\nthe pcaps [mirrored here](https://www.netresec.com/?page=PcapFiles) from\nthe [first.org conference](https://www.first.org/conference/2015/program#phands-on-network-forensics)\n\nA sampling of 1 of each alert from its eve.json  \n[samples/first-org-conf-2015/alerts-only.json](samples/first-org-conf-2015/alerts-only.json)\n\nA list of id's, signature names, and hyperlinks from the rules references section  \n[samples/first-org-conf-2015/references.md](samples/first-org-conf-2015/references.md)\n\nA unique list of sids (Suricata Id's)  \n[samples/first-org-conf-2015/ids-list.txt](samples/first-org-conf-2015/ids-list.txt)\n\nA list of signatures that map 1-1 with the ids-list  \n[samples/first-org-conf-2015/signature-list.txt](samples/first-org-conf-2015/signature-list.txt)\n\n![topology](img/pawned-se.png?raw=true)\n\n## Honey Pot with Modern Honeypot Network\n\n[samples/honeypot-2018](samples/honeypot-2018) was generated from\nusing modern honey pot on digital ocean  \nhttps://threatstream.github.io/mhn/\n\nA sampling of 1 of each alert from its eve.json  \n[samples/honeypot-2018/alerts-only.json](samples/honeypot-2018/alerts-only.json)\n\nA list of id's, signature names, and hyperlinks from the rules references section  \n[samples/honeypot-2018/references.md](samples/honeypot-2018/references.md)\n\nA unique list of sids (Suricata Id's)  \n[samples/honeypot-2018/ids-list.txt](samples/honeypot-2018/ids-list.txt)\n\nA list of signatures that map 1-1 with the ids-list  \n[samples/honeypot-2018/signature-list.txt](samples/honeypot-2018/signature-list.txt)\n\n![topology](img/suricata-sensor.png)\n\nOptionally you can read below on how to (re)create your own data sets from your own\n[pcap](https://en.wikipedia.org/wiki/Pcap) files and Suricata rules.\n\n# Background\n\n[Suricata](https://suricata-ids.org/) is an engine that is capable of real\ntime intrusion detection [IDS](https://en.wikipedia.org/wiki/Intrusion_detection_system),\ninline intrusion prevention [IPS](https://en.wikipedia.org/wiki/Intrusion_detection_system),\nnetwork security monitoring (NSM) and offline pcap processing.\n\nThis repository contains sample `eve.json` log files created by Suricata from\nexisting [pcap](https://en.wikipedia.org/wiki/Pcap) files as well as instructions\non how to create them yourself. This is useful if you want to see what _alerts only_\ndata sets look like or play with the rules yourself and re-create your\nown [eve](https://suricata.readthedocs.io/en/suricata-4.1.2/output/eve/eve-json-output.html)\nfiles for learning purposes to write your own `eve.json` real time simulator.\n\nThe pcaps I found interesting for rich data was the\n[The Western Regional Cyber Defense 2018 (wrccdc)](http://www.wrccdc.org/). wrccdc has a rich set of\ndifferent pcap files in their [archives](https://archive.wrccdc.org/) from various\ncompetitions. I used their [2018 pcap data set](https://archive.wrccdc.org/pcaps/2018/) to\ncreate my `even.json` files for personal use.\n\nwrccdc 2018's [topology](https://archive.wrccdc.org/images/2018/wrccdc2018-topology.pdf) from their\ncompetition is a close to real world scenario.\n\nI also used the [Hands-on Network Forensics - Training PCAP dataset 2015](https://www.netresec.com/?page=PcapFiles)\nfrom this [mirror](https://www.first.org/conference/2015/program#phands-on-network-forensics) and followed along with their [PDF guide](https://download.netresec.com/pcap/FIRST-2015/Hands-on_Network_forensics.pdf)\n\n# How to download all the PCAPS from the 2018 competition\n\nUse [wget](https://www.gnu.org/software/wget/)\n\n```sh\nwget -r -np -k https://archive.wrccdc.org/pcaps/2018/\n```\n\nThis will download over a long period of time all the files to the sub-folder\n\n```sh\narchive.wrccdc.org/pcaps/2018\n```\n\nunzip those using gunzip\n\n```sh\ncd archive.wrccdc.org/pcaps/2018\ngunzip *.gz\n```\n\n# How to make an alerts only configuration\n\nOpen your `suricata.yaml`\n\n```sh\nvim /usr/local/etc/suricata/suricata.yaml\n```\n\nAnd remove the sections of http, dns, tls, files, ssh, stats, and flow events. Also set your\nstats to `enabled: false`. See [conf/suricata.yaml](conf/suricata.yaml) for my example.\n\n# How to write a script to parse each file\n\nSee [scripts/ingest_pcap.sh](scripts/ingest_pcap.sh) for a simple for loop which will run suricata\nand append to your eve.json file for each pcap file in a particular folder.\n\n# SID allocations\n\nFor the signature lists see this page for the allocation of signature ids\nhttps://doc.emergingthreats.net/bin/view/Main/SidAllocation\n\n```\n1000000-1999999 Reserved for Local Use -- Put your custom rules in this range to avoid conflicts\n\nThe following are the reservations for SIDs in the 2000000 space allocated to emerging threats:\n\n2000000-2099999 Emerging Threats Open Rulesets\n\n2100000-2103999 Forked ET Versions of the Original Snort GPL Signatures Originally sids 3464 and prior, forked to be maintained and converted to Suricata\n\n2200000-2200999 Suricata Decoder Events\n\n2210000-2210999 Suricata Stream Events\n\n2220000-2299999 Suricata Reserved\n\n2800000-2899999 Emerging Threats Pro Full Coverage Ruleset -- ETProRules\n\nDynamicly Updated Rules\n\n2400000-2400999 SpamHaus DROP List — Updated Daily -- SpamHausDROPList\n\n2402000-2402299 Dshield Top Attackers Rules — Updated Daily -- DshieldTopAttackers\n\n2403300-2403499 CIArmy.com Top Attackers Rules — Updated Daily - See http://www.ciarmy.com#list -- CiArmy\n\n2404000-2405999 Shadowserver.org Bot C\u0026C List — Updated Daily -- BotCC\n\n2404000-2405999 Shadowserver.org Bot C\u0026C List Grouped by Port — Updated Daily -- BotCC\n\n2406000-2406999 Russian Business Network Known Nets --- OBSOLETED -- RussianBusinessNetwork\n\n2408000-2408499 Russian Business Network Known Malvertisers --- OBSOLETED -- RussianBusinessNetwork\n\n2520000-2521999 Tor Exit Nodes List Updated Daily -- TorRules\n\n2522000-2525999 Tor Relay Nodes List (NOT Exit nodes) Updated Daily -- TorRules\n```\n\n# Command line jq tips and tricks with a eve.json\n\nInstall [jq](https://stedolan.github.io/jq/) and go to a working directory that contains\nan eve file.\n\n```sh\ncd /usr/local/var/log/suricata\n\n# or from the releases zip you can use any of the eve.json\ncd ./release/wrcddc-2018\ncd ./release/first-org-conf-2015\n```\n\nTo get all signatures from a eve.json\n\n```sh\njq '.alert.signature' eve.json\n```\n\nThis will return a list\n\n```sh\n\"ET WEB_SERVER allow_url_include PHP config option in uri\"\n\"ET WEB_SERVER safe_mode PHP config option in uri\"\n\"ET WEB_SERVER suhosin.simulation PHP config option in uri\"\n\"ET WEB_SERVER disable_functions PHP config option in uri\"\n\"ET WEB_SERVER open_basedir PHP config option in uri\"\n```\n\nTo get a single sample signature id from a large `eve.json` using a sid (e.x. `2012647`)\n\n```sh\njq 'select(.alert.signature_id==2012647)' eve.json | jq -s '.[0]'\n```\n\nTo get a list of all uniq and sorted signature id's\n\n```sh\njq 'select(.alert.signature_id)|.alert.signature_id' eve.json | sort | uniq\n```\n\nIt will return sids sorted asc by number:\n\n```sh\n2001219\n2001595\n2001743\n2002157\n```\n\nYou can add that to an array like in a script\n\n```sh\nEVE_FILE=eve.json\nSIGNATURES=(`jq 'select(.alert.signature_id)|.alert.signature_id' ${EVE_FILE} | sort | uniq`)\n```\n\nYou can loop over that array to print a sample of each signature\n\n```sh\nEVE_FILE=eve.json\nfile_list=()\nfor SIGNATURE_ID in \"${SIGNATURES[@]}\"\ndo\n  sample=`jq \"select(.alert.signature_id==$SIGNATURE_ID)\" ${EVE_FILE} | jq -s '.[0]'`\n  file_list=(\"${file_list[@]}\" \"$sample\")\ndone\n\necho \"${file_list[@]}\" | jq -s '.'\n```\n\nTo sort a eve.json object of alerts by timestamp in ascending order:\n\n```sh\njq -s 'sort_by(.timestamp)' eve.json\n```\n\nTo get an ad-hoc timeline of signature strings from an eve.json file of all alerts:\n\n```sh\njq -s 'sort_by(.timestamp)|.[].alert.signature' eve.json\n```\n\nTo get an array of alerts in one of the sample sub-folders sorted by timestamp in ascending order:\n\n```sh\ncd ${ROOT_OF_THIS_PROJECT}\njq 'sort_by(.timestamp)' samples/first-org-conf-2015/alerts-only.json\n```\n\nTo get an ad-hoc timeline of signature strings from one of the samples by timestamp:\n\n```sh\ncd ${ROOT_OF_THIS_PROJECT}\njq 'sort_by(.timestamp)|.[].alert.signature' samples/first-org-conf-2015/alerts-only.json\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FFrankHassanabad%2Fsuricata-sample-data","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FFrankHassanabad%2Fsuricata-sample-data","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FFrankHassanabad%2Fsuricata-sample-data/lists"}