{"id":20749451,"url":"https://github.com/cristianzsh/modsecurity-to-es-parser","last_synced_at":"2025-09-27T20:31:43.469Z","repository":{"id":130227120,"uuid":"192448922","full_name":"cristianzsh/modsecurity-to-es-parser","owner":"cristianzsh","description":"A script to send ModSecurity logs to Elasticsearch","archived":false,"fork":false,"pushed_at":"2019-06-20T18:26:06.000Z","size":450,"stargazers_count":4,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-09T10:01:45.644Z","etag":null,"topics":["elasticsearch","kibana","logs","modsecurity","owasp","owasp-top-10","parser","python","security"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cristianzsh.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-06-18T02:14:12.000Z","updated_at":"2024-09-09T07:21:55.000Z","dependencies_parsed_at":"2023-06-25T20:09:50.321Z","dependency_job_id":null,"html_url":"https://github.com/cristianzsh/modsecurity-to-es-parser","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/cristianzsh/modsecurity-to-es-parser","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cristianzsh%2Fmodsecurity-to-es-parser","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cristianzsh%2Fmodsecurity-to-es-parser/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cristianzsh%2Fmodsecurity-to-es-parser/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cristianzsh%2Fmodsecurity-to-es-parser/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cristianzsh","download_url":"https://codeload.github.com/cristianzsh/modsecurity-to-es-parser/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cristianzsh%2Fmodsecurity-to-es-parser/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":277284824,"owners_count":25792646,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-27T02:00:08.978Z","response_time":73,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elasticsearch","kibana","logs","modsecurity","owasp","owasp-top-10","parser","python","security"],"created_at":"2024-11-17T08:23:04.616Z","updated_at":"2025-09-27T20:31:43.177Z","avatar_url":"https://github.com/cristianzsh.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ModSecurity parser\nA script to send ModSecurity logs to Elasticsearch\n\nModSecurity is a Web Application Firewall (WAF) for Apache and Nginx servers. It has logging capabilities and it is able to monitor HTTP traffic in order to mitigate attacks in real time. Elasticsearch is a search engine which can power extremely fast searches. The objective of this script is to send ModSecurity logs to Elasticsearch in order to be able to visualize them on Kibana.\n\n### ModSecurity logs\n\nThe default logs look like this:\n```\n---dt9xdeik---A--\n[18/Jun/2019:20:52:38 +0000] 156089115890.272782 10.25.3.75 54968 10.25.3.75 8080\n---dt9xdeik---B--\nGET /?command=ping%2010.0.0.1;%20cat%20/etc/passwd HTTP/1.1\nHost: 10.25.3.75:8080\nUser-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:67.0) Gecko/20100101 Firefox/67.0\nCookie: sid=Fe26.2**eea4ff29ce512cebcd40476ba7b3553b115981f39aee4c99a3d596adb0caf114*pxOW22uyKBM3BYacDKNBDQ*48jyCicpWf06ltgoyqqvjKWP3QUZboRfILkiLFEXHLaClA_r9D8FAsXoiUt5-MU7VnaVAm_E3EymUOzOFcTvPA**f97ca8cf9344e7065729e38497d5c562bac756d5dccf35bbf8061fc1828d70b1*C9xs9fCPlcbVu4nWPwb4tfb3Irr7R6QI_kL9p1474-E\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\nAccept-Language: en-US,en;q=0.5\nAccept-Encoding: gzip, deflate\nConnection: keep-alive\nUpgrade-Insecure-Requests: 1\n\n---dt9xdeik---D--\n\n---dt9xdeik---E--\n\u003chtml\u003e\\x0d\\x0a\u003chead\u003e\u003ctitle\u003e403 Forbidden\u003c/title\u003e\u003c/head\u003e\\x0d\\x0a\u003cbody bgcolor=\"white\"\u003e\\x0d\\x0a\u003ccenter\u003e\u003ch1\u003e403 Forbidden\u003c/h1\u003e\u003c/center\u003e\\x0d\\x0a\u003chr\u003e\u003ccenter\u003enginx/1.14.1\u003c/center\u003e\\x0d\\x0a\u003c/body\u003e\\x0d\\x0a\u003c/html\u003e\\x0d\\x0a\n\n---dt9xdeik---F--\nHTTP/1.1 403\nServer: nginx/1.14.1\nDate: Tue, 18 Jun 2019 20:52:38 GMT\nContent-Length: 169\nContent-Type: text/html\nConnection: keep-alive\n\n---dt9xdeik---H--\nModSecurity: Warning. Matched \"Operator `Rx' with parameter `^[\\d.:]+$' against variable `REQUEST_HEADERS:Host' (Value: `10.25.3.75:8080' ) [file \"/usr/local/nginx/conf/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf\"] [line \"762\"] [id \"920350\"] [rev \"\"] [msg \"Host header is a numeric IP address\"] [data \"10.25.3.75:8080\"] [severity \"4\"] [ver \"OWASP_CRS/3.1.0\"] [maturity \"0\"] [accuracy \"0\"] [tag \"application-multi\"] [tag \"language-multi\"] [tag \"platform-multi\"] [tag \"attack-protocol\"] [tag \"OWASP_CRS/PROTOCOL_VIOLATION/IP_HOST\"] [tag \"WASCTC/WASC-21\"] [tag \"OWASP_TOP_10/A7\"] [tag \"PCI/6.5.10\"] [hostname \"10.25.3.75\"] [uri \"/\"] [unique_id \"156089115890.272782\"] [ref \"o0,15v66,15\"]\nModSecurity: Warning. Matched \"Operator `PmFromFile' with parameter `lfi-os-files.data' against variable `ARGS:command' (Value: `ping 10.0.0.1; cat /etc/passwd' ) [file \"/usr/local/nginx/conf/rules/REQUEST-930-APPLICATION-ATTACK-LFI.conf\"] [line \"78\"] [id \"930120\"] [rev \"\"] [msg \"OS File Access Attempt\"] [data \"Matched Data: etc/passwd found within ARGS:command: ping 10.0.0.1; cat /etc/passwd\"] [severity \"2\"] [ver \"OWASP_CRS/3.1.0\"] [maturity \"0\"] [accuracy \"0\"] [tag \"application-multi\"] [tag \"language-multi\"] [tag \"platform-multi\"] [tag \"attack-lfi\"] [tag \"OWASP_CRS/WEB_ATTACK/FILE_INJECTION\"] [tag \"WASCTC/WASC-33\"] [tag \"OWASP_TOP_10/A4\"] [tag \"PCI/6.5.4\"] [hostname \"10.25.3.75\"] [uri \"/\"] [unique_id \"156089115890.272782\"] [ref \"o20,10v14,30t:utf8toUnicode,t:urlDecodeUni,t:normalizePathWin,t:lowercase\"]\nModSecurity: Warning. Matched \"Operator `Rx' with parameter `(?:;|\\{|\\||\\|\\||\u0026|\u0026\u0026|\\n|\\r|\\$\\(|\\$\\(\\(|`|\\${|\u003c\\(|\u003e\\(|\\(\\s*\\))\\s*(?:{|\\s*\\(\\s*|\\w+=(?:[^\\s]*|\\$.*|\\$.*|\u003c.*|\u003e.*|\\'.*\\'|\\\".*\\\")\\s+|!\\s*|\\$)*\\s*(?:'|\\\")*(?:[\\?\\*\\[\\]\\(\\)\\-\\|+\\w'\\\"\\./\\\\\\\\]+/)?[\\\\\\\\'\\\"]*(?: (5210 characters omitted)' against variable `ARGS:command' (Value: `ping 10.0.0.1; cat /etc/passwd' ) [file \"/usr/local/nginx/conf/rules/REQUEST-932-APPLICATION-ATTACK-RCE.conf\"] [line \"99\"] [id \"932100\"] [rev \"\"] [msg \"Remote Command Execution: Unix Command Injection\"] [data \"Matched Data: ; cat /etc/passwd found within ARGS:command: ping 10.0.0.1; cat /etc/passwd\"] [severity \"2\"] [ver \"OWASP_CRS/3.1.0\"] [maturity \"0\"] [accuracy \"0\"] [tag \"application-multi\"] [tag \"language-shell\"] [tag \"platform-unix\"] [tag \"attack-rce\"] [tag \"OWASP_CRS/WEB_ATTACK/COMMAND_INJECTION\"] [tag \"WASCTC/WASC-31\"] [tag \"OWASP_TOP_10/A1\"] [tag \"PCI/6.5.2\"] [hostname \"10.25.3.75\"] [uri \"/\"] [unique_id \"156089115890.272782\"] [ref \"o13,17v14,30\"]\nModSecurity: Warning. Matched \"Operator `Rx' with parameter `(?:^|=)\\s*(?:{|\\s*\\(\\s*|\\w+=(?:[^\\s]*|\\$.*|\\$.*|\u003c.*|\u003e.*|\\'.*\\'|\\\".*\\\")\\s+|!\\s*|\\$)*\\s*(?:'|\\\")*(?:[\\?\\*\\[\\]\\(\\)\\-\\|+\\w'\\\"\\./\\\\\\\\]+/)?[\\\\\\\\'\\\"]*(?:l[\\\\\\\\'\\\"]*(?:s(?:[\\\\\\\\'\\\"]*(?:b[\\\\\\\\'\\\"]*_[\\\\\\\\'\\\"]*r (6252 characters omitted)' against variable `ARGS:command' (Value: `ping 10.0.0.1; cat /etc/passwd' ) [file \"/usr/local/nginx/conf/rules/REQUEST-932-APPLICATION-ATTACK-RCE.conf\"] [line \"448\"] [id \"932150\"] [rev \"\"] [msg \"Remote Command Execution: Direct Unix Command Execution\"] [data \"Matched Data: ping  found within ARGS:command: ping 10.0.0.1; cat /etc/passwd\"] [severity \"2\"] [ver \"OWASP_CRS/3.1.0\"] [maturity \"0\"] [accuracy \"0\"] [tag \"application-multi\"] [tag \"language-shell\"] [tag \"platform-unix\"] [tag \"attack-rce\"] [tag \"OWASP_CRS/WEB_ATTACK/COMMAND_INJECTION\"] [tag \"WASCTC/WASC-31\"] [tag \"OWASP_TOP_10/A1\"] [tag \"PCI/6.5.2\"] [hostname \"10.25.3.75\"] [uri \"/\"] [unique_id \"156089115890.272782\"] [ref \"o0,5v14,30\"]\nModSecurity: Warning. Matched \"Operator `PmFromFile' with parameter `unix-shell.data' against variable `ARGS:command' (Value: `ping 10.0.0.1; cat /etc/passwd' ) [file \"/usr/local/nginx/conf/rules/REQUEST-932-APPLICATION-ATTACK-RCE.conf\"] [line \"481\"] [id \"932160\"] [rev \"\"] [msg \"Remote Command Execution: Unix Shell Code Found\"] [data \"Matched Data: etc/passwd found within ARGS:command: ping 10.0.0.1 cat/etc/passwd\"] [severity \"2\"] [ver \"OWASP_CRS/3.1.0\"] [maturity \"0\"] [accuracy \"0\"] [tag \"application-multi\"] [tag \"language-shell\"] [tag \"platform-unix\"] [tag \"attack-rce\"] [tag \"OWASP_CRS/WEB_ATTACK/COMMAND_INJECTION\"] [tag \"WASCTC/WASC-31\"] [tag \"OWASP_TOP_10/A1\"] [tag \"PCI/6.5.2\"] [hostname \"10.25.3.75\"] [uri \"/\"] [unique_id \"156089115890.272782\"] [ref \"o18,10v14,30t:urlDecodeUni,t:cmdLine,t:normalizePath,t:lowercase\"]\nModSecurity: Access denied with code 403 (phase 2). Matched \"Operator `Ge' with parameter `5' against variable `TX:ANOMALY_SCORE' (Value: `23' ) [file \"/usr/local/nginx/conf/rules/REQUEST-949-BLOCKING-EVALUATION.conf\"] [line \"80\"] [id \"949110\"] [rev \"\"] [msg \"Inbound Anomaly Score Exceeded (Total Score: 23)\"] [data \"\"] [severity \"2\"] [ver \"\"] [maturity \"0\"] [accuracy \"0\"] [tag \"application-multi\"] [tag \"language-multi\"] [tag \"platform-multi\"] [tag \"attack-generic\"] [hostname \"10.25.3.75\"] [uri \"/\"] [unique_id \"156089115890.272782\"] [ref \"\"]\nModSecurity: Warning. Matched \"Operator `Ge' with parameter `5' against variable `TX:INBOUND_ANOMALY_SCORE' (Value: `23' ) [file \"/usr/local/nginx/conf/rules/RESPONSE-980-CORRELATION.conf\"] [line \"76\"] [id \"980130\"] [rev \"\"] [msg \"Inbound Anomaly Score Exceeded (Total Inbound Score: 23 - SQLI=0,XSS=0,RFI=0,LFI=5,RCE=15,PHPI=0,HTTP=0,SESS=0): Remote Command Execution: Unix Shell Code Found; individual paranoia level scores: 23, 0, 0, 0\"] [data \"\"] [severity \"0\"] [ver \"\"] [maturity \"0\"] [accuracy \"0\"] [tag \"event-correlation\"] [hostname \"10.25.3.75\"] [uri \"/\"] [unique_id \"156089115890.272782\"] [ref \"\"]\n\n---dt9xdeik---I--\n\n---dt9xdeik---J--\n\n---dt9xdeik---Z--\n```\n\nAfter executing the script:\n```\n{\"maturity\": \"0\", \"tag\": \"PCI/6.5.10\", \"file\": \"/usr/local/nginx/conf/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf\", \"type\": \"Prevent exploitation\\n\", \"rev\": \"\", \"ver\": \"OWASP_CRS/3.1.0\", \"backup\": \"/root/modsec_logs_backup/2019-06-18 20:56:30.077214.zip\", \"msg\": \"Host header is a numeric IP address\", \"uri\": \"/\", \"id\": \"920350\", \"line\": \"762\", \"severity\": \"4\", \"hostname\": \"10.25.3.75\", \"data\": \"10.25.3.75:8080\", \"ref\": \"o0,15v66,15\", \"accuracy\": \"0\", \"unique_id\": \"156089115890.272782\", \"date\": \"2019/06/18\"}\n\n{\"maturity\": \"0\", \"tag\": \"PCI/6.5.4\", \"file\": \"/usr/local/nginx/conf/rules/REQUEST-930-APPLICATION-ATTACK-LFI.conf\", \"type\": \"Local File Inclusion attack\\n\", \"rev\": \"\", \"ver\": \"OWASP_CRS/3.1.0\", \"backup\": \"/root/modsec_logs_backup/2019-06-18 20:56:30.077214.zip\", \"msg\": \"OS File Access Attempt\", \"uri\": \"/\", \"id\": \"930120\", \"line\": \"78\", \"severity\": \"2\", \"hostname\": \"10.25.3.75\", \"data\": \"Matched Data: etc/passwd found within ARGS:command: ping 10.0.0.1; cat /etc/passwd\", \"ref\": \"o20,10v14,30t:utf8toUnicode,t:urlDecodeUni,t:normalizePathWin,t:lowercase\", \"accuracy\": \"0\", \"unique_id\": \"156089115890.272782\", \"date\": \"2019/06/18\"}\n\n{\"maturity\": \"0\", \"tag\": \"PCI/6.5.2\", \"file\": \"/usr/local/nginx/conf/rules/REQUEST-932-APPLICATION-ATTACK-RCE.conf\", \"rev\": \"\", \"ver\": \"OWASP_CRS/3.1.0\", \"backup\": \"/root/modsec_logs_backup/2019-06-18 20:56:30.077214.zip\", \"msg\": \"Remote Command Execution: Unix Shell Code Found\", \"uri\": \"/\", \"id\": \"932160\", \"line\": \"481\", \"severity\": \"2\", \"hostname\": \"10.25.3.75\", \"data\": \"Matched Data: etc/passwd found within ARGS:command: ping 10.0.0.1 cat/etc/passwd\", \"ref\": \"o18,10v14,30t:urlDecodeUni,t:cmdLine,t:normalizePath,t:lowercase\", \"accuracy\": \"0\", \"unique_id\": \"156089115890.272782\", \"date\": \"2019/06/18\"}\n\n{\"maturity\": \"0\", \"tag\": \"attack-generic\", \"file\": \"/usr/local/nginx/conf/rules/REQUEST-949-BLOCKING-EVALUATION.conf\", \"type\": \"Blocking evaluation\\n\", \"rev\": \"\", \"ver\": \"\", \"backup\": \"/root/modsec_logs_backup/2019-06-18 20:56:30.077214.zip\", \"msg\": \"Inbound Anomaly Score Exceeded (Total Score: 23)\", \"uri\": \"/\", \"id\": \"949110\", \"line\": \"80\", \"severity\": \"2\", \"hostname\": \"10.25.3.75\", \"data\": \"\", \"ref\": \"\", \"accuracy\": \"0\", \"unique_id\": \"156089115890.272782\", \"date\": \"2019/06/18\"}\n\n{\"maturity\": \"0\", \"tag\": \"event-correlation\", \"file\": \"/usr/local/nginx/conf/rules/RESPONSE-980-CORRELATION.conf\", \"type\": \"Correlation\\n\", \"rev\": \"\", \"ver\": \"\", \"backup\": \"/root/modsec_logs_backup/2019-06-18 20:56:30.077214.zip\", \"msg\": \"Inbound Anomaly Score Exceeded (Total Inbound Score: 23 - SQLI=0,XSS=0,RFI=0,LFI=5,RCE=15,PHPI=0,HTTP=0,SESS=0): Remote Command Execution: Unix Shell Code Found; individual paranoia level scores: 23, 0, 0, 0\", \"uri\": \"/\", \"id\": \"980130\", \"line\": \"76\", \"severity\": \"0\", \"hostname\": \"10.25.3.75\", \"data\": \"\", \"ref\": \"\", \"accuracy\": \"0\", \"unique_id\": \"156089115890.272782\", \"date\": \"2019/06/18\"}\n```\n\n### How to use\n\nIn the main file you will need to change the IP and the credentials of Elasticsearch on this line:\n\n```\nself.es = Elasticsearch([\"http://ip:9200\"], http_auth = (\"elastic\", \"password\"))\n```\n\nTo send the logs, just run the script and pass the logs directory as parameter:\n\n```\n$ python3 sendmodseclogs.py -d /path/to/directory/\n```\n\nOutput:\n\n![](prints/terminal.png)\n\nThe script will automatically send the logs every ten seconds.\n\n### Viewing logs on Kibana\n\nCreate a new index pattern called ```modsecurity_logs```.\n\n\n![](prints/kibana1.png)\n\nGo to the discover menu and select the new index pattern.\n\n![](prints/kibana2.png)\n\nYou can also create charts and dashboards to better visualize the data.\n\n![](prints/kibana3.png)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcristianzsh%2Fmodsecurity-to-es-parser","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcristianzsh%2Fmodsecurity-to-es-parser","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcristianzsh%2Fmodsecurity-to-es-parser/lists"}