{"id":26367683,"url":"https://github.com/molu8bits/modsecurity-parser","last_synced_at":"2025-03-16T21:18:14.659Z","repository":{"id":56761894,"uuid":"127528520","full_name":"molu8bits/modsecurity-parser","owner":"molu8bits","description":"modsecurity audit log analyser and parser","archived":false,"fork":false,"pushed_at":"2023-05-04T06:10:55.000Z","size":619,"stargazers_count":55,"open_issues_count":4,"forks_count":19,"subscribers_count":5,"default_branch":"master","last_synced_at":"2023-07-02T00:22:50.561Z","etag":null,"topics":["analyser","analyzer","charts","graphs","modsec-audit","modsecurity","modsecurity-audit-logs","modsecurity-parser","modsecurity3-log","molu8bits","parser"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/molu8bits.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-03-31T12:05:03.000Z","updated_at":"2023-06-30T06:45:53.000Z","dependencies_parsed_at":"2022-08-16T02:00:58.075Z","dependency_job_id":null,"html_url":"https://github.com/molu8bits/modsecurity-parser","commit_stats":null,"previous_names":[],"tags_count":null,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/molu8bits%2Fmodsecurity-parser","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/molu8bits%2Fmodsecurity-parser/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/molu8bits%2Fmodsecurity-parser/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/molu8bits%2Fmodsecurity-parser/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/molu8bits","download_url":"https://codeload.github.com/molu8bits/modsecurity-parser/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243933438,"owners_count":20370988,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["analyser","analyzer","charts","graphs","modsec-audit","modsecurity","modsecurity-audit-logs","modsecurity-parser","modsecurity3-log","molu8bits","parser"],"created_at":"2025-03-16T21:18:14.055Z","updated_at":"2025-03-16T21:18:14.642Z","avatar_url":"https://github.com/molu8bits.png","language":"Python","readme":"# modsecurity audit log parser, analyser and chart maker\r\n\r\n![CI](https://github.com/molu8bits/modsecurity-parser/workflows/CI/badge.svg?branch=develop\u0026event=push)\r\n[![codecov](https://codecov.io/gh/molu8bits/modsecurity-parser/branch/master/graph/badge.svg?token=BY0D5SNBR8)](https://codecov.io/gh/molu8bits/modsecurity-parser)\r\n![Docker Image Size](https://img.shields.io/docker/image-size/molu8bits/modsecurity-parser.svg?sort=date)\r\n![Docker Image Version (latest by date):](https://img.shields.io/docker/v/molu8bits/modsecurity-parser.svg?sort=date)\r\n![Docker Pulls](https://img.shields.io/docker/pulls/molu8bits/modsecurity-parser.svg)\r\n[![Quality Gate Status](https://sonarcloud.io/api/project_badges/measure?project=modsecurity-parser\u0026metric=alert_status)](https://sonarcloud.io/summary/new_code?id=modsecurity-parser)\r\n[![Lines of Code](https://sonarcloud.io/api/project_badges/measure?project=modsecurity-parser\u0026metric=ncloc)](https://sonarcloud.io/summary/new_code?id=modsecurity-parser)\r\n\r\n## TL;DR\r\n\r\nGet the overview of security incidents reported by modsecurity module from modsec_audit.log file.\r\n\r\n## 2023.05.03 update\r\n\r\n- fix showruleid #24\r\n- CI github actions\r\n- address vulnerabilities\r\n\r\n## 2023.01.01 update\r\n\r\n- renamed to modsecurity_parser\r\n- fix for timezone with miliseconds\r\n- linting, testing added\r\n- requirements vulnerabilities fixed\r\n\r\n## 2020.09.20 update\r\n\r\n- added support for logs from timezone \"UTC-...\"\r\n- updated plotting to matplotlib.3.1\r\n- added dockerhub autobuild\r\n- added requirements.txt\r\n\r\n## 2019.04.17 update\r\n\r\n- added support for Modsecurity3 log (Nginx/Apache)\r\n- added feature to read Modsecurity log in JSON format\r\n\r\n## Description\r\n\r\nmodsecurity parser is a python program to read [https://www.modsecurity.org/](https://www.modsecurity.org/)  modsec_audit.log, transform read events into more human and machine readable formats (xlsx/json) and make basic charts.\r\n\r\nFunctionality list:\r\n\r\n- JSON output file with formatting conformed to JSON logging added into Modsecurity 2.9\r\n- XLSX output file which can be analysed further with desktop tools\r\n- PNG file with some basic charts - Timeline nonblocked vs intercepted events, TOP10 IP source address, TOP20 Rule IDs hit, TOP10 Attacks intercepted\r\n\r\n## Graph analysis examples\r\n\r\n\u003cp align=\"left\"\u003e\r\n   \u003cimg src=\"/images/timeline.png\" width=\"950\" /\u003e\r\n\u003c/p\u003e  \r\n\r\n\u003cp align=\"center\"\u003e\r\n\r\n   \u003cimg src=\"/images/top10ipaddresses.png\" width=\"250\" /\u003e\r\n   \u003cimg src=\"/images/top10intercepted.png\" width=\"250\" /\u003e\r\n   \u003cimg src=\"/images/top20ruleID.png\" width=\"250\" /\u003e  \r\n\u003c/p\u003e\r\n\r\n## Installation\r\n\r\nSoftware needs at least Python 3.8.10 with additional libraries:\r\n\r\n- pandas 1.1.3\r\n- Pillow 9.2.0\r\n- matplotlib 3.3.2\r\n- numpy 1.22.4\r\n- openpyxl 2.4.2\r\n  \r\nInstall them with command\r\n\r\n```bash\r\npip3 install -r requirements.txt\r\n```\r\n\r\n## Basic usage\r\n\r\n```bash\r\npython3 modsecurity_parser.py -f /home/user/logs/modsec_audit.log\r\n```\r\n\r\nfor that case results will be recorded into subdirectory \"modsec_output\" where the log to analyse is placed.\r\n\r\n## More options\r\n\r\n```bash\r\npython3 modsecurity_parser.py -h\r\n```\r\n\r\nFilters INCLUDE and EXCLUDE are available for IP source addresses.\r\n\r\n--exclude option ( e.g. \"--exclude 192.168.0.1 10.0.0.1\") just skips events with given IP source addresses\r\n\r\n--include (e.g. \"--include 10.0.5.6\") take precedence over EXCLUDE. INCLUDE process only events with given IP source addresses.\r\n\r\n--jsononeperline  - option recommended for big number of events where e.g. produced JSON is supposed to be read by other SIEM tool. Uses the very same format as modsecurity software when type of logging is set to \"JSON\".\r\n\r\nProcessing Modsecurity3 log\r\n\r\n--version3 (e.g. \"modsecurity_parser.py -f modsec_audit.log --version3\"\r\n\r\nProcessing Modsecurity log in JSON format:\r\n\r\n--jsonaudit (e.g. \"modsecurity_parser.py -f modsec_audit.log --jsonaudit\"\r\n\r\n## Limitations\r\n\r\n- The biggest tested modsec_audit.log was 1GB size with around 70000 records. It took more or less 5 minutes on an 8 years old workstation and memory usage temporarily raised to 2GB of RAM\r\n- modsec_audit.log were taken from Apache web servers with locale set to en-US. Software can except some errors if datatime format is different in the audited log. Adjust LOG_TIMESTAMP_FORMAT and LOG_TIMESTAMP_FORMAT_SHORT accordingly\r\n- To process more than 90000 events just adjust MAXEVENTS\r\n- Tested with modsec_audit.log from version 2.8/2.9/3.0. Anyway Modsecurity3 for some cases produces empty H section and not all information is available to be properly presented in all graphs\r\n\r\n## run via Docker\r\n\r\nCreate a subfolder (e.g. \"modseclogs\") and put into some modsecurity audit logs (by default modsec_audit.log name is processed only).\r\nOutput files will be created inside of ${subfolder}/modsec_output\r\n\r\nRun command\r\n\r\n```bash\r\ndocker run --rm -ti --mount type=bind,source=\"$(pwd)\"/modseclogs,target=/opt/mounted molu8bits/modsecurity-parser:latest\r\n```\r\n\r\nGet some more docker options:\r\n\r\n```bash\r\ndocker run --rm -ti -e HELP=Yes molu8bits/modsecurity-parser:latest\r\n```\r\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmolu8bits%2Fmodsecurity-parser","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmolu8bits%2Fmodsecurity-parser","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmolu8bits%2Fmodsecurity-parser/lists"}