{"id":13591291,"url":"https://github.com/techsneeze/dmarcts-report-parser","last_synced_at":"2025-04-08T14:33:12.814Z","repository":{"id":5355285,"uuid":"52861574","full_name":"techsneeze/dmarcts-report-parser","owner":"techsneeze","description":"A Perl based tool to parse DMARC reports from an IMAP mailbox or from the filesystem, and insert the information into a database. ( Formerly known as imap-dmarcts )","archived":false,"fork":false,"pushed_at":"2023-08-19T12:07:09.000Z","size":196,"stargazers_count":224,"open_issues_count":15,"forks_count":64,"subscribers_count":32,"default_branch":"master","last_synced_at":"2024-11-06T11:46:03.456Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://www.techsneeze.com/how-parse-dmarc-reports-imap/","language":"Perl","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/techsneeze.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2016-03-01T08:42:16.000Z","updated_at":"2024-11-06T09:51:11.000Z","dependencies_parsed_at":"2024-01-16T21:57:21.269Z","dependency_job_id":"67eeb051-ec43-46e5-a132-f666379e2c24","html_url":"https://github.com/techsneeze/dmarcts-report-parser","commit_stats":{"total_commits":143,"total_committers":22,"mean_commits":6.5,"dds":0.7062937062937062,"last_synced_commit":"51ba1de8521559647ebe4b8a1db291c26b572de4"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/techsneeze%2Fdmarcts-report-parser","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/techsneeze%2Fdmarcts-report-parser/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/techsneeze%2Fdmarcts-report-parser/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/techsneeze%2Fdmarcts-report-parser/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/techsneeze","download_url":"https://codeload.github.com/techsneeze/dmarcts-report-parser/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247861149,"owners_count":21008435,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T16:00:55.615Z","updated_at":"2025-04-08T14:33:07.803Z","avatar_url":"https://github.com/techsneeze.png","language":"Perl","funding_links":[],"categories":["Perl","others","Security"],"sub_categories":["DMARC"],"readme":"# dmarcts-report-parser\nA Perl based tool to parse DMARC reports, based on John Levine's [rddmarc](http://www.taugh.com/rddmarc/), but extended by the following features:\n* Allow to read messages from an IMAP server and not only from the local filesystem.\n* Store much more XML values into the database (for example the missing SPF and DKIM results from the policy_evaluated section) and also the entire XML for later reference.\n* Supports MySQL and PostgreSQL.\n* Needed database tables and columns are created automatically, user only needs to provide a database. The database schema is compatible to the one used by rddmarc, but extends it by additional fields. Users can switch from rddmarc to dmarcts-report-parser without having to do any changes to the database by themselves.\n* Due to limitations in stock configurations of MySQL/MariaSQL on some distros, it may be necessary\nto add the following to your configuration (i.e. in /etc/mysql/mariadb.conf.d/50-server.cnf):\n\n```\ninnodb_large_prefix\t= on\ninnodb_file_format\t= barracuda\ninnodb_file_per_table\t= true\n```\n\n## Installation and Configuration\n\nTo install dependencies...\n\n### on Debian:\n```\napt-get install libfile-mimeinfo-perl libmail-imapclient-perl libmime-tools-perl libxml-simple-perl \\\nlibio-socket-inet6-perl libio-socket-ip-perl libperlio-gzip-perl \\\nlibmail-mbox-messageparser-perl unzip\n```\nPlus `libdbd-mysql-perl` for MySQL or `libdbd-pg-perl` for PostgreSQL.\n### on Fedora (Fedora 23):\n```\nsudo dnf install perl-File-MimeInfo perl-Mail-IMAPClient perl-MIME-tools perl-XML-Simple perl-DBI \\\n perl-Socket6 perl-PerlIO-gzip unzip\n```\nPlus `perl-DBD-MySQL` for MySQL or `perl-DBD-Pg` for PostgreSQL.\n### on CentOS (CentOS 7):\n```\nyum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm\nyum install perl-File-MimeInfo perl-Mail-IMAPClient perl-MIME-tools perl-XML-Simple perl-DBI \\\n perl-Socket6 perl-PerlIO-gzip unzip perl-Mail-Mbox-MessageParser\n ```\nPlus `perl-DBD-MySQL` for MySQL or `perl-DBD-Pg` for PostgreSQL.\n### on FreeBSD (FreeBSD 13.2):\n```\nsudo pkg install p5-File-MimeInfo p5-Mail-IMAPClient p5-MIME-tools p5-XML-Simple p5-DBI p5-Socket6 p5-PerlIO-gzip p5-Mail-Mbox-MessageParser unzip\n```\nPlus `p5-DBD-MySQL` for MySQL or `p5-DBD-Pg` for PostgreSQL.\n\nThere is also a port in mail/dmarcts-report-parser that does all this and\ncreates a periodic script that runs the IMAP pull daily:\n```\nsudo pkg install dmarcts-report-parser\n[create the database and IMAP users and edit /usr/local/etc/dmarcts-report-parser.conf]\necho daily_dmarcts_report_parser_enable=yes \u003e\u003e /etc/periodic.conf\n# Test it\ndmarcts-report-parser.pl -i\n# Hide the passwords!\nchmod 600 /usr/local/etc/dmarcts-report-parser.conf\n```\n ### on macOS (macOS 10.13):\n```\nbrew install mysql shared-mime-info\nupdate-mime-database /usr/local/share/mime\nperl -MCPAN -e 'install Mail::IMAPClient'\nperl -MCPAN -e 'install Mail::Mbox::MessageParser'\nperl -MCPAN -e 'install File::MimeInfo'\n```\nPlus `perl -MCPAN -e 'install DBD::mysql'` por MySQL or `perl -MCPAN -e 'install DBD::Pg'` or PostgreSQL.\n\nTo get your copy of the dmarcts-report-parser, you can either clone the repository:\n```\ngit clone https://github.com/techsneeze/dmarcts-report-parser.git\n```\nor download a zip file containg all files from [here](https://github.com/techsneeze/dmarcts-report-parser/archive/master.zip). Once the files have been downloaded, you will need to copy/rename `dmarcts-report-parser.conf.sample` to `dmarcts-report-parser.conf`. Next, edit the configuration options:\n\n```\n################################################################################\n### configuration ##############################################################\n################################################################################\n\n# If IMAP access is not used, config options starting with $imap do not need to\n# be set and are ignored.\n\n$debug = 0;\n$delete_reports = 0;\n\n# Supported types: mysql, Pg. If unset, defaults to mysql\n#$dbtype = 'mysql';\n$dbname = 'dmarc';\n$dbuser = 'dmarc';\n$dbpass = 'password';\n$dbhost = 'dbhost'; # Set the hostname if we can't connect to the local socket.\n$dbport = '3306';\n\n$imapserver       = 'imap.server';\n$imapuser         = 'username';\n$imappass         = 'password';\n$imapport         = '143';\n$imapssl          = '0';        # If set to 1, remember to change server port to 993 and disable imaptls.\n$imaptls          = '0';        # Enabled as the default and best-practice.\n$tlsverify        = '0';        # Enable verify server cert as the default and best-practice.\n$imapignoreerror  = '0';          # set it to 1 if you see an \"ERROR: message_string() \n                                # expected 119613 bytes but received 81873 you may \n                                # need the IgnoreSizeErrors option\" because of malfunction\n                                # imap server as MS Exchange 2007, ...\n$imapreadfolder   = 'dmarc';\n\n# If $imapmovefolder is set, processed IMAP messages will be moved (overruled by\n# the --delete option!)\n$imapmovefolder = 'dmarc/processed';\n\n# maximum size of XML files to store in database, long files can cause transaction aborts\n$maxsize_xml = 50000;\n# store XML as base64 encopded gzip in database (save space, harder usable)\n$compress_xml = 0;\n\n# if there was an error during file processing (message does not contain XML or ZIP parts, \n# or a database error) the parser reports an error and does not delete the file, even if \n# delete_reports is set (or --delete is given). Deletion can be enforced by delete_failed, \n# however not for database errors.\n$delete_failed = 0;\n```\nThe script is looking for `dmarcts-report-parser.conf` in the current working directory. If not found it will look by the calling path. If neither is found than it will abort.\n\nNote: Be sure to use the proper hierarchy separator for your server in all folder specs, and\nif your IMAP server flattens the hierarchy (i.e. Cyrus IMAP with \"altnamespace: yes\") then\nleave \"Inbox\" off of the beginning of such specs.\n\n## Usage\n\n```\n./dmarcts-report-parser.pl [OPTIONS] [PATH]\n```\nPATH can be the filename of a single file or a list of files - wildcard expression are allowed.\n\n**Remember**: This script needs a configurations file called \u003cdmarcts-report-parser.conf\u003e in the current working directory, which defines a database server with credentials and (if used) an IMAP server with credentials.\n\nOne of the following source options must be provided:\n```\n#        -i : Read reports from messages on IMAP server as defined in the config file.\n#        -m : Read reports from mbox file(s) provided in PATH.\n#        -e : Read reports from MIME email file(s) provided in PATH.\n#        -x : Read reports from xml file(s) provided in PATH.\n#        -z : Read reports from zip file(s) provided in PATH.\n```\n\nThe following options are always allowed:\n```\n#        -d : Print debug info.\n#        -r : Replace existing reports rather than failing.\n#  --delete : Delete processed message files (the XML is stored in the\n#             database for later reference).\n```\n\nMore info can currently be found at : [TechSneeze.com](http://www.techsneeze.com/how-parse-dmarc-reports-imap/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftechsneeze%2Fdmarcts-report-parser","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftechsneeze%2Fdmarcts-report-parser","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftechsneeze%2Fdmarcts-report-parser/lists"}