{"id":19390604,"url":"https://github.com/mandiant/goauditparser","last_synced_at":"2025-04-24T00:31:26.621Z","repository":{"id":57568329,"uuid":"341297156","full_name":"mandiant/goauditparser","owner":"mandiant","description":null,"archived":true,"fork":false,"pushed_at":"2023-06-01T13:27:55.000Z","size":2432,"stargazers_count":44,"open_issues_count":0,"forks_count":20,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-03-13T12:32:14.390Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mandiant.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":"auditparser.go","citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-02-22T18:27:13.000Z","updated_at":"2025-01-02T09:09:56.000Z","dependencies_parsed_at":"2024-06-18T22:54:36.616Z","dependency_job_id":"35e22041-277a-4f3a-9ee1-f7d8bb4a5f26","html_url":"https://github.com/mandiant/goauditparser","commit_stats":null,"previous_names":["fireeye/goauditparser"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mandiant%2Fgoauditparser","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mandiant%2Fgoauditparser/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mandiant%2Fgoauditparser/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mandiant%2Fgoauditparser/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mandiant","download_url":"https://codeload.github.com/mandiant/goauditparser/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250539395,"owners_count":21447301,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-10T10:22:08.773Z","updated_at":"2025-04-24T00:31:21.675Z","avatar_url":"https://github.com/mandiant.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"![Logo](etc/GoAuditParser_Logo.png)\n\n## General Usage\n```\n+===================+=============================================================+\n| Basic Parse       | goauditparser -i \u003cin_dir\u003e -o \u003ccsv_dir\u003e                      |\n| Parse \u0026 Timeline  | goauditparser -i \u003cin_dir\u003e -o \u003ccsv_dir\u003e -tl                  |\n| Extract Audits    | goauditparser -i \u003cin_dir\u003e -eo \u003cout_dir\u003e                     |\n| Extract File Acqs | goauditparser -i \u003cin_dir\u003e -efo \u003cout_dir\u003e -ep \u003cpassword\u003e     |\n| Raw Parse         | goauditparser -i \u003cin_dir\u003e -o \u003ccsv_dir\u003e -raw                 |\n+-------------------+-------------------------------------------------------------+\n```\n\nDownload precompiled builds of the latest version for Windows, Mac, and Linux located in the **[Releases](https://github.com/mandiant/goauditparser/releases)** section.\n\n## Features\n* Parse FireEye XML audit data from [FireEye Endpoint Security](https://www.fireeye.com/solutions/hx-endpoint-security-products.html) (previously \"HX\") and [Redline](https://www.fireeye.com/services/freeware/redline.html) into CSV format\n    * Supports most audit data from good old MIR scripts too!\n* Supports FireEye archive extracting and timelining\n* Multi-threaded speedy goodness with optimized memory usage\n    * Can parse XML audits of 100GB or more!\n* Automatically supports the latest FireEye Endpoint Security audit types\n* Automatically caches your progress so you can cancel and resume a parse at any time\n* Adjustable Excel-friendly features\n\n## Recent Version Changes\n\n**April 5, 2021**\n* Added configuration files `config.json` and `timeline.json` to the repository inside the folder `config`. These files must be placed in `~/.MandiantTools/GoAuditParser/` to be used by GoAuditParser, but are still automatically generated if not present at runtime.\n\n**v1.0.0 - February 25, 2021**\n* Initial public release of GoAuditParser!\n\n## Table of Contents\n1. [Usage and Flags](#usage-and-flags)\n2. [Example Usage](#example-usage)\n    1. [FireEye Endpoint Security Comprehensive Investigative Details](#fireeye-endpoint-security-comprehensive-investigative-details)\n    2. [FireEye Endpoint Security File Acquisitions](#fireeye-endpoint-security-file-acquisitions)\n    3. [Redline Collection](#redline-collection)\n    4. [Working With Excel](#working-with-excel)\n        1. [Creating a Table](#creating-a-table)\n        2. [Fixing Timestamps](#fixing-timestamps)\n        3. [Fixing Large Row Sizes](#fixing-large-row-sizes)\n    5. [Timelines](#timelines)\n        1. [Parse and Create a Timeline](#parse-and-create-a-timeline)\n        2. [Create a Timeline After Already Parsed](#create-a-timeline-after-already-parsed)\n        3. [Timeline Filter](#timeline-filter)\n3. [Configuration Files](#configuration-files)\n    1. [Main Configuration](#main-configuration)\n    2. [Timeline Configuration](#timeline-configuration)\n    3. [Parse Cache](#parse-cache)\n4. [All Version Changes](#all-version-changes)\n5. [FAQ \u0026 Support](#faq--support)\n\n## Usage and Flags\nYou can also see this menu by running GoAuditParser with the `-h` or `--help` flags.\n```\n===== [BASICS] ======================================================================================================\n# GoAuditParser can perform multiple tasks, sometimes independent of other steps, but it usually follows this order:\n\n    #  Name       Description                                               Automatic?\n    -- ---------  --------------------------------------------------------  ---------------\n    1) EXTRACT    Extract XML audits and other files from FireEye archives  YES\n    2) SPLIT      Split XML files that are too big into smaller files       YES\n    3) PARSE      Parse XML data to CSV                                     YES\n    4) TIMELINE   Timeline CSV data into an output file                     NO, needs '-tl'\n\n\n===== [REQUIRED] =================================  ===== [NOTES] ====================================================\n  -i \u003cstr\u003e     Directory Input                      ! REQUIRED - (except when '-tlo' used)\n                                                        Can provide multiple comma delimited paths:\n                                                            Ex: -i \"dir/xmldir1,xmldir2\"\n                                                        Works with .xml, .zip, or .mans files in the directory.\n\n===== [EXTRACTING] ===============================  ==================================================================\n# Extract and rename files from triages packages (.mans), bulk data collections (.zip), and file acquisitions (.zip).\n# The standardized naming scheme for XML files is as follows:\n#   \u003chostname\u003e-\u003cagentid\u003e-\u003cEXTRADATA\u003e-\u003caudittype\u003e.xml\n\n  -o \u003cstr\u003e     CSV Directory Output                 ! ONE REQUIRED (1/2) - Parse XML to CSV. Defaults to \"./parsed\".\n  -eo \u003cstr\u003e    Extract Output Directory (Only)      ! ONE REQUIRED (2/2) - Only extract and do not parse audits.\n                                                        Archive files are automatically extracted to input directory\n                                                            if this flag is not used.\n  -ep \u003cstr\u003e    Archive Password                     Provide a password for encrypted archives.\n                                                        Required to extract from file acquisition archives.\n  -efo         Extract File Acquisitions Only       Extract acquired files from archives only, no XML audits.\n                                                        Defaults '-eo' flag to \"files\" if not specified.\n                                                        Does not parse audits if used.\n  -eff \u003cint\u003e   Extract File Acquisition Format      Change how filenames for acquired files are formatted.\n                                                        1: \u003chostname\u003e-\u003cagentid\u003e-\u003cpayloadid\u003e-\u003cfullfilepath\u003e_  (default)\n                                                        2: \u003chostname\u003e-\u003cagentid\u003e-\u003cpayloadid\u003e-\u003cfullfilepath\u003e\n                                                        3: \u003cfullfilepath\u003e_\n                                                        4: \u003cfullfilepath\u003e\n                                                        5: \u003cbasefilename\u003e_\n                                                        6: \u003cbasefilename\u003e\n\n  -exf \u003cint\u003e   Extract XML Format                   Change how filenames for acquired files are formatted.\n                                                        1: \u003chostname\u003e-\u003cagentid\u003e-\u003cpayloadid\u003e-\u003caudittype\u003e.xml  (default)\n                                                        2: \u003chostname\u003e-\u003cagentid\u003e-0-\u003caudittype\u003e.xml\n\n===== [SPLITTING] ================================  ==================================================================\n# Split XML files. This step is automatically included if parsing.\n\n  -xso \u003cstr\u003e   XML Split Output Directory Only      Split XML audits into chunks. Use with '-xsb \u003cint\u003e' if desired.\n                                                        XML files are automatically split to \"\u003cinputdir\u003e/xmlsplit/\".\n                                                        Does not parse audits if a different path is specified.\n                                                        Appends \"_spxml#\" to payload of filename.\n  -xsb \u003cint\u003e   XML Split Byte Size                  Default value is \"300000000\" (300 MB). Not required for '-xso'.\n  -ebs \u003cstr\u003e   Event Buffer Split Output Directory  Split \"eventbuffer\" and \"stateagentinspector\" XML by event types.\n                                                        Provide an output directory.\n                                                        Does not parse audits if used.\n\n===== [PARSING] ==================================  ==================================================================\n# Parse XML audit data to CSV format.\n\n  -o \u003cstr\u003e     CSV Directory Output                 -REQUIRED- Parse XML to CSV. Defaults to \"./parsed\".\n  -r           Recursive Input                      Recursively dive into directories for parsing files.\n  -f           Force                                Force any previously extracted, parsed, or timelined\n                                                        files to be reprocessed.\n  -rn          Replace New-Line Chars with '|'      Useful when grepping through audits like event log messages.\n  -wo          Wipe Output Directory                Delete all files in output directory before parsing.\n                                                        Also enables \"-f\" flag for parsing/timelining only.\n  -c \u003cstr\u003e     Configuration File                   Contains a static order of headers for parsed CSV files.\n                                                        Defaults to \"~/.MandiantTools/GoAuditParser/config.json\".\n  -pcf \u003cint\u003e   Parsed CSV Format                    Change how filenames for acquired files are formatted.\n                                                        1: \u003chostname\u003e-\u003cagentid\u003e-\u003cEXTRADATA\u003e-\u003caudittype\u003e.csv  (default)\n                                                        2: \u003chostname\u003e-\u003cagentid\u003e-0-\u003caudittype\u003e.csv\n  -pah \u003cstr\u003e   Alternate Hostname                   Overwrite Hostname to provided string.\n  -paa \u003cstr\u003e   Alternate AgentID                    Overwrite AgentID to provided string.\n\n===== [TIMELINING] ===============================  ==================================================================\n# Convert parsed CSV audit data in the output directory into a timeline.\n# A static timeline configuration file ('-tlcf \u003cstr\u003e') is required to tell GoAuditParser how to format the timeline.\n\n  -o \u003cstr\u003e     CSV Directory Output                 -REQUIRED- Parse XML to CSV. Defaults to \"./parsed\"\n  -tl          Timeline                             -REQUIRED- Timeline files after parsed from XML to CSV.\n  -tlo         Timeline Only (don't parse)          Only perform timelining with specified CSV directory.\n                                                        Needs output CSV directory specified with \"-o \u003ccsv_dir\u003e\".\n                                                        Does NOT need an input XML directory specified.\n  -tld         Timeline Deduplicate                 Deduplicate timeline lines by entire row.\n  -tlout \u003cstr\u003e Timeline Output Filepath             Defaults to \"\u003ccsv_dir\u003e/_Timeline_\u003cDATE\u003e_\u003cTIME\u003e.csv\".\n  -tlf \u003cstr\u003e   Timeline Filter                      Include only events which match the provided filter(s).\n                                                        Time Filter formats:\n                                                            \"YYYY-MM-DD HH:MM:SS - YYYY-MM-DD HH:MM:SS\"\n                                                            \"YYYY-MM-DD HH:MM:SS +-5m\"\n                                                            \"YYYY-MM-DD - YYYY-MM-DD\"\n                                                            \"YYYY-MM-DD +-5m\"\n                                                        Can provide multiple comma delimited filters:\n                                                            Ex: -tlf \"2019-01-01 - 2020-01-01,2015-01-01 +-3d\"\n  -tlsod       Output IIMS/SOD format               Overwrites default timeline config to match IIMS/SOD format.\n  -tlcf \u003cstr\u003e  Timeline Config Filepath             Defaults to \"~/.MandiantTools/GoAuditParser/timeline.json\".\n\n===== [OTHER] ====================================  =================================================================\n  -c \u003cstr\u003e     Configuration File                   Defaults to \"~/.MandiantTools/GoAuditParser/config.json\".\n  -raw         Disable Excel-Friendly Features      Using this flag will disable the following Excel-Friendly features:\n                                                        1. Truncating cells to 32k chars\n                                                        2. Split CSV files by 1mil rows\n                                                            Appends \"_spcsv#\" to payload of filename.\n  -t \u003cint\u003e     Thread Count                         Defaults to number of existing CPUs.\n  -v[vvv]      Verbose\n  -min         Minimized Output Mode\n  --help       Show this Help Menu\n```\n\n## Example Usage\nThis section explains some of the use cases for GoAuditParser and example command syntaxes for specific situations.\n\n1. [FireEye Endpoint Security Comprehensive Investigative Details](#fireeye-endpoint-security-comprehensive-investigative-details)\n2. [FireEye Endpoint Security File Acquisitions](#fireeye-endpoint-security-file-acquisitions)\n3. [Redline Collection](#redline-collection)\n4. [Working With Excel](#working-with-excel)\n5. [Timelines](#timelines)\n\n- [Back to \"Table of Contents\"](#table-of-contents)\n\n### FireEye Endpoint Security Comprehensive Investigative Details\n[FireEye Endpoint Security](https://www.fireeye.com/solutions/hx-endpoint-security-products.html) is our Enterprise Detection and Response (EDR) solution. From our solution we can request a comprehensive list of artifacts from any connected endpoint. You can acquire a Comprehensive Investigative Details package from any endpoint's \"Hosts\" page by clicking \"ACQUIRE\" \u003e \"Comprehensive Investigative Details\".\n\n![GAP_4_1_1](etc/GAP_4_1_1.png)\n\nOnce the Comprehensive Investigative Details package has been acquired, you can review it in FireEye Endpoint Security's built-in Audit Viewer by clicking \"PROCESS DATA ACQUISITION\", or you can download it by clicking \"Download Full Triage\" which we'll do for use with GoAuditParser.\n\n![GAP_4_1_2](etc/GAP_4_1_2.png)\n\nNotice the package comes in a `.MANS` format. This is the format we use for FireEye archives, but it can be opened with any ZIP extraction utility. If you have Redline installed, you'll see the red Redline icon for MANS files. Let's take a look at the contents of this file below.\n\n![GAP_4_1_3](etc/GAP_4_1_3.png)\n\nThis looks to be quite the obfuscated format, but GoAuditParser knows exactly how to handle these types of files. Let's place the MANS file (not extracted) within its own directory named `zip` and perform a basic parse with GoAuditParser on it specifying the output directory `csv`.\n\n```\ngoauditparser -i zip -o csv\n```\n![GAP_4_1_4](etc/GAP_4_1_4.png)\n\nGoAuditParser extracted the files from the `.MANS` file to the input directory and renamed them based on the contents of the `manifest.json` and `metadata.json` files. Next, for any files were larger than 300 MB, GoAuditParser split them into 300 MB chunks in `zip/xmlsplit/`. Finally, GoAuditParser parsed the XML audit files and wrote them into the provided CSV output directory.\n\n![GAP_4_1_5](etc/GAP_4_1_5.png)\n\nShown below is one of the output CSV files after being formatted using tips mentioned in the [Working With Excel](#working-with-excel) section.\n\n![GAP_4_1_6](etc/GAP_4_1_6.png)\n\nNow we are ready to begin analysis with Excel or perform post-processing / enrichment!\n\n- [Back to top of \"Example Usage\" Section](#example-usage)\n\n### FireEye Endpoint Security File Acquisitions\n[FireEye Endpoint Security](https://www.fireeye.com/solutions/hx-endpoint-security-products.html) is our Enterprise Detection and Response (EDR) solution. From our solution we can request any file on disk. You can acquire a file from any endpoint's \"Hosts\" page by clicking \"ACQUIRE\" \u003e \"FILE\".\n\n![GAP_4_2_1](etc/GAP_4_2_1.png)\n\nOnce you requested files have been acquired, you can download them by clicking \"Download\". Also, take note of the password we'll need to unarchive it later. (The passwords have been removed from these images)\n\n![GAP_4_2_2](etc/GAP_4_2_2.png)\n\nShown below is the contents of one of these files.\n\n![GAP_4_2_6](etc/GAP_4_2_6.png)\n\nEach of the downloaded archives is encrypted and it would be tedious to extract them all one at a time.\n\nLet's extract all of these files at once. First, place them in a directory named `zip`. Next, run the following command with the flags `-eo \u003coutput_dir\u003e` to specify an output directory (optional, default output directory is \"files\" if `-efo` is used), `-efo` to specify that we only want to extract acquired files, and `-ep \u003cpassword\u003e` to provide the password for extracting the encrypted archives.\n```\ngoauditparser -i zip -efo -ep \u003cpassword\u003e\n```\n![GAP_4_2_3](etc/GAP_4_2_3.png)\n\nShown below is the output directory from the command above.\n\n![GAP_4_2_4](etc/GAP_4_2_4.png)\n\nGreat! We have our files extracted, but maybe we would prefer a different filename format. GoAuditParser provides you the following options with the `-eff \u003cint\u003e` flag.\n```\n-eff \u003cint\u003e   Extract File Acquisition Format   Change how filenames for acquired files are formatted.\n                                                   1: \u003chostname\u003e-\u003cagentid\u003e-\u003cpayloadid\u003e-\u003cfullfilepath\u003e_  (default)\n                                                   2: \u003chostname\u003e-\u003cagentid\u003e-\u003cpayloadid\u003e-\u003cfullfilepath\u003e\n                                                   3: \u003cfullfilepath\u003e_\n                                                   4: \u003cfullfilepath\u003e\n                                                   5: \u003cbasefilename\u003e_\n                                                   6: \u003cbasefilename\u003e\n```\n\nLet's use the following command just to get the acquired files extracted with just the base filenames and an underscore `_` at the end to prevent self-pwnage.\n\n```\ngoauditparser -i zip -eo files2 -efo -ep \u003cpassword\u003e -eff 5\n```\n\nShown below is the output directory from the command above.\n\n![GAP_4_2_5](etc/GAP_4_2_5.png)\n\nNow we can analyze these files much quicker than if we had manually extracted and renamed them!\n\n- [Back to top of \"Example Usage\" Section](#example-usage)\n\n### Redline Collection\n\n[Redline](https://www.fireeye.com/services/freeware/redline.html) is a publicly available forensically-sound precursor to FireEye Endpoint Security which lets you collect audit data from a system. Redline lets you create a Collector for Windows, Mac, or Linux. You can copy the Collector to the system you want to perform analysis on and execute it to collect audit data. After it finishes, it places the collected audit data within `./Sessions/AnalysisSession1/Audits/` as shown below.\n\n![GAP_4_4_1](etc/GAP_4_4_1.png)\n\nYou can use Redline itself to review the collected audits, but you may prefer to use Excel or perform post-processes / enrichment on the collected data. GoAuditParser helps you achieve any of these goals.\n\nLet's perform a basic parse with the following command. (a more useful command is shown later)\n```\ngoauditparser -i Sessions/AnalysisSession1/Audits -o csv\n```\n![GAP_4_4_2](etc/GAP_4_4_2.png)\n\nNotice in the console output that the files `platform.xml`, `manifest.json` and `Script.xml` could not be parsed. These files only contain metadata and can be ignored. Also, a file `formhistory.urn...` was found to be empty. This means that while XML file did contain a valid schema, it did not contain any records.\n\nThe output directory `csv` is shown below.\n\n![GAP_4_4_3](etc/GAP_4_4_3.png)\n\nThis XML data doesn't match the standardized Mandiant format of `\u003cHostname\u003e-\u003cAgentID\u003e-\u003cExtraData\u003e-\u003cAuditType\u003e.xml` which is usually the only place that holds the hostname of the system (unless you happen to have a SystemInfoItem audit). Since default Redline XML filenames don't match that format, GoAuditParser will put the original filename into the `\u003cExtraData\u003e` field of the output filename and use placeholders for `\u003cHostname\u003e` and `\u003cAgentID\u003e`. For those unaware, FireEye uses a 22-character AgentID as a unique identifier for that host. The `\u003cAuditType\u003e` for output CSV files is determined by looking inside of the corresponding XML document.\n\nWe can provide our own `\u003cHostname\u003e` with the `-pah \u003cstr\u003e` flag and our own `\u003cAgentID\u003e` with `-paa \u003cstr\u003e`. \n\nLet's see how that would work. We'll set the AgentID to \"0\" since we won't be using it for our purposes. We'll also tag on the `-wo` flag to wipe the output directory of our original files.\n```\ngoauditparser -i Sessions/AnalysisSession1/Audits -o csv -pah OriginalHostname -paa 0 -wo\n```\n![GAP_4_4_4](etc/GAP_4_4_4.png)\n\nThe newly updated output directory `./csv` is shown below.\n\n![GAP_4_4_5](etc/GAP_4_4_5.png)\n\nWith that change, the output CSV files look much nicer and shorter. Also, because we provided the hostname, the output CSV files will also contain it.\n\nShown below is one of the output CSV files after being formatted using tips mentioned in the [Working With Excel](#working-with-excel) section.\n\n![GAP_4_4_6](etc/GAP_4_4_6.png)\n\nNow we are ready to begin analysis with Excel or perform post-processing / enrichment!\n\n- [Back to top of \"Example Usage\" Section](#example-usage)\n\n### Working With Excel\n\nWhile FireEye offers audit-viewing solutions like FireEye Endpoint Security's built-in Audit Viewer and Redline, sometimes you need to work with the data in ways that those offerings weren't designed to support. With Excel, you can use formulas to compute statistics and manipulate the data into the formats you may otherwise need. There are a couple of tricks to getting Excel to work well with audit data, so we're going to try to cover most of those here.\n\n1. [Creating a Table](#creating-a-table)\n2. [Fixing Timestamps](#fixing-timestamps)\n3. [Fixing Large Row Sizes](#fixing-large-row-sizes)\n\n- [Back to top of \"Example Usage\" Section](#example-usage)\n\n#### Creating a Table\n\nLet's use an EventLogItem audit as an example. Upon opening it with Excel, here's what you might be presented with.\n\n![GAP_4_5_1](etc/GAP_4_5_1.png)\n\nFirst thing you may want to do is create a table for your data. This sets you up with the ability to filter and sort columns, and makes the row alternate between two different colors, allowing you to more easily trace data along a row. To easily create a table:\n1. Make sure you have the cell A:1 selected\n2. Hold CTRL and press \"a\" - this selects all of the present data\n  - Don't press CTRL + A twice, or you'll select the entire worksheet!\n3. Hold CTRL and press \"t\" - this prompts you to create a table of the data you have selected\n4. Press ENTER\n\nIf done properly, it should look something like this.\n\n![GAP_4_5_2](etc/GAP_4_5_2.png)\n\n- [Back to top of \"Working with Excel\" Section](#working-with-excel)\n\n#### Fixing Timestamps\n\nNext, we can see there are some problems with the timestamps where they aren't in the a useful format like `yyyy-mm-dd hh:mm:ss`. Maybe they look like `##########` which is caused by Excel trying to display a value but not having the column space to do so. That can be fixed by changing the column size, but unfortunately the` yyyy-mm-dd hh:mm:ss` timestamp format isn't included as the default format in Excel, so we have to fix it ourselves.\n\n1. Holding CTRL, left click on the column letters (Ex: A, B, C) above the headers for each column holding timestamp values\n2. Right click on one of the column letters and select \"Format Cells...\"\n\n![GAP_4_5_3](etc/GAP_4_5_3.png)\n\n3. Click the \"Custom\" Category\n4. Within the \"Type\" field, manually put `yyyy-mm-dd hh:mm:ss`\n5. Click \"OK\"\n\n![GAP_4_5_4](etc/GAP_4_5_4.png)\n\n6. Double click the right-most-edge of any of the selected columns\n\n![GAP_4_5_5](etc/GAP_4_5_5.png)\n\nFinally, you should have columns with proper `yyyy-mm-dd hh:mm:ss` formatted timestamps.\n\n![GAP_4_5_6](etc/GAP_4_5_6.png)\n\n- [Back to top of \"Working with Excel\" Section](#working-with-excel)\n\n#### Fixing Large Row Sizes\n\nSometimes, you may find a cell with multiple lines of data in it.\n\n![GAP_4_5_7](etc/GAP_4_5_7.png)\n\nIf you click in the value and then click out of it, you may cause the whole row to grow to an unusable height.\n\n![GAP_4_5_8](etc/GAP_4_5_8.png)\n\nHere's how you can prevent this and fix any rows that have been impacted by this.\n\n1. Hold CTRL and press \"a\" - this selects all of the present data\n2. Press, *but do not hold*, ALT\n3. Press, *but do not hold*, \"h\", then \"o\", and then \"h\" - this presents a \"Row Height\" menu\n\n![GAP_4_5_9](etc/GAP_4_5_9.png)\n\n4. Type `15`\n5. Press ENTER or click OK\n\nNow all of your rows are fixed back to a normal height!\n\n![GAP_4_5_10](etc/GAP_4_5_7.png)\n\n- [Back to top of \"Working with Excel\" Section](#working-with-excel)\n\n\n### Timelines\n\nAs long as your XML audit data can be parsed to CSV format, it can be timelined. You may want to review the [Timeline Configuration](#timeline-configuration) file for customizing timeline headers and features.\n\n1. [Parse and Create a Timeline](#parse-and-create-a-timeline)\n2. [Create a Timeline After Already Parsed](#create-a-timeline-after-already-parsed)\n3. [Timeline Filter](#timeline-filter)\n\n- [Back to top of \"Example Usage\" Section](#example-usage)\n\n#### Parse and Create a Timeline\n\nFor this example, let's assume we have [a MANS file from FireEye Endpoint Security](#fireeye-endpoint-security-comprehensive-investigative-details) and have not already extracted or parsed it to CSV. Let's put the MANS file in a directory named \"zip\" and run the following command, using `-tl` to timeline the files after they are parsed to CSV. By default, the timeline is written to `\u003cOutputDirectory\u003e/_Timeline_\u003cyyyy-mm-dd\u003e_\u003chhmm\u003e.csv\u003e`, but you can provide an output filepath for the timeline with `-tlout \u003cfilepath\u003e`.\n```\ngoauditparser -i zip -o csv -tl\n```\n![GAP_4_6_1](etc/GAP_4_6_1.png)\n\nThe output directory `csv` is shown below.\n\n![GAP_4_6_2](etc/GAP_4_6_2.png)\n\nShown below is the output timeline file after being formatted using tips mentioned in the [Working With Excel](#working-with-excel) section.\n\n![GAP_4_6_3](etc/GAP_4_6_3.png)\n\n- [Back to top of \"Timelines\" Section](#timelines)\n\n#### Create a Timeline After Already Parsed\n\nSometimes you may want to generate a timeline after you have already parsed files. You can make GoAuditParser generate a timeline without parsing any files with `-tlo` and providing the output directory of CSV files with `-o \u003ccsv_dir\u003e`.\n```\ngoauditparser -o csv -tlo\n```\n\n![GAP_4_6_4](etc/GAP_4_6_4.png)\n\nThe output directory `csv` is shown below.\n\n![GAP_4_6_2](etc/GAP_4_6_2.png)\n\nShown below is the output timeline file after being formatted using tips mentioned in the [Working With Excel](#working-with-excel) section.\n\n![GAP_4_6_3](etc/GAP_4_6_3.png)\n\n- [Back to top of \"Timelines\" Section](#timelines)\n\n#### Timeline Filter\n\nMaybe you're finding that your timeline file is too large or that you only need to focus on a specific timeframe. You can specify a timeline filter with `-tlf \u003ctime_filter\u003e`.\n\n```\n-tlf \u003cstr\u003e   Timeline Filter    Include only events which match the provided filter(s).\n                                    Time Filter formats:\n                                        \"YYYY-MM-DD HH:MM:SS - YYYY-MM-DD HH:MM:SS\"\n                                        \"YYYY-MM-DD HH:MM:SS +-5m\"\n                                        \"YYYY-MM-DD - YYYY-MM-DD\"\n                                        \"YYYY-MM-DD +-5m\"\n```\n\nLet's make a timeline from CSV files that have already been parsed only containing events five minutes around the timeframe `2020-06-27 16:00:00` with the timeline filter `-tlf \"2020-06-27 16:00:00 +-5m\"`\n\n```\ngoauditparser -o csv -tlo -tlf \"2020-06-27 16:00:00 +-5m\"\n```\n\n![GAP_4_6_4](etc/GAP_4_6_5.png)\n\nThe output directory `csv` is shown below.\n\n![GAP_4_6_2](etc/GAP_4_6_6.png)\n\nThe output timeline file is shown below.\n\n![GAP_4_6_3](etc/GAP_4_6_7.png)\n\n- [Back to top of \"Timelines\" Section](#timelines)\n\n## Configuration Files\n\nGoAuditParser uses three (3) different configuration files.\n\n1. [Main Configuration](#main-configuration)\n2. [Timeline Configuration](#timeline-configuration)\n3. [Parse Cache](#parse-cache)\n\n- [Back to \"Table of Contents\"](#table-of-contents)\n\n### Main Configuration\n\nThis configuration file is used for parsing XML audit files to CSV output files. GoAuditParser writes the default main configuration file to `~/.MandiantTools/GoAuditParser/config.json`. You can provide your own main configuration file with `-c path/to/config.json`. If you need a fresh copy of this configuration file, delete the default file and have GoAuditParser attempt to parse XML audit files.\n\n|**Key Name**|**Default Value**|**Explanation**|\n|------------|-----------------|---------------|\n|`Version`|*variable*|The current version of GoAuditParser. If this value is different from the current version of GoAuditParser, the configuration file is updated.|\n|`Dont_Overwrite_With_New_Update`|false|If set to true, GoAuditParser will not update this configuration file if it is outdated.|\n|`Automatically_Split_Big_XML`|true|If set to true, GoAuditParser will split XML files into 300 MB chunks for better memory efficiency.|\n|`Automatically_Extract_Archives`|true|If set to true, GoAuditParser will automatically extract any FireEye archives to the input directory.|\n|`Omit_Nonordered_Headers`|false|If set to true, GoAuditParser will omit any columns whose headers are not specified within `Audit_Header_Configs.#.Header_Order`.|\n|`Mandatory_Headers`|\"Tag\",\u003cbr\u003e\"Notes\",\u003cbr\u003e\"Hostname\",\u003cbr\u003e\"AgentID\"|These specified column headers always come first in CSV output and exist even if these fields aren't present in the audit data.|\n|`Optional_Headers`|\"Audit UID\",\u003cbr\u003e\"UID\",\u003cbr\u003e\"Sequence Number\",\u003cbr\u003e\"FireEyeGeneratedTime\",\u003cbr\u003e\"EventBufferType\"|These specified column headers come after the `Mandatory_Headers` headers in CSV output but don't exist if these fields aren't present in the audit data.|\n|`Audit_Header_Configs`|*variable*|Subconfigurations for each audit type. If an audit type isn't present, its data will be parsed automatically.|\n|`Audit_Header_Configs.#.Name`|*variable*|The name of the audit type. This field is only metadata and doesn't affect parsing.|\n|`Audit_Header_Configs.#.Item_Name`|*variable*|The audit type identifier found within the XML file. If this audit type is found, this subconfiguration is applied. Example: \"FileItem\"|\n|`Audit_Header_Configs.#.Header_Order`|*variable*|These specified column headers come after `Optional_Headers` in CSV output and exist even if these fields aren't present in the audit data. Any non-specified column headers identified by GoAuditParser will be provided after these headers if `Omit_Nonordered_Headers` is set to false and that header is not specified in `Audit_Header_Configs.#.Headers_Omitted`.|\n|`Audit_Header_Configs.#.Headers_Omitted`|*variable*|These specified column headers are removed from CSV output.|\n\n- [Back to top of \"Configuration Files\" Section](#configuration-files)\n\n### Timeline Configuration\n\nThis configuration file is used for timelining CSV files. GoAuditParser writes the default timeline configuration file to `~/.MandiantTools/GoAuditParser/timeline.json`. You can provide your own timeline configuration file with `-tlcf path/to/timeline.json`. If you need a fresh copy of this configuration file, delete the default file and have GoAuditParser attempt to timeline audit CSV files.\n\n|**Key Name**|**Default Value**|**Explanation**|\n|------------|-----------------|---------------|\n|`Version`|*variable*|The current version of GoAuditParser. If this value is different from the current version of GoAuditParser, the configuration file is updated.|\n|`Dont_Overwrite_With_New_Update`|false|If set to true, GoAuditParser will not update this configuration file if it is outdated.|\n|`Include_Summary_Headers`|true|If set to true, values within the \"Summary\" column will have headers prepended to the values like `FullFilePath: C:\\Windows\\Temp\\bad.ps1` instead of just `C:\\Windows\\Temp\\bad.ps1` alone.|\n|`Unique_Row_Per_Timestamp`|false|If set to true, audit entries with multiple timestamp values that are the same will each be put on separate lines instead of all being put into the same timeline row.|\n|`Include_Timestampless_Audits`|true|If set to true, audit entries without a timestamp will be included in the timeline instead of being omitted.|\n|`Extra_Fields_Order`|\"Hostname\",\u003cbr\u003e\"AgentID\",\u003cbr\u003e\"MD5\",\u003cbr\u003e\"Size\",\u003cbr\u003e\"User\",\u003cbr\u003e\"SignatureExists\",\u003cbr\u003e\"SignatureVerified\",\u003cbr\u003e\"SubAuditType\",\u003cbr\u003e\"Extra1\",\u003cbr\u003e\"Extra2\",\u003cbr\u003e\"Extra3\",\u003cbr\u003e\"Tag\",\u003cbr\u003e\"Notes\"|The first columns in a timeline will always be \"Timestamp\", \"Timestamp Description\", \"Summary\", and \"Source\". Anything else you want to include in the timeline as its own column can be specified here. To fill one of these columns, you'll need to specify which columns apply for each audit type in `Audit_Timeline_Configs.#.Extra_Fields`.|\n|`Audit_Timeline_Configs`|*variable*|Subconfigurations for each audit type. If an audit type isn't present, GoAuditParser will inform you at runtime and ignore it.|\n|`Audit_Timeline_Configs.#.Name`|*variable*|The name of the audit type. This field is only metadata and doesn't affect timelining.|\n|`Audit_Timeline_Configs.#.Filename_Suffix`|*variable*|The audit type identifier found within the `\u003cAuditType\u003e` portion of the CSV filename. If this audit type is found, this subconfiguration is applied. Example: \"FileItem\"|\n|`Audit_Timeline_Configs.#.Timestamp_Fields`|*variable*|These specified column headers are what GoAuditParser will look for when creating timeline rows. The timestamp value will fill the cell for the \"Timestamp\" column and the header for this value will fill the cell for the \"Timestamp Description\". If `Unique_Row_Per_Timestamp` is set to false, similar timestamps entries per audit row will be merged.|\n|`Audit_Timeline_Configs.#.Summary_Fields`|*variable*|These specified column headers will fill out the \"Summary\" column of the timeline. If `Include_Summary_Headers` is set to true, the headers will be prepended to each value.|\n|`Audit_Timeline_Configs.#.Extra_Fields`|*variable*|These specified column headers will fill out the fields specified in the `Extra_Fields_Order` column of the timeline. If you want to have a specific header fill out a field of a different name, you can use the syntax `\"auditheader\u003eextrafield\"`. Example: `\"DataLength\u003eSize\"`|\n\n- [Back to top of \"Configuration Files\" Section](#configuration-files)\n\n### Parse Cache\n\nThis cache file is used for keeping track of which files have been parsed. GoAuditParser writes the parse chache file to `\u003cInputPath\u003e/_GAPParseCache.json`.\n\n|**Key Name**|**Default Value**|**Explanation**|\n|------------|-----------------|---------------|\n|`Version`|*variable*|The current version of GoAuditParser. If this value is different from the current version of GoAuditParser, the configuration file is updated.|\n|`OutputDirectories`|*variable*|Subcaches for each output directory specified. Breaking up the output cache by output directory allows you to parse files to different directories without worry of cache conflicts.|\n|`OutputDirectories.#.OutputDirectory`|*variable*|The absolute path of the output directory specified.|\n|`OutputDirectories.#.XMLFiles`|*variable*|Subcaches for each XML audit file identified.|\n|`OutputDirectories.#.XMLFiles.#.Name`|*variable*|The filename of the XML audit file.|\n|`OutputDirectories.#.XMLFiles.#.Size`|*variable*|The file size of the XML audit file.|\n|`OutputDirectories.#.XMLFiles.#.Status`|*variable*|The status of the XML audit file. Can be \"parsed\", \"ignored/issues\", \"ignored/empty\" \"failed/rename\", \"failed/error\", \"failed/notexist\", or \"split\".|\n|`OutputDirectories.#.ArchiveFiles`|*variable*|Subcaches for each archive file (ZIP/MANS) file identified.|\n|`OutputDirectories.#.ArchiveFiles.#.Name`|*variable*|The filename of the archive file.|\n|`OutputDirectories.#.ArchiveFiles.#.Size`|*variable*|The file size of the archive file.|\n|`OutputDirectories.#.ArchiveFiles.#.Status`|*variable*|The status of the XML audit file. Can be \"extracted\", \"partial\", or \"failed\".|\n\n- [Back to top of \"Configuration Files\" Section](#configuration-files)\n\n## All Version Changes\n\n**April 5, 2021**\n* Added configuration files `config.json` and `timeline.json` to the repository inside the folder `config`. These files must be placed in `~/.MandiantTools/GoAuditParser/` to be used by GoAuditParser, but are still automatically generated if not present at runtime.\n\n**v1.0.0 - February 25, 2021**\n* Initial public release of GoAuditParser!\n\n## FAQ \u0026 Support\n\n**What is the AgentID field?**\n- FireEye uses a 22-character AgentID as a unique identifier for a system. This is used in FireEye Endpoint Security but is not used in Redline.\n\n**What is the Audit UID field?**\n- Each record within the XML audit contains a unique identifier generated at the time the data was collected. It is a metadata field only, and does not represent the collected data in any other way.\n\n**What is the FireEyeGeneratedTime timestamp?**\n- This timestamp is when the FireEye tool collected that record of information. It is a metadata field only, and does not represent the collected data in any other way.\n\n**Why are my timestamps appearing malformed in Excel?**\n- Good old Excel. Rest assured, your timestamps are likely fine. You need to tell Excel what format the timestamps should be in (`yyyy-mm-dd hh:mm:ss`) and make sure the column is wide enough to display the value. Check out the [Working With Excel](#working-with-excel) section for more details.\n\n**Why does it say my XML audit file is empty when there is clearly a little data in there?**\n- GoAuditParser reports when an XML audit file as \"empty\" if the XML schema for the audit is present but there are no entries or \"rows\" that can be parsed out of the file. If there is no XML schema present, it reports the file as \"failed\".\n\n**Where are the CSV versions of my \"Issues\" files?**\n- GoAuditParser does not parse Issues files, but it will tell you how many it identified in the Parse Statistics Summary.\n\n**What are these \".csv.incomplete\" files in my output directory?**\n- When GoAuditParser starts parsing an XML file, it attempts to create a temporary `.csv.incomplete` file in the output directory. If it cannot create this file, it skips processing the XML file. This is done to prevent wasted time parsing any particularly large XML file only to not be able to write the CSV output to disk. After it successfully writes the output contents to the temporary file, GoAuditParser makes an operating system call to rename the temporary `.csv.incomplete` file to the finalized `.csv` file. The whole point of the temporary `.csv.incomplete` file is in case you already have the finalized `.csv` file from a previous GoAuditParser parse open in Excel and you go to reparse the same XML files again. Excel locks each open CSV file with a handle, preventing GoAuditParser from overwriting it. If you receive a \"could not rename temp file to finalized file\" error message, you can be rest assured the finalized data is at least in the `.csv.incomplete` file and work with that file instead of needing to reparse everything over again.\n\n**Can I change the order of columns or omit unwanted columns from my CSV output?**\n- You can do both! Locate your main configuration file and set your preferred column orders with the `Mandatory_Headers`, `Optional_Headers`, and `Audit_Header_Configs.#.Header_Order` fields. If you want to omit specific columns from specific audits, set the `Audit_Header_Configs.#.Headers_Omitted` field. If you want to omit all unspecified audit columns, set `Omit_Nonordered_Headers` to true. Check out the [Main Configuration](#main-configuration) section for more details.\n\n**Why do my Hostname and AgentID fields have placeholders?**\n- This occurs when the input XML audit filename does not match the expected Mandiant standardized naming format of `\u003cHostname\u003e-\u003cAgentID\u003e-\u003cExtraData\u003e-\u003cAuditType\u003e`. This is the primary location that GoAuditParser uses to identify the Hostname and AgentID. If the input XML audit filenames do not match this format, the original XML audit filename is put into the `\u003cExtraData\u003e` field in the output CSV filename and the Hostname and the AgentID fields are replaced in the output CSV files with placeholders `HOSTNAMEPLACEHOLDER` and `AGENTIDPLACEHOLDER0000` with those extra four (4) '0' characters padding the AgentID to the expected 22-character length. To overwrite these placeholders, use the flags `-pah \u003cReplacementHostname\u003e` and `-paa \u003cReplacementAgentID\u003e`.\n\n**Why do my filenames contain `\\_spxml#` or `\\_spcsv#`?**\n- The `_spxml#` filename fragment contains the sequence number of an XML audit file that has been split at the XML level into multiple files. By default, GoAuditParser splits files that are larger than 300MB into `\u003cinput_dir\u003e/xmlsplit` and then parses those files instead of the original. The `_spcsv#` filename fragment contains the sequence number of the output CSV that has been split into multiple files. By default, GoAuditParser splits CSV files by one (1) million rows as a compatibility feature for Excel. You can disable automatic XML splitting in the main configuration file by setting `Automatically_Split_Big_XML` to false and you can disable the one (1) million row split by providing the `-raw` flag.\n\n**Why does GoAuditParser split my data into chunks?**\n- GoAuditParser performs two (2) types of splitting, XML splitting and CSV splitting. By default, GoAuditParser splits XML files that are larger than 300MB into `\u003cinput_dir\u003e/xmlsplit` and then parses those files instead of the original. This is because GoAuditParser uses multiple threads (Goroutines) and hashmaps to store parsed XML data before converting it to CSV for a number of efficiency and speed reasons, but threads and hashmaps are very memory expensive. Splitting the XML files before parsing them is the best solution to excessive memory consumption without sacrificing too much speed. Also, by default, GoAuditParser splits output CSV files by one (1) million rows as a compatibility feature for Excel. You can disable automatic XML splitting in the main configuration file by setting `Automatically_Split_Big_XML` to false and you can disable the one (1) million row split by providing the `-raw` flag.\n\n**I got an \"out of memory\" error!**\n- This issue is mostly fixed thanks to file splitting and buffered file reading for larger files, but it may still happen. This issue likely occurs when multiple large files are attempting to be parsed at the same time on two or more threads (Goroutines). Try forcing GoAuditParser to use only one thread with `-t 1`.\n\n**GoAuditParser is making my computer slow!**\n- GoAuditParser automatically uses the same number of threads (Goroutines) as the number of CPUs your computer has to be as speedy as possible. Try forcing GoAuditParser to use fewer threads or only one thread with `-t 1`, but this will slow down your parsing speed.\n\nFor questions, bugs, suggestions, or any other feedback, please contact GoAuditParser's primary developer Daniel Pany at daniel.pany@mandiant.com.\n\n- [Back to \"Table of Contents\"](#table-of-contents)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmandiant%2Fgoauditparser","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmandiant%2Fgoauditparser","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmandiant%2Fgoauditparser/lists"}