{"id":13667354,"url":"https://github.com/jtmoon79/super-speedy-syslog-searcher","last_synced_at":"2025-04-05T17:04:08.676Z","repository":{"id":59529056,"uuid":"407794181","full_name":"jtmoon79/super-speedy-syslog-searcher","owner":"jtmoon79","description":"Speedily search and merge log messages by datetime","archived":false,"fork":false,"pushed_at":"2025-04-04T20:53:58.000Z","size":42932,"stargazers_count":46,"open_issues_count":79,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-05T11:43:52.839Z","etag":null,"topics":["log","log-parser","log-parsing","logging","logs","merge","rust","sort","syslog","syslog-messages","syslog-parser"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jtmoon79.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-09-18T07:50:30.000Z","updated_at":"2025-03-25T03:12:23.000Z","dependencies_parsed_at":"2022-09-18T09:21:34.621Z","dependency_job_id":"15b90f19-b550-4138-9bd7-9bbc12345b9a","html_url":"https://github.com/jtmoon79/super-speedy-syslog-searcher","commit_stats":{"total_commits":2019,"total_committers":2,"mean_commits":1009.5,"dds":"0.055968301139177834","last_synced_commit":"f563b7aa0ebaf45d0b841553eb9ac8e3fe0782a5"},"previous_names":[],"tags_count":75,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jtmoon79%2Fsuper-speedy-syslog-searcher","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jtmoon79%2Fsuper-speedy-syslog-searcher/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jtmoon79%2Fsuper-speedy-syslog-searcher/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jtmoon79%2Fsuper-speedy-syslog-searcher/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jtmoon79","download_url":"https://codeload.github.com/jtmoon79/super-speedy-syslog-searcher/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247332525,"owners_count":20921852,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["log","log-parser","log-parsing","logging","logs","merge","rust","sort","syslog","syslog-messages","syslog-parser"],"created_at":"2024-08-02T07:00:35.570Z","updated_at":"2025-04-05T17:04:08.649Z","avatar_url":"https://github.com/jtmoon79.png","language":"Rust","funding_links":[],"categories":["Rust"],"sub_categories":[],"readme":"# Super Speedy Syslog Searcher! (`s4`) \u003c!-- omit in toc --\u003e\n\nSpeedily search and merge log messages by datetime.\n\n\u003c!--\n* Must update parameters `branch=` and `version=` per release; replace the version\n--\u003e\n\n[![MSRV](https://img.shields.io/crates/msrv/super_speedy_syslog_searcher/0.7.75?logo=rust\u0026logoColor=800000\u0026cacheSeconds=6000)](https://github.com/jtmoon79/super-speedy-syslog-searcher/blob/0.7.75/Cargo.toml#L21)\n[![License](https://img.shields.io/crates/l/super-speedy-syslog-searcher?style=flat-square)](https://github.com/jtmoon79/super-speedy-syslog-searcher/blob/main/LICENSE.txt)\n[![docs.rs](https://img.shields.io/docsrs/super_speedy_syslog_searcher/0.7.75?badge.svg\u0026style=flat-square\u0026logo=docsdotrs)](https://docs.rs/super_speedy_syslog_searcher/0.7.75/)\n\n[![crates.io version](https://img.shields.io/crates/v/super-speedy-syslog-searcher.svg?style=flat-square\u0026logo=rust\u0026logoColor=800000?branch=0.7.75\u0026version=0.7.75)](https://crates.io/crates/super-speedy-syslog-searcher/0.7.75)\n[![crates.io downloads](https://img.shields.io/crates/d/super-speedy-syslog-searcher.svg?style=flat-square\u0026logo=rust\u0026logoColor=800000)](https://crates.io/crates/super-speedy-syslog-searcher#:~:text=Downloads%20all%20time)\n[![crates.io downloads (version)](https://img.shields.io/crates/dv/super_speedy_syslog_searcher/0.7.75?style=flat-square\u0026logo=rust\u0026logoColor=800000)](https://crates.io/crates/super-speedy-syslog-searcher/0.7.75)\n[![CHANGELOG](https://img.shields.io/badge/CHANGELOG-blue?style=flat-square\u0026logo=keep-a-changelog\u0026logoColor=FFFFFF\u0026color=E05735)](https://github.com/jtmoon79/super-speedy-syslog-searcher/blob/main/CHANGELOG.md#0775)\n[![lib.rs](https://img.shields.io/badge/lib.rs-white?style=flat-square\u0026logo=rust\u0026logoColor=202020)](https://lib.rs/crates/super_speedy_syslog_searcher/)\n\n[![Build status](https://img.shields.io/github/actions/workflow/status/jtmoon79/super-speedy-syslog-searcher/rust.yml?branch=0.7.75\u0026style=flat-square\u0026logo=github\u0026logoColor=000000)](https://github.com/jtmoon79/super-speedy-syslog-searcher/actions?query=workflow%3Arust)\n[![coveralls.io](https://img.shields.io/coverallsCoverage/github/jtmoon79/super-speedy-syslog-searcher?style=flat-square\u0026logo=coveralls\u0026logoColor=b94947\u0026branch=0.7.75\u0026version=0.7.75)](https://coveralls.io/github/jtmoon79/super-speedy-syslog-searcher?branch=0.7.75)\n[![Commits since](https://img.shields.io/github/commits-since/jtmoon79/super-speedy-syslog-searcher/0.7.75.svg?logo=github\u0026logoColor=000000)](https://github.com/jtmoon79/super-speedy-syslog-searcher/commits/main)\n\n_Super Speedy Syslog Searcher_ (`s4`) is a command-line tool to search\nand merge varying log messages from varying log files, sorted by datetime.\nDatetime filters may be passed to narrow the search to a datetime range.\n\n`s4` can read standardized log message formats like RFC 3164 and RFC 5424\n(\"syslog\"),\nRed Hat Audit logs, strace output, and can read many non-standardized ad-hoc log\nmessage formats, including multi-line log messages.\nIt also parses binary accounting records acct, lastlog, and utmp\n(`acct`, `pacct`, `lastlog`, `utmp`, `utmpx`, `wtmp`),\nsystemd journal logs (`.journal`), and Microsoft Event Logs (`.evtx`).\n`s4` can read logs that are compressed (`.bz2`, `.gz`, `.lz4`, `.xz`), or archived logs (`.tar`).\n\n`s4` aims to be very fast.\n\n---\n\n\u003c!-- TOC generated by Markdown All In One --\u003e\n\n\u003c!-- toc start --\u003e\n- [Use](#use)\n - [Install `super_speedy_syslog_searcher`](#install-super_speedy_syslog_searcher)\n - [allocator `mimalloc` or `jemalloc`](#allocator-mimalloc-or-jemalloc)\n - [Alpine](#alpine)\n - [Debian and Ubuntu](#debian-and-ubuntu)\n - [OpenSUSE](#opensuse)\n - [Red Hat and CentOS](#red-hat-and-centos)\n - [feature `mimalloc` on Windows](#feature-mimalloc-on-windows)\n - [Run `s4`](#run-s4)\n - [`--help`](#--help)\n- [About](#about)\n - [Why `s4`?](#why-s4)\n - [Features](#features)\n - [File name guessing](#file-name-guessing)\n - [Directory walks](#directory-walks)\n - [Limitations](#limitations)\n - [Hacks](#hacks)\n- [More](#more)\n - [Comparisons](#comparisons)\n - [General Features](#general-features)\n - [Formal Log DateTime Supported](#formal-log-datetime-supported)\n - [Other Log or File Formats Supported](#other-log-or-file-formats-supported)\n - [Archive Formats Supported](#archive-formats-supported)\n - [Speed Comparison](#speed-comparison)\n - [Building locally](#building-locally)\n - [Parsing `.journal` files](#parsing-journal-files)\n - [Requesting Support For DateTime Formats; your particular log file](#requesting-support-for-datetime-formats-your-particular-log-file)\n - [\"syslog\" and other project definitions](#syslog-and-other-project-definitions)\n - [syslog](#syslog)\n - [log message](#log-message)\n- [logging chaos: the problem `s4` solves](#logging-chaos-the-problem-s4-solves)\n - [open-source software examples](#open-source-software-examples)\n - [nginx webserver](#nginx-webserver)\n - [Debian 11](#debian-11)\n - [binary files](#binary-files)\n - [commercial software examples](#commercial-software-examples)\n - [Synology DiskStation](#synology-diskstation)\n - [Mac OS 12](#mac-os-12)\n - [Microsoft Windows 10](#microsoft-windows-10)\n - [Summary](#summary)\n- [Further Reading](#further-reading)\n- [Stargazers](#stargazers)\n\n\u003c!-- toc end --\u003e\n\n---\n\n## Use\n\n### Install `super_speedy_syslog_searcher`\n\nAssuming [rust is installed], run\n\n```lang-text\ncargo install --locked super_speedy_syslog_searcher\n```\n\nA C compiler is required.\n\n[rust is installed]: https://www.rust-lang.org/tools/install\n\n#### allocator `mimalloc` or `jemalloc`\n\nThe default allocator is the System allocator.\n\nAllocator [`mimalloc`] is feature `mimalloc` and allocator [`jemalloc`] is feature `jemalloc`.\nAllocator `mimalloc` [is the fastest according to `mimalloc` project benchmarks].\n`jemalloc` is also very good.\n\n\u003cbr/\u003e\n\n`mimalloc`\n\n```lang-text\ncargo install --locked super_speedy_syslog_searcher --features mimalloc\n```\n\nError `Bus error` is a known issue on some `aarch64-unknown-linux-gnu` systems.\n\n```lang-text\n$ s4 --version\nBus error\n```\n\nEither use `jemalloc` or the default System allocator.\n\n\u003cbr/\u003e\n\n`jemalloc`\n\n```lang-text\ncargo install --locked super_speedy_syslog_searcher --features jemalloc\n```\n\n\u003cbr/\u003e\n\n[`jemalloc`]: http://jemalloc.net/\n[`mimalloc`]: https://microsoft.github.io/mimalloc/bench.html\n[is the fastest according to `mimalloc` project benchmarks]: https://github.com/microsoft/mimalloc#Performance\n\n\u003cbr/\u003e\n\nHere are the packages for building `super_speedy_syslog_searcher` with `jemalloc` or `mimalloc`\non various Operating Systems.\n\n##### Alpine\n\n```lang-text\napk add gcc make musl-dev\n```\n\n##### Debian and Ubuntu\n\n```lang-text\napt install gcc make libc6-dev\n```\n\nor\n\n```lang-text\napt install build-essential\n```\n\n##### OpenSUSE\n\n```lang-text\nzypper install gcc glibc-devel make\n```\n\n##### Red Hat and CentOS\n\n```lang-text\nyum install gcc glibc-devel make\n```\n\n##### feature `mimalloc` on Windows\n\nCompiling `mimalloc` on Windows requires `lib.exe` which is part of _Visual Studio Build Tools_.\nInstructions at [rustup.rs].\n\n[rustup.rs]: https://rustup.rs/\n\n### Run `s4`\n\nFor example, print all the log messages in syslog files under `/var/log/`\n\n```lang-text\ns4 /var/log\n```\n\nOn Windows, print the ad-hoc logs under `C:\\Windows\\Logs`\n\n```lang-text\ns4.exe C:\\Windows\\Logs\n```\n\nOn Windows, print all `.log` files under `C:\\Windows` (with the help of Powershell)\n\n```lang-powershell\nGet-ChildItem -Filter '*.log' -File -Path \"C:\\Windows\" -Recurse -ErrorAction SilentlyContinue `\n | Select-Object -ExpandProperty FullName `\n | s4.exe -\n```\n\n\u003csup style=\"font-size: xx-small\"\u003eβ€’ note that UTF-16 encoded logs cannot be parsed, see [Issue #16]\u003c/sup\u003e\n\u003csup style=\"font-size: xx-small\"\u003eβ€’ note that opening too many files causes error _too many files open_, see [Issue #270], so `Get-ChildItem -Filter` lessens the number of files opened by `s4.exe`\u003c/sup\u003e\n\nOn Windows, print the [Windows Event logs]\n\n```lang-text\ns4.exe C:\\Windows\\System32\\winevt\\Logs\n```\n\nPrint the log messages after January 1, 2022 at 00:00:00\n\n```lang-text\ns4 /var/log -a 20220101\n```\n\nPrint the log messages from January 1, 2022 00:00:00 to January 2, 2022\n\n```lang-text\ns4 /var/log -a 20220101 -b 20220102\n```\n\nor\n\n```lang-text\ns4 /var/log -a 20220101 -b @+1d\n```\n\nPrint the log messages on January 1, 2022, from 12:00:00 to 16:00:00\n\n```lang-text\ns4 /var/log -a 20220101T120000 -b 20220101T160000\n```\n\nPrint the record-keeping log messages from up to a day ago\n(with the help of `find`)\n\n```lang-text\nfind /var -xdev -type f \\( \\\n -name 'lastlog' \\\n -or -name 'wtmp' \\\n -or -name 'wtmpx' \\\n -or -name 'utmp' \\\n -or -name 'utmpx' \\\n -or -name 'acct' \\\n -or -name 'pacct' \\\n \\) \\\n 2\u003e/dev/null \\\n | s4 - -a=-1d\n```\n\nPrint the journal log messages from up to an hour ago,\nprepending the journal file name\n(with the help of `find`)\n\n```lang-text\nfind / -xdev -name '*.journal' -type f 2\u003e/dev/null \\\n | s4 - -a=-1h -n\n```\n\nPrint only the log messages that occurred two days ago\n(with the help of GNU `date`)\n\n```lang-text\ns4 /var/log -a $(date -d \"2 days ago\" '+%Y%m%d') -b @+1d\n```\n\nPrint only the log messages that occurred two days ago during the noon hour\n(with the help of GNU `date`)\n\n```lang-text\ns4 /var/log -a $(date -d \"2 days ago 12\" '+%Y%m%dT%H%M%S') -b @+1h\n```\n\nPrint only the log messages that occurred two days ago during the noon hour in\nBengaluru, India (timezone offset +05:30) and prepended with equivalent UTC\ndatetime (with the help of GNU `date`)\n\n```lang-text\ns4 /var/log -u -a $(date -d \"2 days ago 12\" '+%Y%m%dT%H%M%S+05:30') -b @+1h\n```\n\n[Windows Event logs]: https://github.com/libyal/libevtx/blob/126297f7f0e325f9e2cd27b0b60d3cf02ffdfd04/documentation/Windows%20XML%20Event%20Log%20(EVTX).asciidoc\n[Issue #16]: https://github.com/jtmoon79/super-speedy-syslog-searcher/issues/16\n[Issue #270]: https://github.com/jtmoon79/super-speedy-syslog-searcher/issues/270\n\n### `--help`\n\n```lang-text\nSpeedily search and merge log messages by datetime.\nDateTime filters may be passed to narrow the search.\ns4 aims to be very fast.\n\nUsage: s4 [OPTIONS] \u003cPATHS\u003e...\n\nArguments:\n \u003cPATHS\u003e... Path(s) of log files or directories.\n Directories will be recursed. Symlinks will be followed.\n Paths may also be passed via STDIN, one per line. The user must\n supply argument \"-\" to signify PATHS are available from STDIN.\n\nOptions:\n -a, --dt-after \u003cDT_AFTER\u003e\n DateTime Filter After: print log messages with a datetime that is at\n or after this datetime. For example, \"20200102T120000\" or \"-5d\".\n -b, --dt-before \u003cDT_BEFORE\u003e\n DateTime Filter Before: print log messages with a datetime that is at\n or before this datetime.\n For example, \"2020-01-03T23:00:00.321-05:30\" or \"@+1d+11h\"\n -t, --tz-offset \u003cTZ_OFFSET\u003e\n Default timezone offset for datetimes without a timezone.\n For example, log message \"[20200102T120000] Starting service\" has a\n datetime substring \"20200102T120000\".\n That datetime substring does not have a timezone offset\n so this TZ_OFFSET value would be used.\n Example values, \"+12\", \"-0800\", \"+02:00\", or \"EDT\".\n To pass a value with leading \"-\" use \"=\" notation, e.g. \"-t=-0800\".\n If not passed then the local system timezone offset is used.\n [default: -07:00]\n -z, --prepend-tz \u003cPREPEND_TZ\u003e\n Prepend a DateTime in the timezone PREPEND_TZ for every line.\n Used in PREPEND_DT_FORMAT.\n -u, --prepend-utc\n Prepend a DateTime in the UTC timezone offset for every line.\n This is the same as \"--prepend-tz Z\".\n Used in PREPEND_DT_FORMAT.\n -l, --prepend-local\n Prepend DateTime in the local system timezone offset for every line.\n This is the same as \"--prepend-tz +XX\" where +XX is the local system\n timezone offset.\n Used in PREPEND_DT_FORMAT.\n -d, --prepend-dt-format \u003cPREPEND_DT_FORMAT\u003e\n Prepend a DateTime using the strftime format string.\n If PREPEND_TZ is set then that value is used for any timezone offsets,\n i.e. strftime \"%z\" \"%:z\" \"%Z\" values, otherwise the timezone offset value\n is the local system timezone offset.\n [Default: %Y%m%dT%H%M%S%.3f%z]\n -n, --prepend-filename\n Prepend file basename to every line.\n -p, --prepend-filepath\n Prepend file full path to every line.\n -w, --prepend-file-align\n Align column widths of prepended data.\n --prepend-separator \u003cPREPEND_SEPARATOR\u003e\n Separator string for prepended data.\n [default: :]\n --separator \u003cLOG_MESSAGE_SEPARATOR\u003e\n An extra separator string between printed log messages.\n Per log message not per line of text.\n Accepts a basic set of backslash escape sequences,\n e.g. \"\\0\" for the null character, \"\\t\" for tab, etc.\n --journal-output \u003cJOURNAL_OUTPUT\u003e\n The format for .journal file log messages.\n Matches journalctl --output options.\n [default: short]\n [possible values: short, short-precise, short-iso, short-iso-precise,\n short-full, short-monotonic, short-unix, verbose, export, cat]\n -c, --color \u003cCOLOR_CHOICE\u003e\n Choose to print to terminal using colors.\n [default: auto]\n [possible values: always, auto, never]\n --blocksz \u003cBLOCKSZ\u003e\n Read blocks of this size in bytes.\n May pass value as any radix (hexadecimal, decimal, octal, binary).\n Using the default value is recommended.\n Most useful for developers.\n [default: 65536]\n -s, --summary\n Print a summary of files processed to stderr.\n Most useful for developers.\n -h, --help\n Print help\n -V, --version\n Print version\n\nGiven a file path, the file format will be processed based on a best guess of\nthe file name.\nIf the file format is not guessed then it will be treated as a UTF8 text file.\nGiven a directory path, found file names that have well-known non-log file name\nextensions will be skipped.\n\nDateTime Filters may be strftime specifier patterns:\n \"%Y%m%dT%H%M%S*\"\n \"%Y-%m-%d %H:%M:%S*\"\n \"%Y-%m-%dT%H:%M:%S*\"\n \"%Y/%m/%d %H:%M:%S*\"\n \"%Y%m%d\"\n \"%Y-%m-%d\"\n \"%Y/%m/%d\"\n \"+%s\"\nEach * is an optional trailing 3-digit fractional sub-seconds,\nor 6-digit fractional sub-seconds, and/or timezone.\n\nPattern \"+%s\" is Unix epoch timestamp in seconds with a preceding \"+\".\nFor example, value \"+946684800\" is be January 1, 2000 at 00:00, GMT.\n\nDateTime Filters may be custom relative offset patterns:\n \"+DwDdDhDmDs\" or \"-DwDdDhDmDs\"\n \"@+DwDdDhDmDs\" or \"@-DwDdDhDmDs\"\n\nCustom relative offset pattern \"+DwDdDhDmDs\" and \"-DwDdDhDmDs\" is the offset\nfrom now (program start time) where \"D\" is a decimal number.\nEach lowercase identifier is an offset duration:\n\"w\" is weeks, \"d\" is days, \"h\" is hours, \"m\" is minutes, \"s\" is seconds.\nFor example, value \"-1w22h\" is one week and twenty-two hours in the past.\nValue \"+30s\" is thirty seconds in the future.\n\nCustom relative offset pattern \"@+DwDdDhDmDs\" and \"@-DwDdDhDmDs\" is relative\noffset from the other datetime.\nArguments \"-a 20220102 -b @+1d\" are equivalent to \"-a 20220102 -b 20220103\".\nArguments \"-a @-6h -b 20220101T120000\" are equivalent to\n\"-a 20220101T060000 -b 20220101T120000\".\n\nWithout a timezone, the Datetime Filter is presumed to be the local\nsystem timezone.\n\nCommand-line passed timezones may be numeric timezone offsets,\ne.g. \"+09:00\", \"+0900\", or \"+09\", or named timezone offsets, e.g. \"JST\".\nAmbiguous named timezones will be rejected, e.g. \"SST\".\n\n--prepend-tz and --dt-offset function independently:\n--dt-offset is used to interpret processed log message datetime stamps that\ndo not have a timezone offset.\n--prepend-tz affects what is pre-printed before each printed log message line.\n\n--separator accepts backslash escape sequences:\n \"\\0\", \"\\a\", \"\\b\", \"\\e\", \"\\f\", \"\\n\", \"\\r\", \"\\\\\", \"\\t\", \"\\v\"\n\nResolved values of \"--dt-after\" and \"--dt-before\" can be reviewed in\nthe \"--summary\" output.\n\ns4 uses file naming to determine the file type.\n\ns4 can process files compressed and named .bz2, .gz, .lz4, .xz, and files\narchived within a .tar file.\n\nLog messages from different files with the same datetime are printed in order\nof the arguments from the command-line.\n\nDatetimes printed for .journal file log messages may differ from datetimes\nprinted by program journalctl.\nSee Issue #101\n\nDateTime strftime specifiers are described at\nhttps://docs.rs/chrono/latest/chrono/format/strftime/\n\nDateTimes supported are only of the Gregorian calendar.\n\nDateTimes supported language is English.\n\nFurther background and examples are at the project website:\nhttps://github.com/jtmoon79/super-speedy-syslog-searcher/\n\nIs s4 failing to parse a log file? Report an Issue at\nhttps://github.com/jtmoon79/super-speedy-syslog-searcher/issues/new/choose\n```\n\n---\n\n## About\n\n### Why `s4`?\n\n_Super Speedy Syslog Searcher_ (`s4`) is meant to aid Engineers in reviewing\nvarying log files in a datetime-sorted manner.\nThe primary use-case is to aid investigating problems wherein the time of\na problem occurrence is known and there are many available logs\nbut otherwise there is little source evidence.\n\nCurrently, log file formats vary widely. _Most_ logs are an ad-hoc format.\nEven separate log files on the same system for the same service may have\ndifferent message formats!\nSorting these logged messages by datetime may be prohibitively difficult.\nThe result is an engineer may have to \"hunt and peck\" among many log files,\nlooking for problem clues around some datetime; so tedious!\n\nEnter _Super Speedy Syslog Searcher_ 🦸 β€Ό\n\n`s4` will print log messages from multiple log files in datetime-sorted order.\nA \"window\" of datetimes may be passed, to constrain the period of printed\nmessages. This will assist an engineer that, for example, needs to view all\nlog messages that occurred two days ago between 12:00 and 12:05 among log files taken from multiple\nsystems.\n\nThe ulterior motive for _Super Speedy Syslog Searcher_ was the [primary\ndeveloper](https://github.com/jtmoon79) wanted an excuse to learn rust πŸ¦€,\nand wanted to create an open-source tool for a recurring need of some\nSoftware Test Engineers πŸ˜„\n\nSee the real-world example rationale in the section below,\n[_logging chaos: the problem `s4` solves_].\n\n[_logging chaos: the problem `s4` solves_]: #logging-chaos-the-problem-s4-solves\n\n### Features\n\n- Parses:\n - Ad-hoc log messages using \u003cspan id=\"formal-datetimes\"\u003eformal datetime formats\u003c/span\u003e:\n - [Internet Message Format (RFC 2822)]\u003cbr/\u003ee.g. _Wed, 1 Jan 2020 22:00:00 PST message…_\n - [The BSD syslog Protocol (RFC 3164)]\u003cbr/\u003ee.g. _\\\u003c8\\\u003eJan 1 22:00:00 message…_\n - [Date and Time on the Internet: Timestamps (RFC 3339)]\u003cbr/\u003ee.g. _2020-01-01T22:00:00-08:00 message…_\n - [The Syslog Protocol (RFC 5424)]\u003cbr/\u003ee.g. _2020-01-01T22:00:00-08:00 message…_\n - [ISO 8601]\u003cbr/\u003ee.g. _2020-01-01T22:00:00-08:00 message…_, _20200101T220000-0800 message…_, etc. \u003csup\u003e\u003ca href=\"#f1\"\u003e\\[1\\]\u003c/a\u003e\u003c/sup\u003e\n - [Red Hat Audit Log] files\n - [strace] output files with options `-ttt` or `--timestamps`,\n i.e. Unix epoch plus optional milliseconds, microseconds, or nanoseconds\n - binary user accounting records files\n ([`acct`, `pacct`], [`lastlog`], [`utmp`, `utmpx`])\n from multiple Operating Systems and CPU architectures\n - binary [Windows Event Log] files\n - binary [systemd journal] files with printing options matching [`journalctl`]\n - many varying text log messages with ad-hoc datetime formats\n - multi-line log messages\n- Inspects `.tar` archive files for parseable log files \u003csup\u003e\u003ca href=\"#f2\"\u003e\\[2\\]\u003c/a\u003e\u003c/sup\u003e\n- Can process `.bz2`, `.gz`, `.lz4`, or `.xz` containing log files.\n- Tested against \"in the wild\" log files from varying sources\n (see project path [`./logs/`])\n- Prepends datetime and file paths, for easy programmatic parsing or\n visual traversal of varying log messages\n- [Comparable speed as GNU `grep` and `sort`](#speed-comparison)\n- Processes invalid UTF-8\n- Accepts arbitrarily large files \u003csup\u003e\u003ca href=\"#hacks\"\u003esee _Hacks_\u003c/a\u003e\u003c/sup\u003e\n\n[`acct`, `pacct`]: https://www.man7.org/linux/man-pages/man5/acct.5.html\n[`lastlog`]: https://man.netbsd.org/lastlog.5\n[`utmp`, `utmpx`]: https://en.wikipedia.org/w/index.php?title=Utmp\u0026oldid=1143684808#utmpx,_wtmpx_and_btmpx\n[Internet Message Format (RFC 2822)]: https://www.rfc-editor.org/rfc/rfc2822#section-3.3\n[The BSD syslog Protocol (RFC 3164)]: https://www.rfc-editor.org/rfc/rfc3164#section-4.1.2\n[Date and Time on the Internet: Timestamps (RFC 3339)]: https://www.rfc-editor.org/rfc/rfc3339#section-5.8\n[The Syslog Protocol (RFC 5424)]: https://www.rfc-editor.org/rfc/rfc5424#section-6.2.3\n[ISO 8601]: https://en.wikipedia.org/w/index.php?title=ISO_8601\u0026oldid=1113067353#General_principles\n[Red Hat Audit Log]: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/security_guide/sec-understanding_audit_log_files\n[strace]: https://www.man7.org/linux/man-pages/man1/strace.1.html\n[Windows Event Log]: https://learn.microsoft.com/en-us/windows/win32/wes/windows-event-log\n[systemd journal]: https://systemd.io/JOURNAL_FILE_FORMAT/\n[`journalctl`]: https://www.man7.org/linux/man-pages/man1/journalctl.1.html\n[`./logs/`]: https://github.com/jtmoon79/super-speedy-syslog-searcher/tree/main/logs\n\n#### File name guessing\n\nGiven a file path, `s4` will attempt to parse it. The type of file must be in\nthe name. Guesses are made about files with non-standard names.\n\nFor example, standard file name `utmp` will always be treated as a `utmp` record\nfile. But non-standard name `log.utmp.1` is guessed to be a `utmp` record file.\nSimilar guesses are applied to `lastlog`, `wtmp`, `acct`, `pacct`,\n`journal`, and `evtx` files.\nWhen combined with compression or archive file name extensions,\ne.g. `.bz2`, `.gz`, `.lz4`, or `.xz`, then `s4` makes a best attempt at\nguessing the compression or archive type and the file within the archive based\non the name.\nFor example, `user.journal.gz` is guessed to be a systemd journal file within a\ngzip compressed file. However, if that same file is named something unusual like\n`user.systemd-journal.gz` then it is guessed to be a text log file within a gzip\ncompressed file.\n\nWhen a file type cannot be guessed then it is treated as a UTF8 text log file.\nFor example, a file name just `unknown` is not any obvious type so it is attempted\nto be parsed as a UTF8 ad-hoc text log file.\n\n`tar` files are inspected for parseable files.\u003csup\u003e\u003ca href=\"#f2\"\u003e\\[2\\]\u003c/a\u003e\u003c/sup\u003e\n\n#### Directory walks\n\nGiven a directory path, `s4` will walk the directory and all subdirectories and\nfollow symbolic links and cross file system paths.\n`s4` will ignore files with extensions that are known to be non-log files.\nFor example, files with extensions `.dll`, `.mp3`, `.png`, or `.so`, are\nunlikely to be log files and so are not processed.\n\nSo given a file `/tmp/file.mp3`, an invocation of `s4 /tmp` will not attempt\nto process `file.mp3`. An invocation of `s4 /tmp/file.mp3` will attempt to\nprocess `file.mp3`. It will be treated as a UTF8 text log file.\n\n### Limitations\n\n- Only processes UTF-8 or ASCII encoded syslog files. ([Issue #16])\n- Cannot process multi-file `.gz` files (only processes first stream found).\n ([Issue #8])\n- Cannot process multi-file `.xz` files (only processes first stream found).\n ([Issue #11])\n- Cannot process `.zip` archives ([Issue #39])\n- \u003cspan id=\"f1\"\u003e\u003csup\u003e\\[1\\]\u003c/sup\u003e\u003c/span\u003e ISO 8601\n - ISO 8601 forms recognized (using [ISO descriptive format])\n - `YYYY-MM-DDThh:mm:ss`\n - `YYYY-MM-DDThhmmss`\n - `YYYYMMDDThhmmss`\n (may use date-time separator character `'T'` or character blank space `' '`)\n - ISO 8601 forms not recognized:\n - Absent seconds\n - [_Ordinal dates_], i.e. \"day of the year\", format `YYYY-DDD`, e.g. `\"2022-321\"`\n - [_Week dates_], i.e. \"week-numbering year\", format `YYYY-Www-D`, e.g. `\"2022-W25-1\"`\n - times [without minutes and seconds] (i.e. only `hh`)\n- \u003cspan id=\"f2\"\u003e\u003csup\u003e\\[2\\]\u003c/sup\u003e\u003c/span\u003e Cannot process archive files or compressed files within\n other archive files or compressed files ([Issue #14])\u003cbr/\u003e\n e.g. cannot process `logs.tar.xz`, nor file `log.gz` within `logs.tar`\n\n[Issue #8]: https://github.com/jtmoon79/super-speedy-syslog-searcher/issues/8\n[Issue #11]: https://github.com/jtmoon79/super-speedy-syslog-searcher/issues/11\n[Issue #14]: https://github.com/jtmoon79/super-speedy-syslog-searcher/issues/14\n[Issue #12]: https://github.com/jtmoon79/super-speedy-syslog-searcher/issues/12\n[Issue #39]: https://github.com/jtmoon79/super-speedy-syslog-searcher/issues/39\n[Issue #86]: https://github.com/jtmoon79/super-speedy-syslog-searcher/issues/86\n[ISO descriptive format]: https://en.wikipedia.org/w/index.php?title=ISO_8601\u0026oldid=1114310323#Calendar_dates\n[_Ordinal dates_]: https://en.wikipedia.org/w/index.php?title=ISO_8601\u0026oldid=1114310323#Ordinal_dates\n[_Week dates_]: https://en.wikipedia.org/w/index.php?title=ISO_8601\u0026oldid=1114310323#Week_dates\n[without minutes and seconds]: https://en.wikipedia.org/w/index.php?title=ISO_8601\u0026oldid=1114310323#Times\n\n### Hacks\n\n- Entire `.bz2` files are read once before processing ([Issue #300])\n- Entire `.lz4` files are read once before processing ([Issue #293])\n- Entire `.xz` files are read into memory before printing ([Issue #12])\n- Entire `.evtx` files are read into memory before printing ([Issue #86])\n- Entire files within a `.tar` file are read into memory before printing ([Issue #13])\n- Entire [user accounting record files are read into memory] before printing\n- Compressed `.journal` and `.evtx` files are extracted to a temporary file ([Issue #284])\n\n[user accounting record files are read into memory]: https://docs.rs/super_speedy_syslog_searcher/0.6.70/s4lib/readers/fixedstructreader/struct.FixedStructReader.html#summary-of-operation\n[Issue #13]: https://github.com/jtmoon79/super-speedy-syslog-searcher/issues/13\n[Issue #284]: https://github.com/jtmoon79/super-speedy-syslog-searcher/issues/284\n[Issue #293]: https://github.com/jtmoon79/super-speedy-syslog-searcher/issues/293\n[Issue #300]: https://github.com/jtmoon79/super-speedy-syslog-searcher/issues/300\n\n\u003cbr/\u003e\n\n---\n\n## More\n\n### Comparisons\n\nAn overview of features of varying log mergers including GNU tools.\n\n- GNU `grep` piped to GNU `sort`\n- _Super Speedy Syslog Searcher_; `s4`\n- [_lnav_](https://github.com/tstack/lnav); `lnav`\n- [_logmerger_](https://github.com/ptmcg/logmerger); `logmerger`\n- [_Toolong_](https://github.com/Textualize/toolong); `tl`\n- [_logdissect_](https://github.com/dogoncouch/logdissect); `logdissect.py`\n\n|Symbol| |\n|- |-|\n|βœ” |_Yes_ |\n|⬀ |_Most_ |\n|β—’ |_Some_ |\n|βœ— |_No_ |\n|☐ |_with an accompanying GNU program_ |\n|! |_with user input_ |\n|β€Ό |_with complex user input_ |\n\n---\n\n#### General Features\n\n|Program |Source|CLI|TUI|Interactive|live tail|merge varying log formats|datetime search range|\n|- |- |- |- |- |- |- |- |\n|`grep \\| sort` |C |βœ” |βœ— |βœ— |☐ `tail`|βœ— |β€Ό |\n|`s4` |Rust |βœ” |βœ— |βœ— |βœ— |βœ” |βœ” |\n|`lnav` |C++ |βœ” |βœ” |βœ” |βœ” |βœ” |β€Ό |\n|`logmerger` |Python|βœ” |βœ” |βœ” |βœ— |β€Ό |βœ” |\n|`tl` |Python|βœ” |βœ” |βœ” |βœ” |βœ— |βœ— |\n|`logdissect.py`|Python|βœ” |βœ— |βœ— |βœ— |βœ— |βœ— |\n\n---\n\n#### Formal Log DateTime Supported\n\n|Program |RFC 2822|RFC 3164|RFC 3339|RFC 5424|ISO 8601|\n|- |- |- |- |- |- |\n|`grep \\| sort` |βœ— |β€Ό |! |! |! |\n|`s4` |βœ” |βœ” |βœ” |βœ” |βœ” |\n|`lnav` |β€Ό |βœ” |βœ” |βœ” |βœ” |\n|`logmerger` |βœ— |βœ— |! |! |β—’ |\n|`tl` |βœ— |βœ— |βœ” |βœ” |βœ” |\n\n\u003c!--\n TODO: confirm lnav does not or does not _merge_ these RFC \"out of the box\"\n--\u003e\n\n\u003csup style=\"font-size: xx-small\"\u003eβ€’ [RFC 2822]: _Internet Message Format: Date and Time Specification_; e.g. _Wed, 1 Jan 2020 22:00:00 PST message…_\u003c/sup\u003e\n\n\u003csup style=\"font-size: xx-small\"\u003eβ€’ [RFC 3164]: _The BSD syslog Protocol: HEADER Part of a syslog Packet_; e.g. _\\\u003c8\\\u003eJan 1 22:00:00 message…_\u003c/sup\u003e\n\n\u003csup style=\"font-size: xx-small\"\u003eβ€’ [RFC 3339]: _Date and Time on the Internet: Internet Date/Time Format_; e.g. e.g. _2020-01-01T22:00:00-08:00 message…_\u003c/sup\u003e\n\n\u003csup style=\"font-size: xx-small\"\u003eβ€’ [RFC 5424]: _The Syslog Protocol: TIMESTAMP_; e.g. _2020-01-01T22:00:00-08:00 message…_\u003c/sup\u003e\n\n\u003csup style=\"font-size: xx-small\"\u003eβ€’ [ISO 8601]: _Data elements and interchange formats – Information interchange – Representation of dates and times_; e.g. _2020-01-01T22:00:00-08:00 message…_, _20200101T220000-0800 message…_, etc.\u003c/sup\u003e\n\n[RFC 2822]: https://www.rfc-editor.org/rfc/rfc2822#section-3.3\n[RFC 3164]: https://www.rfc-editor.org/rfc/rfc3164#section-4.1.2\n[RFC 3339]: https://www.rfc-editor.org/rfc/rfc3339#section-5.8\n[RFC 5424]: https://www.rfc-editor.org/rfc/rfc5424#section-6.2.3\n\n\u003c!--\nXXX: I could not get `logdissect.py` to work for any \"parser\" for any standard RFC log file.\n```bash\n for logfile in ./logs/standards/*.log ; do\n for lp in ciscoios emerge linejson sojson syslog syslogiso syslognohost tcpdump webaccess windowsrsyslog ; do\n (set -x;\n logdissect -p $lp $logfile) 2\u003e/dev/null\n done\n done\n```\n--\u003e\n\n---\n\n#### Other Log or File Formats Supported\n\nBinary formats supported:\n\n|Program |journal|`acct`/`lastlog`/`utmp`|`.evtx`|`.pcap`/`.pcapng`|`.jsonl`|\n|- |- |- |- |- |- |\n|`grep \\| sort` |βœ— |βœ— |βœ— |βœ— |βœ— |\n|`s4` |βœ” |βœ” |βœ” |[βœ—](https://github.com/jtmoon79/super-speedy-syslog-searcher/issues/255)|βœ” |\n|`lnav` |βœ— |βœ— |βœ— |βœ” |βœ— |\n|`logmerger` |βœ— |βœ— |βœ— |βœ” |βœ— |\n|`tl` |βœ— |βœ— |βœ— |βœ— |βœ” |\n\nAd-hoc text formats:\n\n|Program |Ad-hoc text formats|Red Hat Audit Log|strace|Apache Common Log Format|\n|- |- |- |- |- |\n|`grep \\| sort` |β€Ό |! |βœ” |β€Ό |\n|`s4` |βœ” |βœ” |βœ” |βœ” |\n|`lnav` |β€Ό |β€Ό |β€Ό |βœ” |\n|`logmerger` |β€Ό |β€Ό |βœ” |β€Ό |\n|`tl` |βœ— |βœ— |βœ— |βœ” |\n\nAll programs besides `s4` fail to merge different text log formats.\n\n---\n\n#### Archive Formats Supported\n\n|Program |`.gz` |`.lz` |`.lz4` |`.bz` |`.bz2` |`.xz` |`.tar`|`.zip`|\n|- |- |- |- |- |- |- |- |- |\n|`grep \\| sort` |☐ `zgrep`|☐ `lz`|☐ `lz4` |☐ `bzip`|☐ `bzip2` |☐ `xz` |βœ— |βœ— |\n|`s4` |βœ” |βœ— |βœ” |βœ— |βœ” |βœ” |βœ” |[βœ—](https://github.com/jtmoon79/super-speedy-syslog-searcher/issues/39)|\n|`lnav` |βœ” |βœ” |βœ” |? |βœ” |βœ” |βœ” |βœ” |\n|`logmerger` |βœ” |βœ— |βœ— |βœ— |βœ— |βœ— |βœ— |βœ— |\n|`tl` |βœ” |βœ— |βœ— |βœ” |βœ” |βœ— |βœ— |βœ— |\n|`logdissect.py`|βœ” |βœ— |βœ— |βœ— |βœ— |βœ— |βœ— |βœ— |\n\n---\n\n#### Speed Comparison\n\nA comparison of merging three large log files running on Ubuntu 22 on WSL2.\nThe three log files have 5000 lines, 2158138 bytes (β‰ˆ2.1 MB) each, with high-plane unicode.\nEach program had 30 runs except `toolong`.\n\n|Command |Mean (ms) |Min (ms) |Max (ms) |Max RSS (KB)|CPU %|\n|:--- |---: |---: |---: |---: |---: |\n|`grep \\| sort` |16.5 Β± 0.6 |15.7 |18.6 |5512 |41% |\n|`s4 (system)` |37.0 Β± 1.8 |34.3 |40.9 |48060 |182% |\n|`s4 (jemalloc)`|37.2 Β± 2.0 |33.9 |43.0 |71536 |165% |\n|`s4 (mimalloc)`|32.0 Β± 2.1 |27.4 |36.1 |75776 |182% |\n|`lnav` |155.9 Β± 1.8 |153.0 |162.7 |37320 |94% |\n|`logmerger` |779.3 Β± 10.4 |760.3 |803.2 |55288 |99% |\n|`toolong` | | | |53208 |40% |\n\n\u003csup style=\"font-size: xx-small\"\u003eβ€’ _Mean_ is mean runtime in milliseconds\u003c/sup\u003e\n\u003csup style=\"font-size: xx-small\"\u003eβ€’ _Min_ is minimum runtime in milliseconds\u003c/sup\u003e\n\u003csup style=\"font-size: xx-small\"\u003eβ€’ _Max_ is maximum runtime in milliseconds\u003c/sup\u003e\n\u003csup style=\"font-size: xx-small\"\u003eβ€’ _Max RSS_ is maximum Resident Set Size in Kilobytes\u003c/sup\u003e\n\u003csup style=\"font-size: xx-small\"\u003eβ€’ _CPU %_ is an average of CPU used over the runtime\u003c/sup\u003e\n\nPrograms tested:\n\n- GNU `grep` 3.7, GNU `sort` 8.32\n- `s4` 0.7.75\n- `logmerger` 0.9.0 on Python 3.10.12\n- `tl` 1.5.0 on Python 3.10.12\n\nUsing `hyperfine` to measure timing and GNU `time` to measure RSS and CPU.\n\nSee directory results in [`compare-log-mergers.txt`].\n\n[`compare-log-mergers.txt`]: https://github.com/jtmoon79/super-speedy-syslog-searcher/tree/0.7.75/releases/0.7.75\n\n---\n\n### Building locally\n\nSee section [_Install `super_speedy_syslog_searcher`_].\n\n[_Install `super_speedy_syslog_searcher`_]: #install-super_speedy_syslog_searcher\n\n### Parsing `.journal` files\n\nRequires `libsystemd` to be installed to use `libsystemd.so` at runtime.\n\n### Requesting Support For DateTime Formats; your particular log file\n\nIf you have found a log file that _Super Speedy Syslog Searcher_ does not parse\nthen you may create a [new Issue type _Feature request (datetime format)_].\n\nHere is [an example user-submitted Issue].\n\n[new Issue type _Feature request (datetime format)_]: https://github.com/jtmoon79/super-speedy-syslog-searcher/issues/new/choose\n[an example user-submitted Issue]: https://github.com/jtmoon79/super-speedy-syslog-searcher/issues/81\n\n### \"syslog\" and other project definitions\n\n#### syslog\n\nIn this project, the term \"_syslog_\" is used generously to refer to any\nlog message that has a datetime stamp on the first line of log text.\n\nTechnically, \"_syslog_\" is [defined among several RFCs]\nproscribing fields, formats, lengths, and other technical constraints.\nIn this project, the term \"_syslog_\" is interchanged with \"_log_\".\n\nThe term \"_sysline_\" refers to a one log message which may comprise\nmultiple text lines.\n\nSee [docs section _Definitions of data_] for more project definitions.\n\n[defined among several RFCs]: https://en.wikipedia.org/w/index.php?title=Syslog\u0026oldid=1219545533#Internet_standard_documents\n[docs section _Definitions of data_]: https://docs.rs/super_speedy_syslog_searcher/latest/s4lib/data/index.html\n\n#### log message\n\nA \"log message\" is a single log entry for any type of logging scheme;\nan entry in a utmpx file, an entry in a systemd journal, an entry in a\nWindows Event Log, a formal RFC 5424 syslog message, or an ad-hoc log message.\n\n---\n\n## logging chaos: the problem `s4` solves\n\nIn practice, most log file formats are an ad-hoc format. And among formally\ndefined log formats, there are many variations. The result is merging varying\nlog messages by datetime is prohibitively tedious.\nIf an engineer is investigating a problem that is symptomatic among many log\nfiles then the engineer must \"hunt and peck\" among those many log files.\nLog files can not be merged for a single coherent view.\n\nThe following real-world example log files are available in project directory\n`./logs`.\n\n### open-source software examples\n\n#### nginx webserver\n\nFor example, the open-source nginx web server\n[logs access attempts in an ad-hoc format] in the file `access.log`\n\n```text\n192.168.0.115 - - [08/Oct/2022:22:26:35 +0000] \"GET /DOES-NOT-EXIST HTTP/1.1\" 404 0 \"-\" \"curl/7.76.1\" \"-\"\n```\n\nwhich is an entirely dissimilar log format to the neighboring nginx log file,\n`error.log`\n\n```text\n2022/10/08 22:26:35 [error] 6068#6068: *3 open() \"/usr/share/nginx/html/DOES-NOT-EXIST\" failed (2: No such file or directory), client: 192.168.0.115, server: _, request: \"GET /DOES-NOT-EXIST HTTP/1.0\", host: \"192.168.0.100\"\n```\n\nnginx is following the bad example set by the apache web server.\n\n#### Debian 11\n\nHere are log snippets from a Debian 11 host.\n\nfile `/var/log/alternatives.log`\n\n```text\nupdate-alternatives 2022-10-10 23:59:47: run with --quiet --remove rcp /usr/bin/ssh\n```\n\nfile `/var/log/dpkg.log`\n\n```text\n2022-10-10 15:15:02 upgrade gpgv:amd64 2.2.27-2 2.2.27-2+deb11u1\n```\n\nfile `/var/log/kern.log`\n\n```text\nOct 10 23:07:16 debian11-b kernel: [ 0.10034] Linux version 5.10.0-11-amd64\n```\n\nfile `/var/log/unattended-upgrades/unattended-upgrades-shutdown.log`\n\n```text\n2022-10-10 23:07:16,775 WARNING - Unable to monitor PrepareForShutdown() signal, polling instead.\n```\n\n#### binary files\n\nAnd then there are binary files, such as the `wtmp` file on Linux and other\nUnix Operating Systems.\nUsing tool `utmpdump`, a `utmp` record structure is converted to text like:\n\n```text\n[7] [12103] [ts/0] [user] [pts/0] [172.1.2.1] [172.1.2.2] [2023-03-05T23:12:36,270185+00:00]\n```\n\nAnd from a _systemd_ `.journal` file, read using `journalctl`\n\n```text\nMar 03 10:26:10 host systemd[1]: Started OpenBSD Secure Shell server.\nβ–‘β–‘ Subject: A start job for unit ssh.service has finished successfully\nβ–‘β–‘ Defined-By: systemd\nβ–‘β–‘ Support: http://www.ubuntu.com/support\nβ–‘β–‘\nβ–‘β–‘ A start job for unit ssh.service has finished successfully.\nβ–‘β–‘\nβ–‘β–‘ The job identifier is 120.\nMar 03 10:31:23 host sshd[4559]: Accepted login for user1 from 172.1.2.1 port 51730 ssh2\n```\n\nTry merging those two log messages by datetime using GNU `grep`, `sort`, `sed`,\nor `awk`!\n\nAdditionally, if the `wtmp` file is from a different architecture\nor Operating System, then the binary record structure is likely not parseable\nby the resident `utmpdump` tool. What then!?\n\n### commercial software examples\n\nCommercial software and computer hardware vendors nearly always use\nad-hoc log message formatting that is even more unpredictable among each log\nfile on the same system.\n\n#### Synology DiskStation\n\nHere are log file snippets from a Synology DiskStation host.\n\nfile `DownloadStation.log`\n\n```text\n2019/06/23 21:13:34\t(system) trigger DownloadStation 3.8.13-3519 Begin start-stop-status start\n```\n\nfile `sfdisk.log`\n\n```text\n2019-04-06T01:07:40-07:00 dsnet sfdisk: Device /dev/sdq change partition.\n```\n\nfile `synobackup.log`\n\n```text\ninfo\t2018/02/24 02:30:04\tSYSTEM:\t[Local][Backup Task Backup1] Backup task started.\n```\n\n(yes, those are tab characters)\n\n#### Mac OS 12\n\nHere are log file snippets from a Mac OS 12.6 host.\n\nfile `/var/log/system`\n\n```text\nOct 11 15:04:55 localhost syslogd[110]: Configuration Notice:\n\tASL Module \"com.apple.cdscheduler\" claims selected messages.\n\tThose messages may not appear in standard system log files or in the ASL database.\n```\n\nfile `/var/log/wifi`\n\n```text\nThu Sep 21 23:05:35.850 Usb Host Notification NOT activated\n```\n\nfile `/var/log/fsck_hs.log`\n\n```text\n/dev/rdisk2s2: fsck_hfs started at Thu Sep 21 21:31:05 2023\n QUICKCHECK ONLY; FILESYSTEM CLEAN\n```\n\nfile `/var/log/anka.log`\n\n```text\nFri Sep 22 00:06:05 UTC 2023: Checking /Library/Developer/CoreSimulator/Volumes/watchOS_20S75...\n```\n\nfile `/var/log/displaypolicyd.log`\n\n```text\n2023-09-15 04:26:56.330256-0700: Started at Fri Sep 15 04:26:56 2023\n```\n\nfile `/var/log/com.apple.xpc.launchd/launchd.log.1`\n\n```text\n2023-10-26 16:56:23.287770 \u003cNotice\u003e: swap enabled\n```\n\nfile `/var/log/asl/logs/aslmanager.20231026T170200+00`\n\n```text\nOct 26 17:02:00: aslmanager starting\n```\n\nDid you also notice how the log file _names_ differ in unexpected ways?\n\n#### Microsoft Windows 10\n\nHere are log snippets from a Windows 10 host.\n\nfile `${env:SystemRoot}\\debug\\mrt.log`\n\n```text\nMicrosoft Windows Malicious Software Removal Tool v5.83, (build 5.83.13532.1)\nStarted On Thu Sep 10 10:08:35 2020\n```\n\nfile `${env:SystemRoot}\\comsetup.log`\n\n```text\nCOM+[12:24:34]: ********************************************************************************\nCOM+[12:24:34]: Setup started - [DATE:05,27,2020 TIME: 12:24 pm]\n```\n\nfile `${env:SystemRoot}\\DirectX.log`\n\n```text\n11/01/19 20:03:40: infinst: Installed file C:\\WINDOWS\\system32\\xactengine2_1.dll\n```\n\nfile `${env:SystemRoot}/Microsoft.NET/Framework/v4.0.30319/ngen.log`\n\n```text\n09/15/2022 14:13:22.951 [515]: 1\u003eWarning: System.IO.FileNotFoundException: Could not load file or assembly\n```\n\nfile `${env:SystemRoot}/Performance/WinSAT/winsat.log`\n\n```text\n68902359 (21103) - exe\\logging.cpp:0841: --- START 2022\\5\\17 14:26:09 PM ---\n68902359 (21103) - exe\\main.cpp:4363: WinSAT registry node is created or present\n```\n\n(yes, it reads hour `14`, and `PM`… πŸ™„)\n\n### Summary\n\nThis chaotic logging approach is typical of commercial and open-source software,\n__*AND IT'S A MESS!*__\nAttempting to merge log messages by their natural sort mechanism,\na datetime stamp, is difficult to impossible.\n\nHence the need for _Super Speedy Syslog Searcher_! 🦸\n`s4` merges varying log files into a single coherent datetime-sorted log.\n\n[logs access attempts in an ad-hoc format]: https://docs.nginx.com/nginx/admin-guide/monitoring/logging/#setting-up-the-access-log\n\n---\n\n## Further Reading\n\n- [`CHANGELOG.md`]\n- [`Extended-Thoughts.md`]\n\n[`CHANGELOG.md`]: ./CHANGELOG.md\n[`Extended-Thoughts.md`]: ./Extended-Thoughts.md\n\n## Stargazers\n\n[![Stargazers over time](https://starchart.cc/jtmoon79/super-speedy-syslog-searcher.svg?variant=adaptive)](https://starchart.cc/jtmoon79/super-speedy-syslog-searcher)\n\n---\n\n\u003ca href=\"https://stackexchange.com/users/216253/\"\u003e\n\u003cimg src=\"https://stackexchange.com/users/flair/216253.png\" width=\"208\" height=\"58\" alt=\"profile for @JamesThomasMoon on Stack Exchange, a network of free, community-driven Q\u0026amp;A sites\" title=\"profile for @JamesThomasMoon on Stack Exchange, a network of free, community-driven Q\u0026amp;A sites\" /\u003e\n\u003c/a\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjtmoon79%2Fsuper-speedy-syslog-searcher","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjtmoon79%2Fsuper-speedy-syslog-searcher","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjtmoon79%2Fsuper-speedy-syslog-searcher/lists"}