{"id":13520285,"url":"https://github.com/cmichi/check_timed_logs_fast","last_synced_at":"2025-04-07T07:32:54.307Z","repository":{"id":57549802,"uuid":"150306371","full_name":"cmichi/check_timed_logs_fast","owner":"cmichi","description":"A blazingly fast Rust reimplementation of the nagios plugin.","archived":false,"fork":false,"pushed_at":"2020-09-02T14:39:18.000Z","size":616,"stargazers_count":2,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-22T15:02:02.692Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cmichi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-09-25T17:49:24.000Z","updated_at":"2019-04-23T14:31:09.000Z","dependencies_parsed_at":"2022-09-26T18:41:44.389Z","dependency_job_id":null,"html_url":"https://github.com/cmichi/check_timed_logs_fast","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cmichi%2Fcheck_timed_logs_fast","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cmichi%2Fcheck_timed_logs_fast/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cmichi%2Fcheck_timed_logs_fast/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cmichi%2Fcheck_timed_logs_fast/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cmichi","download_url":"https://codeload.github.com/cmichi/check_timed_logs_fast/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247612565,"owners_count":20966775,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T05:02:16.568Z","updated_at":"2025-04-07T07:32:52.600Z","avatar_url":"https://github.com/cmichi.png","language":"Rust","funding_links":[],"categories":["Rust"],"sub_categories":[],"readme":"# check_timed_logs_fast\n\n[![Build Status](https://travis-ci.org/cmichi/check_timed_logs_fast.svg?branch=master)](https://travis-ci.org/cmichi/check_timed_logs_fast)\n[![codecov](https://codecov.io/gh/cmichi/check_timed_logs_fast/branch/master/graph/badge.svg)](https://codecov.io/gh/cmichi/check_timed_logs_fast)\n[![Crates.io](https://img.shields.io/crates/v/check_timed_logs_fast.svg)](https://crates.io/crates/check_timed_logs_fast)\n[![docs.rs](https://docs.rs/check_timed_logs_fast/badge.svg)](https://docs.rs/check_timed_logs_fast)\n\nThis is a blazingly fast reimplementation of the [check_timed_logs](https://exchange.nagios.org/directory/Plugins/Log-Files/check_timed_logs/details)\nnagios plugin in Rust (the original is in Perl). The API stayed the same,\nso you can just replace the original perl script with the binary.\n\nThe purpose of the plugin is to monitor log files and alert if there\nare more than X occurrences of a regex in the last Y minutes (e.g. more\nthan one exception in the last minute or more than 5 warnings in the last\ntwo minutes).  \n\nThis rewrite was triggered by problems which occur with very large\nand verbose log files. The original plugin takes a long time for parsing\nand nagios times out after a few seconds of getting no reaction from a\ncheck — this then falsely shows up as a critical incident in\nmonitoring.\n\n\n## Performance\n\n| Log File Size     | 1.2M      | 37M        | 5.7G       |\n| ------------------|-----------|------------|----------- |\n| Original          | 0.878 sec | 20.287 sec | \u003e30 min    |\n| Rust Rewrite      | 0.031 sec | 0.676 sec  | 83.088 sec |\n| Improvement       | 96.4 %    | 96.6 %     |            |\n\nThese metrics provide a rough idea, I haven't looked in detail at the\nexact difference in RAM usage (it doesn't seem to have increased\nthough). The performance is also dependent on the complexity of the\nregular expression.\n\nI did the benchmarks using the following command on a high performance server\n(which shouldn't really matter since only one core is utilized anyway and the\nRAM fingerprint is low).\n\n\tperf stat\n\t\t-r 10\n\t\t-d ./check_timed_logs_fast -pattern '.*nonExistentPattern.*' -i 9999999 -c 1 -logfile ./log\n\nThe command executs the check ten times and parses the entire file, the\nresulting average execution time is the duration in the table above.\n\nThe crazy rate of improvement comes from Rust and using `memmap` to read the\nfile backwards. At the moment the implementation is pretty straight forward\n— one process which blocks with the i/o operations and the parsing.\nI suspect that there is room for more improvement and would like to implement\ntwo additional strategies:\n\n1. split work into worker threads\n2. asynchronous processing\n\nFurthermore, I know for sure (because I benchmarked it) that the `fancy-regex`\ncrate is a slowing factor. The `regex` crate had better performance, but doesn't\nsupport advanced regex features like look-ahead. The original `check_timed_logs`\nscript supports these features and since I want to stay compatible, I have to\nuse a (slower) crate which supports these features.\n\n\n## Installation/Usage\n\n\tcargo install check_timed_logs_fast\n\n\t# a warning should be issued if there is \u003e= 1 occurrence of either the string\n\t# \"timeout\" or \"closed\" in the last ten minutes. if there are \u003e= 5 matches\n\t# issue a critical incident.\n\tcheck_timed_logs_fast -logfile /var/log/app.log -pattern \"timeout|closed\" -interval 10 -w 1 -c 5\n\nYou can use MUSL to compile a generic, static binary for some unknown linux:\n\n\trustup target add x86_64-unknown-linux-musl\n\tcargo build --release --target x86_64-unknown-linux-musl\n\n\n## Building a Debian package\n\nBuild the package like this:\n\nClone and compile with `cargo build --release --target=x86_64-unknown-linux-musl`.\nThe resulting executable is `target/x86_64-unknown-linux-musl/release/check_timed_logs_fast`.\nIf you `cd target/x86_64-unknown-linux-musl/release/` you can build the debian\npackage with this `fpm` command:\n\n\tfpm -s dir \\\n\t\t-t deb \\\n\t\t--iteration 1plugins1 \\\n\t\t--architecture all \\\n\t\t--deb-ignore-iteration-in-dependencies \\\n\t\t--maintainer \"Michael Mueller \u003cmich@elmueller.net\u003e\" \\\n\t\t--name check_timed_logs_fast \\\n\t\t--verbose \\\n\t\t--version 0.0.8 \\\n\t\t./check_timed_logs_fast=/usr/lib/nagios/plugins/check_timed_logs_fast\n\n\n## License\n\n\tCopyright (c)\n\n\t\t2018 Michael Mueller, http://micha.elmueller.net/\n\n\tPermission is hereby granted, free of charge, to any person obtaining\n\ta copy of this software and associated documentation files (the\n\t\"Software\"), to deal in the Software without restriction, including\n\twithout limitation the rights to use, copy, modify, merge, publish,\n\tdistribute, sublicense, and/or sell copies of the Software, and to\n\tpermit persons to whom the Software is furnished to do so, subject to\n\tthe following conditions:\n\n\tThe above copyright notice and this permission notice shall be\n\tincluded in all copies or substantial portions of the Software.\n\n\tTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND,\n\tEXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF\n\tMERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND\n\tNONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE\n\tLIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION\n\tOF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION\n\tWITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcmichi%2Fcheck_timed_logs_fast","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcmichi%2Fcheck_timed_logs_fast","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcmichi%2Fcheck_timed_logs_fast/lists"}