{"id":16538050,"url":"https://github.com/multicatch/java-logger-benchmark","last_synced_at":"2025-03-03T21:10:38.902Z","repository":{"id":137900624,"uuid":"380457128","full_name":"multicatch/java-logger-benchmark","owner":"multicatch","description":"A simple benchmark of popular Java logging libraries with a report of results","archived":false,"fork":false,"pushed_at":"2021-06-26T08:57:34.000Z","size":733,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-01-14T05:11:30.625Z","etag":null,"topics":["benchmark","comparison","java","jmh","logging","report"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/multicatch.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-06-26T08:52:21.000Z","updated_at":"2021-06-29T20:52:25.000Z","dependencies_parsed_at":null,"dependency_job_id":"aa6144f8-3f05-4810-be6d-0a7c9cec3e13","html_url":"https://github.com/multicatch/java-logger-benchmark","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/multicatch%2Fjava-logger-benchmark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/multicatch%2Fjava-logger-benchmark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/multicatch%2Fjava-logger-benchmark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/multicatch%2Fjava-logger-benchmark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/multicatch","download_url":"https://codeload.github.com/multicatch/java-logger-benchmark/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241739430,"owners_count":20012103,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark","comparison","java","jmh","logging","report"],"created_at":"2024-10-11T18:44:18.449Z","updated_at":"2025-03-03T21:10:38.875Z","avatar_url":"https://github.com/multicatch.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Java Logging Benchmark\n\nA simple benchmark of some Java logging libraries.\n\n## Introduction\n\nThe purpose of this project is to benchmark the most popular Java logging libraries.\nLogging is one of the most important parts of modern software, \nas it allows to easily track the significant ongoing processes in a running piece of software.\n\nIf logging is done well enough, it usually provides enough information to reproduce or fix a bug without excessive debugging.\nApart from debugging, logging serves other purposes:\n* usage statistics and analytics,\n* system auditing,\n* resource monitoring.\n\nHowever, logging is a tool that is required depending on context and current needs.\nAdministrators should be free to opt out of some logs if they serve no significant purpose,\nand they should be able to control how detailed the messages should be.\n\nBecause of this, logging should not be a bottleneck of a production grade system.\nIt should be able to provide as much information as it can possibly provide depending on a context,\nand it should not have any negative impact on any algorithms and logic.\n\n## Logger Transparency\n\nLogging should be as transparent as possible to ensure that it does not cause any performance or stability issues in the long run.\nIf logging is causing any side effects or issues, \nit is likely that some programmers will decide not to log some events to avoid such problems.\n\nLogging usually proves to be a useful tool if used well. \nEven some less descriptive messages might give a lot of context to the internal processes of a system, \nbut it should be easy to filter them out.\nToo many messages might cause \"information noise\", which impacts the readability and usefulness of logs.\n\nIf the logs cannot be effectively filtered and are unreadable, then some people decide to turn them off,\nwhich defeats the purpose of logs in the first place.\nIt is also necessary to organize logging in a way that it is easy to filter logs or spot important events \n(e.g. severe errors or data inconsistency).\n\nThis project, however, does not aim to show all good practices of how to implement logging.\nThe purpose of this project is to show data that can help with the decision of choosing a logging library.\nAs logging should be as transparent as possible, the impact on performance is very important.\n\n## Overview of Java Logging Libraries\n\nThe most commonly used libraries for logging in Java are Logback and Log4j. \nBoth are usually used with Slf4j, which is an API for logging that allows to easily switch between implementations that handle logging.\nFor example, thanks to Slf4j, it is very easy to switch between Logback, Log4j and JUL (Java Util Logging) without major code changes. \n\nThis project includes benchmarks for the following libraries:\n* Java Util Logging (JUL) - it is the standard logging library of Java.\n  It is included with every Java release since JDK 1.4.\n  Despite being a logging mechanism provided by a standard library, it is not popular and used very rarely.\n* [Logback](https://logback.qos.ch/) - the project page says that \"Logback is intended as a successor to the popular log4j project\".\n  It is actually a bit different from current Log4j release, albeit shares some similarities with it.\n  Overall Logback is a very popular choice and is more flexible than JUL, but no new stable release has been published for 4 years \n  (as of 2021-06-24 the latest stable is 1.2.3, published in Mar, 2017).\n* [Log4j 2](https://logging.apache.org/log4j/2.x/) - it is an Apache project, which is an improved version of Log4j. \n  This is a library that is actually more mature than Logback, because of the longer development history.\n  It is also very configurable and commonly used. \n  The another benefit is that it is still actively maintained (as of 2021-06-24 there were 3 releases in the past year).\n\n## The Benchmark Setup\n\nThis benchmark was prepared to measure how fast the logging is. \nIt contains the typical use cases:\n* logging a message without any parameters,\n* logging a message with parameters,\n* logging an exception.\n\nTo reduce side effects caused by IO which can affect the results, \nthe loggers are configured to use a custom OutputStream that does nothing.\nThis OutputStream is called *BlackholeOutputStream* and is written in a way \nthat it satisfies the following needs:\n* it does not do any IO,\n* it does not any CPU or memory intensive operations,\n* it does not block or slow down any thread,\n* JIT compiler doesn't see it as dead code.\n\nIf the first three requirements are not satisfied, \nthen the results would be worse not because of how the libraries work, \nbut because of this OutputStream, which is not the subject of our benchmark.\n\nThe last requirement is important as if the OutputStream did not actually do anything,\nthen the JIT compiler would notice that after a few iterations of `write` method.\nIt would mark this as *dead code* and skip whole call stack in the future iterations \nas running code that does nothing and has no side effects is redundant. \nThis means that the results would be better than the actual performance of a logging library.\n\nTo run and measure the logging performance, a framework named [Java Microbenchmark Harness (JMH)](https://github.com/openjdk/jmh) was used.\nThis project builds a fat jar that can be run independently. \nIt is important to run this as a standalone app, \nas this allows to stop any redundant background programs and processes which may affect the available resources of the machine\nand affect the results.\n\nEach library has 4 following benchmarks:\n* logging a simple message (`...LoggerWithoutParams`) - tests how the library performs when appending a message to an OutputStream without the need to parse the message,\n* logging a message with parameter evaluation (`...Logger`) - tests how the library performs when parsing a message and replacing placeholders with values,\n* logging a throwable (`...LoggerThrowable`) - tests how the library performs when formatting a stack trace,\n* logging with a disabled logging level (`...LoggerDisabled`) - tests how the library performs when there is no need to log the message \n  (i.e. minimum level is INFO, but message was logged with TRACE or FINEST level).\n\n## The Results\n\nThe benchmarks were run on a MacBookPro8,2 with Intel(R) Core(TM) i7-2720QM CPU @ 2.20GHz, 16 GB of RAM and Java 11.\nResults may vary between runs and between different machines and current run environment.\n\n```\njava --version\njava 11.0.10 2021-01-19 LTS\nJava(TM) SE Runtime Environment 18.9 (build 11.0.10+8-LTS-162)\nJava HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.10+8-LTS-162, mixed mode)\n```\n\nResults of the benchmarks were saved as JSON and TXT files, \nwhere the JSON files were generated by JMH and TXT files contain the program output.\nThe files are available in the [results](results) directory of this repository.\n\nJSON results can be visualized with [JMH Visualizer](https://jmh.morethan.io/). \nThanks to this tool, we can see the throughput summary as the following bar chart.\n\n![Bar chart of throughput benchmark results](results/throughput.png)\n\nWhat is the most noticeable is the fact that all 3 libraries performed very well \nwhen a message was logged with a level lower than the minimum configured level (the `...LoggerDisabled` benchmarks).\nNumber in the range of 433k-812k ops/ms suggest that the code was optimized by JIT compiler - using the aforementioned dead code elimination mechanism.\n\nThe effects of dead code elimination mechanism can be seen by comparing the performance of loggers during the warmups -\nit was less stable as JIT compiler was \"learning\" what code is redundant. \nFirst iteration may be slower, as some invocations are not optimized yet \nand later warmups are faster as dead code elimination was applied.\n\n```text\n# Benchmark: io.github.multicatch.LoggerBenchmark.javaLoggerDisabled\n\n# Run progress: 8,33% complete, ETA 00:27:41\n# Fork: 1 of 1\n# Warmup Iteration   1: 737907,992 ops/ms\n# Warmup Iteration   2: 811645,680 ops/ms\n# Warmup Iteration   3: 810355,901 ops/ms\n# Warmup Iteration   4: 784816,809 ops/ms\n# Warmup Iteration   5: 806521,710 ops/ms\n```\n\nThis is a very good information as this suggests that those libraries do not cause a lot of side effects when logging\nand can be optimized by JIT compiler when needed.\n\nThe next thing to notice is that Log4j 2 has the best performance when logging messages (without any parameterization) - 2228 ops/ms.\nLogback performs almost as well (ca. 60% of Log4j 2 performance), but JUL performs almost 100 times worse.\n\nThe result of 23 operations/ms means that the performance of JUL allows only for 23 invocations of `logger.log(Level.INFO, \"Message\")` per millisecond,\nwhich may affect performance of the system if it relies on JUL too much (one invocation needed about 0.04 ms to complete). \nIO can slow this down more, but the most important is the fact that is has such poor performance because of its implementation only,\nwhich means that with IO it might be even slower.\n\nIt is worth mentioning that both Logback and Log4j perform almost as well when there is parameterization in the message (ca. 900 ops/ms).\nThe difference in performance is so little that it is irrelevant. \n\nThe fact that JUL performs almost the same with and without parameterization is interesting, \nbut it is still poor performance overall.\nIt seems that parameter substitution is done no matter how many parameters were supplied\nand that might slow it down some more.\n\nThe last thing to compare was how the loggers perform when a stack trace of a throwable is logged.\nJUL performs the worst, with score of 14 ops/ms. \nLogback performs the best (185 ops/ms), \nbut surprisingly, Log4j 2 is a lot slower - about 1/5 of Logback's performance in this case.\n\n## Conclusion\n\nThe matter of choosing the right logging library should be a matter of a developer - \nbased on their opinion on what API is the most comfortable to use, \nwhat are their needs and what is already used in a project.\nThis benchmark should be used as a suggestion and not as an ultimate argument for/against given library.\n\nThe measured throughput and times describe the internal implementation only, \nwithout any additional slowdowns from IO or external mechanisms.\n\nBased on these results, it is very clear that JUL is the slowest.\nThe benchmark does not show the direct cause of this, so it is not known why is that.\n\nLogback and Log4j 2 have almost comparable performance - \nwith Log4j 2 performing better in most cases except throwable logging.\nHowever, I believe that it should not be very significant unless the whole system flow is exception-based\n(which, in my opinion, is quite inefficient, as JVM handles exception throwing slower than a simple return).\nExceptions that indicate some problems usually happen less frequently than standard log messages.\n\nThe results are overall interesting, but the final conclusion about choosing the right tool for this job\nshould be up to a developer/team. \nBefore you make a decision, please make sure to review all [results](results).\n\n## Contributing\n\nIf you have seen any errors or see some room for improvement, please create an issue on GitHub or make a Pull Request.\nI can run benchmarks again on my machine for consistency if your change adds or modifies them.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmulticatch%2Fjava-logger-benchmark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmulticatch%2Fjava-logger-benchmark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmulticatch%2Fjava-logger-benchmark/lists"}