Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/vy/log4j2-logstash-layout

Log4j 2.x plugin for customizable and Logstash-friendly JSON layout.
https://github.com/vy/log4j2-logstash-layout

java json log4j log4j2 logging logstash

Last synced: about 1 month ago
JSON representation

Log4j 2.x plugin for customizable and Logstash-friendly JSON layout.

Awesome Lists containing this project

README

        

**`log4j2-logstash-layout` is not maintained anymore! Since Log4j 2.14.0, it is
superseded by `log4j-layout-template-json` shipping
[`JsonTemplateLayout`](https://logging.apache.org/log4j/2.x/manual/json-template-layout.html),
which is a successor of `LogstashLayout`. We strongly advise all `LogstashLayout`
users to migrate to `JsonTemplateLayout`. We will do our best to bring bug fixes
to `LogstashLayout`, but all the new development efforts will be focused on
`JsonTemplateLayout`.**

[![Actions Status](https://github.com/vy/log4j2-logstash-layout/workflows/CI/badge.svg)](https://github.com/vy/log4j2-logstash-layout/actions)
[![Maven Central](https://img.shields.io/maven-central/v/com.vlkan.log4j2/log4j2-logstash-layout-parent.svg)](https://search.maven.org/#search%7Cga%7C1%7Cg%3A%22com.vlkan.log4j2%22)
[![License](https://img.shields.io/github/license/vy/log4j2-logstash-layout.svg)](https://www.apache.org/licenses/LICENSE-2.0.txt)

`LogstashLayout` is **the fastest** [Log4j 2](https://logging.apache.org/log4j/2.x/)
JSON layout allowing schema customization and [Logstash](https://www.elastic.co/products/logstash)-friendly
output.

By default, `LogstashLayout` ships the official `JSONEventLayoutV1` stated by
[log4j-jsonevent-layout](https://github.com/logstash/log4j-jsonevent-layout)
Log4j 1.x plugin. Compared to
[JSONLayout](https://logging.apache.org/log4j/2.x/manual/layouts.html#JSONLayout)
included in Log4j 2 and `log4j-jsonevent-layout`, `LogstashLayout` provides
the following additional features:

- Superior [performance](#performance)
- Customizable JSON schema (see `eventTemplate[Uri]` and `stackTraceElementTemplate[Uri]` parameters)
- Customizable timestamp formatting (see `dateTimeFormatPattern` and `timeZoneId` parameters)

# Table of Contents

- [Usage](#usage)
- [Predefined Templates](#templates)
- [Features](#features)
- [Appender Support](#appender-support)
- [Performance](#performance)
- [F.A.Q.](#faq)
- [Contributors](#contributors)
- [License](#license)

# Usage

Add the `log4j2-logstash-layout` dependency to your POM file

```xml

com.vlkan.log4j2
log4j2-logstash-layout
${log4j2-logstash-layout.version}

```

in combination with a valid `log4j-core` dependency:

```xml

org.apache.logging.log4j
log4j-core
${log4j2.version}

```

(Note that the Java 9 module name is `com.vlkan.log4j2.logstash.layout`.)

Below you can find a sample `log4j2.xml` snippet employing `LogstashLayout`.

```xml










```

Or using the `log4j2.properties` file instead:

```ini
status = warn

appender.console.name = CONSOLE
appender.console.type = CONSOLE
appender.console.target = SYSTEM_OUT

appender.console.logstash.type = LogstashLayout
appender.console.logstash.dateTimeFormatPattern = yyyy-MM-dd'T'HH:mm:ss.SSSZZZ
appender.console.logstash.eventTemplateUri = classpath:LogstashJsonEventLayoutV1.json
appender.console.logstash.prettyPrintEnabled = true
appender.console.logstash.stackTraceEnabled = true

rootLogger.level = info
rootLogger.appenderRef.stdout.ref = CONSOLE
```

This generates an output as follows:

```json
{
"exception": {
"exception_class": "java.lang.RuntimeException",
"exception_message": "test",
"stacktrace": "java.lang.RuntimeException: test\n\tat com.vlkan.log4j2.logstash.layout.demo.LogstashLayoutDemo.main(LogstashLayoutDemo.java:11)\n"
},
"line_number": 12,
"class": "com.vlkan.log4j2.logstash.layout.demo.LogstashLayoutDemo",
"@version": 1,
"source_host": "varlik",
"message": "Hello, error!",
"thread_name": "main",
"@timestamp": "2017-05-25T19:56:23.370+02:00",
"level": "ERROR",
"file": "LogstashLayoutDemo.java",
"method": "main",
"logger_name": "com.vlkan.log4j2.logstash.layout.demo.LogstashLayoutDemo"
}
```

`LogstashLayout` is configured with the following parameters:

| Parameter Name | Type | Description |
|----------------|------|-------------|
| `prettyPrintEnabled` | boolean | enables pretty-printer (defaults to `false`) |
| `locationInfoEnabled` | boolean | includes the filename and line number in the output (defaults to `false`) |
| `stackTraceEnabled` | boolean | includes stack traces (defaults to `false`) |
| `emptyPropertyExclusionEnabled` | boolean | exclude empty and null properties (defaults to `false`) |
| `dateTimeFormatPattern` | String | timestamp formatter pattern (defaults to `yyyy-MM-dd'T'HH:mm:ss.SSSZZZ`) |
| `timeZoneId` | String | time zone id (defaults to `TimeZone.getDefault().getID()`) |
| `locale` | String | locale in one of the following forms: ``, `_`, or `__` (defaults to `Locale.getDefault()`) |
| `mdcKeyPattern` | String | regex to filter MDC keys (does not apply to direct `mdc:key` access) |
| `ndcPattern` | String | regex to filter NDC items |
| `eventTemplate` | String | inline JSON template for rendering `LogEvent`s (has priority over `eventTemplateUri`) |
| `eventTemplateUri` | String | JSON template for rendering `LogEvent`s (defaults to [`classpath:LogstashJsonEventLayoutV1.json`](layout/src/main/resources/LogstashJsonEventLayoutV1.json)) |
| `eventTemplateAdditionalFields`1 | KeyValuePair | additional key-value pairs appended to the root of the event template |
| `stackTraceElementTemplate` | String | inline JSON template for rendering `StackTraceElement`s (has priority over `stackTraceElementTemplateUri`) |
| `stackTraceElementTemplateUri` | String | JSON template for rendering `StackTraceElement`s (defaults to [`classpath:Log4j2StackTraceElementLayout.json`](layout/src/main/resources/Log4j2StackTraceElementLayout.json)) |
| `lineSeparator` | String | used to separate log outputs (defaults to `System.lineSeparator()`) |
| `maxByteCount` | int | used to cap the internal `byte[]` buffer used for serialization (defaults to 16 KiB) |
| `maxStringLength`2 | int | truncate string values longer than the specified limit (defaults to 0) |
| `objectMapperFactoryMethod` | String | custom object mapper factory method (defaults to `com.fasterxml.jackson.databind.ObjectMapper.new`) |
| `mapMessageFormatterIgnored` | boolean | as a temporary work around for [LOG4J2-2703](https://issues.apache.org/jira/browse/LOG4J2-2703), serialize `MapMessage`s using Jackson rather than `MapMessage#getFormattedMessage()` (defaults to `true`) |

1 One can configure additional event template fields as follows:

```xml




```

2 Note that string value truncation via `maxStringLength` can take
place both in object keys and values, and this operation does not leave any
trace behind. `maxStringLength` is intended as a soft protection against bogus
input and one should always rely on `maxByteCount` for a hard limit.

`eventTemplateUri` denotes the URI pointing to the JSON template that will be used
while formatting the `LogEvent`s. By default, `LogstashLayout` ships
[`LogstashJsonEventLayoutV1.json`](layout/src/main/resources/LogstashJsonEventLayoutV1.json)
providing [the official Logstash `JSONEventLayoutV1`](https://github.com/logstash/log4j-jsonevent-layout).

```json
{
"mdc": "${json:mdc}",
"ndc": "${json:ndc}",
"exception": {
"exception_class": "${json:exception:className}",
"exception_message": "${json:exception:message}",
"stacktrace": "${json:exception:stackTrace:text}"
},
"line_number": "${json:source:lineNumber}",
"class": "${json:source:className}",
"@version": 1,
"source_host": "${hostName}",
"message": "${json:message}",
"thread_name": "${json:thread:name}",
"@timestamp": "${json:timestamp}",
"level": "${json:level}",
"file": "${json:source:fileName}",
"method": "${json:source:methodName}",
"logger_name": "${json:logger:name}"
}
```

Similarly, `stackTraceElementUri` denotes the URI pointing to the JSON template
that will be used while formatting the `StackTraceElement`s. By default,
`LogstashLayout` ships [`classpath:Log4j2StackTraceElementLayout.json`](layout/src/main/resources/Log4j2StackTraceElementLayout.json)
providing an identical stack trace structure produced by Log4j 2 `JSONLayout`.

```json
{
"class": "${json:stackTraceElement:className}",
"method": "${json:stackTraceElement:methodName}",
"file": "${json:stackTraceElement:fileName}",
"line": "${json:stackTraceElement:lineNumber}"
}
```

In case of need, you can create your own templates with a structure tailored
to your needs. That is, you can add new fields, remove or rename existing
ones, change the structure, etc. Please note that `eventTemplateUri` parameter
only supports `file` and `classpath` URI schemes.

Below is the list of allowed `LogEvent` template variables that will be replaced
while rendering the JSON output.

| Variable Name | Description |
|---------------|-------------|
| `endOfBatch` | `logEvent.isEndOfBatch()` |
| `exception:className` | `logEvent.getThrown().getClass().getCanonicalName()` |
| `exception:message` | `logEvent.getThrown().getMessage()` |
| `exception:stackTrace` | `logEvent.getThrown().getStackTrace()` (inactive when `stackTraceEnabled=false`) |
| `exception:stackTrace:text` | `logEvent.getThrown().printStackTrace()` (inactive when `stackTraceEnabled=false`) |
| `exceptionRootCause:className` | the innermost `exception:className` in causal chain |
| `exceptionRootCause:message` | the innermost `exception:message` in causal chain |
| `exceptionRootCause:stackTrace[:text]` | the innermost `exception:stackTrace[:text]` in causal chain |
| `level` | `logEvent.getLevel()` |
| `level:severity` | [Syslog severity](https://en.wikipedia.org/wiki/Syslog#Severity_levels) keyword of `logEvent.getLevel()` |
| `level:severity:code` | [Syslog severity](https://en.wikipedia.org/wiki/Syslog#Severity_levels) code of `logEvent.getLevel()` |
| `logger:fqcn` | `logEvent.getLoggerFqcn()` |
| `logger:name` | `logEvent.getLoggerName()` |
| `main:` | performs [Main Argument Lookup](https://logging.apache.org/log4j/2.0/manual/lookups.html#AppMainArgsLookup) for the given `key` |
| `map:` | performs [Map Lookup](https://logging.apache.org/log4j/2.0/manual/lookups.html#MapLookup) for the given `key` |
| `marker:name` | `logEvent.getMarker.getName()` |
| `mdc` | Mapped Diagnostic Context `Map` returned by `logEvent.getContextData()` |
| `mdc:` | Mapped Diagnostic Context `String` associated with `key` (`mdcKeyPattern` is discarded) |
| `message` | `logEvent.getFormattedMessage()` |
| `message:json` | if `logEvent.getMessage()` is of type `MultiformatMessage` and supports JSON, its read value, otherwise, `{"message": }` object |
| `ndc` | Nested Diagnostic Context `String[]` returned by `logEvent.getContextStack()` |
| `source:className` | `logEvent.getSource().getClassName()` |
| `source:fileName` | `logEvent.getSource().getFileName()` (inactive when `locationInfoEnabled=false`) |
| `source:lineNumber` | `logEvent.getSource().getLineNumber()` (inactive when `locationInfoEnabled=false`) |
| `source:methodName` | `logEvent.getSource().getMethodName()` |
| `thread:id` | `logEvent.getThreadId()` |
| `thread:name` | `logEvent.getThreadName()` |
| `thread:priority` | `logEvent.getThreadPriority()` |
| `timestamp` | `logEvent.getTimeMillis()` formatted using `dateTimeFormatPattern` and `timeZoneId` |
| `timestamp:epoch` | epoch nanoseconds derived from `logEvent.getInstant()` |
| `timestamp:epoch:divisor=` | epoch nanoseconds derived from `logEvent.getInstant()` divided by provided `divisor` (of type `double`) |
| `timestamp:epoch:divisor=,integral` | epoch nanoseconds derived from `logEvent.getInstant()` divided by provided `divisor` (of type `double`) and casted to `long` |

JSON field lookups are performed using the `${json:}` scheme
where `` is defined as `[:]`.
Characters following colon (`:`) are treated as the `resolver-key`.

[Log4j 2 Lookups](https://logging.apache.org/log4j/2.0/manual/lookups.html)
(e.g., `${java:version}`, `${env:USER}`, `${date:MM-dd-yyyy}`) are supported
in templates too. Though note that while `${json:...}` template variables are
expected to occupy an entire field, that is, `"level": "${json:level}"`, a
lookup can be mixed within a regular text: `"myCustomField": "Hello, ${env:USER}!"`.

Similarly, below is the list of allowed `StackTraceElement` template variables:

| Variable Name | Description |
|---------------|-------------|
| `stackTraceElement:className` | `stackTraceElement.getClassName()` |
| `stackTraceElement:methodName` | `stackTraceElement.getMethodName()` |
| `stackTraceElement:fileName` | `stackTraceElement.getFileName()` |
| `stackTraceElement:lineNumber` | `stackTraceElement.getLineNumber()` |

As in `LogEvent` templates, `StackTraceElement` templates support Log4j 2
lookups too.

See [`layout-demo`](layout-demo) directory for a sample application
demonstrating the usage of `LogstashLayout`.

# Predefined Templates

`log4j2-logstash-layout` artifact contains the following predefined templates:

- [`EcsLayout.json`](layout/src/main/resources/EcsLayout.json) described by
[the Elastic Common Schema (ECS) specification](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html)

- [`LogstashJsonEventLayoutV1.json`](layout/src/main/resources/LogstashJsonEventLayoutV1.json)
described in [log4j-jsonevent-layout](https://github.com/logstash/log4j-jsonevent-layout)

- [`GelfLayout.json`](layout/src/main/resources/GelfLayout.json) described by
[the Graylog Extended Log Format (GELF) payload specification](https://docs.graylog.org/en/3.1/pages/gelf.html#gelf-payload-specification)
with additional `_thread` and `_logger`fields similar to
[`GelfLayout` of Log4j 2](https://logging.apache.org/log4j/2.0/manual/layouts.html#GELFLayout)
(Here it is advised to override the obligatory `host` field with a user
provided constant via `eventTemplateAdditionalFields` to avoid `hostName`
property lookup at runtime, which incurs an extra cost.)

# Features

Below is a feature comparison matrix between `LogstashLayout` and alternatives.

| Feature | `LogstashLayout` | `JsonLayout` | `EcsLayout` |
|---------|------------------|--------------|-------------|
| Java version | 8 | 73 | 6 |
| Dependencies | Jackson | Jackson | None |
| Full schema customization? | ✓ | ✕ | ✕ |
| Timestamp customization? | ✓ | ✕ | ✕ |
| (Almost) garbage-free? | ✓ | ✕ | ✓ |
| Custom typed `Message` serialization? | ✓ | ✕ | ✓4 |
| Custom typed `MDC` value serialization? | ✓ | ✕ | ✕ |
| Rendering stack traces as array? | ✓ | ✓ | ✓ |
| Enabling/Disabling JSON pretty print? | ✓ | ✓ | ✕ |
| Additional fields? | ✓ | ✓ | ✓ |

3 Log4j 2.4 and greater requires Java 7, versions 2.0-alpha1 to 2.3
required Java 6.

4 Only for `ObjectMessage`s and if Jackson is in the classpath.

# Fat JAR

Project also contains a `log4j2-logstash-layout-fatjar` artifact which
includes all its transitive dependencies in a separate shaded package (to
avoid the JAR Hell) with the exception of `log4j-core`, that you need to
include separately.

This might come handy if you want to use this plugin along with already
compiled applications, e.g., Elasticsearch 5.x and 6.x versions, which
requires Log4j 2.

# Appender Support

`log4j2-logstash-layout` is all about providing a highly customizable JSON
schema for your logs. Though this does not necessarily mean that all of its
features are expected to be supported by every appender in the market. For
instance, while `prettyPrintEnabled=true` works fine with
[log4j2-redis-appender](/vy/log4j2-redis-appender), it should be turned off
for Logstash's `log4j-json` file input type. (See
[Pretty printing in Logstash](/vy/log4j2-logstash-layout/issues/8) issue.)
Make sure you configure `log4j2-logstash-layout` properly in a way that
is aligned with your appender of preference.

# Performance

The source code contains a benchmark comparing `LogstashLayout` performance with
[`JsonLayout`](https://logging.apache.org/log4j/2.0/manual/layouts.html#JSONLayout)
(shipped by default in Log4j 2) and
[`EcsLayout`](https://github.com/elastic/java-ecs-logging/tree/master/log4j2-ecs-layout)
(shipped by Elastic). There [JMH](https://openjdk.java.net/projects/code-tools/jmh/)
is used to assess the rendering performance of these layouts. In the tests,
different `LogEvent` profiles are employed:

- **full**: `LogEvent` contains MDC, NDC, and an exception.
- **lite:** `LogEvent` has no MDC, NDC, or exception attachment.

To give an idea, we ran the benchmark with the following settings:

- **CPU:** Intel i7 2.70GHz (x86-64, confined `java` process to a single core
using [`taskset -c 0`](http://www.man7.org/linux/man-pages/man1/taskset.1.html))
- **JVM:** OpenJDK 64-Bit, AdoptOpenJDK, build 25.232-b09
- `-XX:+TieredCompilation`
- `-Dlog4j2.garbagefreeThreadContextMap=true`
- `-Dlog4j2.enableDirectEncoders=true`
- `-Dlog4j2.enable.threadlocals=true`
- `-Dlog4j2.is.webapp=false`
- **OS:** Xubuntu 18.04.3 (4.15.0-70-generic, x86-64)
- **`LogstashLayout4{Ecs,Json,Gelf}Layout`** used default settings with the
following exceptions:
- **`stackTraceEnabled`:** `true`
- **`maxByteCount`:** (4096) 4KiB
- **`JsonLayout`** used in two different flavors:
- **`DefaultJsonLayout`:** default settings
- **`CustomJsonLayout`:** default settings with an additional `"@version": 1`
field (this forces instantiation of a wrapper class to obtain the necessary
Jackson view)
- **`EcsLayout`** used with the following configurations:
- **`serviceName`:** `benchmark`
- **`additionalFields`:** `new KeyValuePair[0]`
- **`GelfLayout`** used with the following configurations:
- **`compressionType`:** `off`

The figures for serializing 1,000 `LogEvent`s at each operation are as follows.
(See [`layout-benchmark`](layout-benchmark) directory for the full report.)





Benchmark
ops/sec5
B/op5




liteLogstashLayout4GelfLayout
1,517,062
▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ (100%)
0.0


liteLogstashLayout4EcsLayout
1,196,255
▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ (79%)
0.0


liteGelfLayout
1,184,922
▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ (78%)
0.0


liteLogstashLayout4JsonLayout
870,012
▉▉▉▉▉▉▉▉▉▉▉ (57%)
0.0


liteEcsLayout
836,648
▉▉▉▉▉▉▉▉▉▉▉ (55%)
0.0


liteDefaultJsonLayout
506,985
▉▉▉▉▉▉▉ (33%)
5,331,680.0


liteCustomJsonLayout
446,243
▉▉▉▉▉▉ (29%)
5,740,400.0


fullLogstashLayout4JsonLayout
118,294
▉▉ (8%)
104,000.1


fullLogstashLayout4GelfLayout
73,102
▉ (5%)
35,663,200.3


fullLogstashLayout4EcsLayout
60,569
▉ (4%)
35,631,200.4


fullEcsLayout
27,887
▉ (2%)
46,479,200.5


fullGelfLayout
21,458
▉ (1%)
58,911,200.7


fullDefaultJsonLayout
13,513
▉ (1%)
234,102,401.5


fullCustomJsonLayout
13,511
▉ (1%)
234,238,401.5




5 99th percentile


Let us try to answer some common questions:

- **How come `log4j2-logstash-layout` can yield superior performance compared
to Log4j 2 `JsonLayout`?** Log4j 2 `JsonLayout` employs a single Jackson view
to generate JSON, XML, and YAML outputs. For this purpose, it uses Jackson
`ObjectMapper`, which needs to walk over the class fields via reflection and
perform heavy branching and intermediate object instantiation. On the
contrary, `log4j2-logstash-layout` parses the given template once and
compiles a (mostly) garbage- and (to a certain extent) branching-free
JSON generator employing Jackson `JsonGenerator`.

- **Why is `log4j2-logstash-layout` is not totally garbage-free?**

- Since `Throwable#getStackTrace()` clones the original
`StackTraceElement[]`, accesses to (and hence rendering) stack traces can
never be garbage-free.

- Rendering of context data (that is, MDC) field values is garbage-free if
the value is either `null`, or of type `String`, `Short`, `Integer`,
`Long`, or `byte[]`.

- **How can one run the benchmark on his/her machine?** After a fresh
`mvn clean verify` within the source directory, run
`layout-benchmark/benchmark.py`.

- **What about thread-local allocations?** Even though Log4j 2 exposes a
`log4j2.enable.threadlocals` flag to toggle TLAs, neither `EcsLayout`, nor
Log4j 2 `JsonLayout` and `GelfLayout` honor it. Historically, `LogstashLayout`
used to have TLAs taking the `log4j2.enable.threadlocals` flag into account.
In version 0.18, we switched to simple memory-efficient object pools, though
that incurred extra synchronization costs. Since version 0.22,
`LogstashLayout` switched back to TLAs taking `log4j2.enable.threadlocals`
into account.

# F.A.Q.

- **How can one enable thread-local allocations?** For performance reasons,
it is highly recommended to turn TLAs on. For this purpose, you need to make
sure `log4j2.enable.threadlocals=true` and `log4j2.is.webapp=false`.

- **Is there a fat JAR of the plugin?** Project also contains a
`log4j2-logstash-layout-fatjar` artifact which includes all its transitive
dependencies in a separate shaded package (to avoid the JAR Hell) with the
exception of `log4j-core`, that you need to include separately. Fat JAR might
come handy if you want to use this plugin along with certain applications,
e.g., Elasticsearch 5.x and 6.x versions, which requires Log4j 2.

- **Why do I get irrelevant stack traces from
`exception[RootCause]:stackTrace[:text]` directives?** `LogstashLayout`
uses `Throwable#printStackTrace(PrintWriter)` and `Throwable#getStackTrace()`
methods to resolve these directives. Under certain circumstances, these
methods may throw exception as well. Put another way, trying to access the
stack trace of an exception might cause another exception. In such a case,
`LogstashLayout` continues the directive resolution using the new exception
and keeps on repeating until it manages to resolve a stack trace to at least
provide some insight into the underlying cause.

# Contributors

- [bakomchik](https://github.com/bakomchik)
- [chrissydee](https://github.com/chrissydee)
- [Daniel Lundsgaard Skovenborg](https://github.com/waldeinburg)
- [Eric Schwartz](https://github.com/emschwar)
- [Felix Barnsteiner](https://github.com/felixbarny)
- [Johann Schmitz](https://github.com/ercpe)
- [John Ament](https://github.com/johnament)
- [Jonathan Guéhenneux](https://github.com/Achaaab)
- [justinsaliba](https://github.com/justinsaliba)
- [Maurice Zeijen](https://github.com/mzeijen)
- [Michael K. Edwards](https://github.com/mkedwards)
- [Mikael Strand](https://github.com/MikaelStrand)
- [Rafa Gómez](https://github.com/rgomezcasas)
- [Yaroslav Skopets](https://github.com/yskopets)

# License

Copyright © 2017-2020 [Volkan Yazıcı](https://vlkan.com/)

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.