https://github.com/puppetlabs/structured-logging

Write data structures to your logs from clojure
https://github.com/puppetlabs/structured-logging

Last synced: 3 months ago
JSON representation

Write data structures to your logs from clojure

• Host: GitHub
• URL: https://github.com/puppetlabs/structured-logging
• Owner: puppetlabs
• License: apache-2.0
• Created: 2015-09-28T15:38:30.000Z (over 9 years ago)
• Default Branch: main
• Last Pushed: 2025-02-04T02:23:19.000Z (5 months ago)
• Last Synced: 2025-03-29T15:07:41.794Z (3 months ago)
• Language: Clojure
• Homepage:
• Size: 29.3 KB
• Stars: 29
• Watchers: 142
• Forks: 8
• Open Issues: 2
• Metadata Files:
  • Readme: README.md
  • License: LICENSE
  • Codeowners: CODEOWNERS

Awesome Lists containing this project

README

        
# puppetlabs/structured-logging [![Clojars Project](https://img.shields.io/clojars/v/puppetlabs/structured-logging.svg)](https://clojars.org/puppetlabs/structured-logging) [![Build Status](https://travis-ci.org/puppetlabs/structured-logging.svg?branch=master)](https://travis-ci.org/puppetlabs/structured-logging)

`structured-logging` is a library that helps you to:

 - write arbitrary JSON to your logs, making it easier to interface with log analysis tools.

 - write messages to arbitrarily named loggers, instead of just the one named

   after the namespace in which the log statement appears.

It is built on `clojure.tools.logging`, but it only works with logback. 

## Usage

### maplog

Structured log entries are created by calling `maplog` like this:

    (maplog :warn {:user-service "https://...", :status 503, :elapsed 27}

            #(format "Failed to query user-service %s. Response: status %d"

                     (:user-service %) (:status %)))

The second parameter is the structured data you want to log, and any

clojure map is ok, as long as

[Cheshire](https://github.com/dakrone/cheshire) can handle it.

The log message itself is generated by passing the structured data to

the function provided as the final argument.  This provides a general

way to support arbitrary formatters, including some like those in

[clj-i18n](https://github.com/puppetlabs/clj-i18n), which require

that their invocations appear literally, at each translation call

site, i.e.:

    (maplog :info {:temperature t :stamp (now)}

            #(i18n/trs "Temperature has exceeded {0} at {1}"

                       (:temperature %) (:stamp %)))

### Named loggers

When you would pass a log-level parameter, you may instead supply a vector of

`[:custom-logger :level]`. This will log a message to the logger named

"custom-logger". Then you can specifically address that logger in your logback

configuration (to redirect it to a different log file, for example).

## Common fields

If you use the recommended logger configuration, as described below, you will

see the following fields in each JSON log message:

* @timestamp

* message

* logger_name

* thread_name

* level

* level_value (numeric, suitable for sorting)

* stack_trace

Additional relevant fields may be added in any given message, but this base set

will always be present.

## Configuration

### JSON text

If you want to log JSON data where you would otherwise log regular text, replace the `encoder` element in

your `logback.xml` with this one:

    

      

        

        

        

        

        

        

        

        

      

    

Even though this says 'logstash' on it, it works completely independently from

any log aggregation system. The final `` element is

important - that inserts our custom properties into each json message.

## Logstash integration

You can also log directly to logstash with an appender configured like this:

    

      my.great.logstash.server.com

      4560

      

        

          

          

          

          

          

          

          

          

        

      

    

You will also need to add a reference to the appender from the `` element:

    

      ... 

      

    

## Log Output

We are using a json encoder that comes with the `logstash-logback` integration

library. (https://github.com/logstash/logstash-logback-encoder) This can be used

in conjunction with the logstash tcp appender, or in conjunction with a typical

file appender. In either case, the json for the above log message looks

something like this:

    {

      "@timestamp": "...",

      "level": "WARN",

      "user-service": "https://...",

      "status": 503,

      "elapsed": 27

    }

The exact fields that show up depend on how you've configured the encoder;

'@timestamp' is a logstash convention.

## Answers to expected questions

*Can I embed arbitrary structures in there?*

You can log anything that cheshire can serialize, but you may wish to stick

to simple key-value formats to keep your logs easy to analyze.

*What happens when I log a map with :level as the key?*

Don't do that. It's not yet clear how this case should be handled. For now,

avoid such keys. Namespacing your map keys may be prudent if you're worried

about collisions.

*Does this library require logstash?*

No. It does depend on the json encoder that comes with the logback-logstash

integration library, but you can point the json it spits out at any old logback

appender. Critically, it has a feature that allows us to edit the json before it

gets written, which is where we add our custom information. We *could* write

such an encoder ourselves, but they already wrote it.

*Why didn't you use one of the other logback-json encoders?*

Because those just encode the normal log event fields as json; the useful

feature here is the ability to include arbitrary data as part of the json.

## Tips

Structured logging is a slippery slope and should be applied where it has

clear benefits. While we've streamlined it a lot, this is heavier than regular

logging, and it's still non-application code that you have to scatter throughout

your program.

## License

Copyright © 2015 Puppet Labs

Distributed under the [Apache License, Version 2.0](http://www.apache.org/licenses/LICENSE-2.0.html)