https://github.com/ulule/python-logstash-formatter

python JSON log formatter with a logstash compatible schema
https://github.com/ulule/python-logstash-formatter

Last synced: 10 months ago
JSON representation

python JSON log formatter with a logstash compatible schema

• Host: GitHub
• URL: https://github.com/ulule/python-logstash-formatter
• Owner: ulule
• License: mit
• Created: 2013-05-01T21:08:53.000Z (over 13 years ago)
• Default Branch: master
• Last Pushed: 2019-08-12T16:08:29.000Z (almost 7 years ago)
• Last Synced: 2025-09-29T01:29:31.018Z (10 months ago)
• Language: Python
• Size: 42 KB
• Stars: 158
• Watchers: 12
• Forks: 61
• Open Issues: 3
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# logstash_formatter: JSON logs for logstash

This library is provided to allow standard python logging to output log data

as json objects ready to be shipped out to logstash.

This project has been originally open sourced by [exoscale](https://www.exoscale.ch/) (which is a great hosting service btw),

thanks to them.

## Installing

Using `pip` ([PyPI](https://pypi.python.org/pypi/logstash_formatter)):

~~~

pip install logstash_formatter

~~~

Manual:

~~~

git clone https://github.com/ulule/python-logstash-formatter.git

cd python-logstash-formatter

python setup.py install

~~~

## Usage

Json outputs are provided by the LogstashFormatter logging formatter.

~~~python

import logging, sys

from logstash_formatter import LogstashFormatterV1

# configure logging

handler = logging.StreamHandler(stream=sys.stdout)

handler.setFormatter(LogstashFormatterV1())

logging.basicConfig(handlers=[handler], level=logging.INFO)

# use it

logging.info("my log")

~~~

The LogstashFormatter may take the following named parameters:

* `fmt`: Config as a JSON string that supports:

  * `extra`: provide extra fields always present in logs.

  * `source_host`: override source host name.

* `json_cls`: JSON encoder to forward to `json.dump`.

* `json_default`: Default JSON representation for unknown types,

  by default coerce everything to a string.

`LogstashFormatterV1` adheres to the more 1.2.0 schema and will not update

fields, apart from a special handling of `msg` which will be updated to

`message` when applicable.

You can also add extra fields to your json output by specifying a dict in place of message, or by specifying

the named argument `extra` as a dictionary. When supplying the `exc_info` named argument with a truthy value,

and if an exception is found on the stack, its traceback will be attached to the payload as well.

~~~python

logger.info({"account": 123, "ip": "172.20.19.18"})

logger.info("classic message for account: {account}", extra={"account": account})

try:

    h = {}

    h['key']

except:

    logger.info("something unexpected happened", exc_info=True)

~~~

## Sample output for LogstashFormatter

The following keys will be found in the output JSON:

* `@source_host`: source hostname for the log

* `@timestamp`: ISO 8601 timestamp

* `@message`: short message for this log

* `@fields`: all extra fields

~~~json

{

    "@fields": {

        "account": "pyr",

        "args": [],

        "created": 1367480388.013037,

        "exception": [

            "Traceback (most recent call last):\n",

            "  File \"test.py\", line 16, in \n    k['unknown']\n",

            "KeyError: 'unknown'\n"

        ],

        "filename": "test.py",

        "funcName": "",

        "levelname": "WARNING",

        "levelno": 30,

        "lineno": 18,

        "module": "test",

        "msecs": 13.036966323852539,

        "name": "root",

        "pathname": "test.py",

        "process": 1819,

        "processName": "MainProcess",

        "relativeCreated": 18.002986907958984,

        "thread": 140060726359808,

        "threadName": "MainThread"

    },

    "@message": "TEST",

    "@source_host": "phoenix.spootnik.org",

    "@timestamp": "2013-05-02T09:39:48.013158"

}

~~~

## Sample output for LogstashFormatterV1

The following keys will be found in the output JSON:

* `@timestamp`: ISO 8601 timestamp

* `@version`: Version of the schema

~~~json

{"@version": 1,

 "account": "pyr",

 "lineno": 1,

 "levelno": 30,

 "filename": "test.py",

 "thread": 140566036444928,

 "@timestamp": "2015-03-30T09:46:23.000Z",

 "threadName": "MainThread",

 "relativeCreated": 51079.52117919922,

 "process": 10787,

 "source_host": "phoenix.spootnik.org",

 "processName": "MainProcess",

 "pathname": "test.py",

 "args": [],

 "module": "test",

 "msecs": 999.9005794525146,

 "created": 1427708782.9999006,

 "name": "root",

 "stack_info": null,

 "funcName": "",

 "levelname": "WARNING",

 "message": "foo"}

~~~