{"id":15008840,"url":"https://github.com/bobbui/json-logging-python","last_synced_at":"2025-04-05T08:06:52.470Z","repository":{"id":42589678,"uuid":"114656836","full_name":"bobbui/json-logging-python","owner":"bobbui","description":"Cloud-native distributed Python logging library to emit JSON log that can be easily indexed by logging infrastructure ","archived":false,"fork":false,"pushed_at":"2024-07-18T20:02:10.000Z","size":316,"stargazers_count":303,"open_issues_count":21,"forks_count":60,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-03-29T07:05:04.679Z","etag":null,"topics":["elasticsearch","elk","elk-stack","json","kibana","logging","logging-library","logstash","python","python2","python3"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bobbui.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["bobbui"]}},"created_at":"2017-12-18T15:27:46.000Z","updated_at":"2025-01-16T09:27:23.000Z","dependencies_parsed_at":"2024-02-15T08:30:16.846Z","dependency_job_id":"e4e951af-d648-4e6a-9f43-a1ad733d9ff0","html_url":"https://github.com/bobbui/json-logging-python","commit_stats":{"total_commits":140,"total_committers":26,"mean_commits":5.384615384615385,"dds":0.7857142857142857,"last_synced_commit":"403d8221795a6808f96be9d24a9d97250fcd293e"},"previous_names":["thangbn/json-logging"],"tags_count":31,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bobbui%2Fjson-logging-python","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bobbui%2Fjson-logging-python/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bobbui%2Fjson-logging-python/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bobbui%2Fjson-logging-python/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bobbui","download_url":"https://codeload.github.com/bobbui/json-logging-python/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247305934,"owners_count":20917208,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elasticsearch","elk","elk-stack","json","kibana","logging","logging-library","logstash","python","python2","python3"],"created_at":"2024-09-24T19:20:54.863Z","updated_at":"2025-04-05T08:06:52.436Z","avatar_url":"https://github.com/bobbui.png","language":"Python","readme":"# json-logging\n\nPython logging library to emit JSON log that can be easily indexed and searchable by logging infrastructure such\nas [ELK](https://www.elastic.co/elk-stack)\n, [EFK](https://www.digitalocean.com/community/tutorials/how-to-set-up-an-elasticsearch-fluentd-and-kibana-efk-logging-stack-on-kubernetes)\n, AWS Cloudwatch, GCP Stackdriver, etc.\n\nIf you're using Cloud Foundry, it might worth to check out the\nlibrary [SAP/cf-python-logging-support](https://github.com/SAP/cf-python-logging-support) which I'm also original\nauthor.\n\n# Content\n\n1. [Features](#1-features)\n2. [Usage](#2-usage)   \n   2.1 [Non-web application log](#21-non-web-application-log)  \n   2.2 [Web application log](#22-web-application-log)  \n   2.3 [Get current correlation-id](#23-get-current-correlation-id)  \n   2.4 [Log extra properties](#24-log-extra-properties)  \n   2.5 [Root logger](#25-root-logger)  \n   2.6 [Custom log formatter](#26-custom-log-formatter)  \n   2.7 [Exclude certain URL from request instrumentation](#27-exclude-certain-url-from-request-instrumentation)\n3. [Configuration](#3-configuration)\n4. [Python References](#4-python-references)\n5. [Framework support plugin development](#5-framework-support-plugin-development)\n6. [FAQ \u0026 Troubleshooting](#6-faq--troubleshooting)\n7. [References](#7-references)\n\n# 1. Features\n\n1. Emit JSON logs ([format detail](#0-full-logging-format-references))\n2. Lightweight, no dependencies, minimal configuration needed (1 LoC to get it working)\n3. Seamlessly integrate with Python native **logging** module. Support both Python 2.7.x and 3.x\n4. Auto extract **correlation-id** for distributed tracing [\\[1\\]](#1-what-is-correlation-idrequest-id)\n5. Support HTTP request instrumentation. Built in support for [FastAPI](https://fastapi.tiangolo.com/)\n   , [Flask](https://github.com/pallets/flask/), [Sanic](https://github.com/channelcat/sanic)\n   , [Quart](https://gitlab.com/pgjones/quart), [Connexion](https://github.com/zalando/connexion). Extensible to support\n   other web frameworks. PR welcome :smiley: .\n6. Highly customizable: support inject arbitrary extra properties to JSON log message, override logging formatter, etc.\n7. Production ready, has been used in production since 2017 with more than 8 millions downloads\n\n# 2. Usage\n\nInstall by running this command:\n\u003e pip install json-logging\n\nBy default log will be emitted in normal format to ease the local development. To enable it on production set enable_json in **json_logging.init_\u003c framework_name \u003e()** method call.\nOnce configured library will try to configure all loggers (existing and newly created) to emit log in JSON format. See following use cases for more detail.\n\n## 2.1 Non-web application log\n\nThis mode don't support **correlation-id**.\n\n```python\nimport json_logging, logging, sys\n\n# log is initialized without a web framework name\njson_logging.init_non_web(enable_json=True)\n\nlogger = logging.getLogger(\"test-logger\")\nlogger.setLevel(logging.DEBUG)\nlogger.addHandler(logging.StreamHandler(sys.stdout))\n\nlogger.info(\"test logging statement\")\n```\n\n## 2.2 Web application log\n\n### FastAPI\n\n```python\nimport datetime, logging, sys, json_logging, fastapi, uvicorn\n\napp = fastapi.FastAPI()\njson_logging.init_fastapi(enable_json=True)\njson_logging.init_request_instrument(app)\n\n# init the logger as usual\nlogger = logging.getLogger(\"test-logger\")\nlogger.setLevel(logging.DEBUG)\nlogger.addHandler(logging.StreamHandler(sys.stdout))\n\n\n@app.get('/')\ndef home():\n    logger.info(\"test log statement\")\n    logger.info(\"test log statement with extra props\", extra={'props': {\"extra_property\": 'extra_value'}})\n    correlation_id = json_logging.get_correlation_id()\n    return \"Hello world : \" + str(datetime.datetime.now())\n\n\nif __name__ == \"__main__\":\n    uvicorn.run(app, host='0.0.0.0', port=5000)\n```\n\n### Flask\n\n```python\nimport datetime, logging, sys, json_logging, flask\n\napp = flask.Flask(__name__)\njson_logging.init_flask(enable_json=True)\njson_logging.init_request_instrument(app)\n\n# init the logger as usual\nlogger = logging.getLogger(\"test-logger\")\nlogger.setLevel(logging.DEBUG)\nlogger.addHandler(logging.StreamHandler(sys.stdout))\n\n\n@app.route('/')\ndef home():\n    logger.info(\"test log statement\")\n    logger.info(\"test log statement with extra props\", extra={'props': {\"extra_property\": 'extra_value'}})\n    correlation_id = json_logging.get_correlation_id()\n    return \"Hello world : \" + str(datetime.datetime.now())\n\n\nif __name__ == \"__main__\":\n    app.run(host='0.0.0.0', port=int(5000), use_reloader=False)\n```\n\n### Sanic\n\n```python\nimport logging, sys, json_logging, sanic\n\napp = sanic.Sanic(name=\"sanic-web-app\")\njson_logging.init_sanic(enable_json=True)\njson_logging.init_request_instrument(app)\n\n# init the logger as usual\nlogger = logging.getLogger(\"sanic-integration-test-app\")\nlogger.setLevel(logging.DEBUG)\nlogger.addHandler(logging.StreamHandler(sys.stdout))\n\n\n@app.route(\"/\")\nasync def home(request):\n    logger.info(\"test log statement\")\n    logger.info(\"test log statement with extra props\", extra={'props': {\"extra_property\": 'extra_value'}})\n    # this will be faster\n    correlation_id = json_logging.get_correlation_id(request=request)\n    # this will be slower, but will work in context you cant get a reference of request object\n    correlation_id_without_request_obj = json_logging.get_correlation_id()\n\n    return sanic.response.text(\"hello world\")\n\n\nif __name__ == \"__main__\":\n    app.run(host=\"0.0.0.0\", port=5000)\n```\n\n### Quart\n\n```python\nimport asyncio, logging, sys, json_logging, quart\n\napp = quart.Quart(__name__)\njson_logging.init_quart(enable_json=True)\njson_logging.init_request_instrument(app)\n\n# init the logger as usual\nlogger = logging.getLogger(\"test logger\")\nlogger.setLevel(logging.DEBUG)\nlogger.addHandler(logging.StreamHandler(sys.stdout))\n\n\n@app.route('/')\nasync def home():\n    logger.info(\"test log statement\")\n    logger.info(\"test log statement with extra props\", extra={'props': {\"extra_property\": 'extra_value'}})\n    correlation_id = json_logging.get_correlation_id()\n    return \"Hello world\"\n\n\nif __name__ == \"__main__\":\n    loop = asyncio.get_event_loop()\n    app.run(host='0.0.0.0', port=int(5000), use_reloader=False, loop=loop)\n```\n\n### Connexion\n\n```python\nimport logging, sys, json_logging, connexion\n\napp = connexion.FlaskApp(__name__)\njson_logging.init_connexion(enable_json=True)\njson_logging.init_request_instrument(app)\n\napp.add_api('api.yaml')\n\n# init the logger as usual\nlogger = logging.getLogger(\"test-logger\")\nlogger.setLevel(logging.DEBUG)\nlogger.addHandler(logging.StreamHandler(sys.stdout))\n\nif __name__ == \"__main__\":\n    app.run()\n```\n\n### Custom handler for request instrumentation\n\nyou need to explicitly set JSONRequestLogFormatter as default formatter for any extra handler that is added to\nrequest_logger\n\n```python\nimport json_logging.formatters\n\nrequest_logger = json_logging.get_request_logger()\nhandler = logging.handlers.RotatingFileHandler(filename='log_req.log', maxBytes=5000000, backupCount=10)\nhandler.setFormatter(json_logging.formatters.JSONRequestLogFormatter())\nrequest_logger.addHandler(handler)\n```\n\n## 2.3 Get current correlation-id\n\nCurrent request correlation-id can be retrieved and pass to downstream services call as follow:\n\n```python\n# this will be faster\ncorrelation_id = json_logging.get_correlation_id(request=request)\n# this will be slower, but will work in context where you couldn't get a reference of request object\ncorrelation_id_without_request_obj = json_logging.get_correlation_id()\n# use correlation id for upstream service calls here\n```\n\nIn request context, if one is not present, a new one might be generated depends on CREATE_CORRELATION_ID_IF_NOT_EXISTS\nsetting value.\n\n\n## 2.4 Log extra properties\n\nExtra property can be added to logging statement as follows:\n\n```python\nlogger.info(\"test log statement\", extra={'props': {'extra_property': 'extra_value'}, 'tags': ['app:name']})\n```\n\n_For backwards compatibility, properties added in `props` are extracted to the root and take precedence over those set directly in the root of `extra`. This is relevant only for logs formatted by `JSONLogFormatter`, other formatters never printed the `props` details._\n\n### Forced overriding correlation-id\nA custom correlation id can be passed to logging statement as follow:\n```\nlogger.info(\"test log statement\", extra={'props': {'correlation_id': 'custom_correlation_id'}})\n```\n\n## 2.5 Root logger\n\nIf you want to use root logger as main logger to emit log. Make sure you call **config_root_logger()** after initialize\nroot logger (by logging.basicConfig() or logging.getLogger()) [\\[2\\]](#2-python-logging-propagate)\n\n```python\nlogging.basicConfig()\njson_logging.init_ \u003c framework_name \u003e ()\njson_logging.config_root_logger()\n```\n\n## 2.6 Custom log formatter\n\nCustomer JSON log formatter can be passed to init method. see examples for more\ndetail: [non web](https://github.com/thangbn/json-logging-python/blob/master/example/custom_log_format.py),\n[web](https://github.com/thangbn/json-logging-python/blob/master/example/custom_log_format_request.py)\n\n## 2.7 Exclude certain URl from request instrumentation\n\nCertain URL can be excluded from request instrumentation by specifying a list of regex into **init_request_instrument**\nmethod like below:\n\n```python\njson_logging.init_request_instrument(app, exclude_url_patterns=[r'/exclude_from_request_instrumentation'])\n```\n\n# 3. Configuration\n\nlogging library can be configured by setting the value in json_logging, all configuration must be placed before\njson_logging.init method call\n\nName | Description | Default value\n --- | --- | ---\nENABLE_JSON_LOGGING | **\nDEPRECATED** Whether to enable JSON logging mode.Can be set as an environment variable, enable when set to to either one in following list (case-insensitive) **['true', '1', 'y', 'yes']** , this have no effect on request logger | false\nCORRELATION_ID_FIELD | Name of field in generated log messages containing the correlation-id value | 'correlation_id'\nCORRELATION_ID_HEADERS | List of HTTP headers that will be used to look for correlation-id value. HTTP headers will be searched one by one according to list order| ['X-Correlation-ID','X-Request-ID']\nEMPTY_VALUE | Default value when a logging record property is None |  '-'\nCORRELATION_ID_GENERATOR | function to generate unique correlation-id | uuid.uuid1\nJSON_SERIALIZER | function to encode object to JSON | json.dumps\nCOMPONENT_ID | Uniquely identifies the software component that has processed the current request | EMPTY_VALUE\nCOMPONENT_NAME | A human-friendly name representing the software component | EMPTY_VALUE\nCOMPONENT_INSTANCE_INDEX | Instance's index of horizontally scaled service | 0\nCREATE_CORRELATION_ID_IF_NOT_EXISTS |  Whether to generate a new correlation-id in case one is not present| True\n\n# 4. Python References\n\nTODO: update Python API docs on Github page\n\n# 5. Framework support plugin development\n\nTo add support for a new web framework, you need to extend following classes in [**\nframework_base**](/blob/master/json_logging/framework_base.py) and register support using [**\njson_logging.register_framework_support**](https://github.com/thangbn/json-logging-python/blob/master/json_logging/__init__.py#L38)\nmethod:\n\nClass | Description | Mandatory\n--- | --- | ---\nBaseRequestInfoExtractor | Helper class help to extract logging-relevant information from HTTP request object | yes\nBaseResponseInfoExtractor | Helper class help to extract logging-relevant information from HTTP response object | yes\nBaseFrameworkConfigurator |  Class to perform logging configuration for given framework as needed | no\nBaseAppRequestInstrumentationConfigurator | Class to perform request instrumentation logging configuration | no\n\nTake a look at [**json_logging/base_framework.py**](json_logging/framework_base.py), [**\njson_logging.flask**](json_logging/framework/flask) and [**json_logging.sanic**](json_logging/framework/sanic) packages\nfor reference implementations.\n\n# 6. FAQ \u0026 Troubleshooting\n\n1. I configured everything, but no logs are printed out?\n\n    - Forgot to add handlers to your logger?\n    - Check whether logger is disabled.\n\n2. Same log statement is printed out multiple times.\n\n    - Check whether the same handler is added to both parent and child loggers [2]\n    - If you using flask, by default option **use_reloader** is set to **True** which will start 2 instances of web\n      application. change it to False to disable this behaviour [\\[3\\]](#3-more-on-flask-use-reloader)\n\n# 7. References\n\n## [0] Full logging format references\n\n2 types of logging statement will be emitted by this library:\n\n- Application log: normal logging statement e.g.:\n\n```\n{\n\t\"type\": \"log\",\n\t\"written_at\": \"2017-12-23T16:55:37.280Z\",\n\t\"written_ts\": 1514048137280721000,\n\t\"component_id\": \"1d930c0xd-19-s3213\",\n\t\"component_name\": \"ny-component_name\",\n\t\"component_instance_idx\": 0,\n\t\"logger\": \"test logger\",\n\t\"thread\": \"MainThread\",\n\t\"level\": \"INFO\",\n\t\"line_no\": 22,\n\t\"filename\": \"/path/to/foo.py\"\n\t\"exc_info\": \"Traceback (most recent call last): \\n  File \"\u003cstdin\u003e\", line 1, in \u003cmodule\u003e\\n ValueError: There is something wrong with your input\",\n\t\"correlation_id\": \"1975a02e-e802-11e7-8971-28b2bd90b19a\",\n\t\"extra_property\": \"extra_value\",\n\t\"msg\": \"This is a message\"\n}\n```\n\n- Request log: request instrumentation logging statement which recorded request information such as response time,\n  request size, etc.\n\n```\n{\n\t\"type\": \"request\",\n\t\"written_at\": \"2017-12-23T16:55:37.280Z\",\n\t\"written_ts\": 1514048137280721000,\n\t\"component_id\": \"-\",\n\t\"component_name\": \"-\",\n\t\"component_instance_idx\": 0,\n\t\"correlation_id\": \"1975a02e-e802-11e7-8971-28b2bd90b19a\",\n\t\"remote_user\": \"user_a\",\n\t\"request\": \"/index.html\",\n\t\"referer\": \"-\",\n\t\"x_forwarded_for\": \"-\",\n\t\"protocol\": \"HTTP/1.1\",\n\t\"method\": \"GET\",\n\t\"remote_ip\": \"127.0.0.1\",\n\t\"request_size_b\": 1234,\n\t\"remote_host\": \"127.0.0.1\",\n\t\"remote_port\": 50160,\n\t\"request_received_at\": \"2017-12-23T16:55:37.280Z\",\n\t\"response_time_ms\": 0,\n\t\"response_status\": 200,\n\t\"response_size_b\": \"122\",\n\t\"response_content_type\": \"text/html; charset=utf-8\",\n\t\"response_sent_at\": \"2017-12-23T16:55:37.280Z\"\n}\n```\n\nSee following tables for detail format explanation:\n\n- Common field\n\nField | Description | Format | Example\n --- | --- | --- | ---\nwritten_at | The date when this log message was written. | ISO 8601 YYYY-MM-DDTHH:MM:SS.milliZ | 2017-12-23T15:14:02.208Z\nwritten_ts | The timestamp in nano-second precision when this request metric message was written. | long number | 1456820553816849408\ncorrelation_id | The timestamp in nano-second precision when this request metric message was written. | string | db2d002e-2702-41ec-66f5-c002a80a3d3f\ntype | Type of logging. \"logs\" or \"request\"  | string |\ncomponent_id | Uniquely identifies the software component that has processed the current request | string | 9e6f3ecf-def0-4baf-8fac-9339e61d5645\ncomponent_name | A human-friendly name representing the software component | string | my-fancy-component\ncomponent_instance_idx | Instance's index of horizontally scaled service  | string | 0\n\n- application logs\n\nField | Description | Format | Example\n --- | --- | --- | ---\nmsg | The actual message string passed to the logger.  | string | This is a log message\nlevel | The log \"level\" indicating the severity of the log message.  | string | INFO\nthread | Identifies the execution thread in which this log message has been written.  | string | http-nio-4655\nlogger | The logger name that emits the log message. |string | requests-logger\nfilename | The file name where an exception originated | string | /path/to/foo.py\nexc_info | Traceback information about an exception | string | \"Traceback (most recent call last): \\n  File \"\u003cstdin\u003e\", line 1, in \u003cmodule\u003e\\n ValueError: There is something wrong with your input\"\n\n- request logs:\n\nField | Description | Format | Example\n --- | --- | --- | ---\nrequest | request path that has been processed. | string | /get/api/v2\nrequest_received_at | The date when an incoming request was received by the producer.| ISO 8601 YYYY-MM-DDTHH:MM:SS.milliZ   The precision is in milliseconds. The timezone is UTC.  | 2015-01-24 14:06:05.071Z\nresponse_sent_at | The date when the response to an incoming request was sent to the consumer.  | ditto | 2015-01-24 14:06:05.071Z\nresponse_time_ms | How many milliseconds it took the producer to prepare the response.  | float | 43.476\nprotocol | Which protocol was used to issue a request to a producer. In most cases, this will be HTTP (including a version specifier), but for outgoing requests reported by a producer, it may contain other values. E.g. a database call via JDBC may report, e.g. \"JDBC/1.2\"  | string | HTTP/1.1\nmethod | The corresponding protocol method. | string | GET\nremote_ip |  IP address of the consumer (might be a proxy, might be the actual client) | string | 192.168.0.1\nremote_host |  host name of the consumer (might be a proxy, might be the actual client) | string | my.happy.host\nremote_port | Which TCP port is used by the consumer to establish a connection to the remote producer. | string | 1234\nremote_user | The username associated with the request | string | user_name\nrequest_size_b | The size in bytes of the requesting entity or \"body\" (e.g., in case of POST requests). | long | 1234\nresponse_size_b | The size in bytes of the response entity | long | 1234\nresponse_status | The status code of the response. | long | 200\nresponse_content_type | The MIME type associated with the entity of the response if available/specified | long | application/json\nreferer | For HTTP requests, identifies the address of the webpage (i.e. the URI or IRI) that linked to the resource being requested. | string |  /index.html\nx_forwarded_for | Comma-separated list of IP addresses, the left-most being the original client, followed by proxy server addresses that forwarded the client request. | string |  192.0.2.60,10.12.9.23\n\n## [1] What is correlation-id/request id\n\nhttps://stackoverflow.com/questions/25433258/what-is-the-x-request-id-http-header\n\n## [2] Python logging propagate\n\nhttps://docs.python.org/3/library/logging.html#logging.Logger.propagate\nhttps://docs.python.org/2/library/logging.html#logging.Logger.propagate\n\n## [3] more on flask use_reloader\n\nhttp://flask.pocoo.org/docs/0.12/errorhandling/#working-with-debuggers\n","funding_links":["https://github.com/sponsors/bobbui"],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbobbui%2Fjson-logging-python","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbobbui%2Fjson-logging-python","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbobbui%2Fjson-logging-python/lists"}