{"id":15044046,"url":"https://github.com/danyi1212/dans-log-formatter","last_synced_at":"2025-10-23T20:30:18.106Z","repository":{"id":228233860,"uuid":"765477739","full_name":"danyi1212/dans-log-formatter","owner":"danyi1212","description":"Extensible log formatter for Python logging library","archived":false,"fork":false,"pushed_at":"2025-01-27T21:23:54.000Z","size":137,"stargazers_count":4,"open_issues_count":8,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-01-30T20:40:50.072Z","etag":null,"topics":["celery","datadog","django","fastapi","flask","logging","python","python-logging"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/danyi1212.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-03-01T01:54:06.000Z","updated_at":"2024-10-29T14:45:56.000Z","dependencies_parsed_at":"2024-03-17T19:31:36.216Z","dependency_job_id":"2cda11be-305f-4626-9c3b-f55fdcaf47a4","html_url":"https://github.com/danyi1212/dans-log-formatter","commit_stats":{"total_commits":42,"total_committers":2,"mean_commits":21.0,"dds":"0.38095238095238093","last_synced_commit":"fa3aeae98d8b6601b87b936e96bc346eb730411e"},"previous_names":["danyi1212/dans-log-formatter"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danyi1212%2Fdans-log-formatter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danyi1212%2Fdans-log-formatter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danyi1212%2Fdans-log-formatter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danyi1212%2Fdans-log-formatter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/danyi1212","download_url":"https://codeload.github.com/danyi1212/dans-log-formatter/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":237882172,"owners_count":19381176,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["celery","datadog","django","fastapi","flask","logging","python","python-logging"],"created_at":"2024-09-24T20:49:59.896Z","updated_at":"2025-10-23T20:30:17.677Z","avatar_url":"https://github.com/danyi1212.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003eDan's Log Formatter\u003c/h1\u003e\n\u003cp align=\"center\"\u003e\n  \u003cem\u003eExtensible, reusable, awesome log formatter for Python\u003c/em\u003e\n\u003c/p\u003e\n\u003cp align=\"center\"\u003e\n\u003ca href=\"https://github.com/danyi1212/dans-log-formatter/actions/workflows/test.yml\" target=\"_blank\"\u003e\n  \u003cimg src=\"https://github.com/danyi1212/dans-log-formatter/actions/workflows/test.yml/badge.svg\" alt=\"Test\"\u003e\n\u003c/a\u003e\n\u003ca href=\"https://codecov.io/gh/danyi1212/dans-log-formatter\" \u003e\n \u003cimg src=\"https://codecov.io/gh/danyi1212/dans-log-formatter/graph/badge.svg?token=0NR3NUEFDV\" alt=\"Coverage\" /\u003e\n\u003c/a\u003e\n\u003ca href=\"https://pypi.org/project/dans-log-formatter\" target=\"_blank\"\u003e\n  \u003cimg src=\"https://img.shields.io/pypi/v/dans-log-formatter?color=%2334D058\u0026label=version\" alt=\"Package version\"\u003e\n\u003c/a\u003e\n\u003ca href=\"https://pypi.org/project/dans-log-formatter\" target=\"_blank\"\u003e\n  \u003cimg src=\"https://img.shields.io/pypi/pyversions/dans-log-formatter.svg?color=%2334D058\" alt=\"Supported Python versions\"\u003e\n\u003c/a\u003e\n\u003c/p\u003e\n\n---\n\nYou too are tiered of rewriting log handling for each project?\nHere's my simple, extensible formatter, designed so we never have to write it again.\n\nThis log formatter ships with commonly used features, like JSON serialization, attribute injection, error handling and more.\n\nAdding log attributes beside the message is made simple, like contextual data, runtime information, request information, basicly whatever you may need.\nThose attribute providers can easily be shared between your services, streamlining the development experince between your services.\n\n## Features\n\n- **Extensible** - Add attributes to logs with ease, including simple error handling\n- **Reusable** - Share your attribute providers across projects\n- **Contextual** - Automatically adds useful context to logs\n- **Out-of-the-box** - Include common providers for HTTP data, runtime, and more\n\nMy log record's default attributes are mostly compatible\nwith [DataDog's Standard Attributes](https://docs.datadoghq.com/logs/log_configuration/attributes_naming_convention/#standard-attributes).\n\n#### Integrations\n\n- **Django** - Automatically adds request context\n- **FastAPI** - Automatically adds request context (including Starlette support)\n- **Flask** - Automatically adds request context\n- **Celery** - Automatically adds task context\n- **orjson** - Uses orSON for serialization\n- **ujson** - Uses uJSON for serialization\n\n## Usage\n\nInstall my package using pip:\n\n```shell\npip install dans-log-formatter\n```\n\nThen set up your logging configuration:\n\n```python\nimport logging.config\n\nfrom dans_log_formatter.providers.context import ContextProvider\n\nlogging.config.dictConfig({\n  \"version\": 1,\n  \"formatters\": {\n    \"json\": {\n      \"()\": \"dans_log_formatter.JsonLogFormatter\",\n      \"providers\": [\n        ContextProvider(),\n      ],  # optional\n    }\n  },\n  \"handlers\": {\n    \"console\": {\n      \"class\": \"logging.StreamHandler\",\n      \"formatter\": \"json\",\n    }\n  },\n  \"root\": {\n    \"handlers\": [\"console\"],\n    \"level\": \"INFO\",\n  },\n})\n```\n\nThen, use it in your project:\n\n```python\nimport logging\n\nlogger = logging.getLogger(__name__)\n\n\ndef main():\n  logger.info(\"Hello, world!\")\n\n\nif __name__ == \"__main__\":\n  main()\n\n# STDOUT: {'timestamp': '2025-01-01T00:00:00', 'status': 'INFO', 'message': 'hello world!', 'location': 'my_module-main#4', 'file': '/Users/danyi1212/projects/my-project/my_module.py'}\n```\n\n## Providers\n\nProviders add attributes to logs. You can use the built-in providers or create your own.\n\n### Context Provider\n\nInject context into logs using decorator or context manager.\n\n```python\nfrom dans_log_formatter import JsonLogFormatter\nfrom dans_log_formatter.providers.context import ContextProvider\n\nformatter = JsonLogFormatter(providers=[ContextProvider()])\n```\n\nThen use the `inject_log_context()` as a context manager\n\n```python\nimport logging\nfrom dans_log_formatter.providers.context import inject_log_context\n\nlogger = logging.getLogger(__name__)\n\nwith inject_log_context({\"user_id\": 123}):\n  logger.info(\"Hello, world!\")\n\n# STDOUT: {'timestamp': '2025-01-01T00:00:00', 'status': 'INFO', 'message': 'hello world!', 'user_id': 123, ...}\n```\n\nAlternatively, use it as `@inject_log_context()` decorator\n\n```python\nimport logging\nfrom dans_log_formatter.providers.context import inject_log_context\n\nlogger = logging.getLogger(__name__)\n\n\n@inject_log_context({\"custom_context\": \"value\"})\ndef my_function():\n  logger.info(\"Hello, world!\")\n\n# STDOUT: {'timestamp': '2025-01-01T00:00:00', 'status': 'INFO', 'message': 'hello world!', 'custom_context': 'value', ...}\n```\n\n### Extra Provider\n\nAdd `ExtraProvider()` from `dans_log_formatter.providers.extra`, then use the `extra={}` argument in your log calls\n\n```python\nimport logging\n\nlogger = logging.getLogger(__name__)\nlogger.info(\"Hello, world!\", extra={\"user_id\": 123})\n# STDOUT: {'timestamp': '2025-01-01T00:00:00', 'status': 'INFO', 'message': 'hello world!', 'user_id': 123, ...}\n```\n\n### Runtime Provider\n\nAdd `RuntimeProvider()` from `dans_log_formatter.providers.runtime` to add runtime information to logs.\n\n#### Attributes\n\n* `process` - Current process name and ID (e.g. `main (12345)`)\n* `thread` - Current thread name and ID (e.g. `MainThread (12345)`)\n* `task` - Current asyncio task name (e.g. `my_corrutine`)\n\n### Create your own provider\n\n```python\nfrom logging import LogRecord\nfrom typing import Any\n\nfrom dans_log_formatter.providers.abstract import AbstractProvider\n\n\nclass MyProvider(AbstractProvider):\n  \"\"\"Add 'my_attribute' to all logs\"\"\"\n\n  def get_attributes(self, record: LogRecord) -\u003e dict[str, Any]:\n    return {\"my_attribute\": \"some value\"}\n```\n\nYou can also use the abstract context provider to add data from contextvars\n\n```python\nfrom contextvars import ContextVar\nimport logging\nfrom typing import Any\nfrom dataclasses import dataclass\n\nfrom dans_log_formatter.providers.abstract_context import AbstractContextProvider\n\n\n@dataclass\nclass User:\n  id: int\n  name: str\n\n\ncurrent_user_context: ContextVar[User | None] = ContextVar(\"current_user_context\", default=None)\n\n\nclass MyContextProvider(AbstractContextProvider):\n  \"\"\"Add user.id and user.name context to logs\"\"\"\n\n  def __init__(self):\n    super().__init__(current_user_context)  # Pass the context\n\n  def get_context_attributes(self, record: logging.LogRecord, current_user: User) -\u003e dict[str, Any]:\n    return {\"user.id\": current_user.id, \"user.name\": current_user.name}\n\n\nlogger = logging.getLogger(__name__)\n\ntoken = current_user_context.set(User(id=123, name=\"John Doe\"))\nlogger.info(\"Hello, world!\")\ncurrent_user_context.reset(token)\n# STDOUT: {'timestamp': '2025-01-01T00:00:00', 'status': 'INFO', 'message': 'Hello, world!', 'user.id': 123, 'user.name': 'John Doe', ...}\n```\n\n## Integrations\n\n### Django Request Provider\n\nInstall using 'pip install dans-log-formatter[django]'\n\nAdd the 'LogContextMiddleware' to your Django middlewares at the very beginning.\n\n```python\n# settings.py\nMIDDLEWARE = [\n  \"dans_log_formatter.contrib.django.middleware.LogContextMiddleware\",\n  ...\n]\n```\n\nThen, add `DjangoRequestProvider()` to your formatter.\n\n```python\n# settings.py\nfrom dans_log_formatter.contrib.django.provider import DjangoRequestProvider\n\nLOGGING = {\n  \"version\": 1,\n  \"formatters\": {\n    \"json\": {\n      \"()\": \"dans_log_formatter.JsonLogFormatter\",\n      \"providers\": [\n        DjangoRequestProvider(),\n      ],\n    }\n  },\n  # ...\n}\n```\n\n#### Attributes\n\n* `resource` - View route (e.g. `POST /api/users/\u003cint:user_id\u003e/delete`)\n* `http.url` - Full URL (e.g. `https://example.com/api/users/123/delete`)\n* `http.method` - HTTP method (e.g. `POST`)\n* `http.referrer` - `Referrer` header (e.g. `https://example.com/previous-page`)\n* `http.user_agent` - `useragent` header\n* `http.remote_addr` - `X-Forwarded-For` or `REMOTE_ADDR` header\n* `user.id` - User ID\n* `user.name` - User's username\n* `user.email` - User email\n\n\u003e Note: The `user` attributes available only inside the `django.contrib.auth.middleware.AuthenticationMiddleware`\n\u003e middleware.\n\n### FastAPI Request Provider\n\nInstall using 'pip install dans-log-formatter[fastapi]'\n\nAdd the 'LogContextMiddleware' to your FastAPI app.\n\n```python\nfrom fastapi import FastAPI\nfrom dans_log_formatter.contrib.fastapi.middleware import LogContextMiddleware\n\napp = FastAPI()\napp.add_middleware(LogContextMiddleware)\n```\n\nThen, add `FastAPIRequestProvider()` to your formatter.\n\n```python\nimport logging.config\nfrom dans_log_formatter.contrib.fastapi.provider import FastAPIRequestProvider\n\nlogging.config.dictConfig({\n  \"version\": 1,\n  \"formatters\": {\n    \"json\": {\n      \"()\": \"dans_log_formatter.JsonLogFormatter\",\n      \"providers\": [\n        FastAPIRequestProvider(),\n      ],\n    }\n  },\n  # ...\n})\n```\n\n#### Attributes\n\n* `resource` - Route path (e.g. `POST /api/users/{user_id}/delete`)\n* `http.url` - Full URL (e.g. `https://example.com/api/users/123/delete`)\n* `http.method` - HTTP method (e.g. `POST`)\n* `http.referrer` - `Referrer` header (e.g. `https://example.com/previous-page`)\n* `http.user_agent` - `useragent` header\n* `http.remote_addr` - `X-Forwarded-For` header or the `request.client.host` attribute\n\n### Flask Request Provider\n\nInstall using 'pip install dans-log-formatter[flask]'\n\nAdd the 'FlastRequestProvider' to your formatter, and its magic!\n\n```python\nimport logging.config\nfrom dans_log_formatter.contrib.flask.provider import FlaskRequestProvider\n\nlogging.config.dictConfig({\n  \"version\": 1,\n  \"formatters\": {\n    \"json\": {\n      \"()\": \"dans_log_formatter.JsonLogFormatter\",\n      \"providers\": [\n        FlaskRequestProvider(),\n      ],\n    }\n  },\n  # ...\n})\n```\n\n#### Attributes\n\n* `resource` - URL path (e.g. `POST /api/users/123/delete`)\n* `http.url` - Full URL (e.g. `https://example.com/api/users/123/delete`)\n* `http.method` - HTTP method (e.g. `POST`)\n* `http.referrer` - `Referrer` header (e.g. `https://example.com/previous-page`)\n* `http.user_agent` - `useragent` header\n* `http.remote_addr` - `request.remote_addr` attribute\n\n### Celery Task Provider\n\nInstall using 'pip install dans-log-formatter[celery]'\n\nAdd the 'CeleryTaskProvider' to your formatter, and its magic!\n\n```python\nimport logging.config\nfrom dans_log_formatter.contrib.celery.provider import CeleryTaskProvider\n\nlogging.config.dictConfig({\n  \"version\": 1,\n  \"formatters\": {\n    \"json\": {\n      \"()\": \"dans_log_formatter.JsonLogFormatter\",\n      \"providers\": [\n        CeleryTaskProvider(),  # optional include_args=True\n      ],\n    }\n  },\n  # ...\n})\n```\n\n#### Attributes\n\n* `resource` - Task name (e.g. `my_project.tasks.my_task`)\n* `task.id` - Task ID\n* `task.retries` - Number of retries\n* `task.root_id` - Root task ID\n* `task.parent_id` - Parent task ID\n* `task.origin` - Producer host name\n* `task.delivery_info` - Delivery info (\n  e.g. `{\"exchange\": \"my_exchange\", \"routing_key\": \"my_routing_key\", \"queue\": \"my_queue\"}`)\n* `task.worker` - Worker hostname\n* `task.args` - Task arguments (if `include_args=True`)\n* `task.kwargs` - Task keyword arguments (if `include_args=True`)\n\n\u003e Warning: Including task arguments can expose sensitive information, and may result in very large logs.\n\n### ujson Formatter\n\nInstall using 'pip install dans-log-formatter[ujson]'\n\nUses `ujson` for JSON serialization of the log records.\n\n```python\nimport logging.config\n\nlogging.config.dictConfig({\n  \"version\": 1,\n  \"formatters\": {\n    \"json\": {\n      \"()\": \"dans_log_formatter.contrib.ujson.UJsonLogFormatter\",\n      \"providers\": [],  # optional\n    }\n  },\n  \"handlers\": {\n    \"console\": {\n      \"class\": \"logging.StreamHandler\",\n      \"formatter\": \"json\",\n    }\n  },\n  \"root\": {\n    \"handlers\": [\"console\"],\n    \"level\": \"INFO\",\n  },\n})\n```\n\n### orjson Serializer Formatter\n\nInstall using 'pip install dans-log-formatter[orjson]'\n\nUses `orjson` for JSON serialization of the log records.\n\n```python\nimport logging.config\n\nlogging.config.dictConfig({\n  \"version\": 1,\n  \"formatters\": {\n    \"json\": {\n      \"()\": \"dans_log_formatter.contrib.orjson.OrJsonLogFormatter\",\n      \"providers\": [],  # optional\n    }\n  },\n  \"handlers\": {\n    \"console\": {\n      \"class\": \"logging.StreamHandler\",\n      \"formatter\": \"json\",\n    }\n  },\n  \"root\": {\n    \"handlers\": [\"console\"],\n    \"level\": \"INFO\",\n  },\n})\n```\n\n## Available Formatters\n\nBy default, all formatter includes the following attributes:\n\n* `timestamp` - Unix timestamp (same as the `record.created` attribute, or the value returned\n  by `time.time()`. [See the docs](https://docs.python.org/3/library/logging.html#logrecord-attributes))\n* `status` - Log level name (e.g. `INFO`, `ERROR`, `CRITICAL`)\n* `message` - Log message\n* `location` - Location of the log call (e.g. `my_module-my_func#4`)\n* `file` - File path of the log call (e.g. `/Users/danyi1212/projects/my-project/my_module.py`)\n* `error` - Exception message and traceback (when `exec_info=True`)\n* `stack_info` - Stack trace (when `stack_info=True`)\n* `formatter_errors` - Errors from the formatter or providers (when an error occurs)\n\nBy default, the `message` value is truncated to 64k characters, and the `error`, 'stack_info', and `formatter_errors`\nvalues are truncated to 128k characters.\n\nYou can override the default truncation using:\n```python\nimport logging.config\n\nlogging.config.dictConfig({\n  \"version\": 1,\n  \"formatters\": {\n    \"json\": {\n      \"()\": \"dans_log_formatter.JsonLogFormatter\",\n      \"message_size_limit\": 1024,  # Set None to unlimited\n      \"stack_size_limit\": 1024,  # Set None to unlimited\n    }\n  },\n  # ...\n})\n```\n\n### JsonLogFormatter\n\nFormat log records as JSON using `json.dumps()`.\n\n### TextLogFormatter\n\nFormat log records as human-readable text using `logging.Formatter` ([See the docs](https://docs.python.org/3/library/logging.html#formatter-objects)).\n\nAll attributes are available to use in the format string.\n\nThe `timestamp` attribute is formatted using the `datefmt` like in the `logging.Formatter`.\n\n```python\nimport logging.config\n\nfrom dans_log_formatter.providers.context import ContextProvider, inject_log_context\n\nlogging.config.dictConfig({\n  \"version\": 1,\n  \"formatters\": {\n    \"text\": {\n      \"()\": \"dans_log_formatter.TextLogFormatter\",\n      \"providers\": [ContextProvider()],\n      \"fmt\": \"{timestamp} {status} | {user_id} - {message}\",\n      \"datefmt\": \"%H:%M:%S\",\n      \"style\": \"{\"\n    }\n  },\n  # ...\n})\n\nlogger = logging.getLogger(__name__)\n\nwith inject_log_context({\"user_id\": 123}):\n  logger.info(\"Hello, world!\")\n\n# STDOUT: 12:00:42 INFO | 123 - Hello, world!\n```\n\n## Extending your own formatter\n\nYou can extend the `JsonLogFormatter` to modify the default attributes, add new ones, use other log record serializer or anything else.\n\n```python\nimport socket\nfrom logging import LogRecord\nimport xml.etree.ElementTree as ET\n\nfrom dans_log_formatter import JsonLogFormatter\n\n\nclass MyCustomFormatter(JsonLogFormatter):\n  root_tag = \"log\"\n\n  def format(self, record: LogRecord) -\u003e str:\n    # Serialize to XML instead of JSON\n    return self.attributes_to_xml(self.get_attributes(record))\n\n  def attributes_to_xml(self, attributes: dict[str, str]) -\u003e str:\n    root = ET.Element(self.root_tag)\n    for key, value in attributes.items():\n      element = ET.SubElement(root, key)\n      element.text = value\n    return ET.tostring(root, encoding=\"unicode\")\n\n  def format_status(self, record: LogRecord) -\u003e int:\n    return record.levelno  # Use the level number instead of the level name\n\n  def format_location(self, record: LogRecord) -\u003e str:\n    return f\"{record.module}-{record.funcName}\"  # Use only the module and function name, without the line number\n\n  def format_exception(self, record: LogRecord) -\u003e str:\n    return f\"{record.exc_info[0].__name__}: {record.exc_info[1]}\"  # Use only the exception name and message\n\n  def get_attributes(self, record: LogRecord) -\u003e dict:\n    attributes = super().get_attributes(record)\n    attributes[\"hostname\"] = socket.gethostname()  # Add an extra hostname default attribute\n    return attributes\n```\n\n\u003e Note: Creating a custom `HostnameProvider` is a better way to add the hostname attribute.\n\n### Error handling\n\nWhen an error occurs in the formatter or providers, the `formatter_errors` attribute is added to the log record.\n\nSilent errors can be added to the `formatter_errors` attribute using the `record_error()` method.\n\n```python\nfrom dans_log_formatter.providers.abstract import AbstractProvider\n\n\nclass MyProvider(AbstractProvider):\n  def get_attributes(self, record: LogRecord) -\u003e dict[str, Any]:\n    self.record_error(\"Something went wrong\")  # Add an error to the formatter_errors attribute\n    return {'my_attribute': 'some value'}\n```\n\nException traceback context is automatically added to the recorded error or caught exceptions described in the `formatter_errors` attribute.\n\n\n## Contributing\n\nBefore contributing, please read the [contributing guidelines](CONTRIBUTING.md) for guidance on how to get started.\n\n\n### License\n\nThis project is licensed under the [MIT License](LICENSE).\n\n\n# Happy logging!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdanyi1212%2Fdans-log-formatter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdanyi1212%2Fdans-log-formatter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdanyi1212%2Fdans-log-formatter/lists"}