Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/calcite/loggate
The complex logging system with support of tagging and delivery to Grafana Loki.
https://github.com/calcite/loggate
Last synced: 3 months ago
JSON representation
The complex logging system with support of tagging and delivery to Grafana Loki.
- Host: GitHub
- URL: https://github.com/calcite/loggate
- Owner: calcite
- License: mit
- Created: 2022-09-08T11:11:45.000Z (over 2 years ago)
- Default Branch: master
- Last Pushed: 2024-08-08T08:51:14.000Z (6 months ago)
- Last Synced: 2024-10-08T14:07:47.794Z (4 months ago)
- Language: Python
- Size: 202 KB
- Stars: 4
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Loggate
[](https://pypi.org/project/loggate/)
The complex logging system with support of log metadata and delivery to [Grafana Loki](https://grafana.com/oss/loki/).
This library supports threading & asyncio modules.
## Simple stdout/stderr colorized output
One example is more than a thousand words.
```python
from loggate import setup_logging, get_logger
setup_logging(level='DEBUG')
logger = get_logger('component', meta={'version': '1.0.0'})
logger.debug('Loading resources for the component')
logger.info('Initialize of the component')
logger.warning('The component is not ready')
logger.error('The component failed.',
meta={'inputs': {'A': 1, 'B': 2}})
logger.critical('The component unexpected failed.',
meta={'attrs': {'A': 1, 'B': 2}})
```
*Console output:*
### Exceptions
```python
from loggate import setup_logging, get_logger
setup_logging()
logger = get_logger('component', meta={'version': '1.0.0'})
try:
raise Exception('Some error')
except Exception as ex:
logger.error('The component failed.', exc_info=True)
```
*Console output:*
## Advanced configuration
The Loggate supports a declarative configuration alike as [logging.config](https://docs.python.org/3/library/logging.config.html).
But this support profiles as well. It's mean we can declare many logging profiles and switch between them. Default profile is called `default`.
We can use yaml file as configuration file (see next):
```yaml
profiles:
default:
# Default profile
filters:
warning:
# This is a filter for stdout logger, that enable only logs with level lower than WARNING.
# For logs with WARNING and higher we use stderr logger.
class: loggate.LowerLogLevelFilter
level: WARNING
formatters:
colored:
# This is stdout/sterr formatter.
class: loggate.LogColorFormatter
loki:
# This is formatter of loki messages.
class: loggate.loki.LokiLogFormatter
handlers:
stdout:
# This is a stdout handler
class: logging.StreamHandler
stream: ext://sys.stdout
formatter: colored
filters:
- warning
stderr:
# This is a stderr handler
class: logging.StreamHandler
stream: ext://sys.stderr
formatter: colored
level: WARNING
loki:
# This is a loki handler
class: loggate.loki.LokiThreadHandler # for asyncio use loggate.loki.LokiHandler
formatter: loki
max_queue_size: 1000 # Default is 0 = unlimit
# send_retry: [5, 5, 10, 10, 30, 30, 60, 60, 120]
urls:
- "http://loki1:3100/loki/api/v1/push"
- "http://loki2:3100/loki/api/v1/push"
- "http://loki3:3100/loki/api/v1/push"
meta:
# loggate handlers accept metadata as well. Standard logging handlers do not!
stage: dev
ip: 192.168.1.10
loggers:
root:
level: INFO
handlers:
- stdout
- stderr
- loki
'urllib3.connectionpool':
level: WARNING
```
```python
import yaml
from loggate import setup_logging, get_logger
def get_yaml(filename):
with open(filename, 'r+') as fd:
return yaml.safe_load(fd)
schema = get_yaml('test.yaml')
setup_logging(profiles=schema.get('profiles'))
logger = get_logger('component')
logger.debug('Loading resources for the component')
logger.info('Initialize of the component')
logger.warning('The component is not ready')
```
The console output is the same as above, but now we send logs to Loki as well.
*Loki output:*
# Description
## Methods
- `get_logger`
- `name` - Return logger for this name. Empty name returns root logger.
- `meta` - Metadata (dict), which are sent only by this logger.
- `getLogger` - only alias for `get_logger`
- `setup_logging` - init setup of application logging.
- `profiles` - Profiles (dict) of logging profiles. When we do not set this parameter, application use predefined profile with log `INFO` level (this level can be set by parameter `level`).
- `default_profile` - name of the default profile (default: `default`)
- `level` - This is special parameter for situation when application use predefined profile (default `INFO`).
## Filters
### Class `loggate.LowerLogLevelFilter`
This filters out all logs which are higher than `level`.
- `level` - log level
## Formatters
### Class `loggate.LogColorFormatter`
Colorized formatter for stdout/stderr.
- `fmt` - message format (default: `%(LEVEL_COLOR)s%(asctime)s\t [%(levelname)s] %(name)s:%(COLOR_RESET)s %(message)s`)
- `datefmt` - datetime format (default: `%Y-%m-%d %H:%M:%S`)
- `style` - style of templating (default: `%`).
- `validate` - validate the input format (default: True)
- `INDENTATION_TRACEBACK` - default: `\t\t\t`
- `INDENTATION_METADATA` - default: `\t\t\t\t`
- `COLOR_DEBUG`, ..., `COLOR_CRITICAL` - set color of this levels (e.g. `\x1b[1;31m`, see [more colors](https://dev.to/ifenna__/adding-colors-to-bash-scripts-48g4)).
- `COLOR_METADATA` - set color metadata
- `COLOR_TRACEBACK` - set color of tracebacks
- `COLOR_...` - set custom color
### Class `loggate.loki.LokiLogFormatter`
This is special loki formatter, this converts log records to jsons.
## Handlers
### Class `loggate.loki.LokiHandler`
This handler send log records to Loki server. This is blocking implementation of handler.
It means, when we call log method (`debug`, ... `critical`) the message is sent in the same thread. We should use
this only for tiny scripts where other ways have a big overhead.
- `level` - This handler sends only log records with log level equal or higher than this (default: all = `logging.NOTSET`).
- `urls` - List of loki entrypoints.
- `strategy` - Deploy strategy (default: `random`).
- `random` - At the beginning the handler choose random Loki server and others are fallbacks.
- `fallbacks` - The handler uses the first Loki server and others are fallbacks.
- `all` - The handler send the log record to all loki servers.
- `auth` - The Loki authentication, the list with two items (`username`, `password`).
- `timeout` - Timeout for one delivery try (default: 5s).
- `ssl_verify` - Enable ssl verify (default: True).
- `max_queue_size` - Size of sending queue. The default is 0 = unlimited. Privileged messages have got a limit 110% of `max_queue_size`.
- `send_retry` - Comma separated list of seconds for resend. The last item of this list is used as default for all other sending.
- `loki_tags` - the list of metadata keys, which are sent to Loki server as label (defailt: [`logger`, `level`]).
- `meta` - Metadata (dict), which are sent only by this handler.
### Class `loggate.loki.LokiAsyncioHandler`
This is non-bloking extending of LokiHandler. We register an extra asyncio task for sending messages to the Loki server.
Parameters are the same as `loggate.loki.LokiHandler`. This handler uses `urllib.requests` module in default ([aiohttp](https://pypi.org/project/aiohttp/) as optional).
Unfortunately `urllib.requests` module does not support asyncio, it means the sending itself is blocking.
The `loggate.loki.Loki AsyncioHandler` can use the optional dependency [aiohttp](https://pypi.org/project/aiohttp/) for non-bloking sending.
### Class `loggate.loki.LokiThreadHandler`
This is non-bloking extending of LokiHandler. We register and start an extra thread for sending messages to the Loki server.
Parameters are the same as `loggate.loki.LokiHandler`.
## Profiles
The structure of profiles (parameter `profiles` of `setup_logging`).
```yaml
:
filters:
:
class:
:
formatters:
:
class:
:
handlers:
:
class:
:
loggers:
|root: # definition of root logger
level:
handlers:
- |
disabled: True|False # default: False
propagate: True|False # default: True
meta:
```