https://github.com/diegojromerolopez/backgroundlog
Thread-based log handler for better performance
https://github.com/diegojromerolopez/backgroundlog
library log logger logging non-blocking performance python threading
Last synced: 5 months ago
JSON representation
Thread-based log handler for better performance
- Host: GitHub
- URL: https://github.com/diegojromerolopez/backgroundlog
- Owner: diegojromerolopez
- License: mit
- Created: 2025-07-06T07:15:47.000Z (12 months ago)
- Default Branch: main
- Last Pushed: 2025-07-10T05:22:42.000Z (12 months ago)
- Last Synced: 2025-07-10T10:19:12.355Z (12 months ago)
- Topics: library, log, logger, logging, non-blocking, performance, python, threading
- Language: Python
- Homepage:
- Size: 28.3 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
Awesome Lists containing this project
README
# backgroundlog
Thread-based log handler for better performance
[](https://opensource.org/licenses/MIT)
[](https://github.com/diegojromerolopez/backgroundlog/graphs/commit-activity)
[](https://www.python.org/)
[](https://github.com/psf/black)
[](https://pycqa.github.io/isort/)
[](http://mypy-lang.org/)
[](https://pypi.python.org/pypi/backgroundlog/)
[](https://pypi.python.org/pypi/backgroundlog/)
[](https://pypi.python.org/pypi/backgroundlog/)
[](https://pypi.python.org/pypi/backgroundlog/)
Do not have your python program slowed down by your logging.
## Introduction
Most of the time we log on disk, and all I/O is time consuming.
By leveraging a thread, it is possible to speed up your python application
considerably.
## Use
### Default use
```python
import logging
from backgroundlog.handlers.thread_handler import ThreadHandler
# Setting up the logging thread handler
file_handler = logging.FileHandler('/var/log/myapp.log', mode="a", encoding="utf-8")
thread_handler = ThreadHandler(file_handler)
# Creating a new logger
bg_logger = logging.getLogger('bg_logger')
bg_logger.setLevel(logging.INFO)
# Adding the thread handler
bg_logger.addHandler(thread_handler)
# Using the logger
bg_logger.info('This is a log message')
```
### Options
#### Set a queue size
```python
from backgroundlog.handlers.thread_handler import ThreadHandler
thread_handler = ThreadHandler(file_handler, queue_size=5000)
```
By default, the queue size is 1000.
#### Set a blocking policy by logging record levels
When putting the records in the queue, it could reach the queue size.
We provide a way to deal with this issue: set a blocking policy
by logging record level, and in the case of a non-blocking policy,
the record will be discarded and we will increment a dropped log record.
##### Only info, error and critical records are blocking:
```python
from backgroundlog.handlers.thread_handler import ThreadHandler
from logging import CRITICAL, ERROR, INFO
thread_handler = ThreadHandler(file_handler, blocking_levels={INFO, ERROR, CRITICAL})
```
##### Only error and critical records are blocking
```python
from backgroundlog.handlers.thread_handler import ThreadHandler
from logging import CRITICAL, ERROR
thread_handler = ThreadHandler(file_handler, blocking_levels={ERROR, CRITICAL})
```
##### Only critical records are blocking
```python
from backgroundlog.handlers.thread_handler import ThreadHandler
from logging import CRITICAL
thread_handler = ThreadHandler(file_handler, blocking_levels={CRITICAL})
```
##### No records are blocking
```python
from backgroundlog.handlers.thread_handler import ThreadHandler
thread_handler = ThreadHandler(file_handler, blocking_levels=None)
```
By default, the error and critical records are blocking, the rest are not.
## Performance testing
We have done several local testing with different logging handlers.
See the file
[run_performance_comparison.py](/backgroundlog/performance/run_performance_comparison.py) for
a full catalog of the performance tests we run.
We used two versions of Python depending on if they had or not the global interpret lock:
- Python 3.15.3 (with GIL)
- Python 3.15.3t (without GIL)
All tests are 100_000 iterations of creating the same logging message,
and were run in a Macbook Pro M1 with 16 GB of RAM.
### Python 3.15.3 (with GIL)
| Logging Handler | Spent Time | | vs. Baseline |
|-------------------------------|----------------|--------------|--------------|
| | Mean Time (ms) | Std Dev (ms) | |
| StreamHandler | 0.685 | 0.006 | baseline |
| FileHandler | 0.685 | 0.018 | +0.03% |
| ThreadHandler (StreamHandler) | 0.487 | 0.03 | -28.911% |
| ThreadHandler (FileHandler) | 0.475 | 0.002 | -30.66% |
There is a ~30% of improvement when running the thread handler.
### Python 3.15.3t (without GIL)
| Logging Handler | Spent Time | | vs. Baseline |
|-------------------------------|----------------|--------------|--------------|
| | Mean Time (ms) | Std Dev (ms) | |
| StreamHandler | 0.539 | 0.004 | baseline |
| FileHandler | 0.545 | 0.013 | +1.109% |
| ThreadHandler (StreamHandler) | 0.344 | 0.002 | -36.301% |
| ThreadHandler (FileHandler) | 0.339 | 0.001 | -37.118% |
There is a ~36% of improvement when running the thread handler. +6% with respect of the
Python version with GIL.
### Conclusions
Not blocking the main flow of your program by making use of backgroundlog
gives you a 30% speed gain.
## Dependencies
This package has no dependencies.
## Python version support
Minimum version support is 3.10.
## License
[MIT](LICENSE) license, but if you need any other contact me.