https://github.com/jruere/multiprocessing-logging

Handler for logging from multiple processes
https://github.com/jruere/multiprocessing-logging

linux logging multiprocessing python

Last synced: about 14 hours ago
JSON representation

Handler for logging from multiple processes

• Host: GitHub
• URL: https://github.com/jruere/multiprocessing-logging
• Owner: jruere
• License: lgpl-3.0
• Created: 2015-01-22T14:49:01.000Z (almost 10 years ago)
• Default Branch: master
• Last Pushed: 2023-10-25T19:50:43.000Z (about 1 year ago)
• Last Synced: 2024-04-24T03:23:50.678Z (7 months ago)
• Topics: linux, logging, multiprocessing, python
• Language: Python
• Size: 75.2 KB
• Stars: 378
• Watchers: 15
• Forks: 60
• Open Issues: 7
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# multiprocessing-logging

[![Supported Python versions](https://img.shields.io/pypi/pyversions/multiprocessing-logging.svg)](https://pypi.python.org/pypi/multiprocessing-logging/)

[![License](https://img.shields.io/pypi/l/multiprocessing-logging.svg)](https://pypi.python.org/pypi/multiprocessing-logging/)

When using the `multiprocessing` module, logging becomes less useful since

sub-processes should log to individual files/streams or there's the risk of

records becoming garbled.

This simple module implements a `Handler` that when set on the root

`Logger` will handle tunneling the records to the main process so that

they are handled correctly.

It's currently tested in Linux and Python 2.7 & 3.6+.

Pypy3 hangs on the tests so I don't recommend using it.

Pypy appears to be working, recently.

Only works on POSIX systems and only Linux is supported. It does not work on Windows.

# Origin

This library was taken verbatim from a [StackOverflow post](http://stackoverflow.com/questions/641420/how-should-i-log-while-using-multiprocessing-in-python)

and extracted into a module so that I wouldn't have to copy the code in every

project.

Later, several improvements have been contributed.

# Usage

Before you start logging but after you configure the logging framework (maybe with `logging.basicConfig(...)`), do the following:

```py

import multiprocessing_logging

multiprocessing_logging.install_mp_handler()

```

and that's it.

## With multiprocessing.Pool

When using a Pool, make sure `install_mp_handler` is called before the Pool is instantiated, for example:

```py

import logging

from multiprocessing import Pool

from multiprocessing_logging import install_mp_handler

logging.basicConfig(...)

install_mp_handler()

pool = Pool(...)

```

# Problems

The approach of this module relies on

[fork](https://docs.python.org/3.9/library/multiprocessing.html#multiprocessing.set_start_method)

being used to create new processes. This start method

[is basically unsafe when also using threads](https://bugs.python.org/issue37429),

as this module does.

The consequence is that there's a low probability of the application hanging

when creating new processes.

As a palliative, don't continuously create new processes. Instead, create a

Pool once and reuse it.